Optimizing Memory Usage, Improved TTL Management, and Vector-String Conversion | Greptime Biweekly Report

Summary ​

Together with our global community of contributors, GreptimeDB continues to evolve and flourish as a growing open-source project. We are grateful to each and every one of you.

Below are the highlights among recent commits:

• Implement LIMIT in PartSort to Reduce Memory Usage.

• Alter Database TTL(Time To Live).

• Asynchronous Recovery of File Cache Indexes.

• Enable RPC Compression for Metasrv Client.

• Add Conversion between Vector and String.

Contributors ​

For the past two weeks, our community has been super active with a total of 72 PRs merged. Of these, 1 PR were contributed by 1 individual contributor, with many more pending to be merged.

• @lyang24 (db#5046)

A big THANK YOU to all our members and contributors! It is people like you who are making GreptimeDB a great product. Let's build an even greater community together.

Highlights of Recent PRs ​

db#5018 Implement LIMIT in PartSort to Reduce Memory Usage ​

We introduced a TopK heap-based LIMIT feature, significantly reducing memory usage during time-indexed queries. For example:

SELECT * FROM t ORDER BY ts LIMIT 5;

db#5035 Alter Database TTL (Time To Live) ​

With this update, you can now change the database TTL and other options using ALTER DATABASE <database_name> SET <options> and ALTER DATABASE <database_name> UNSET <options>.

db#5087 Asynchronous Recovery of File Cache Indexes ​

Restoring large numbers of file cache indexes synchronously can be time-consuming. For instance, restoring 500,000 cache files may take 10-20 seconds. This update addresses the issue by making the cache index recovery process asynchronous, reducing Datanode startup time by eliminating the need to wait for cache recovery.

db#5078 Enable RPC Compression for Metasrv Client ​

Previously, MetaSrv RPC did not have compression enabled, which could result in the following error message: Error, message length too large: found 4228291 bytes, the limit is: 4194304 bytes. This PR enables the default zstd compression for the MetaSrv client, thereby resolving the issue.

db#5029 Add Conversion Between Vector and Strings ​

We added parse_vec and vec_to_string functions, making it easier to convert between vectors and strings. Example:

=> SELECT vec_to_string(parse_vec('[1.0, 2.0]'));
+----------------------------------------------+
| vec_to_string(parse_vec(Utf8("[1.0, 2.0]"))) |
+----------------------------------------------+
| [1,2]                                        |
+----------------------------------------------+

Good First Issue ​

db#5085 Add the Backup and Restore API for Metasrv ​

• Keywords: Metasrv, API

• Difficulty: Medium

db#5084 Add the HTTP API for Querying Pipelines ​

• Keywords: Pipeline, API

• Difficulty: Simple

About Greptime ​

Greptime offers industry-leading time series database products and solutions to empower IoT and Observability scenarios, enabling enterprises to uncover valuable insights from their data with less time, complexity, and cost.

GreptimeDB is an open-source, high-performance time-series database offering unified storage and analysis for metrics, logs, and events. Try it out instantly with GreptimeCloud, a fully-managed DBaaS solution—no deployment needed!

The Edge-Cloud Integrated Solution combines multimodal edge databases with cloud-based GreptimeDB to optimize IoT edge scenarios, cutting costs while boosting data performance.

Star us on GitHub or join GreptimeDB Community on Slack to get connected.