Committer +1! GreptimeDB Sorting Performance Optimization | Greptime Biweekly Report

Summary ​

Together with our global community of contributors, GreptimeDB continues to evolve and flourish as a growing open-source project. We are grateful to each and every one of you.

Below are the highlights among recent commits:

• Sorting Performance Optimization: Introduced the Part Sort physical operator and Windowed Sort rule.

• Resource Usage Limitation: Introducing a resource-limiting mechanism can be useful in resource-constrained environments (especially embedded devices).

• Storage Engine Performance Enhancement: Accelerated queries by caching row groups for a time range.

• Dynamic Table Option Changes: Allow the table TTL to be dynamically modified.

Contributors ​

For the past two weeks, our community has been super active with a total of 88 PRs merged. Of these, 5 PRs were contributed by 3 individual contributors, with many more pending to be merged.

• @CookiePieWw (db#4933 db#4919 db#4893)

• @Kev1n8（db#4864）

• @ActivePeter（db#4782）

👏 Welcome @ActivePeter to the community as a new contributor with a successfully merged PR, and more PRs from other individual contributors are waiting to be merged.

🎉 According to the result of the Committer promotion vote in GreptimeDB community 2024.11.1, @lyang24 is officially promoted as a new Committer due to his high quality PRs and long term contributions, looking forward to more of your contributions in the community!

😄 Warmly welcome @lyang24, an individual contributor, to the community as a new Committer, thank you for your efforts and hope we can continue to work hand in hand to build the community.

A big THANK YOU to all our members and contributors! It is people like you who are making GreptimeDB a great product. Let's build an even greater community together.

Highlights of Recent PRs ​

db#4847 db#4922 Introduce PartSort Physical Operator ​

Added the PartSort physical operator to sort data within PartitionRange boundaries, and implemented a simple limit mechanism in the PartSort operator.

db#4874 Introduce Windowed Sort Rule ​

The optimizer has been improved by introducing a windowed sort rule, utilizing the PartSort physical operator to enhance overall sorting performance.

db#4859 OpenTelemetry Data Writing Enhancement ​

Enhanced OpenTelemetry writing capabilities by allowing fields to be extracted from attributes for indexing, increasing data processing flexibility.

db#4782 CPU Usage Limitation Mechanism ​

Implemented a CPU usage limitation mechanism to operate efficiently in resource-constrained environments, such as embedded devices.

db#4864 New JSON User-Defined Function json_path_match ​

Introduced a new JSON user-defined function, json_path_match, to expand JSON data handling capabilities.

db#4871 Separate Schedulers in Storage Engine ​

Optimized the storage engine by separating the schedulers for flush and compaction tasks, addressing scheduler blockage issues.

db#4869 Cache Row Groups over Time Range in Storage Engine ​

Enhanced query performance by caching row groups over time ranges and retrieving them in subsequent queries.

db#4848 Support for Dynamic Table Option Changes ​

Enabled dynamic modification of table options (such as TTL) through SQL syntax, enhancing table flexibility.

The syntax is:

ALTER TABLE <table_name> SET <option_name>=<option_value>[, ...];

db#4909 Refine Region Statistics Table ​

Added new columns disk_size, region_rows, and index_size to information_schema.region_statistics for more detailed regional statistics.

db#4908 Insert JSON Data via gRPC Protocol ​

Implemented the ability to insert JSON data via the gRPC protocol, expanding JSON data writing methods.

Good First Issue ​

db#4935 Fix the region_stats test ​

Fix the test case tests::instance_test::test_use_database.

• Keywords: Region

• Difficulty: Simple

db#4932 Unify Datanode options in Frontend and Metasrv ​

Two Datanode options with the same functionality are located in frontend::service_config::datanode and meta_srv::metasrv, which need to be unified.

• Keywords: Config

• Difficulty: Simple

About Greptime ​

We help industries that generate large amounts of time-series data, such as Connected Vehicles (CV), IoT, and Observability, to efficiently uncover the hidden value of data in real-time.

Visit the latest version from any device to get started and get the most out of your data.

• GreptimeDB, written in Rust, is a distributed, open-source, time-series database designed for scalability, efficiency, and powerful analytics.
• Edge-Cloud Integrated TSDB is designed for the unique demands of edge storage and compute in IoT. It tackles the exponential growth of edge data by integrating a multimodal edge-side database with cloud-based GreptimeDB Enterprise. This combination reduces traffic, computing, and storage costs while enhancing data timeliness and business insights.
• GreptimeCloud is a fully-managed cloud database-as-a-service (DBaaS) solution built on GreptimeDB. It efficiently supports applications in fields such as observability, IoT, and finance.

Star us on GitHub or join GreptimeDB Community on Slack to get connected. Also, you can go to our contribution page to find some interesting issues to start with.