Unlocking the Potential of Unstructured Data with New JSON Type Support | Greptime Biweekly Report

Summary ​

Together with our global community of contributors, GreptimeDB continues to evolve and flourish as a growing open-source project. We are grateful to each and every one of you.

Below are the highlights among recent commits:

• Support JSON data type, and added some commonly used UDFs: get_by_path and json_is.

• Added more h3 scalar functions, for processing and analyzing spatial data.

• Added procedure_info and region_statistics tables in information_schema to improve system observability.

• Use batch mode to optimize Flow performance.

• Added slow query logging.

• Updated pgwire to resolve potential issues with connection establishment.

Contributors ​

For the past two weeks, our community has been super active with a total of 180 PRs merged. Of these, 11 PRs were contributed by 6 individual contributors, with many more pending to be merged.

• @CookiePieWw (db#4770 db#4742 db#4726 db#4515 docs#1196)

• @haoqixu (db-ingester-go#39)

• @J0HN50N133 (db#4774 db#4729)

• @Kev1n8 (db#4287)

• @lyang24 (db#4699)

• @poltao (db#4675)

Congrats on becoming our most active contributors in the past 2 weeks!

👏 Welcome contributor @haoqixu @Kev1n8 join to the community as the new individual contributor, and congratulations on successfully merging their first PR!

A big THANK YOU to all our members and contributors! It is people like you who are making GreptimeDB a great product. Let's build an even greater community together.

Highlights of Recent PRs ​

db#4619 db#4515 db#4720 JSON data type support ​

Progress has been made on JSON data type support.

1. Both the RFC and the implementation have been merged into the main branch.
2. Added some UDFs for JSON type: get_by_path and json_is.

db#4719 Implemented slow query logging ​

The slow query logging is mainly used to record query statements that take a long time to execute.

By analyzing the slow query log, developers can find out which query statements are inefficient and optimize them accordingly.

db#4675 Improve observability for procedure ​

Before this PR, we could not view the status of each procedure, which made it difficult to debug or verify the content related to a specific procedure. Therefore, this pr adds a procedure_info table to information_schema. We can query this table to get the status of the procedure, including procedure_id, procedure_type, start_time, end_time, lock_keys etc.

The following is an example:

select * from information_schema.procedure_info \G

The output will be displayed:

*************************** 1. row ***************************
  procedure_id: 140ae3dd-da22-4f5d-a447-debe55e6bbb1
procedure_type: metasrv-procedure::DropTable
    start_time: 2024-09-13 14:42:29.298000
      end_time: 2024-09-13 14:42:29.309000
        status: Done
     lock_keys: Share("__catalog_lock/greptime"),Share("__schema_lock/greptime.public"),Exclusive("__table_lock/1044")
*************************** 2. row ***************************
  procedure_id: 2a6d3155-b2c0-4503-81d7-a981393b08fb
procedure_type: metasrv-procedure::CreateTable
    start_time: 2024-09-13 14:42:33.490000
      end_time: 2024-09-13 14:42:33.498000
        status: Done
     lock_keys: Share("__catalog_lock/greptime"),Share("__schema_lock/greptime.public"),Exclusive("__table_name_lock/greptime.public.temperatures")
2 rows in set (0.01 sec)

db#4771 Improve observability for Region ​

A new region_statistics table has been added to information_schema. We can query this table to get the status of the region, including table_id, region_number, memtable_size etc.

The following is an example:

select * from information_schema.region_statistics;

The output will be displayed:

+---------------+----------+---------------+---------------+---------------+-----------+--------+-------------+
| region_id     | table_id | region_number | memtable_size | manifest_size | sst_size  | engine | region_role |
+---------------+----------+---------------+---------------+---------------+-----------+--------+-------------+
| 4440996184064 |     1034 |             0 |       3586817 |         11704 | 301876450 | mito   | Leader      |
| 4432406249472 |     1032 |             0 |     269686817 |         12508 | 279053485 | mito   | Leader      |
| 4445291151360 |     1035 |             0 |      56686817 |          9232 | 297195520 | mito   | Leader      |
| 4436701216768 |     1033 |             0 |     239386817 |          9139 | 281388451 | mito   | Leader      |
+---------------+----------+---------------+---------------+---------------+-----------+--------+-------------+
4 rows in set (0.01 sec)

Good First Issue ​

db#3262 Provide an official Grafana configuration ​

Provides an intuitive out-of-the-box grafana dashboard for GreptimeDB.

• Keywords: Grafana

• Difficulty: Medium

db#4805 Add more JSON UDFs ​

Add more UDFs about JSON data type to achieve powerful query capabilities.

• Keywords: JSON, UDF

• Difficulty: Simple

About Greptime ​

We help industries that generate large amounts of time-series data, such as Connected Vehicles (CV), IoT, and Observability, to efficiently uncover the hidden value of data in real-time.

Visit the latest version from any device to get started and get the most out of your data.

• GreptimeDB, written in Rust, is a distributed, open-source, time-series database designed for scalability, efficiency, and powerful analytics.
• Edge-Cloud Integrated TSDB is designed for the unique demands of edge storage and compute in IoT. It tackles the exponential growth of edge data by integrating a multimodal edge-side database with cloud-based GreptimeDB Enterprise. This combination reduces traffic, computing, and storage costs while enhancing data timeliness and business insights.
• GreptimeCloud is a fully-managed cloud database-as-a-service (DBaaS) solution built on GreptimeDB. It efficiently supports applications in fields such as observability, IoT, and finance.

Star us on GitHub or join GreptimeDB Community on Slack to get connected. Also, you can go to our contribution page to find some interesting issues to start with.