Enhancing Flow Usability! Easier Task and Metadata Management | Greptime Biweekly Report

Summary ​

Together with our global community of contributors, GreptimeDB continues to evolve and flourish as a growing open-source project. We are grateful to each and every one of you.

Below are the highlights among recent commits:

• Improve usability of Flow: Managing Flow and performing metadata checks have become much easier.

• Enhance data backup functionality: Exporting and backup data now become more functional.

Contributors ​

For the past two weeks, our community has been super active with a total of 87 PRs merged. 6 PRs from 6 individual contributors merged successfully and lots pending to be merged.

• @besrabasant (docs#1100)

• @J0HN50N133 (db#4428)

• @leaf-potato (db#4427 db#4426 db#4412)

• @realtaobo (db#4405)

Congrats on becoming our most active contributors in the past 2 weeks!

👏 Welcome contributor @besrabasant @leaf-potato join to the community as the new individual contributor, and congratulations on successfully merging their first PR, more PRs are waiting to be merged.

A big THANK YOU to all our members and contributors! It is people like you who are making GreptimeDB a great product. Let's build an even greater community together.

Highlights of Recent PRs ​

db#4386 db#4416 Improve Usability of Flow ​

We have focused on making Flow easier to use in the past two weeks. Including implementing SHOW FLOWS clause:

public=> SHOW FLOWS;
     Flows      
----------------
 filter_numbers
(1 row)

There is also a new table FLOWS in the information schema to show Flow's metadata:

public=> select * from INFORMATION_SCHEMA.FLOWS;
   flow_name    | flow_id | catalog_name |                      raw_sql                       | comment | expire_after | source_table_ids |       sink_table_name       | flownode_ids | options 
----------------+---------+--------------+----------------------------------------------------+---------+--------------+------------------+-----------------------------+--------------+---------
 filter_numbers |    1024 | greptime     | SELECT number FROM numbers_input WHERE number > 10 |         |              | [1024]           | greptime.public.out_num_cnt | {"0":0}      | {}
(1 row)

In situations where a manual update is necessary, the flush_flow function can now be used to facilitate this:

select flush_flow('test_numbers_df_func');

db#4382 db#4405 Enhanced Data Backup Capabilities ​

Starting from the latest version, our built-in CLI tool, greptime cli, has a new option for the export subcommand to facilitate database data export:

greptime cli export --help
Usage: greptime cli export [OPTIONS] --addr <ADDR> --output-dir <OUTPUT_DIR> --target <TARGET>

Options:
  -t, --target <TARGET>
          Things to export

          Possible values:
          - create-table:  Corresponding to `SHOW CREATE TABLE`
          - table-data:    Corresponding to `EXPORT TABLE`
          - database-data: Corresponding to `EXPORT DATABASE`

Additionally, it has replaced the old table-data target option.

For the COPY TABLE clause, we can now specify a time range for export. This feature enables incremental backups:

copy table x from datasource with (start_time='2022-06-15 07:02:37', end_time='2022-06-15 07:02:38');

Good First Issue ​

db#3072 Merge Small IO to Optimize for Cloud Storage ​

Excessive small requests can lead to costly bills; we can optimize object storage, such as S3, to reduce unnecessary expenses. If the ranges are almost continuous, we can merge these ranges into a large chunk and fetch this chunk in the preferred size concurrently.

• Keywords: Cloud, Object Store

• Difficulty: Medium