On this page
Since Greptime released its open-source project GretptimeDB in November last year, four months have passed. Although it may not seem like a long time, with the help from the community, GreptimeDB is steadily moving towards its goals.
We split previously set goals into three upcoming releases before this May: v0.1, v0.2, and v0.3, the intermediate goal is to deliver a stable, distributed Time Series Database (TSDB) with v0.3. After spending more than a week finalizing the documentation and testing processes, we are delighted to announce the release of GreptimeDB v0.1. This post provides an overview of our accomplishments for the past few months and what we intend to do next.
Community โ
Ever since GreptimeDB open-sourced on GitHub, it has received more attention from the wider community than we originally anticipated. Companies from various industries, such as Finance, Utilities, Autonomous Vehicles, Manufacturing, Wholesale, Logistics, and more, have approached our team and expressed a strong interest in becoming seed users.
We would like to take this chance to express our gratitude to everyone who has supported, helped and grown with GreptimeDB throughout our journey. It is because of all of your efforts that we have achieved these impressive numbers:
- 2.3k GitHub stars: more than 80% of GitHub users who starred GreptimeDB are from China, USA, Russia, Germany, and France
- 491 PRs merged: 60 of them are contributors outside of our team, accounting for 12.2% of the total
- 11 open-sourced projects in the GreptimeDB ecosystem: GreptimeDB, dashboard, promql-parse, greptimedb-operator, etc
- 16 articles: including 6 original technical articles written by our talented developers
- 18 contributors outside of Greptime's team
- 1977 downloads
Behind these numbers are the joint efforts of the team and community, and the valuable time invested by each contributor towards developing new features, improving existing ones, and fixing issues in GreptimeDB.
Next, let's take a look at the highlights of v0.1.
GreptimeDB v0.1 โ
Features โ
- Compaction
Compaction is necessary for an LSM-Tree (log-structured merge-tree) architecture to buffer time series data. V0.1 now supports Compaction which enables data retention policies based on TTL.
- Support Object Storage
OpenDAL allows users to effortlessly access data. v0.1 is now able to support S3 and OSS object storage through OpenDAL.
- Native Support for Python
Enabling Python scripting capabilities and providing support for DataFrame API and third-party Python libraries, which can be used as coprocessors and UDF(User Defined Functions).
- Native support for PromQL
PromQL is almost an established standard for cloud-native Observability. We intend to support PromQL natively in GreptimeDB, which is a huge challenge. Currently, we have implemented native PromQL support preliminarily, which will be released with GreptimeDB version 0.1. Although it hasn't been fully compatible with the official compatibility tests, it is already available for basic uses. For this compatibility test, the plan is to pass more than half of the test cases in GreptimeDB v0.2 and more than 70% in v0.3.
Protocol โ
- New version of high-performance communication protocol
Built on the Arrow Flight RPC, it is now much simpler and more efficient than the original gRPC private protocol, and it is easy for multiple languages to communicate directly with GreptimeDB through Arrow Flight's SDK. It is also more convenient for supporting Stream. See docs.
- MySQL & PostgreSQL support TLS
It is essential for MySQL and PostgreSQL to support TLS (Transport Layer Security) to facilitate privacy and data security. In addition, unlike HTTP or gRPC, database protocols have their own TLS handshake process. As a result, we have integrated TLS support at the database level.
Clients โ
- Java SDK based on Arrow Flight RPC communication
- Go SDK based on Arrow Flight RPC still under development
Refactor โ
- Datafusion & Arrow Reconstruction
GreptimeDB initially used Arrow2 heavily, but the Arrow2 branch of datafusion is no longer maintained, so we decided to switch to Arrow so we could keep up with the latest datafusion version, another challenging task completed by our developers.
In addition to the contents mentioned above, we have 491 PRs merged for various feature preparations, refactoring, bug fixes and documentation. Besides, we have also contributed some of our experience back to the open source community, including contributing code to DataFusion, sqlness, Parquet2, DataFusion-Substrait, RustPython, OpenDAL and other external projects.
Future plans โ
GreptimeDB โ
We will release v0.3 in May as planned, with the goal of reaching a reliable standalone version with initial availability for distributed capabilities. The reliability will be mainly reflected in performance and stability, but since it is a standalone version, there are certain limits and we will gradually improve the distributed version.
We plan to provide a distributed GA (General Availability) version by the end of the year. However, v0.3 should be enough to satisfy the requirements of a standalone TSDB and we encourage users to follow the guidance to download and give it a try.
In terms of performance optimization, we will continue to focus on the life cycle of data ingestion, storage and analysis, with an emphasis on query efficiency, storage costs and distributed architecture, including:
- Performance optimization for PromQL
- Python scripts support the MapReduce framework, enabling higher efficiency distributed computing processing
- Optimization for query engines including vectorized queries, smart indexing, cost-based optimizer, etc.
- Separation of storage and computing, automatic scaling enabled
- Adopting adaptive compression algorithms, supporting a mixed workload of time series models and analysis models with a single set of data for lower storage costs
The rationale for giving priority to these features is based on feedback and comments we have collected from end-users. Of course, if you have other ideas or suggestions, please feel free to communicate with us via Slack, email, etc.
Community โ
A good open-source project cannot be created without the help from the community. We will continue to operate the community based on the principles of "openness", "respect" and "transparency", and treat every contributor and feedback seriously.
We believe that the key to keep an open-source project sustainable is to ensure it delivers value for every contributor. This value can be provided by addressing their business challenges, enhancing their productivity or lowering associated costs with our tools. At the same time, we will foster a prosperous ecosystem by partnering with other organizations both upstream and downstream, as well as contributing and reciprocating within the wider community by participating in other events.
Some activities include:
Increase development visibility, provide developer services, including development documentation projects, usage documentation in multiple languages, project progress kanban and gantt charts, etc.
Interactive activities such as online and offline Office Hour, non-profit hackathon in partnership with institutions, Open Demo Day, etc.
Continuously share knowledge among enthusiasts in the field through various digital media platforms
Organize Meetups with the wider community to exchange ideas, experiences and technologies or tools related to open-source projects
If you are interested in taking part in any of the activities mentioned above or have suggestions for improving the user experience within the community, please do not hesitate to join us on Slack or email us at [email protected].
