Unlimited Performance in a Resource-Limited Environment - GreptimeDB Edge Enhances In-Vehicle Data Processing

The rapid development of the smart automotive industry has brought about an explosion in vehicle data volume, creating new demands for data storage, querying, and management. To tackle these challenges, we propose an Edge-Cloud Integrated Solution, aiming to optimize in-vehicle data management and application.

At the core of this solution lies GreptimeDB Edge, a database tailored for in-vehicle systems. Designed to excel in both storage and computation, it achieves highly efficient data processing with minimal resource consumption. This allows onboard systems to fully leverage their computational power to meet demanding data collection and processing needs. Moreover, GreptimeDB Edge supports synchronizing high-compression data files to cloud object storage for query purposes, significantly reducing bandwidth costs.

In resource-constrained environments like in-vehicle systems, where resources allocated to foundational services remain limited despite advances in chipset capabilities, performance becomes a key factor. This article highlights GreptimeDB Edge's performance under heavy write loads and its compression efficiency in real-world use cases.

Test Environment ​

Hardware Platform ​

Software ​

Edge Configuration Highlights ​

1. Write-Ahead Logging (WAL) Disabled: To enhance IO performance and minimize disk degradation, WAL was disabled during testing. However, WAL can be enabled for critical data to enhance reliability.

2. Memory Optimization: Data is written to memory first and flushed to disk once it reaches a specified threshold. For this test, individual table memory usage was capped at 1MB, while total memory usage was limited to 50MB.

Test Data ​

• Tables: 30
• Fields Per Table: 10
• Field Types: int32, uint32, bool (non-timestamp fields designated as fields rather than tags).

Write Methodology ​

• A simulated write program using a GreptimeDB Edge SDK (based on shared memory communication) generated the write load to GreptimeDB Edge.
• The write program can generate corresponding write loads based on the configuration.

Test Results: ​

350K Points per Second (350K PPS) ​

At a sustained write load of 350K points per second over 30 minutes:

• Average CPU Usage: Single-core 3% (peak below 15%).
• Resident Memory Size: Averaged at 132MB, fluctuating between 120MB and 166MB.

The Performance of Write Rate:

The Performance of CPU Usage:

The Performance of Resident Memory Size:

700K Points per Second (700K PPS) ​

At 700K points per second over 30 minutes:

• Average CPU Usage: Single-core 5.7% (peak below 15%).
• Resident Memory Size: Averaged at 135MB, fluctuating between 130MB and 150MB.

The Performance of Write Loads:

The Performance of CPU Usage:

The Performance of Resident Memory Size:

Compression Efficiency ​

Efficient compression is vital for minimizing network traffic and reducing data transmission costs. It also allows for extended local data retention or lower storage usage.

In a real-world application with a leading EV manufacturer:

• Scenario:
  • GreptimeDB Edge was integrated into the infotainment system (Head Unit, HU) as a system service.
  • Custom services collected CAN signals, writing approximately 250K PPS to GreptimeDB Edge via an SDK.
  • Data was exported every 3 minutes and uploaded to a cloud-based GreptimeDB cluster.
• Results:
  • 10 exported files totaled 42MB, compared to 1.3GB in the ASC Log format.

The ASC log format is an ASCII text file format provided by Vector for recording and analyzing CAN bus data. For more details, refer to the CAN_LOG_TRIGGER_ASC_Format.pdf.

This represents a 30–40x compression ratio, doubling the efficiency of previous methods employed by the manufacturer.

Key Factor: GreptimeDB Edge leverages columnar storage layouts and applies optimal encoding and compression algorithms for each column, achieving this remarkable efficiency.

Balancing Performance Trade-offs

A trade-off exists between CPU, memory, and compression efficiency. For instance:

• Increasing compression efficiency may demand more CPU and memory resources.
• Reducing CPU usage could sacrifice compression and memory efficiency. These trade-offs can be adjusted according to specific application needs.

Conclusion ​

• Performance Under LoadGreptimeDB Edge demonstrates exceptional performance under heavy write loads. At both 350K and 700K PPS, CPU usage averaged 3% and 5.7%, respectively, with stable memory usage around 130MB. This positions GreptimeDB Edge as a reliable in-vehicle data management solution.

• Compression Excellence Real-world tests with CAN data highlight its 30–40x compression efficiency, halving data costs for EV manufacturers.

Looking ahead, we will continue to enhance GreptimeDB Edge, driving innovation in the smart and electric vehicle industries.

Learn More:

• GreptimeDB vs. SQLite - A Performance Comparison Report on the Qualcomm 8155 Platform
• GreptimeDB vs. ClickHouse vs. ElasticSearch — Log Engine Performance Benchmark
• GreptimeDB vs. InfluxDB Performance Benchmark
• GreptimeDB vs. Grafana Mimir - First Official Benchmark for High Volume Write In Performance

About Greptime ​

Greptime offers industry-leading time series database products and solutions to empower IoT and Observability scenarios, enabling enterprises to uncover valuable insights from their data with less time, complexity, and cost.

GreptimeDB is an open-source, high-performance time-series database offering unified storage and analysis for metrics, logs, and events. Try it out instantly with GreptimeCloud, a fully-managed DBaaS solution—no deployment needed!

The Edge-Cloud Integrated Solution combines multimodal edge databases with cloud-based GreptimeDB to optimize IoT edge scenarios, cutting costs while boosting data performance.

Star us on GitHub or join GreptimeDB Community on Slack to get connected.