On this page
Overview
With the rapid adoption of Internet of Things (IoT) technology, businesses are increasingly challenged by the need to manage vast amounts of real-time data. IoT devices generate extensive data that must be transmitted and stored efficiently, securely, and with minimal latency to support timely business decisions.
In response, we propose an integrated solution for data collection, transmission, and storage that leverages the powerful capabilities of the YoMo edge computing framework alongside the GreptimeDB time-series database. This integration is designed to optimize data processing efficiency, minimize network latency, and enhance data storage and analysis. By implementing this solution, businesses can not only maximize their data utilization but also significantly bolster their real-time data processing and analysis capabilities, providing timely and accurate insights for informed decision-making.
YoMo: A Low-Latency Edge Computing Framework
YoMo is an open-source, geo-distributed edge computing framework focused on real-time data processing. Its architecture is designed to transmit device data through a secure, low-latency network to the cloud, where real-time processing occurs using Stateful Serverless technology.
- Low-Latency Transmission: Especially in cellular networks, YoMo effectively reduces data transmission latency from IoT devices, enhancing throughput.
- Geo-Distributed Architecture: Tailored for large-scale enterprise IoT applications, YoMo supports multi-region deployments, ensuring data is always routed to the nearest data center for real-time processing.
- High Security: YoMo allows individual TLS v1.3 certificates for each IoT device, ensuring secure data transmission.
GreptimeDB: A High-Performance Time-Series Database
GreptimeDB is a cloud-native time-series database designed to manage substantial volumes of time-series data. Its primary strength lies in efficient data storage and real-time analysis, making it ideal for complex IoT data scenarios.
Exceptional Data Compression and Low Storage Costs: In edge computing environments, GreptimeDB offers 19 times better data compression than traditional SQLite, significantly reducing storage and transmission costs. It integrates seamlessly with mainstream cloud services and provides flexible resource configuration, minimizing computational and storage expenses, making it ideal for large-scale IoT deployments.
High-Performance Writes and Horizontal Scalability: Compared to Grafana Mimir, GreptimeDB supports at least 100 nodes per cluster under similar conditions while utilizing only one-fifth of the CPU and memory resources. Its efficient horizontal scalability ensures outstanding performance when managing large volumes of IoT time-series data.
Comprehensive SQL Query Support: GreptimeDB offers a complete SQL query syntax, making it user-friendly for data engineers. It supports complex time-series data analysis and aggregation, delivering rapid query responses and efficient analysis from straightforward queries to intricate business evaluations.
Case Study
HS, a provider of technical support for sugar factories in the U.S. and Asia, encountered two significant challenges with its traditional IoT system:
1. Unstable IoT Network
Approximately 90% of HS’s IoT devices rely on cellular networks for data transmission. Among these, 28% experience packet loss rates exceeding 10%, while 8% face rates over 20%. Additionally, 50% of devices report round-trip times (RTT) greater than 30ms. This high packet loss and latency environment led to an 80% reduction in the performance of traditional IoT systems, severely impacting data transmission reliability and system availability. By rewriting the data acquisition module using the YoMo framework and utilizing the efficient transmission capabilities of the QUIC protocol, HS achieved low-latency data transmission even in challenging network conditions, with 99% of data written to the cloud within 50ms, greatly enhancing system stability.
2. Storing and Analyzing Massive IoT Data
HS's IoT system collects over 2TB of time-series data daily from sugar factories worldwide, encompassing metrics such as temperature, humidity, and production line efficiency. As data volumes surged, the traditional system faced bottlenecks in storage and query efficiency, particularly during peak production hours, where data write speeds and query latencies became critical, affecting real-time production management. To address these challenges, HS adopted GreptimeDB to optimize its storage and analysis architecture, yielding impressive results:
- Efficient Data Ingestion: GreptimeDB supports high-concurrency writes, processing up to 500,000 records per second during peak hours. It scaled horizontally, doubling write capacity while maintaining reasonable resource consumption, thereby preventing system overload.
- Storage Optimization: GreptimeDB's time-series data compression algorithms reduced storage costs by approximately 50%, enabling HS to significantly lower both hardware investments and cloud storage expenses.
- Real-Time Querying and Analysis: Leveraging GreptimeDB's query optimization technology, HS processes millions of time-series records in under one second and performs complex SQL aggregation queries, ensuring real-time monitoring and analysis of factory operations.
By utilizing GreptimeDB's storage optimization and high-performance writing capabilities, HS significantly enhanced its IoT data processing efficiency, ensuring system stability and analytical capacity in high-volume data environments.
Solution Architecture
The architecture of this integrated solution is centered around the following key components:
1. Efficient Data Transmission: Data collected from IoT devices undergoes pre-processing on edge devices via the YoMo framework. The optimized data is then transmitted to GreptimeDB using a low-latency, high-bandwidth network protocol, minimizing bandwidth usage while ensuring efficient and secure data transfer.
2. Centralized Data Storage and Analysis: GreptimeDB serves as the core data storage and analysis platform, adeptly managing data transmitted from edge devices and supporting high-throughput writes and low-latency queries for large-scale data analysis.
3. Decision Support: The integration of YoMo and GreptimeDB empowers businesses to swiftly capture key events and respond promptly, optimizing operational processes and enhancing decision-making efficiency.
Conclusion
In large-scale IoT deployments, the generation, collection, transmission, and storage of data necessitate extreme efficiency. The integration of YoMo and GreptimeDB provides businesses with a comprehensive solution, facilitating everything from data collection to real-time analysis. This solution enhances data processing efficiency, network transmission performance, and overall system operational efficacy, meeting the high demands of managing substantial data in IoT environments.
Appendix
You can find our sample code on the GitHub repository, demonstrating how to collect real-time GPU load data from AI servers, transmit it through YoMo, and store and analyze it in GreptimeDB.
Below is an example of a real-time GPU server performance monitoring dashboard:
The YoMo team manages hundreds of GPU servers across more than a dozen data centers globally. This real-time monitoring system ensures efficient resource scheduling and optimizes the distribution of computing tasks worldwide.
GreptimeDB, tailored to the needs of domestic and international clients, addresses market pain points in database management, leveraging a cloud-native architecture and strong horizontal scalability to provide comprehensive support for IoT device data storage and analysis.
About Greptime
We help industries that generate large amounts of time-series data, such as Connected Vehicles (CV), IoT, and Observability, to efficiently uncover the hidden value of data in real-time.
Visit the latest version from any device to get started and get the most out of your data.
- GreptimeDB, written in Rust, is a distributed, open-source, time-series database designed for scalability, efficiency, and powerful analytics.
- Edge-Cloud Integrated TSDB is designed for the unique demands of edge storage and compute in IoT. It tackles the exponential growth of edge data by integrating a multimodal edge-side database with cloud-based GreptimeDB Enterprise. This combination reduces traffic, computing, and storage costs while enhancing data timeliness and business insights.
- GreptimeCloud is a fully-managed cloud database-as-a-service (DBaaS) solution built on GreptimeDB. It efficiently supports applications in fields such as observability, IoT, and finance.
Star us on GitHub or join GreptimeDB Community on Slack to get connected. Also, you can go to our contribution page to find some interesting issues to start with.
