On this page
In the past few years, the rising popularity of the Internet of Things (IoT) and the need for real-time data has resulted in a significant surge in the adoption of time series databases (TSDBs). According to the DB-Engines ranking, TSDBs have experienced greater growth in popularity than any other type of database, second only to Graph DBMS.
As a critical tool for storage, management, and analysis of time-sensitive data, the popularity of TSDBs is expected to persist in the near future. If you are unfamiliar with TSDBs, this article provides a comprehensive introduction to what they are and why specialized databases are necessary for time-series data.
Source: DB-Engine
What is time series data (TSD)
To understand what a time series database is, we must initially grasp the concept of time series data and distinguish it from other types of data.
Definition
Time series data is a type of data that is collected over time and is ordered chronologically. Typically, these data points track changes of a particular variable and consist of successive measurements over a fixed time interval made from the same source.
Time series data can be represented graphically using line charts, scatter plots, or other visualization methods to highlight trends, patterns, and anomalies in the data over time.
The graph above displays a typical example of a time series data plot. The x-axis is always reserved for time while the y-axis depicts the measured variable value, namely temperature and humidity over the timeframe of January to March in 2013.
Characteristics
The Time Series data type is distinct from other types of data. Numerical and categorical data types do not have an inherent temporal dimension. Numerical data can take on any continuous or discrete value, whereas categorical data is composed of a group of distinct categories or labels.
Particularly, some of the characteristics of time series data include:
- Time dependency: Every data point has a timestamp, and the time series data is sequential, with each observation depending on the previous observation in some way.
- High Volume: Time series data can be generated at high frequencies and can quickly accumulate into very large datasets.
- Trend: Time series data may exhibit a long-term trend, such as an increasing or decreasing pattern over time.
- Seasonality: Time series data may also exhibit a pattern that repeats at regular intervals, such as daily, weekly, or yearly seasonality.
- Irregularity: Time series data may also contain irregular or unpredictable fluctuations that are not related to any specific pattern or trend.
Understanding these characteristics is important for analyzing and modelling time series data, as it allows for appropriate statistical methods and models to be used.
Examples
Time series data is utilized for analyzing and forecasting historical or real-time trends, and to predict patterns over time. Here are some common examples of time series applications in everyday life.
- Stock Prices: Time series plots are helpful because they allow stock analysts and traders to understand the trend and direction of a certain stock price.
- Weather: predict what the temperatures will be during different months and seasons throughout the year.
- Health monitoring: Time series analysis is also used in the medical field to monitor the heart rate or other health measurements of patients who may be on certain medications.
- Website traffic data: The number of visitors to a website or social media followers over time, reflecting a trend for the corresponding popularity of a certain media.
- Sensor data: Data collected by sensors such as temperature sensors, humidity sensors, and motion sensors over time, commonly used on the industrial internet of things.
Why is time series data important?
Even though time series data is not at all a new data type, its popularity and usage have significantly increased in the past few years, as depicted in the first graph of this article. Several factors have contributed to this trend, including:
The growth of the internet and the digitalization of many industries have led to the collection of vast amounts of time-stamped data, such as website traffic, social media activity, and sensor readings.
The development of machine learning algorithms that are well-suited for time series data analysis, such as recurrent neural networks (RNNs) and long short-term memory (LSTM) networks, has made it easier to extract valuable insights from this type of data.
The rise of predictive analytics has made time series data an essential tool for forecasting trends and making accurate predictions about future outcomes.
The increasing need for real-time decision-making in fields such as finance, healthcare, and transportation has made time series data analysis crucial for understanding and responding to rapidly changing conditions.
What are time series databases (TSDB)
In order to efficiently manage and analyze large volumes of time series data, specialized databases are needed. These databases are optimized to provide better performance, scalability, and flexibility for managing time series data compared to traditional databases, given the unique characteristics of time series data mentioned above.
Time series data is almost always appended in comparison to updates or deletions. That means databases can have huge workloads, and users would want statistics or aggregates collected over a time period.
Time series databases vs Traditional databases
Time series databases are specifically designed to handle time-stamped data, which has unique characteristics that set it apart from other types of data. Traditional relational databases, such as MySQL, are not as well-suited for time series data due to several reasons:
Data structure: Relational databases use tables to store data, which is ideal for structured data with fixed schema. Time series data, on the other hand, is typically semi-structured or unstructured, with variable schema that may evolve over time. Time series databases are designed to handle these dynamic schemas efficiently.
Time-based indexing: Time series data is primarily indexed and queried based on time, which can lead to performance issues in traditional databases as they are not optimized for time-based indexing. Time series databases are designed to handle time-based indexing efficiently, allowing for fast data ingestion and querying.
Data retention and compression: Time series data often requires specialized data retention policies and compression techniques to manage data storage effectively. Traditional databases may not provide efficient or flexible data retention and compression options, while time series databases are designed with these requirements in mind.
Time-based aggregations and analysis: Analyzing time series data often involves time-based aggregations and statistical calculations. Traditional databases are not optimized for these types of operations, making them less efficient and less performant for time series data analysis. Time series databases, however, provide built-in functions and optimizations specifically for time-based analysis.
Time series databases features
And to address the challenges associated with time series data, TSDBs commonly employ some technologies. Some of the typical features include:
Log-Structured Merge-tree (LSM-tree): A disk-based data structure optimized for write-heavy workloads, LSM-trees enable efficient data ingestion and storage by merging and compacting data in a series of levels. This reduces write amplification and provides better write performance compared to traditional B-trees.
Time-based partitioning: Time series databases often partition data based on time intervals, enabling faster and more efficient queries, as well as easier data retention and management. This approach helps isolate recent, frequently accessed data from older, less frequently accessed data, optimizing storage and query performance.
Data compression: Time series databases employ various compression techniques, such as delta encoding, Gorilla compression, or dictionary encoding, to reduce storage space requirements. These techniques exploit the temporal and value-based patterns in time series data, allowing for efficient storage without significant loss of data fidelity.
Built-in time-based functions and aggregations: Time series databases provide native support for time-based functions, such as moving averages, percentiles, and time-based aggregations. These built-in functions enable users to perform complex time series analysis more efficiently and with less computational overhead compared to traditional databases.
In summary, while traditional databases like MySQL can store and manage time series data to some extent, they are not optimized for its unique characteristics. Time series databases are specifically designed to address the challenges associated with time series data, offering better performance, scalability, and functionality for time-based analysis.
Conclusion
Time series data is a unique form of data characterized by its time-based nature, which is becoming increasingly important in various industries and applications. As the volume and frequency of time series data continue to grow, it is crucial to have the right tools to store, manage, and analyze it efficiently.
Time series databases, specifically designed to handle the challenges associated with time series data, provide a powerful solution for these needs. TSDB has already been used in applications such as IoT (Internet of Things), financial data analysis, monitoring and alerting systems, energy management, healthcare applications and other time-sensitive use cases. By using them, organizations can unlock valuable insights from their data, driving informed decision-making and gaining a competitive edge.
Reference: Wikipedia, Time series database
