A Practical Toolkit for Time Series Anomaly Detection, Using Python

fascinating aspects of time series is the intrinsic complexity of such an apparently simple kind of data.

At the end of the day, in time series, you have an x axis that usually represents time (t), and a y axis that represents the quantity of interest (stock price, temperature, traffic, clicks, etc…). This is significantly simpler than a video, for example, where you might have thousands of images, and each image is a tensor of width, height, and three channels (RGB).

However, the evolution of the quantity of interest (y axis) over time (x axis) is where the complexity is hidden. Does this evolution present a trend? Does it have any data points that clearly deflect from the expected signal? Is it stable or unpredictable? Is the average value of the quantity larger than what we would expect? Those can all somehow be defined as anomalies.

This article is a collection of multiple anomaly detection techniques. The goal is that, given a dataset of multiple time series, we can detect which time series is anomalous and why.

These are the 4 time series anomalies we are going to detect:

1. We are going to detect any trend in our time series (trend anomaly)
2. We are going to evaluate how volatile the time series is (volatility anomaly).
3. We are going to detect the point anomalies within the time series (single-point anomaly).
4. We are going to detect the anomalies within our bank of signals, to understand what signal behaves differently from our set of signals (dataset-level anomaly).

Image made by author

We are going to theoretically describe each anomaly detection method from this collection, and we are going to show the Python implementation. The whole code I used for this blog post is included in the PieroPaialungaAI/timeseriesanomaly GitHub folder

0. The dataset

In order to build the anomaly collector, we need to have a dataset where we know exactly what anomaly we are searching for, so that we know if our anomaly detector is working or not. In order to do that, I have created a data.py script. The script contains a DataGenerator object that:

1. Reads the configuration of our dataset from a config.json* file.
2. Creates a dataset of anomalies
3. Gives you the ability to easily store the data and plot them.

This is the code snippet:

Image made by author

So we can see that we have:

1. A shared time axis, from 0 to 100
2. Multiple time series that form a time series dataset
3. Each time series presents one or many anomalies.

The anomalies are, as expected:

1. The trend behavior, where the time series have a linear or polynomial degree behavior
2. The volatility, where the time series is more volatile and changing than normal
3. The level shift, where the time series has a higher average than normal
4. A point anomaly, where the time series has one anomalous point.

Now our goal will be to have a toolbox that can identify each one of these anomalies for the whole dataset.

*The config.json file allows you to modify all the parameters of our dataset, such as the number of time series, the time series axis and the kind of anomalies. This is how it looks like:

1. Trend Anomaly Identification

1.1 Theory

When we say “a trend anomaly”, we are looking for a structural behavior: the series moves upward or downward over time, or it bends in a consistent way. This matters in real data because drift often means sensor degradation, changing user behavior, model/data pipeline issues, or another underlying phenomenon to be investigated in your dataset.

We consider two kinds of trends:

• Linear regression: we fit the time series with a linear trend
• Polynomial regression: we fit the time series with a low-degree polynomial.

In practice, we measure the error of the Linear Regression model. If it is too large, we fit the Polynomial Regression one. We consider a trend to be “significant” when the p value is lower than a set threshold (commonly p < 0.05).

1.2 Code

The AnomalyDetector object in anomaly_detector.py will run the code described above using the following functions:

• The detector, which will load the data we have generated in DataGenerator.
• detect_trend_anomaly and detect_all_trends detect the (eventual) trend for a single time series and for the whole dataset, respectively
• get_series_with_trend returns the indices that have a significant trend.

We can use plot_trend_anomalies to display the time series and see how we are doing:

Image made by author

Good! So we are able to retrieve the “trendy” time series in our dataset without any bugs. Let’s move on!

2. Volatility Anomaly Identification

2.1 Theory

Now that we have a global trend, we can focus on volatility. What I mean by volatility is, in plain English, how all over the place is our time series? In more precise terms, how does the variance of the time series compare to the average one of our dataset?

This is how we are going to test this anomaly:

1. We are going to remove the trend from the timeseries dataset.
2. We are going to find the statistics of the variance.
3. We are going to find the outliers of these statistics

Pretty simple, right? Let’s dive in with the code!

2.2 Code

Similarly to what we have done for the trends, we have:

• detect_volatility_anomaly, which checks if a given time series has a volatility anomaly or not.
• detect_all_volatilities, and get_series_with_high_volatility, which check the whole time series datasets for volatility anomaly and return the anomalous indices, respectively.

This is how we display the results:

Image made by author

3. Single-point Anomaly

3.1 Theory

Ok, now let’s ignore all the other time series of the dataset and let’s focus on each time series at a time. For our time series of interest, we want to see if we have one point that is clearly anomalous. There are many ways to do that; we can leverage Transformers, 1D CNN, LSTM, Encoder-Decoder, etc. For the sake of simplicity, let’s use a very simple algorithm:

1. We are going to adopt a rolling window approach, where a fixed sized window will move from left to right
2. For each point, we compute the mean and standard deviation of its surrounding window (excluding the point itself)
3. We calculate how many standard deviations the point is away from its local neighborhood using the Z-score

We define a point as anomalous when it exceeds a fixed Z-score value. We are going to use Z-score = 3 which means 3 times the standard deviations.

3.2 Code

Similarly to what we have done for the trends and volatility, we have:

• detect_point_anomaly, which checks if a given time series has any single-point anomalies using the rolling window Z-score method.
• detect_all_point_anomalies and get_series_with_point_anomalies, which check the whole time series dataset for point anomalies and return the indices of series that contain at least one anomalous point, respectively.

And this is how it is performing:

Image made by author

4. Dataset-level Anomaly

4.1 Theory

This part is intentionally simple. Here we are not looking for weird points in time, we are looking for weird signals in the bank. What we want to answer is:

Is there any time series whose overall magnitude is significantly larger (or smaller) than what we expect given the rest of the dataset?

To do that, we compress each time series into a single “baseline” number (a typical level), and then we compare those baselines across the whole bank. The comparison will be done in terms of the median and Z score.

4.2 Code

This is how we do the dataset-level anomaly:

1. detect_dataset_level_anomalies(), finds the dataset-level anomaly across the whole dataset.
2. get_dataset_level_anomalies(), finds the indices that present a dataset-level anomaly.
3. plot_dataset_level_anomalies(), displays a sample of time series that present anomalies.

This is the code to do so:

5. All together!

Ok, it’s time to put it all together. We will use detector.detect_all_anomalies() and we will evaluate anomalies for the whole dataset based on trend, volatility, single-point and dataset-level anomalies. The script to do this is very simple:

The df will give you the anomaly for each time series. This is how it looks like:

If we use the following function we can see that in action:

Image made by author

Pretty impressive right? We did it. 🙂

6. Conclusions

Thank you for spending time with us, it means a lot. ❤️ Here’s what we have done together:

• Built a small anomaly detection toolkit for a bank of time series.
• Detected trend anomalies using linear regression, and polynomial regression when the linear fit is not enough.
• Detected volatility anomalies by detrending first and then comparing variance across the dataset.
• Detected single-point anomalies with a rolling window Z-score (simple, fast, and surprisingly effective).
• Detected dataset-level anomalies by compressing each series into a baseline (median) and flagging signals that live on a different magnitude scale.
• Put everything together in a single pipeline that returns a clean summary table we can inspect or plot.

In many real projects, a toolbox like the one we built here gets you very far, because:

• It gives you explainable signals (trend, volatility, baseline shift, local outliers).
• It gives you a strong baseline before you move to heavier models.
• It scales well when you have many signals, which is where anomaly detection usually becomes painful.

Keep in mind that the baseline is simple on purpose, and it uses very simple statistics. However, the modularity of the code allows you to easily add complexity by just adding the functionality in the anomaly_detector_utils.py and anomaly_detector.py.

7. Before you head out!

Thank you again for your time. It means a lot ❤️

My name is Piero Paialunga, and I’m this guy here:

Image made by author

I’m originally from Italy, hold a Ph.D. from the University of Cincinnati, and work as a Data Scientist at The Trade Desk in New York City. I write about AI, Machine Learning, and the evolving role of data scientists both here on TDS and on LinkedIn. If you liked the article and want to know more about machine learning and follow my studies, you can:

A. Follow me on Linkedin, where I publish all my stories
B. Follow me on GitHub, where you can see all my code
C. For questions, you can send me an email at piero.paialunga@hotmail