… like OS load, and Java Beans, and packet counts and pings.
The Evolution of TSDB
Here at Betfair, we’ve been enthusiastic users of OpenTSDB for a couple of years now. We use it to gather and store metrics across the entirety of our production estate, and the graphing that it can produce is frequently the first place that our engineers turn when trying to diagnose a fault.
Of course, now that people have seen how useful it can be, they want more from it, and they want it faster. And so we’re developing one particular component of it in new and (hopefully) exciting directions.
But, before we get into the details, a very quick recap of the basic architecture of OpenTSDB
TSDB itself is an HBase database which sits on top of an HDFS cluster and stores the metrics sent to it by various instances of tcollector.
tcollector is a python script which sits on the servers themselves and runs various plugins to generate the metrics which it then passes on to the database.
The OS plugin, surprisingly enough, gathers metrics from the base OS. The JMX plugin gathers metrics from JVMs running on the server, and there are various other application plugins available as required. Statse and poller are slightly different – poller is used to gather information from devices which can’t run the plugins directly (routers, netscalers, etc), and Statse turns event based data into time series type data (it turns a series of ‘this thing just happened messages’ into ‘this thing happened this many times’ metrics)
Now that that’s all clear (I told you it would be quick – you can get much more information from the docs at the OpenTSDB website linked above), let’s move on to why Betfair decided to put the time and effort into redeveloping one of the components.
We can break down the motivation into two basic sections:
- Performance Enhancements
- Additional functionality
The performance side is fairly simple – tcollector is written in Python, and while it’s certainly well written python, once we started running up the number of plugins and metrics we wanted to collect, it started placing a noticeable, and unacceptable, load on our production systems.
On the other side, enhancing the functionality, once TSDB started becoming entrenched into our engineers workflow, people started asking about whether they could use the data, not just for historical analysis, but also for real-time monitoring and alerting. Unfortunately, with the vanilla architecture of TSDB, this is not easy to do – having a large number of monitoring tools continually polling the database would have resulted in an unacceptable load.
These, along with some other ideas, are what led to the development of the forwarder. In its basic form, the forwarder is a drop-in replacement for tcollector, but it has several new advantages. It has been written in Go, which means it has a much reduced memory and CPU usage profile, and it supports some new features
The first and most obvious difference is that the forwarder will output metrics to multiple destinations.
In addition to this, the forwarder now supports some basic metric filtering and augmentation (for example, adding a tag to each metric to report on which cluster the host reporting it is in), and these can all be controlled from a single centralised configuration host.
While the diagram above shows the collector forwarding to several different endpoints, the usage that we have settled on for our particular use case forwards them all to two separate locations:
- The TSDB itself, as with vanilla tcollector
- The aggregator
The aggregator is simply another instance of the forwarder, running with a different set of options. It accepts input streams of metrics and forwards them all as a single stream to various different outputs.
Among the endpoints that we currently feed our metric stream into are:
- Riemann (for tracking and alerting on user defined triggers)
- Kale (for anomaly detection)
- Real time plotting
- A separately developed plugin for monitoring the health of the metric feed itself
There are many, many other things that we could do with this: it’s a feed of all of the metrics gathered from all of our production estate, and all arriving within a fraction of a second of the metric being gathered.
This combination of forwarder & aggregator is shortly to be released to the community as Open Source code under a BSD licence as the Time Series Pipeline (OpenTSP), and we hope that it proves as useful to other people as it has been to us.
About /me: James is part of Betfair’s SRE team.