We use RRDTool for the weather station! 6


We completed our weather station a few weeks ago. Thats when I started working on a simple website, to display the received data. My first approach was to store the received samples in an SQL-database and to plot the data via Javascript. It worked fine for a few days, but when I tried to displayed all samples of a week / a month or a year I realized that this approach, was not going to work at all.

Rendering all the samples for 1 year (every 5 seconds so 6311520) via Javascript does not work of course. Just omitting data and taking one random value per day is also not a satisfying way of solving this.

To solve this properly you would have to calculate min/max/average for longer timeframes – ideal would be to have more then 1 length for these periods (for example 30 min for a week and 1 day for a year).. Other problems are handling periods, when no data is received.

I realized if i continued to use sql this was going to be a quite a big project. And probably quite messy.

I was disappointed. As I am lazy I had a look at Cacti, and thats when I discovered RRDTool – that Cacti uses to store and render the measurements.

What is RRD-Tool?

RRD stands for Round Robin Database. The concept is quite simple.

Defining the Database

Imagine you had 100 simple containers that you can fill with values. You start at the first one, then put the second value into the second one and continue until you reach container 100. After you are there you start at the first one again and fill until you are at container 100 again. Round Robin – you get the idea 😉

This would be quite useless in my case – because I want to store a whole year off weatherdata. Thats were the magic happens.

With rrdtool you can create a database that contains more then one RRA (Round Robin Archives).

The definition for my Database looks like this:

[code lang=“bash“]
rrdtool create weatherdata.rrd –step 60 \
DS:temperature:GAUGE:120:-50:100 \
DS:voltage:GAUGE:120:0:15 \
DS:humidity:GAUGE:120:0:100 \
DS:light:GAUGE:120:0:1024 \
DS:wind:GAUGE:120:0:100 \
DS:pressure:GAUGE:120:50000:200000 \
RRA:AVERAGE:0.5:1:1440 \
RRA:MIN:0.5:1:1440 \
RRA:MAX:0.5:1:1440 \
RRA:AVERAGE:0.5:30:17520 \
RRA:MIN:0.5:30:17520 \
RRA:MAX:0.5:30:17520
[/code]

Lets disect this:

The database will be stored in a weatherdata.rdd and excepts data every 60 seconds.

The lines like [code lang=“bash“]DS:humidity:GAUGE:120:0:100[/code] specify which values should be included in the RRD. In this case its the humidity, which can be between 0 and 100 percent.120 means, that after 120 seconds the value is filled out with an unknown.

The RRA lines define the different Round Robin Archives.

I use an RRA that contains a sample every 60 seconds for 1 week.
Then I have 3 bigger RRA’s that store the weather data for up to one year. Every 30 minutes a new entry in this RRA is created. One for the maximal-value off the 30 samples, 1 for the minimal value and 1 for the average.

Thats quite nice – because RRD-Tool handles all the uglyness for me. For longer periods of time it stores less data and after a year it will not get any bigger.

Putting stuff into the database

After the database file is created you can put data into it.

[code lang=“bash“]
rrd update weatherdata.rdd N:10.55:11.452:88:20:5:95654
[/code]

The N means now and the you need to append a „:“-seperated list of values. The order is the same as you ordered your values in the database definition.

Graph values

The last part is drawing some graphs. I found this part the most confusing one.

There are very nice examples on the rrdtool page, but there is no code next to them. The documentation is kind of confusing for me as a beginner.

[code]
rrdtool graph temperature.png \
–height=200 –width=750 \
–color=BACK#FFFFFF \
–start end-1d –vertical-label "°C" –title "Temperatur letzte 24 Stunden" \
DEF:temperature=weatherdata.rdd:temperature:AVERAGE \
LINE2:temperature#3366CC
[/code]

But after a little while I managed to draw some basic graphs.

The result

Graphs using RRDTool

You can view the project on GitHub. After I discovered rrdtool I dropped Rails and replaced it with Sinatra and Rake.

I still use SQL as alternative if I want to have the full resolution of my samples in a few years from now.

The graphs are generated with a cron-job every 5 minutes. With rrdtool a complex problem was reduced to a simple and manageable task. Thanks to the creator for such a wonderfull tool. It will remain in my toolbox, to solve similar problems.

Ressources

Also I found these ressources quite helpful.

Ressources


Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

6 Gedanken zu “We use RRDTool for the weather station!

  • JJ

    hi, im doing a project on an arduino based weather station, i just wanted to know if you could tell me how to capture the temperature value into a database, i was thinking sqlite. i already have the code for displaying the temperature in the arduino, but i want to store that data into a database so i can get that data and display it on a local website.
    thanks

  • Rocu

    Hey JJ,

    we don’t store the data on the Arduino.. We transfer it with onto a computer and process it there.

    I also started with a SQLite database but it has some drawbacks. You have to render all the graphs yourself… Its quite involved to do this because after you have a few months of data in your database you have to start to calculate averages for every day etc.. Thats why we used a Round Robin Database.

    A round robin database has the drawback, that it looses data after some time or looses precision if the data goes to containers with a lower resolution.

    But back to your question. If you want to try it with SQLite you have to get it onto a computer.

    Some ideas how to do it.

    1) Use the serial interface via USB
    2) Use a Zigbee Shield
    3) Bluetooth
    4) Also you could use the Ethernet shield to push the data to a simple PHP script
    5) Pull the data from a server on the arduino

    On the computer you need a script that stores the values in SQLite..

    Here is an example how I did it with ruby. Its a hybrid approach with SQL + RRD tool.. The SQL database is only used to not loose values.

    https://github.com/shostakovich/ZWeather/blob/master/tasks/fetch_serial_data.rake

    The complete project you can find here:

    https://github.com/shostakovich/ZWeather

    Regars,
    Robert

  • jamores

    Hi,
    correct me if I’m wrong, but in the example database, you’re collecting 60-second resolution samples for a single day, not an entire week:

    RRA:AVERAGE:0.5:1:1440 –> 60secs * 1440 = 1 day

    to collect 60-second for a week

    RRA:AVERAGE:0.5:1:10080 –> 60secs * 10080 = 60secs * 1440 * 7 = 1week

    Regards

  • Mikhail Stolyarov

    Hello!
    Great post!
    I’m working on same type project on arduino and looking to rrdtool because of it’s graphs are easy to create.
    But one problem still there – I want to store endless data from sensors (for example from now to infinite =). But if you use rrdtool archives – it’s impossible? Am I right?