November 23rd, 2023
Hi, this is Hendrik. You won’t have seen me writing Weekly Charts very often, because I’m one of Datawrapper’s backend developers.
As a backend developer you literally drown in data.
Everything I work with creates data, be it some kind of user input, server logs, or any other output our cloud services generate. It’s gigabytes every day. So when it comes to my Weekly Chart I’m always a bit like: Wait, the data isn’t already there? The data journalism part of the task is always a bit hard for me.
More often, my work involves data transformation. When data is output by one system and fed into another, most of the time the format needs to be changed to make it readable for the second process.
This is actually something that you can also apply to Datawrapper. Sometimes the most obvious chart or map type in Datawrapper might not be what you want — and to use your data for another chart type, you may need to transform it.
To show off how far this can go, I decided to transform a three-dimensional globe into a two-dimensional scatterplot. And then animate it.
Probably the most far-fetched thing you could do, which makes it interesting for me.
To create this map, I wanted to take some world scale polygons from the data backing our locator maps and feed them into a Datawrapper scatterplot.
Input: EPSG:4326 formatted polygons
Output: Styled dots in a two dimensional X-Y plane
So what do I need to do to make that transformation?
At the end of that, I had a scatterplot showing one frame of my animation.
Having verified that the above approach works, I used Blender’s animation features to spin the globe by 360°, creating 450 new views, and generated a Datawrapper-friendly CSV file for each of them.
Now all that remained was to update the chart with each of the CSVs and export a PNG of each version.
It’s obvious that no one would enjoy repeating the same procedure over and over again, so I used our own API to do it. I have to confess that although I’ve worked on parts of the API, I never really worked with the API, so it took me a while to automate the process.
It turned out my first try was flawed. Because of the way I calculated the 2D projection, some dots were dropped due to rounding. But this effect was so random between the frames that the animation did not look good. Dots were flickering.
I fixed that by doubling the resolution and exporting 2x2 clusters. That way at least something of the 2x2 cluster will survive the projection.
Instead of relying on all these graphics processing tools, I could also just have transformed the polygons in code, as it only involves two rotations to put a point given in latitude/longitude onto a sphere. But then I couldn't have seen the output until it was already in the final Datawrapper chart, and manually tweaking the data (which I did a lot) would have hardly been possible.
Sure, this is not the normal tooling I typically use for backend development, but it shows one thing: To get your data into the format you want, you sometimes need to walk away from well-known paths!
Even a backend developer has to use unusual tools sometimes to get his job done.
That's it from me, watch out for more serious stuff from our skilled data journalists in the coming weeks.