Animating a globe with a scatterplot

Hendrik Bartusch

Hi, this is Hendrik. You won’t have seen me writing Weekly Charts very often, because I’m one of Datawrapper’s backend developers.

As a backend developer you literally drown in data.

Everything I work with creates data, be it some kind of user input, server logs, or any other output our cloud services generate. It’s gigabytes every day. So when it comes to my Weekly Chart I’m always a bit like: Wait, the data isn’t already there? The data journalism part of the task is always a bit hard for me.

More often, my work involves data transformation. When data is output by one system and fed into another, most of the time the format needs to be changed to make it readable for the second process.

This is actually something that you can also apply to Datawrapper. Sometimes the most obvious chart or map type in Datawrapper might not be what you want — and to use your data for another chart type, you may need to transform it.

To show off how far this can go, I decided to transform a three-dimensional globe into a two-dimensional scatterplot. And then animate it.

Probably the most far-fetched thing you could do, which makes it interesting for me.

The transformation

To create this map, I wanted to take some world scale polygons from the data backing our locator maps and feed them into a Datawrapper scatterplot.

Input: EPSG:4326 formatted polygons

Output: Styled dots in a two dimensional X-Y plane

So what do I need to do to make that transformation?

1. Import the map polygons into a PostGIS database and do a little cleaning. I usually use this database for preparing locator map tiles.
2. Export the polygons via ST_AsSVG.
3. Import the polygons into InkScape, a graphics editor.
4. Export as high-res PNG.
5. Import the PNG as the texture of a sphere in Blender, which is specifically for 3D graphics.
6. Export the sphere to a format readable in MeshLab, another tool for 3D.
7. Use Meshlab to transfer the colors into vertex properties and fine-tune the vertices for later use.
8. Re-import into Blender to render a 2D projection of the sphere.
9. Use some self-written stuff to map the colors to labels and write everything out to a Datawrapper-ready CSV.

At the end of that, I had a scatterplot showing one frame of my animation.

Having verified that the above approach works, I used Blender’s animation features to spin the globe by 360°, creating 450 new views, and generated a Datawrapper-friendly CSV file for each of them.

Now all that remained was to update the chart with each of the CSVs and export a PNG of each version.

It’s obvious that no one would enjoy repeating the same procedure over and over again, so I used our own API to do it. I have to confess that although I’ve worked on parts of the API, I never really worked with the API, so it took me a while to automate the process.

It turned out my first try was flawed. Because of the way I calculated the 2D projection, some dots were dropped due to rounding. But this effect was so random between the frames that the animation did not look good. Dots were flickering.

I fixed that by doubling the resolution and exporting 2x2 clusters. That way at least something of the 2x2 cluster will survive the projection.

Instead of relying on all these graphics processing tools, I could also just have transformed the polygons in code, as it only involves two rotations to put a point given in latitude/longitude onto a sphere. But then I couldn't have seen the output until it was already in the final Datawrapper chart, and manually tweaking the data (which I did a lot) would have hardly been possible.

Sure, this is not the normal tooling I typically use for backend development, but it shows one thing: To get your data into the format you want, you sometimes need to walk away from well-known paths!

Even a backend developer has to use unusual tools sometimes to get his job done.

That's it from me, watch out for more serious stuff from our skilled data journalists in the coming weeks.

---

Hendrik Bartusch

(he/him, @eXo_X5) is a backend developer at Datawrapper. He takes care of our servers and the map data used in Datawrapper locator maps. Hendrik believes that only code that you write yourself is code you truly understand, and likes coding late at night. He lives in Magdeburg.

Liked this article? Maybe your friends will too:

Twitter Facebook