As I add more data to the NYC Chaos Dashboard, a website that maps live urban activity, I have been looking for a more efficient way to render the map. Since I collect all of the data in one process and return the Dashboard as one HTML file, I kept wondering how I could optimize the map’s loading time by pre-processing the data as much as possible in the backend. This is where vector tiles come in.
The code shown in this post is written in Go.
Why generate tiles?
Initially, all of the map’s data was passed to the rendering library in GeoJSON format (embedded directly in the HTML file). For those who don’t know, GeoJSON is a JSON based standard to represent geographic information. You can go see the full RFC here, but here’s a quick preview of what it looks like so you can get an idea:
{ " type " : " Feature " , " geometry " : { " type " : " Point " , " coordinates " : [ -74.04452395542852 , 40.68987850656795 ] }, " properties " : { " name " : " Statue of Liberty " , " status " : " open " } }
Now, I’m definitely not JSON’s greatest fan. It’s all text, meaning that a number is stored in a base 10 ASCII representation, where a number like 42 gets stored as "4" and "2" . I could go on, but I think you see the problem: it’s not the most efficient way to store data. Nonetheless, JSON has a lot of merits: it’s human-readable and easy to share between systems, so I find myself using it more than I’d like to - more often than not, simplicity is the way to go, and a simple format like GeoJSON just gets the job done, and that alone makes it a worthy geographic standard.
So what happens when I start adding more layers to the map? It gets slow. I’m working on adding flood sensor data (thank you Floodnet for granting me access to the API), LIRR and MetroNorth data, NYISO power data, and many more datasets which will start adding a lot of layers to the map. I can already see that the HTML file, at the time of writing, is 4.5Mb (once decompressed) and takes ~770ms to transfer from Cloudflare’s CDN to my browser. This seems pretty reasonable for now, but it won’t scale as the future datasets are much larger, and running a quick check on the website, chrome is already telling me that the site has performance issues:
And if I look more closely, I can see that it’s the result of a long rendering (over 2s!), which I can see here:
Now, I suspect this is the result of a few things:
As mentioned above, all of the embedded data is GeoJSON. This means the rendering library (MapLibre GL JS) needs to parse the JSON. There’s a lot, with a lot of properties, so this takes time. MapLibre GL needs to then take the coordinates and then place the lines, points and polygons on the map accordingly. This takes time, and it all happens on your browser.
... continue reading