Integrating own map tiles into the blog

I added the “outdoor” category to this blog a few months ago. The main feature is the possibility to read GPX files I recorded on outdoor trips. After reading the GPX file, it is displayed on a map (via leaflet) and some metadata is displayed. This is how the first try looked like:

Maps in the browser

Maps in the browser usually work like this: a mapping library loads tiles for specific geo areas and zoom levels on demand, normally from an external server, and the library then renders/places the tiles on the map. A well-known example is openstreetmap, which uses tiles hosted by openstreetmap.

Hosting map tiles can lead to serious traffic. Imagine providing the possibility to download a sheer infinite amount of PNGs, where a new download is triggered on each pan or zoom on the map. That’s why tile hosters typically require registration or have non-free or restricted usage policies. See here in the openstreetmap wiki for examples.

What do I require?

One could argue that I could just select one of the ‘free’ services, but those normally fall into the ‘usage policy’ category. I simply do not want to bother great open-source projects with traffic from my personal blog so that they can use their funds for better purposes. Another point is the possibility of styling - the free tiles are usually raster tiles, which are prerendered PNGs already styled for you. If you do not like the style provided, you have to look elsewhere. Typically, that leads to another, non-free service.

In software engineering, the requirements play an important role in the success of a project. They basically define how the system should behave, which in turn should be based on the user requirements for this application.

So what are my requirements?

• I want a cool style for the map.
• I want to host the map without much hassle.

That could not be too difficult, could it?

Protomaps to the rescue

Some time ago, I read about a mapping file format that boils down a vector map dataset into a single file. Doing some research, I found it again - its name was pmtiles.

This file type is part of the protomaps ecosystem. It can integrate the whole world into a single file, which can directly be queried from the browser. That means you can put the file onto your own server and query it from your own page without much hassle (except using some additional client-side JS library)! Also, as it supports vector format, I can define my own styles for my map. This fulfills all my requirements!

Refining the Dataset

Protomaps provides the daily (?) updated dataset for me to download, which I can then use for my maps. There is one catch though - the file is almost 120GB. It contains the whole world, of course it will be large… But the way the data is stored within the file makes it easy to split it up into geo-specific areas, e.g., defined by a bounding box. You will still have a world outline view, but as you zoom in, you recognize missing data at some places. But that’s okay, as I exactly know where my outdoor tours are happening. Also, I do not need to provide more data for anybody else.

I found a great blog post by Simon Willison on his blog about this. He provides a great description of his tinkering with the file format, including where to get the data and how to crop it. Check it out for an in-depth description!

TLDR: use the pmtiles cli to:

• load dataset from protomaps.com
• limit the downloaded dataset to the bbox you know or found out on bboxfinder.com

Displaying and styling the Dataset

Having downloaded the dataset, I needed to find out how to display it on the leaflet map. That was not difficult; it just required integrating another library for leaflet, called protomaps-leaflet.

This is a very simple integration where you can select one of a few flavors for the style of the basemap. For more info, check this blogpost. The cool thing is you can choose one of the default flavors as your starting point and adapt the styling of specific layers to your liking.

I changed some of the settings but didn’t delve too deep into it. This is how the end result looks like now:

For now, the result is fine, but I will surely continue to improve the style. When directly comparing it to the state before, I see tweaking the city labels, changing zoom ranges for smaller streets, or adapting the map colors to the blog’s color scheme, etc.

Deploying the tiles

The blog is basically a git repo with some files that are fed into hugo, which then generates the pages per my theme and settings. Large binary files are not great in git repos, and the pmtiles can be very large and very binary.

I let an LLM write me a small script that does that for me so I can use it locally to get the tile data. However, the main reason why I have the script is so that it can by run in the CI, as I do not want to pollute the blogs git repo with massive binaries. Now every time I push a change the CI will load and crop the latest maps (I defined in a single json file) and deploy them together with the rest of the blog. I just have to make sure that they are copied in the right location on the server. Pretty neat!

package main

import (
	"encoding/json"
	"flag" // Import the flag package for command line arguments
	"fmt"
	"os"
	"os/exec"
)

func main() {
	//todo: add a flag to specify the path of the pmtiles binary
	// Define command line flags
	helpFlag := flag.Bool("h", false, "Display help message")
	outputDir := flag.String("o", "./static/tiles", "Output directory for .pmtiles files")
	pmtilesBinary := flag.String("b", "pmtiles", "Path to the pmtiles binary")
	flag.Parse() // Parse the command line arguments

	if *helpFlag {
		fmt.Println("Usage: go run your_program_name.go [options] <path_to_json_file>")
		fmt.Println("\nOptions:")
		flag.PrintDefaults() // Print default flag usage
		os.Exit(0)
	}

	// Check if the JSON file path is provided
	if flag.NArg() < 1 {
		fmt.Println("Please provide the path to the JSON file as a command line argument.")
		fmt.Println("Usage: go run your_program_name.go <path_to_json_file>")
		fmt.Println("Use -h for help.")
		os.Exit(1)
	}

	filePath := flag.Arg(0) // Get the first non-flag argument as the file path

	// Read the content of the file
	jsonData, err := os.ReadFile(filePath)
	if err != nil {
		fmt.Printf("Error reading file: %v\n", err)
		os.Exit(1)
	}

	// Create a map to store the parsed JSON data
	var dataMap map[string]string

	// Unmarshal the JSON data into the map
	err = json.Unmarshal([]byte(jsonData), &dataMap)
	if err != nil {
		fmt.Println("Error parsing JSON:", err)
		return
	}

	// Use the output directory specified by the command line flag
	for key, bbox := range dataMap {
		fmt.Printf("Fetching %v.pmtiles with bbox %v\n", key, bbox)
		outputFilePath := fmt.Sprintf("%s/%s.pmtiles", *outputDir, key)
		// Use the pmtilesBinary flag value to specify the command
		cmd := exec.Command(*pmtilesBinary, "extract", "https://data.source.coop/protomaps/openstreetmap/v4.pmtiles", outputFilePath, "--bbox="+bbox)
		cmd.Stdout = os.Stdout
		cmd.Stderr = os.Stderr
		if err := cmd.Run(); err != nil {
			fmt.Printf("Error loading %v: %v", key, err)
			os.Exit(1)
		}
	}
}