mlthlschr

Integrating own map tiles into the blog

Sat, 19 Apr 2025 19:33:11 +0000

I added the “outdoor” category to this blog a few months ago. The main feature is the possibility to read GPX files I recorded on outdoor trips. After reading the GPX file, it is displayed on a map (via leaflet) and some metadata is displayed. This is how the first try looked like:

Maps in the browser

Maps in the browser usually work like this: a mapping library loads tiles for specific geo areas and zoom levels on demand, normally from an external server, and the library then renders/places the tiles on the map. A well-known example is openstreetmap, which uses tiles hosted by openstreetmap.

Hosting map tiles can lead to serious traffic. Imagine providing the possibility to download a sheer infinite amount of PNGs, where a new download is triggered on each pan or zoom on the map. That’s why tile hosters typically require registration or have non-free or restricted usage policies. See here in the openstreetmap wiki for examples.

What do I require?

One could argue that I could just select one of the ‘free’ services, but those normally fall into the ‘usage policy’ category. I simply do not want to bother great open-source projects with traffic from my personal blog so that they can use their funds for better purposes. Another point is the possibility of styling - the free tiles are usually raster tiles, which are prerendered PNGs already styled for you. If you do not like the style provided, you have to look elsewhere. Typically, that leads to another, non-free service.

In software engineering, the requirements play an important role in the success of a project. They basically define how the system should behave, which in turn should be based on the user requirements for this application.

So what are my requirements?

I want a cool style for the map.
I want to host the map without much hassle.

That could not be too difficult, could it?

Protomaps to the rescue

Some time ago, I read about a mapping file format that boils down a vector map dataset into a single file. Doing some research, I found it again - its name was pmtiles.

This file type is part of the protomaps ecosystem. It can integrate the whole world into a single file, which can directly be queried from the browser. That means you can put the file onto your own server and query it from your own page without much hassle (except using some additional client-side JS library)! Also, as it supports vector format, I can define my own styles for my map. This fulfills all my requirements!

Refining the Dataset

Protomaps provides the daily (?) updated dataset for me to download, which I can then use for my maps. There is one catch though - the file is almost 120GB. It contains the whole world, of course it will be large… But the way the data is stored within the file makes it easy to split it up into geo-specific areas, e.g., defined by a bounding box. You will still have a world outline view, but as you zoom in, you recognize missing data at some places. But that’s okay, as I exactly know where my outdoor tours are happening. Also, I do not need to provide more data for anybody else.

I found a great blog post by Simon Willison on his blog about this. He provides a great description of his tinkering with the file format, including where to get the data and how to crop it. Check it out for an in-depth description!

TLDR: use the pmtiles cli to:

load dataset from protomaps.com
limit the downloaded dataset to the bbox you know or found out on bboxfinder.com

Displaying and styling the Dataset

Having downloaded the dataset, I needed to find out how to display it on the leaflet map. That was not difficult; it just required integrating another library for leaflet, called protomaps-leaflet.

This is a very simple integration where you can select one of a few flavors for the style of the basemap. For more info, check this blogpost. The cool thing is you can choose one of the default flavors as your starting point and adapt the styling of specific layers to your liking.

I changed some of the settings but didn’t delve too deep into it. This is how the end result looks like now:

For now, the result is fine, but I will surely continue to improve the style. When directly comparing it to the state before, I see tweaking the city labels, changing zoom ranges for smaller streets, or adapting the map colors to the blog’s color scheme, etc.

Deploying the tiles

The blog is basically a git repo with some files that are fed into hugo, which then generates the pages per my theme and settings. Large binary files are not great in git repos, and the pmtiles can be very large and very binary.

I let an LLM write me a small script that does that for me so I can use it locally to get the tile data. However, the main reason why I have the script is so that it can by run in the CI, as I do not want to pollute the blogs git repo with massive binaries. Now every time I push a change the CI will load and crop the latest maps (I defined in a single json file) and deploy them together with the rest of the blog. I just have to make sure that they are copied in the right location on the server. Pretty neat!

Show code

package main

import (
	"encoding/json"
	"flag" // Import the flag package for command line arguments
	"fmt"
	"os"
	"os/exec"
)

func main() {
	//todo: add a flag to specify the path of the pmtiles binary
	// Define command line flags
	helpFlag := flag.Bool("h", false, "Display help message")
	outputDir := flag.String("o", "./static/tiles", "Output directory for .pmtiles files")
	pmtilesBinary := flag.String("b", "pmtiles", "Path to the pmtiles binary")
	flag.Parse() // Parse the command line arguments

	if *helpFlag {
		fmt.Println("Usage: go run your_program_name.go [options] <path_to_json_file>")
		fmt.Println("\nOptions:")
		flag.PrintDefaults() // Print default flag usage
		os.Exit(0)
	}

	// Check if the JSON file path is provided
	if flag.NArg() < 1 {
		fmt.Println("Please provide the path to the JSON file as a command line argument.")
		fmt.Println("Usage: go run your_program_name.go <path_to_json_file>")
		fmt.Println("Use -h for help.")
		os.Exit(1)
	}

	filePath := flag.Arg(0) // Get the first non-flag argument as the file path

	// Read the content of the file
	jsonData, err := os.ReadFile(filePath)
	if err != nil {
		fmt.Printf("Error reading file: %v\n", err)
		os.Exit(1)
	}

	// Create a map to store the parsed JSON data
	var dataMap map[string]string

	// Unmarshal the JSON data into the map
	err = json.Unmarshal([]byte(jsonData), &dataMap)
	if err != nil {
		fmt.Println("Error parsing JSON:", err)
		return
	}

	// Use the output directory specified by the command line flag
	for key, bbox := range dataMap {
		fmt.Printf("Fetching %v.pmtiles with bbox %v\n", key, bbox)
		outputFilePath := fmt.Sprintf("%s/%s.pmtiles", *outputDir, key)
		// Use the pmtilesBinary flag value to specify the command
		cmd := exec.Command(*pmtilesBinary, "extract", "https://data.source.coop/protomaps/openstreetmap/v4.pmtiles", outputFilePath, "--bbox="+bbox)
		cmd.Stdout = os.Stdout
		cmd.Stderr = os.Stderr
		if err := cmd.Run(); err != nil {
			fmt.Printf("Error loading %v: %v", key, err)
			os.Exit(1)
		}
	}
}

Another one bites the dust

Sun, 13 Apr 2025 18:14:12 +0000

Ever since I was doing longer bike tours I always had a very good companion with me. Not only there, also on hiking tours they were accompanying me like a trusted old friend. In the last years they have changed a bit and got more annoying than I knew them, but as every good friend has their quirks, in the end the pros always outweighed the cons.

Once in a while I was wondering what would happen if the companion changed so drastically that I could not bear the quirks anymore. In the end I always told myself that they have been standing on the right side all along and they assured me of that by promising not to change.

Actually on one of my last trips I was wondering about that. I wish I hadn’t. The trusted companion decided to sell their soul.

Komoot has been acquired by an Italian investor having a reputation for not necessarily improving their products. Do you still remember evernote? Yeah, exactly. One of the first things they like to do is to lay off people. Apparently that has already happened for Komoot as well. I am sorry for the people that created that great product.

For the app, that most probably means it will only get worse from now on. For me, it seems like that I have to prepare for a breakup.

Trip to North Sea

Sun, 23 Mar 2025 18:00:00 +0000

General

22.03.25 (Saturday)
- sunny - cloudy, 12C .. 13C, 25-40kn E’ly
- during night still windy, 3C-5C
23.03.25 (Sunday)
- cloudy - overcast, 10C .. 11C, 20-30kn, NE’ly

Saturday

The weekend before I wanted to go to Plön but due to a cold forecasted night I postponed it to this weekend. As you can see, again this trip was not towards Plön but in the opposite direction. What was the excuse this time?

Again, the weather. Not the weather in the night but rather the weather at daytime. Strong winds were forecasted from easterly directions, even a wind warning was posted by the german met office. As Plön is to the east of Kiel, that would have meant having beautiful gusts of 40kn directly in my face.

I am not quite there yet to “weather” such situations by simply going, but I also wanted to go out. So I did, but rather in the other direction, which also meant having those winds to my favour now. At least Saturday…

So I went westward. After 2.5km I already decided to turn around, as I forgot some equipment that was essential for the night. That mistake handsomely increased my planned distance by about 7km. But as it was early in the day I simply went back and proceeded after getting my stuff.

The first part was through rural Kiel, then the more natural paths began near Westensee. Almost the whole way to the beautiful wild moor south of Rendsburg was in nature. After that, Passing through Jevenstedt, I reached the Kiel Canal (is it still called like that in English, even near Rendsburg?), which led me 10km westwards. With the wind in my back that was one hell of a ride, the speed making up for the less interesting nature compared to before.

The last part leading to Albersdorf was again dominated by agricultural planes, just until right before the sleeping spot, which was on the edge of a forest.

I checked the spot, planned for the night, bought some rations at Lidl in the town of Albersdorf and started to set up the tent.

At some point another hiker came along, an interesting guy from Passau who was just five days into hiking the German part of E1 from Flensburg to Konstanz. What a trip! We talked a bit about his tour, his gear, his life and his plans. He had some interesting and inspiring things to tell.

Sleeping Spot

The sleeping spot was okay
One tent can have a good spot, the others have to live with more slopy parts
No table or bench available
Lidl and Edeka available in town
Toilets available in stone age museum, open from tuesdays to sundays

Night

A bit windy but not too cold
Took some time to get warm feet but in the end everything was snuggly
Had to turn around in the tent, which i set up in the direction of the wind (but not in the direction of the slope…)

Sunday

Lying in the tent I checked out where I could go to take the train back to Kiel. Before departing, I though about going to Büsum. But after the exhausting but fulfilling day I decided that I could make some more way on this Sunday. So in the end I decided going to Husum, which is to the North of Albersdorf.

The wind has turned a bit in northerly directions, which meant that I had the wind from a less favourable direction than yesterday. Anyway, the trip was really nice, mainly through agricultural areas in the beginning but later on more the classical north sea marsh lands. It was quite cool to see how the landscape changes, even if you only go 100km from Kiel to the West.

Arrived in Husum I took the chance to see the north sea, but when I arrived it was the time where the moon decided to retreat the north sea. So only intertidal mudflat¹ and tidal creeks¹ as far as you could see, but that’s also what makes this part special!

Notes to self/Stuff for nerds

Gear

Almost same as the previous overnighter, only that

Vans shoes
Thermal shirt instead of the trekking shirt

and I did not forget anything important (after going back once, haha).

To be improved

Down jacket was used throughout Sunday, as the wind was really chilly; so maybe a wind breaker would be nice to have
next time take some tissues for the runny nose

I like the German words “Watt” and “Priel” more as they sound less generic and somehow more special for what they describe. It’s not just a mudflat! But maybe thats comparable to languages where you have 18 different words for Snow or something… ↩︎ ↩︎

Git workflow - fixing messed up commits

Sun, 16 Mar 2025 15:00:49 +0000

Sometimes you recognize that your change history is a bit of a mess. One commit contains multiple different types of changes, possibly even some refactoring.

In this post I describe my (current) workflow to handle that. Git being git, this might not be the only one, possibly not even the best way to do it!

First you need to know which commit is affected.

Last commit

If you happen to remember that the disastrous commit was the one you just did, you are in luck, as the process is quite easy.

git reset HEAD^
add the changes cleaner this time
commit as required
optional: stash other changes to run build/run tests
repeat until working tree clean

Last commit~{>=1}

Old commits are a bit more tricky, as they require a rebase (you change the history). If you have not pushed the changes or are able to force push, that should not be a problem.

Finding out the commit

First of all we need to find out which commit is affected and how we can reference it exactly. Usually it is sufficient to simply git log and find the hash, or at least an approximation of how many commits have happened since your critical change. The three functions below are sufficient for the majority of the cases:

git log or git log --oneline to find the commit by commit message only
git log --stat to find the commit by additionally checking the files affected
git log -p -- ${FILENAME} to find all commits with changes of ${FILENAME} (incl. diff)

Again, there are various different ways to describe a commit in git, but this is content for another post.

Changing the commit

Besides the rebase, we have the same workflow as for the last commit: ¹

git rebase -i ${COMMIT_HASH}^, rebases from HEAD until the parent commit of ${COMMIT_HASH}
vim/your editor opens showing all the commits since ${COMMIT_HASH}^
mark ${COMMIT_HASH} as edit/e
apply same steps as if it were the last commit
git rebase --continue

In the first step you do not have to rebase all commits since the affected commit hash. As long as you define a range of commits in which the affected critical commit is in, you should be fine. Remember that the way rebasing works the history (incl. hashes) of all commits from your change on is going to be changed.

Git reference: rebase ↩︎

Southeast Kiel Ride

Sat, 15 Mar 2025 00:00:00 +0000

I actually wanted to do another overnighter but the forecast was too dire (-5C in the night). That plan is not canceled but instead postponed to next weekend.

This alternative ride was quite beautiful. As usual, I tried to stay in nature as much as possible. The previous ride I already went in a south-westerly direction. This time I took the same start but after Veloroute 10 ended, I proceeded more south-easterly in the direction of Meimersdorf, Moorsee and Rönne. I have never been there before, so it was exciting to see that more rural part of Kiel as well.

In Wellsee there are some hiking paths which are also quite nice to ride. Of course, here you always have to yield to pedestrians but today there were not too many to care about.

Weather was a bit cold, about 4-5C. Again, my feet were the first ones that disliked those temperatures. Definitely gonna need some proper shoes for the future.

Overnighter in Olendiekssau

Sat, 08 Mar 2025 00:00:00 +0000

General

08.03.25 (Saturday)
- evening ride to sleeping spot @ 54.196389, 9.920639
- sunny, 12C .. 15C, calm
- during night 3C-5C
09.03. (Sunday)
- return home in the morning
- sunny, 8C .. 12C, calm

Misc

First really spring like weekend, took the chance to have a nice overnighter
I almost forgot how easy it is in Kiel to get out into nature within a couple of minutes
The final part of the first leg was very nice, through the forest
forgot lots of stuff: plate, cutlery, TP etc, so had to eat with hands…

Sleeping Spot

The sleeping spot was okayish, a bit small but sufficient for my small tent; could be too small in summer
Two picnic tables and some space to pitch small tents
I was quite tired from the previous days and thus started to sleep at 20:00
at around 21:30 a car came, and a couple of minutes later another one
second car was cop car checking the other, which were some finns on their voyage to France

Night

a bit cold on the back in the night, could have been the sleeping pad which lost a bit of air
feet very warm, had to remove socks at one point

Day 2

breakfast was prepared sandwich, so didnt need to much preparation
was hoping to have a talk with the finns but they were still snoring in their hastily pitched tent
time from wake up til departure ~1:10
way home started again with very nice forests, later on more agricultural

Clothing

Riding Clothes

Fjallraven/specialized pants
Trekking T-shirt
Fleece pullover
fleece mittens
nike runners
normal socks

Sleepwear

Sleeping gear (joggers, long sleeve)
Beanie
extra pair of socks, underwear

Misc

down jacket (used in the morning)

Camping Gear

Decathlon Quickhike 2 Person tent
Iso mat (R=2)
Inflatable mat (R=1)
down sleeping bag (0C comfort)
Inflatable pillow

To be improved

take some appropriate entertainment, e.g. book
dont forget TP, cutlery etc…
shovel for feces

Git workflow - how to keep commits clean

Sun, 28 Nov 2021 00:00:00 +0000

When commiting my changes, I like to keep the commits as simple and atomic as possible. My basic workflow for that is to check all the changed files, choose the respective lines which have been changed, add them to the commit and commit all of my selected changes.

Jetbrains IDEs incorporate a beautiful commit GUI, where you can do all of that. But as there are days where I am not using a Jetbrains IDE, I asked myself how such a workflow might be possible using the tools at hand - the tools offered by the git command.

File lifecycle

In order to be able to follow the mentioned workflow, it is good to have a better understanding of the git file lifecycle.

Untracked    Unmodified     Modified        Staged
|= add file ===============================>|
|            |= edit file =>|               |
|            |              |= stage file =>|
|            |<==================== commit =|

Pro Git, Chapter 2.2, Figure 8.¹

Edited files are in the state of ‘modified’, but are part of a commit only when they have been added to the staging area.

Adding files to staging area

There are several ways to add files and their content to the staging area.

Shotgun

The easiest and fastest way to add files to the staging area is via

git add file

or even easier

git add .

This is the shotgun method - you hit your target but might also hit something else. You have to make sure that your files contain only the information you also want to have in your commit.

More specific

Most of the times, you need to select single lines for a certain commit.

You can also do that via CLI

git add --patch file
git add -p file

Consecutive hunks² of your changes whithin that file will be shown to you. You can select y or n to add the specific hunk to the staging area or leave it in the modified area.

In case the given hunk is too big, you can split it up by choosing s. Depending on the hunk, the given hunks could contain a single line or a few. The workflow with these smaller hunks is the same as before.

Line by line

If the given choice is still too broad, e.g. you have multiple consecutive lines changed together, an additional split might not be possible. What if you still want to split these lines into different commits? This is where the the manual edit comes into play, by choosing e.

This opens your favorite text editor with the contents of the hunk, which looks more or less like this:

-deleted line
+added instead
 unchanged line
+we want to keep this addition here

Let’s assume you want to add the last line to the commit, but not the deletion and the addition of the first two lines. We have to bring the document to the state it has been before - thus replacing the - of the deleted line with a space and removing the added line completely. Why a space? Space means unchanged (see the unchanged line, it also has a space in front), and exactly thats what we want.

That’s how it looks now:

 deleted line
 unchanged line
+we want to keep this addition here

We can now save and close the editor to apply our custom hunk. There will be a validity check when you save, e.g. you deleted a deletion or added some text which has not been there before. In case there was a problem, you get another chance to edit your selections.

Removing files from staging area

In case you noticed that you added too many files to the staging area, you can simply

git reset filename

This will remove the file from the staging area, but not revert your local changes (as it just changes the file’s lifecycle state from staged back to modified). Now you can start to add it to staging area again or continue committing the rest.

Committing

Before committing, I always like to check what I am about to commit. The best way to do that, is of course via git status, connected with git diff for specific files in the staging area. If you keep your commits light and atomic, then you even might want to try the verbose status

git status -v

to skip the diff part. This will show you all textual changes of all files.

Too much text? Maybe your commit is too big :-)

If you are happy, go ahead and commit!

Losing overview?

I have to admit, you can quickly lose overview over your hunks and you find yourself in a loop of git status and git add -p.

This is where git gui comes in handy. What, leaving CLI? Using GUI? Only noobs use GUI?! Come on, in this case a good visualisation comes in handy.

You can do all the above mentioned steps in the gui by either adding whole files with CTRL+T, adding single hunks by right clicking and adding them, or by marking single lines and adding/removing them.

When you are done with that, you can check the staged items and commit them conveniently.

Commit, push, run!

Pro Git ↩︎
Gnu diffutils: hunks are groups of differing lines in between sequences of lines common to two files. ↩︎

Contributing to Nextcloud with Docker

Thu, 07 May 2020 00:00:00 +0000

The other day I wanted to debug a Nextcloud app which I am using quite frequently. To have a local environment would be a good start, however I do not like to install everything on my system (as it is described in the Nextcloud docs). So what can we do? Docker, of course. I found two repos on github which were promoting docker Nextcloud development environments, this one and this one.

To make it short: the first one seemed a bit outdated and didn’t work anyway. The second one was working after a few hiccups.

What you need:

docker
docker-compose
git

Have it already? Then let’s go.

1. Clone the project

git clone https://github.com/juliushaertl/nextcloud-docker-dev.git

2. Clone Nextcloud server

cd nextcloud-docker-dev
git clone https://github.com/nextcloud/server.git
cd server
git submodule update --init
pwd
cd ..

3. Set up env for Nextcloud server

Create an .env file in the root folder of nextcloud-docker-dev containing some specific variables:

COMPOSE_PROJECT_NAME=master

REPO_PATH_SERVER=/home/jus/repos/nextcloud/server
ADDITIONAL_APPS_PATH=/home/jus/repos/nextcloud/server/apps-extra

NEXTCLOUD_AUTOINSTALL_APPS="viewer activity"

# can be used to run separate setups besides each other
DOCKER_SUBNET=192.168.15.0/24
PORTBASE=815

# Main dns names for ssl proxy
# This can be used to append a custom domain name to the container names
DOMAIN_SUFFIX=.local

Enter your REPO_PATH_SERVER as per the pwd from step 2. The ADDITIONAL_APPS_PATH is the place where you can put the apps you want to work on.

4. Set up hosts

The reverse proxy is resolving the domain nextcloud.local, add it to your /etc/hosts/ to reach it

127.0.0.1 nextcloud.local

If you forget this step, you will not be able to open Nextcloud from your browser.

5. Spin up the containers

docker-compose up proxy nextcloud

This command starts the revproxy (nginx), nextcloud, the database (postgres) and mailhog (whatever that is for, lol). After a small installation process you are ready to go.

There are some more containers in the docker-compose, which I do not need for developing. That’s why I am ok with the command mentioned above.

6. Open Nextcloud in browser

As set up in step 4, you can simply open nextcloud.local and you are good to go. Log in with one of the user names mentioned in the set up process. I do not remember all of them, but some are these:

jane
admin
alice
bob
user1

Password is the username.

7. Start developing

Pull the apps you want to develop to the folder described in ADDITIONAL_APPS_PATH.

Make sure you enable them in the Nexctloud app menu using the admin login.

Happy contributing!