Oops!

You may have noticed that some (or all) of the images in every post before this one are just completely gone.

TL;DR: I ran docker system prune -af instead of docker image prune -a. Oops!

So I host a video game server for a few friends, and it crashed. Looking at the logs, it was pretty clear what the issue was. I ran out of storage space. a quick df -h confirmed that one. This often happens when watchtower automatically applies new image updates, but doesn't remove the old ones. Every couple months I go through Portainer and manually delete any images marked Unused. I quick search for a cli command to do it lead to this StackOverflow answer:

screenshot of Stack Overflow, "How to remove old and unused Docker images Asked 8 years, 4 months ago Modified 3 months ago Viewed 736k times 1103 upvotes When running Docker for a long time, there are a lot of images in system. How can I remove all unused Docker images at once safety to free up the storage? In addition, I also want to remove images pulled months ago, which have the correct TAG. So, I'm not asking for removing untagged images only. I'm searching for a way to remove general unused images, which includes both untagged and other images such as pulled months ago with correct TAG."

The top answer seemed pretty clear:

screenshot of StackOverflow, "Update Sept. 2016: Docker 1.13: PR 26108 and commit 86de7c0 introduce a few new commands to help facilitate visualizing how much space the docker daemon data is taking on disk and allowing for easily cleaning up "unneeded" excess. docker system prune will delete all dangling data (containers, networks, and images). You can remove all unused volumes with the --volumes option and remove all unused images (not just dangling) with the -a option. You also have: docker container prune docker image prune docker network prune docker volume prune For unused images, use docker image prune -a (for removing dangling and ununsed images). Warning: 'unused' means "images not referenced by any container": be careful before using -a."

Unfortunately I did not read the line I'm pointing to with a green arrow. I saw the orange and didn't bother reading anything else. I mean it seems pretty clear:

docker system prune will delete all dangling data [...] remove all unused images (not just dangling) with the -a option

Seems perfect!

In hindsight, they clearly meant that in addition to containers, networks, and images, unused images would be deleted.

I am one dumb idiot.

Anyway, the volume wasn't touched, just the container. I backed up the volume... ish images didn't come along, but crucially the database was backed up. I also didn't have a compose file. I guess I did it from CLI, which is dumb. The ghost version I had wouldn't mix well with the latest image's partly-dumped volume, so I decided to scp the dumped volume's database to my computer, open it in DB Browser for SQLite, and export the posts table to JSON. By some MIRACLE, it was EXACTLY the syntax used by the official post export JSON file. I took an ancient JSON backup of the blog and stuck the jumped posts JSON into the posts array inside the backup file. I restored from JSON, and my posts were back!

...minus the images, of course.

Thankfully, my posts are still readable! I added an alt text to every image, initially for those who are blind or have poor eyesight, but they have an excellent unintended use. When the image is missing, the alt text is shown by the browser instead.

The real moral of the story: anything you don't have backed up, expect to lose.