The mysterious disapperance of Docker images
A node hosts a Gitlab runner and a small k3s cluster which runs a few services as regular kubernetes deployments. A CI job pinned to that runner builds Docker images for these services services, updates the image of the corresponding deployments, and starts a few system and acceptance tests. The CI job does not push those images to the in-house registry; to avoid polluting the registry with hundreds of images it just builds locally.
Each test then scales each deployment to zero replicas to effectively stop all services, clears the system’s underlying database, and scales the service deployments back to a small number of replicas sufficient for testing.
The whole thing runs fine until one day the replicas randomly fail to start.
The first symptom is that the tests time out while waiting for replicas to start. The services remain unreachable after system reset. Inspecting the deployment reveals that the Docker image doesn’t exist:
$ kubectl describe pod foo-68fbd8c7c5-8grqd
Name: foo-68fbd8c7c5-8grqd
[…]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned default/foo-68fbd8c7c5-8grqd to node.example.com
Normal BackOff 11m (x6 over 13m) kubelet Back-off pulling image "registry.example.com/foo/foo:1.2.3-64-affa5225"
Normal Pulling 11m (x4 over 13m) kubelet Pulling image "registry.example.com/foo/foo:1.2.3-64-affa5225"
Warning Failed 11m (x4 over 13m) kubelet Error: ErrImagePull
Warning Failed 11m (x4 over 13m) kubelet Failed to pull image "registry.example.com/foo/foo:1.2.3-64-affa5225": rpc error: code = Unknown desc = Error response from daemon: manifest for registry.example.com/foo/foo:1.2.3-64-affa5225 not found: manifest unknown: manifest unknown
Warning Failed 3m11s (x43 over 13m) kubelet Error: ImagePullBackOff
Docker doesn’t find the image in the registry because it was never pushed, but it shouldn’t even try to ask the registry: The image was built locally and should already exist on the node. So why is it gone? Perhaps a docker build failure?
However a docker image ls thrown into the pipeline before the test reveals that the image builds fine; in fact it’s there minutes before the test starts and then randomly disappears while the tests are running. Clearly something’s deleting images from the Docker daemon.
Some careful search for the right keywords (“docker images disappear” wasn’t good enough) reveals a Stack Overflow answer pointing to Kubernetes’ own garbage collection which runs every minute, and, under disk space pressure, removes unused Docker images. And indeed, on the affected node kubelet wanted to delete even more:
[…] node.example.com k3s[…]: E0809 […] Image garbage collection failed multiple times in a row: wanted to free 311382016 bytes, but freed 573253440 bytes space with errors in image deletion
Why does kubelet feel pressured to delete images? And why just about 500MiB? That’s not a lot, by Docker’s standards.
Turns out, the system had a very very small /var/lib/docker:
$ dh -h
Filesystem Size Used Avail Use% Mounted on
[…]
/dev/sda6 7.0G 2.3G 4.8G 33% /var
Mystery solved: That’s nowhere near enough to satisfy docker’s demand for space, and explains why kubelet feels so much pressure to delete images in almost every build.
As a stop-gap measure all images were pruned; later the day /var/lib/docker was given much more space. The tests haven’t failed since.