Launch: Local Docker registry mirrors in Blacksmith

We at Blacksmith have the privilege of observing patterns in thousands of GitHub Action workflow jobs a day. Docker image pulls are a ubiquitous part of most CI workloads — running service containers for integration tests, pulling dependencies for Docker builds, all pull images from a Docker registry. In a typical GitHub Action workflow, common service container dependencies like Postgres, Redis, MySQL, etc are all directly pulled from Docker Hub over the internet.

This status quo is not great because we have seen our customers run into a few issues:

• In November 2020, Docker Hub started aggressively rate limiting unauthenticated Docker image pulls to a maximum of 100 every 6 hours. Authenticated but free user accounts were rate limited to 200 pulls every 6 hours. For any reasonably sized engineering org, this is untenable.

• A Pareto distribution emerges in workflow jobs, where 80% of them depend on the same 20% of Docker images, and hitting Docker Hub over the public internet for every invocation is slow and redundant.

GitHub’s official hosted runners for GitHub Actions have a deal with Docker Hub where they are not subject to these rate limits (or the rate limits are high enough that no single user is likely to ever hit them). This was a problem for us at Blacksmith, as we provide managed high-performance runners for GitHub Actions — we had to ask our bigger customers to authenticate with Docker Hub to avoid hitting these rate limits.

Today, we’re launching local Docker registry mirrors across our fleet, which act as a “pull through” cache for public Docker images. All Docker pulls on Blacksmith are routed through these mirrors and they only need to hit Docker Hub once, to hydrate the cache. This pull hydrates that particular image into the registry mirror and subsequent pulls, across organizations, are served through a node-local registry that runs on its own dedicated disk. This means common image pulls are substantially faster.

The other main benefit is that CI jobs across an organization will now not be pulling as frequently from the Docker Hub and will therefore not run into Docker Hub’s rate limits. Here are the results:

Before

After

This is all happening transparently for our users, with no code changes. If you’re already a Blacksmith customer, you should see your Docker pulls get noticeably faster. If you’re not, consider this an additional reason to become one! In addition to way better performance, we can cut your CI spending down by 50-75%.

Reach out to us at hello@blacksmith.sh for all things GitHub Actions and CI.