Push, Cache, Repeat: Amazon ECR as a remote Docker cache for GitHub Actions

This is a follow-up to our previous blog post on Docker layer caching for GitHub Actions. In the previous post, we used a registry cache to store cached artifacts in a docker registry. We recently had a call with a customer where we helped them set up a remote cache with AWS ECR, and we decided to share our learnings.

A Docker image is composed of various layers. During a Docker build, each layer is built one after the other. If a layer doesn’t change between builds, having a cache helps retrieve an already built and unchanged layer - this can drastically speed up your build.

Ideally, you want to store these cached layers in your runner building the Docker image. However, due to the ephemeral nature of these runners, this is not always possible when you’re using GitHub-hosted runners.

A way to overcome this limitation is to store these cached layers in AWS ECR, separate from the built and pushed image.

First, let’s look at how to do it plain and simple without the remote cache.

name: Docker Build and Push to ECR Private

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: bs-blog-code

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3

    - name: Login to ECR
      uses: docker/login-action@v3
      with:
        registry: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com
        username: ${{ secrets.AWS_ACCESS_KEY_ID }}
        password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        file: ./Dockerfile
        push: true
        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.ECR_REPOSITORY }}:${{ github.sha }}

We are doing the usual steps of logging in to ECR and building and pushing the image to ECR using the docker/build-push-action@v5 action.

The remote cache feature is not supported by default in Docker and requires you to use Docker Buildx, an advanced Docker build tool. You must set it up using docker/setup-buildx-action@v3 and create a builder instance.

Once you’ve set it up, you need to populate the cache-to and cache-from parameters to the build command like this, as showcased in the official documentation from AWS on this feature.

docker build -t <account-id>.dkr.ecr.<my-region>.amazonaws.com/buildkit-test:image \
--cache-to mode=max,image-manifest=true,oci-mediatypes=true,type=registry,ref=<account-id>.dkr.ecr.<my-region>.amazonaws.com/buildkit-test:cache \
--cache-from type=registry,ref=<account-id>.dkr.ecr.<my-region>.amazonaws.com/buildkit-test:cache .

The cache-from specifies the source to use as a cache when building your Docker image, and the cache-to specifies the destination for storing the build cache once a Docker image has been built.

This is the final workflow file for building and pushing with a remote cache in ECR once we convert it to the format that docker/build-push-action@v5 expects.

name: Docker Build and Push to ECR Private with remote cache

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: bs-blog-code

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3

    - name: Login to ECR
      uses: docker/login-action@v3
      with:
        registry: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com
        username: ${{ secrets.AWS_ACCESS_KEY_ID }}
        password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Create and use a new builder instance
      run: |
        docker buildx create --name mybuilder --use

    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        file: ./Dockerfile
        push: true
        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.ECR_REPOSITORY }}:${{ github.sha }}
        builder: mybuilder
        cache-from: type=registry,ref=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.ECR_REPOSITORY }}:cache
        cache-to: type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.ECR_REPOSITORY }}:cache

Performance Improvements

Workflow without the remote cache

Workflow with the remote cache rebuilding the same Docker file

At Blacksmith, we’re always trying to optimize customer interactions with Docker - be it through local Docker registry mirrors or out-of-the-box Docker layer caching (coming soon!). You should give us a shot if you’re not already a customer. We can run your GitHub Actions twice as fast while reducing your spending by 50-75%.

Notes

The above code block was for pushing to a private ECR. If you wish to push to a public registry, you should change the ref in the cache-to and cache-from to

public.ecr.aws/${{ secrets.AWS_ACCOUNT_ID }}/${{ env.ECR_REPOSITORY }}:cache instead of ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.ECR_REPOSITORY }}:cache

If you’re getting a 403 forbidden, it might be that your IAM does not have permissions for these ECR operations. You can solve this by attaching these policies to your IAM user.
- ecr:GetAuthorizationToken
- ecr:BatchCheckLayerAvailability
- ecr:GetDownloadUrlForLayer
- ecr:BatchGetImage
- ecr:PutImage
- ecr:InitiateLayerUpload
- ecr: UploadLayerPart
- ecr:CompleteLayerUpload

‍

Thank you! We will notify you when we publish new posts!

Oops! Something went wrong while submitting the form.

Push, Cache, Repeat: Amazon ECR as a remote Docker cache for GitHub Actions

Recent Posts

Launch: GitHub Actions Analytics

Deployments still suck, but maybe they don’t have to

Blacksmith Achieves SOC 2 Type 2 Compliance

Recent Guides

Best Practices for Managing Secrets in GitHub Actions

How to reduce spend in GitHub Actions

Matrix Builds with GitHub Actions

Push, Cache, Repeat: Amazon ECR as a remote Docker cache for GitHub Actions

Get notified about new posts

Recent Posts

Launch: GitHub Actions Analytics

Deployments still suck, but maybe they don’t have to

Blacksmith Achieves SOC 2 Type 2 Compliance

Recent Guides

Best Practices for Managing Secrets in GitHub Actions

How to reduce spend in GitHub Actions

Matrix Builds with GitHub Actions