How to run Jest tests faster in GitHub Actions

If you’re writing a lot of frontend code or have a node backend, there’s a good chance you’re using Jest for testing. We recently spoke to a customer who had to wait 20 mins for their Jest unit tests to succeed. We helped them cut it down to 3 minutes. We’re going to share some of what we learned in this post.

Start with the simplest case: running your GitHub Actions on a 2vCPU runner is all you do.

name: Frontend CI

jobs:
  jest:
    runs-on: blacksmith-2vcpu-ubuntu-2204
    steps:
      - uses: actions/checkout@v3

      - name: Install pnpm
        uses: pnpm/action-setup@v2
        with:
          version: 8.x.x

      - name: Set up Node.js
        uses: useblacksmith/setup-node@v5
        with:
          node-version: 18.12.1
          cache: pnpm

      - name: Install package.json dependencies with pnpm
        run: pnpm install --frozen-lockfile

      - name: Test with Jest
        run: npm jest --silent --maxWorkers=2

The —maxWorkers flag specifies the maximum number of workers the worker pool will spawn for running tests. The default is n-1 because one core is meant to be reserved for the coordinator/runner thread.

It took ~19 minutes, which is not great. There are two ways to make it substantially faster.

Adding more cores

Since Jest can effectively parallelize jobs, running it on a machine with a higher core count definitely helps. I ran the same workflow across runners with 2, 4, 8, and 16 vCPUs while specifying the maxWorker count.

name: Frontend CI

jobs:
  jest-benchmark:
    strategy:
      matrix:
        runner: [ blacksmith-2vcpu-ubuntu-2204, blacksmith-4vcpu-ubuntu-2204, blacksmith-8vcpu-ubuntu-2204, blacksmith-16vcpu-ubuntu-2204 ]
        maxWorkers: [ 2, 4, 8, 16 ]

    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v3

      - name: Install pnpm
        uses: pnpm/action-setup@v2
        with:
          version: 8.x.x

      - name: Set up Node.js
        uses: useblacksmith/setup-node@v5
        with:
          node-version: 18.12.1
          cache: pnpm

      - name: Install package.json dependencies with pnpm
        run: pnpm install --frozen-lockfile

      - name: Test with Jest
        run: npm jest --silent --maxWorkers=${{ matrix.maxWorkers }}

‍

It should come as no surprise that it was a lot faster when we threw more cores at it. Here are the results

| Runner                        | Duration  |
|-------------------------------|-----------|
| blacksmith-2vcpu-ubuntu-2204  | 18m 16s   |
| blacksmith-4vcpu-ubuntu-2204  | 9m 37s    |
| blacksmith-8vcpu-ubuntu-2204  | 5m 31s    |
| blacksmith-16vcpu-ubuntu-2204 | 3m 34s    |

However, you can only scale so much vertically. Since all tests use the same machine, resource contention for I/O and memory could occur, leading to diminishing marginal returns after a point where adding additional cores does not help with performance. If you have a large test suite, you likely need to shard it across multiple runners to get your CI time under control.

Sharding

The other option is to scale horizontally by sharding these tests into smaller shards and running the shards in parallel on individual machines.

Jest has the —shard flag and specifies the shard index and the shard count in the format of shardIndex/shardCount. To split the test into three shards, one has to write

jest --shard=1/3
jest --shard=2/3
jest --shard=3/3

In our case, let’s split it into 8 shards.

name: Frontend CI

jobs:
  jest-shard:
    runs-on: blacksmith-2vcpu-ubuntu-2204
    name: Jest test ${{ matrix.chunk }}

    strategy:
      fail-fast: true
      matrix:
        chunk: [1, 2, 3, 4, 5, 6, 7, 8]

    steps:
      - uses: actions/checkout@v3

      - name: Install pnpm
        uses: pnpm/action-setup@v2
        with:
          version: 8.x.x

      - name: Set up Node.js
        uses: useblacksmith/setup-node@v5
        with:
          node-version: 18.12.1
          cache: pnpm

      - name: Install package.json dependencies with pnpm
        run: pnpm install --frozen-lockfile

      - name: Test with Jest
        run: npm jest --shard=${{ matrix.chunk }}/8 --maxWorkers=2 --silent

Here are the results

Each shard ran on a 2vCPU machine. The longest shard took 2m 44s to run, which is much faster than running on a 16vCPU machine, which took 3m 34s. By effectively sharding and parallelizing, we reduced the jest test time from 19 minutes to under 3 minutes. Sharding allows you to scale with the number of tests by adding more shards as the number of tests increases in your codebase while ensuring that the wait time for the workflow stays the same.

Now let’s compare the costs associated with each of these scenarios.

| Scenario                         | Duration | Cost   |
|----------------------------------|----------|--------|
| 2vCPU runner                     | 18m 16s  | $0.076 |
| 16vCPU runner                    | 3m 34s   | $0.128 |
| 8 shards, each on a 2vCPU runner | 2m 44s   | $0.096 |

In this case, throwing more cores would make it substantially faster, but sharding allows you to make it even faster and cheaper than throwing more cores. Sharding allows you to cut your CI time by almost 85%, but you’ll need to pay ~25% more than running it on a 2vCPU runner.

Here’s something to pay attention to when thinking about sharding,

Time for each shard to run = Setup time + test execution time

where setup time refers to how long it takes to run all the steps before running the actual tests, like installing dependencies.

In this case, the setup time was 20s, which is negligible. However, if your setup time is high, there might be a tradeoff between the number of shards and how much you want to pay for the extra speedups. Sharding works when you have a large set of tests and the test execution time is significantly greater than the setup time.

If you’re sharding, I would recommend using the `fail-fast: true` flag in your workflow. This flag says that all jobs in the matrix should be canceled if any one of them fails. This can help reduce unnecessary CI spending.

If you're not already a customer, you should consider using Blacksmith for your GitHub Actions. We offer unlimited concurrency, which means you can run as many shards in parallel as possible to cut down your CI time. Blacksmith runs GitHub Actions on cutting-edge gaming CPUs with a colocated cache. This means your GitHub Actions are twice as fast while reducing your spending by 50-75%.

Thank you! We will notify you when we publish new posts!

Oops! Something went wrong while submitting the form.

How to run Jest tests faster in GitHub Actions

Recent Posts

How The Economics of Multitenancy Work

Blacksmith Raises $3.5M led by GV and Y Combinator to Build a High-Performance CI Cloud.

You have 5 days before the new DockerHub limits f*ck you over.

Recent Guides

Best Practices for Managing Secrets in GitHub Actions

How to reduce spend in GitHub Actions

Matrix Builds with GitHub Actions

How to run Jest tests faster in GitHub Actions

Get notified about new posts

Recent Posts

How The Economics of Multitenancy Work

Blacksmith Raises $3.5M led by GV and Y Combinator to Build a High-Performance CI Cloud.

You have 5 days before the new DockerHub limits f*ck you over.

Recent Guides

Best Practices for Managing Secrets in GitHub Actions

How to reduce spend in GitHub Actions

Matrix Builds with GitHub Actions