If you’re writing a lot of frontend code or have a node backend, there’s a good chance you’re using Jest for testing. We recently spoke to a customer who had to wait 20 mins for their Jest unit tests to succeed. We helped them cut it down to 3 minutes. We’re going to share some of what we learned in this post.
name: Frontend CI
jobs:
jest:
runs-on: blacksmith-2vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v3
- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 8. x.x
- name: Set up Node.js
uses: useblacksmith/setup-node@v5
with:
node-version: 18.12 .1
cache: pnpm
- name: Install package.json dependencies with pnpm
run: pnpm install --frozen-lockfile
- name: Test with Jest
run: npm jest --silent --maxWorkers=2
The —maxWorkers flag specifies the maximum number of workers the worker pool will spawn for running tests. The default is n-1 because one core is meant to be reserved for the coordinator/runner thread.
It took ~19 minutes, which is not great. There are two ways to make it substantially faster.
Adding more cores Since Jest can effectively parallelize jobs, running it on a machine with a higher core count definitely helps. I ran the same workflow across runners with 2, 4, 8, and 16 vCPUs while specifying the maxWorker count.
name: Frontend CI
jobs:
jest-benchmark:
strategy:
matrix:
runner: [ blacksmith-2vcpu-ubuntu-2204 , blacksmith-4vcpu-ubuntu-2204 , blacksmith-8vcpu-ubuntu-2204 , blacksmith-16vcpu-ubuntu-2204 ]
maxWorkers: [ 2 , 4 , 8 , 16 ]
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v3
- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 8. x.x
- name: Set up Node.js
uses: useblacksmith/setup-node@v5
with:
node-version: 18.12 .1
cache: pnpm
- name: Install package.json dependencies with pnpm
run: pnpm install --frozen-lockfile
- name: Test with Jest
run: npm jest --silent --maxWorkers=${{ matrix.maxWorkers }}
It should come as no surprise that it was a lot faster when we threw more cores at it. Here are the results
| Runner | Duration |
|-------------------------------|-----------|
| blacksmith-2vcpu-ubuntu-2204 | 18m 16s |
| blacksmith-4vcpu-ubuntu-2204 | 9m 37s |
| blacksmith-8vcpu-ubuntu-2204 | 5m 31s |
| blacksmith-16vcpu-ubuntu-2204 | 3m 34s |
However, you can only scale so much vertically. Since all tests use the same machine, resource contention for I/O and memory could occur, leading to diminishing marginal returns after a point where adding additional cores does not help with performance. If you have a large test suite, you likely need to shard it across multiple runners to get your CI time under control.
Sharding The other option is to scale horizontally by sharding these tests into smaller shards and running the shards in parallel on individual machines.
Jest has the —shard flag and specifies the shard index and the shard count in the format of shardIndex/shardCount. To split the test into three shards, one has to write
jest --shard=1/3
jest --shard=2/3
jest --shard=3/3
In our case, let’s split it into 8 shards.
name: Frontend CI
jobs:
jest-shard:
runs-on: blacksmith-2vcpu-ubuntu-2204
name: Jest test ${{ matrix.chunk }}
strategy:
fail-fast: true
matrix:
chunk: [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ]
steps:
- uses: actions/checkout@v3
- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 8. x.x
- name: Set up Node.js
uses: useblacksmith/setup-node@v5
with:
node-version: 18.12 .1
cache: pnpm
- name: Install package.json dependencies with pnpm
run: pnpm install --frozen-lockfile
- name: Test with Jest
run: npm jest --shard=${{ matrix.chunk }}/8 --maxWorkers=2 --silent
Here are the results
Each shard ran on a 2vCPU machine. The longest shard took 2m 44s to run, which is much faster than running on a 16vCPU machine, which took 3m 34s. By effectively sharding and parallelizing, we reduced the jest test time from 19 minutes to under 3 minutes. Sharding allows you to scale with the number of tests by adding more shards as the number of tests increases in your codebase while ensuring that the wait time for the workflow stays the same.
Now let’s compare the costs associated with each of these scenarios.
| Scenario | Duration | Cost |
|----------------------------------|----------|--------|
| 2vCPU runner | 18m 16s | $0.076 |
| 16vCPU runner | 3m 34s | $0.128 |
| 8 shards, each on a 2vCPU runner | 2m 44s | $0.096 |
In this case, throwing more cores would make it substantially faster, but sharding allows you to make it even faster and cheaper than throwing more cores. Sharding allows you to cut your CI time by almost 85%, but you’ll need to pay ~25% more than running it on a 2vCPU runner.
Here’s something to pay attention to when thinking about sharding,
Time for each shard to run = Setup time + test execution time where setup time refers to how long it takes to run all the steps before running the actual tests, like installing dependencies.
In this case, the setup time was 20s, which is negligible. However, if your setup time is high, there might be a tradeoff between the number of shards and how much you want to pay for the extra speedups. Sharding works when you have a large set of tests and the test execution time is significantly greater than the setup time.
If you’re sharding, I would recommend using the `fail-fast: true` flag in your workflow. This flag says that all jobs in the matrix should be canceled if any one of them fails. This can help reduce unnecessary CI spending.
If you're not already a customer, you should consider using Blacksmith for your GitHub Actions. We offer unlimited concurrency , which means you can run as many shards in parallel as possible to cut down your CI time. Blacksmith runs GitHub Actions on cutting-edge gaming CPUs with a colocated cache . This means your GitHub Actions are twice as fast while reducing your spending by 50-75%.
