Clerk’s SDK Infrastructure team was facing performance and flakiness issues with their browser-based integration tests, which were ultimately impacting their deployment frequency.
Now, Clerk’s developers can reliably and efficiently run their integration tests before publishing to tens of thousands of developers, without dealing with flakiness caused by the underlying infrastructure.
Clerk is the most comprehensive user management platform for developers. They are trusted by fast-growing companies around the world like Browserbase, Inngest, Turso, and many more.
As a tool designed to help developers build user experiences for sign-ups, sign-ins, and profile management, Clerk’s SDKs need to integrate smoothly into a variety of application stacks such as Next.js, Astro, Express, React, and various others that are popular amongst developers. This requires the SDK Infrastructure team to test across all of these stacks to ensure compatibility and reliability. Moreover, they need to account for not just one, but multiple Next.js versions, each introducing its own quirks and behaviors.
All of this immediately turns into a complex testing matrix, where the number of possible configurations explodes rather quickly, making it an extremely resource intensive operation to validate Clerk’s SDKs before publishing them.
The Problem
When Infra causes slow, flaky tests.
Before Blacksmith, Clerk was relying on a single GitHub-hosted runner with an 8-core CPU and 32GB of RAM. Back then, their browser-based integration tests, which ran over 20 browsers in parallel, started hitting some serious limitations. As Dylan Staley, Software Engineer at Clerk, succinctly puts it, “the tests were taking forever.”
Not only were test dragging on for up to 40 minutes, they were also running out of memory, causing browser instances to become unresponsive. This led to bizarre timeouts and flaky tests, where local runs would pass, but the CI wouldn’t. At one point, Dylan had to run the CI pipeline ten times just to get the test to pass due to these infrastructure instabilities. The underlying infrastructure simply didn’t have enough CPU and memory resources to handle the load, leading to slow and inconsistent results.
At that point, enough was enough. It was time to find a faster, more powerful solution.
We just kept experiencing significant issues with GitHub Actions, including long test execution times and errors due to capacity limitations.
— Dylan Staley, Software Engineer, Clerk.
The Solution
Selecting Blacksmith over GitHub.
Dylan and the team at Clerk initially considered scaling horizontally, and adding more runners to the pool. But they quickly realized there’s only so much you can parallelize a CI test before you’re bumping up against the limitations of single-threaded performance on a CPU. They knew that boosting single-threaded performance wouldn’t just speed up their individual tests—it would help them finally tackle those pesky flaky tests too.
By the time they started exploring other options, Blacksmith caught their attention with our fleet of gaming CPUs with 2x faster performance, all at a fraction of the cost of GitHub. It was a value proposition they couldn’t ignore. Sure, they looked into other competitors, but the alternatives involved more complexity and riskier migrations. In the end, it was Blacksmith’s CPU performance and drop-in migration that sealed the deal for Clerk!
Shortly after selecting Blacksmith, Dylan created a pull request, which migrated all of their testing workflows onto Blacksmith!
We were worried about the resources needed to migrate to a different vendor, but switching to Blacksmith was incredibly easy.
— Dylan Staley, Software Engineer, Clerk.
The Result
So fast you just know.
Today, Clerk’s flagship JavaScript repository runs all its integration tests on Blacksmith runners. This not only eliminated all flaky tests caused by infrastructure issues overnight, but it also resulted in a 70% cost reduction compared to the GitHub-hosted runners they were previously using.
With a 40-60% performance boost per test, reducing the total time from 9 minutes and 43 seconds to 5 minutes and 56 seconds, the difference was so noticeable that one of the team members pointed out on the company Slack that the CI had suddenly become much more reliable and faster.
I can’t recall a single instance where a test failure was a result of Blacksmith infra. It is very reliable. We can now rely on test failures being actual failures, and not an infra issue.
— Dylan Staley, Software Engineer, Clerk.
If you’re currently dealing with slow, flaky tests due to your CI infrastructure, let us help you too. We know a better way.
