benchmarks
Same code. Same night. Two computers.
Every night we build the same open-source projects on GitHub's cloud computers and on small computers running Codemany. Then we compare the times.
Black is GitHub's cloud computer. Green is Codemany on a small computer you own.
Loading the latest results…
The numbers on this page load live from the bench repository (github.com/codemanyhq/bench), which updates after every nightly run at 03:17 UTC. The raw data: results.json (every build: project, computer, cold or warm, seconds, and a link to its GitHub job), projects.json (the projects, their pinned commits and build commands) and hardware.json (the computers, and GitHub's prices with their source).
No results yet
First results arrive after tonight's run at 03:17 UTC.
Build times
Shorter bars are faster. Every project is built on both computers, every night.
Point at a bar to see its time. Click it to open that build on GitHub.
How we test
We keep it fair. When we have to guess, we guess in GitHub's favour.
- Same code. Each project is pinned to one exact commit. Both computers build that commit.
- Same night. One run starts every build, each night at 03:17 UTC.
- Same tools. Both sides set up the same versions of Rust, Go, Node and the rest.
- One timer. We time one step, called "build". Setting up and waiting in line are not in it.
- Cold and warm. Cold: nothing saved from before, on both sides. Warm: GitHub uses its cache, and Codemany keeps what the computer already has.
- Speed favours GitHub. GitHub builds on its 4-core Linux computer, which public code gets for free. Private code gets only 2 cores.
- Price favours GitHub. We price GitHub at its cheaper 2-core rate: $0.006 a minute for Linux, $0.062 for macOS. Public code builds free on GitHub; the price is what private code would pay.
- Check it yourself. Every number links to the GitHub run that made it. All the code is open: codemanyhq/bench.
Night by night
Pick a project to see its build time on every night.
The computers
Codemany ran on small computers you can buy. GitHub ran on its own cloud computers.
All the numbers
Every typical build time, with a link to the GitHub job that measured it.
Every run
The newest nightly runs. Each one opens all its builds on GitHub.