= Why Use a Monorepo Build Tool?
// tag::header[]
:author: Li Haoyi
:revdate: 17 December 2024

_{author}, {revdate}_




Software build tools mostly fall into two categories:

1. Single-language build tools, e.g.
   https://maven.apache.org/[Maven] (Java), https://python-poetry.org/[Poetry] (Python),
   https://doc.rust-lang.org/cargo/[Cargo] (Rust)

2. Monorepo build tools targeting large codebases, e.g. https://bazel.build/[Bazel],
   https://www.pantsbuild.org/[Pants], https://buck.build/[Buck], and https://mill-build.org/[Mill]

One question that comes up constantly is why do people use Monorepo build tools? Tools
like Bazel are orders of magnitude more complicated and hard to use than tools
like Poetry or Cargo, so why do people use them at all?

// end::header[]



It turns out that Monorepo build tools like Bazel or Mill do a lot of non-obvious things that
other build tools don't, that become important in larger codebases (100-10,000 active developers).
These features are generally irrelevant for smaller projects, which explains why most people
do not miss them. But past a certain size of codebase and engineering organization these
features become critical. We'll explore some of the core features of "Monorepo Build Tools"
below, from the perspective of Bazel (which I am familiar with) and Mill (which this
technical blog is about).


== Support for Multiple Languages

While small software projects usually start in one programming language, larger ones
inevitably grow more heterogeneous over time. For example, you may be building a Go binary
and Rust library which are both used in a Python executable, which is then tested using a
Bash script and deployed as part of a Java backend server. The Java backend server may also
server a front-end web interface compiled from Typescript, and the whole deployment again
tested using Selenium in Python or Playwright in Javascript.

The reality of working in any large codebase and organization, such rube-goldberg
code paths _do_ happen on a regular basis, and any monorepo build tool has to accommodate them.
If the build tool does not accommodate multiple languages, then what ends up happening is you
end up having lots of small build tools wired together. Taking the example above,
you may have:

- A simple Maven build for your backend server,
- A simple Webpack build for the Web frontend
- A simple Poetry build for your Python executable
- A simple Cargo build for your Rust library
- A simple Go build for your Go binary

Although each tool does its job, none of those tools are sufficient to build/test/deploy
your project! Instead, you end up having a `bin/` or `build/` folder full of `.sh` scripts
that wire up these simpler per-language build tools in ad-hoc ways. And while the individual
language-specific build tools may be clean and simple, the rats nest of shell scripts that
you also need usually ends up being a mess that is impossible to work with.

That is why monorepo build tools like Bazel and Mill try to be _language agnostic_.
Although they may come with some built in functionality (e.g. Bazel comes with C/C++/Java
support built in, Mill comes with Java/Scala/Kotlin), monorepo build tools must make
it extending them to support additional languages _easy_. Bazel via its ecosystem
of `rules_*` libraries, Mill via it's extensibility APIs which make it easy to
implement your own support for additional languages like
xref:mill:ROOT:extending/example-python-support.adoc[Python] or
xref:mill:ROOT:extending/example-typescript-support.adoc[Typescript]. That means that when
the built-in language support runs out - which is inevitable in large growing monorepos -
the user can smoothly extend the build tool to keep going rather than falling back to
ad-hoc shell scripts.

== Support for Custom Build Tasks

As projects get large, they also get more unique. Every hello-world Java or Python or
Javascript project looks about the same, but larger projects start having unusual
requirements that no-one else does, for example:

- Invoking a bespoke code-generator to integrate with your company's internal RPC system


- Generating custom deployment artifact formats to support that one legacy datacenter you
  need to get your code running in

- Downloading third-party dependency sources, patching them, and building them from source
  to work around issues that you have fixed but not yet succeeded in upstreaming

- Compiling the compiler you need to compile the rest of your codebase, again perhaps
  to make use of compiler bugfixes that you have not yet managed to get into an upstream release.

The happy paths in build tools are usually great, and the slightly-off-the-beaten-path
workflows usually have third-party plugins supporting them: things like linting, generating
docker containers, and so on. But as any growing software organization quickly finds itself
with build-time use cases that nobody else in the world has. At that point the paved paths
have ended and the build engineers will need to implement the custom build tasks themselves

Every build tool allows some degree of customization, but how easy and flexible they are
differs from tool to tool. e.g. a build tool like Maven requires its plugins to fit into
a very restricted Build Lifecycle (https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html[link]):
this is good when compiling Java source code is all you need to do, but can be problematic when
you need to do something more far afield. The alternative is the aforementioned rats-nest
of shell scripts - either wrapping or wrapped by the traditional build tools - that implement
the custom build tasks you require.

That is why monorepo build tools like Bazel and Mill make it easy to write custom tasks. In
Bazel a custom task is just a https://bazel.build/reference/be/general#genrule[genrule()], in Mill
just `def foo = Task { ... }` with a block of code doing what you need,
and you can even use any third-party JVM library
you are already familiar with as part of your custom task. This helps ensure your custom
tasks are written in concise type-checked code with automatic caching and parallelism,
which are all things that are lacking if you start implementing your logic outside of
your build tool in ad-hoc scripts.

== Automatically Caching and Parallelizing Everything

In most build tools, caching is opt in, so the core build/compile tasks usually end up getting
cached but everything else is not and ends up being wastefully recomputed all the time. In
monorepo build tools like Bazel or Mill, everything is cached and everything is parallelized.
Even tests are cached so if you run a test twice on the same code and inputs (transitively),
the second time it is skipped.

The importance of caching and parallelism grows together with the codebase:

- For smaller codebases, you do not need to cache or parallelize at all: compilation and
  testing are fast enough that you can just run them every time from a clean slate
  without inconvenience

- For medium-sized codebases, caching and parallelizing the slowest tasks (e.g. compilation
  or testing) is usually enough. Most build tools have some support for manually opting-in to
  some kind of caching or parallelization framework, and although you will likely miss out
  on many "ad-hoc" build tasks that still run un-cached and sequentially, those are few
  enough not to matter

- For large codebases, you want everything to be cached and parallelized. Caching just the
  "core" build tasks is no longer enough, and any non-cached or non-parallel build tasks
  results in noticeable slowness and inconvenience.

Take ad-hoc source code generation as an example: a small codebase may not have any. A
medium-sized codebase may have some but little enough that it doesn't matter if it runs
sequentially un-cached. But a large codebase may have multiple RPC IDL
code generators (e.g. https://protobuf.dev/[protobuf], https://thrift.apache.org/[thrift],
static resource pre-processors, and other custom tasks that not caching and parallelizing
these causes visible slowdowns and inconvenience.

In monorepo build tools like Mill or Bazel, caching and parallelism are automatic and
enabled by default. That means that it doesn't matter what you are running - whether
it's the core compilation workflows or some ad-hoc custom tasks - you always get the
benefits of caching and parallelization to keep your build system fast and responsive.

== Seamless Remote Caching

"Remote caching" means I can compile something on my laptop, you download it to your laptop
for usage. "Seamless" means I don't need to do anything to get this behavior - no manual
commands to upload and download - so the distribution of build outputs from my laptop to
yours happens completely automatically.

This also applies to tests: e.g. if TestFoo was run in CI on master, if I pull
master and run all tests without changing the Foo code, TestFoo is skipped and uses the
CI result.

Bazel, Pants, and many other monorepo build tools support this out of the box, with
open source back-end servers such as https://github.com/buchgr/bazel-remote[Bazel Remote].
The clients and servers all conform to a https://github.com/bazelbuild/remote-apis[standardize
protocol], so you can easily drop in a new server or new build client and have it work
with all your existing infrastructure. Mill does not yet support remote caching, but there
are some https://github.com/com-lihaoyi/mill/pull/2777[prototypes] and
https://github.com/com-lihaoyi/mill/pull/4065[work in progress] that will hopefully
add support in the not-too-distant future.

== Remote Execution

"Remote execution" means that I can run "compile" on my laptop and have it automatically
happen in the cloud on 96 core machines, or I run a lot of tests (e.g. after a big refactor)
on my laptop and it seamlessly gets farmed out to run 1024x parallel on a large
compute cluster.

Remote execution is valuable for two reasons:

1. *Better Parallelism*:
   The largest cloud machines you can get are typically around 96 cores, whereas if you farm
   out the execution to a cluster you can easily run on many 1024 or more cores in parallel

2. *Better Utilization*: e.g. If you
   give every individual a 96 core devbox, most of the time when they are not actively running
   anything (e.g. they are thinking, typing, talking to someone, etc.) those 96 cores are
   completely idle. It's not usual for utilization on devboxes to be <1% while you are still
   paying for the other 99% of idle CPU time. In contrast, an auto-scaling remote execution
   cluster can spin down machines that are not in use, and achieve >50% utilization rates

One surprising thing is that remote execution can be both faster _and_ cheaper_than running
things locally on a laptop or devbox! Running 256 cores for 1 minute doesn't cause any more
cloud spending than running 16 cores for 16 minutes, even though the former finishes 16x
faster! And due to the improved utilization from remote execution clusters, the total savings
can be significant.

Monorepo build tools like Bazel, Pants, and Buck all support remote execution out of the box.
Mill does not support it, which means it might not be suitable for the largest monorepos
with >10,000 active developers.

== Dependency based test selection

When using Bazel to build a large project, you can use bazel query to determine the possible
targets and tests affected by a code change, allowing you to easily set up pull-request validation
to only run tests downstream of a PR diff and skip unrelated ones. The Mill build tool also supports
this, as xref:mill:ROOT:large/selective-execution.adoc[Selective Execution], letting you snapshot
your code before and after a code change and only run tasks that are downstream of those changes.

Fundamentally, running "all tests" in CI is wasteful when you know from the build tool
that only some tests are relevant to the code change being tested. If every pull request always
runs every single test in a monorepo, then it's natural for PR validation times to grow unbounded
as the monorepo grows. Sooner or later this will start causing issues.


Any large codebase that doesn't use a monorepo build tool ends up re-inventing this manually, e.g.
consider this code in apache/spark that re-implements this in a Python script that wraps
`mvn` or `sbt` (https://github.com/apache/spark/blob/290b4b31bae2e02b648d2c5ef61183f337b18f8f/dev/sparktestsupport/modules.py#L108-L126[link]).
With a proper monorepo build tool, such functionality comes for free out-of-the-box with better
precision and correctness than anything you could hack together manually.

== Build Task Sandboxing

There are two kinds of sandboxing that monorepo build tools like Bazel do:

1. *Semantic sandboxing*: this ensures your build tasks do not make use of un-declared files,
   or write to places on disk that can affect other tasks. In most build tools, this
   kind of mistake results in confusing nondeterministic parallelism and cache invalidation
   problems down the road, where e.g. your build step may rely on a file on disk but not realize
   it needs to re-compute when the file changes. In Bazel, these mis-configurations result in a
   deterministic error up front, enforced via a https://bazel.build/docs/sandboxing[variety of mechanisms]
   (e.g. https://en.wikipedia.org/wiki/Cgroups[CGroups] on Linux,
   https://www.chromium.org/developers/design-documents/sandbox/osx-sandboxing-design/[Seatbelt Sandboxes] on Mac-OSX).

1. *Resource sandboxing*: Bazel also has the ability to limit CPU/Memory usage
  (https://github.com/bazelbuild/bazel/pull/21322), which eliminates the noisy neighbour
   problem and ensures a build step or test gets the same compute footprint whether run alone
   during development or 96x parallel on a CI worker.
   Otherwise, it's common for tests to pass when run alone during manual development, then timeout
   or OOM when run in CI under resource pressure from other tests hogging the CPU or RAM

Both kinds of sandboxing have the same goal: to make sure your build tasks behave the same
way no matter how they are run sequentially or in parallel with one another. Even Bazel's
sandboxes aren't 100% hermetic, but are hermetic enough

xref:mill:ROOT:depth/sandboxing.adoc[The Mill build tool's sandboxing] is less powerful
than Bazel's CGroup/Seatbelt sandboxes, and simply runs tasks and subprocesses in
sandbox directories to try and limit cross-task interference. But it has the same goal
of adding best-effort guardrails to mitigate race conditions and non-determinism.

== Who Needs Monorepo Build Tools?

Most small projects never need the features listed above: small projects build quickly
without any optimizations, use a single language toolchain without customization, and
any bugs related to non-determinism or resource footprint can usually be investigated
and dealt with manually. Any missing build-tool features can be papered over with shell
scripts.

That is how every small project starts, and as most small projects never grow big you
can go quite a distance without needing anything more. While the features above would be
nice to have, they are _wants_ rather than _needs_.

But once in a while, a project _does_ grow large. Sometimes the rocket-ship really _does_
take off! In such cases, as the number of developers grows from 1 to 10 to 1,000,
you will inevitably start feeling pain:

1. Local build times slowing to a crawl on your laptop, using 1 out of 16 available CPUs
2. Pull-request validation taking 4 hours to run mostly-unnecessary tests with a 50% flake rate
3. An unmaintainable multi-layer jungle of shell, Python, and Make scripts layered on
   top of your classic build tools like Maven/Poetry/Cargo, that everyone knows should be
   cleaned up but nobody knows how.

Monorepo build tools bring performance optimizations to
bring down CI times, sandboxing improvements to reduce flakiness, and structured way
of replacing the ubiquitous folder-full-of-bash-scripts. It is these features that really
let a codebase _scale_, allowing you to grow your developer team from 100 to 1,000 developers
and beyond without everything grinding to a halt. That is why people use "monorepo build tools"
like Mill (most suitable for projects 10-1,000 active developers) or Bazel
(most suitable for larger projects 100-10,000 active developers) .

