So, What is a Build System Anyway?

For several years, I have seen build and language ecosystem tools being misused to create haphazardly-maintained builds, deployments, and interfaces between projects. Across projects I have seen drastically different approaches to this (google3, Google's internal monorepo, is notoriously biased toward "philosophically maintained" builds, while small projects tend to be much scrappier and rely on shell scripts and duct tape). I want to describe some of the background for what build tools are used for and how they apply to hardware infrastructure. Going back several decades, the canonical starting point for most engineers is the venerable GNU Make, whose purpose is to orchestrate the execution of language toolchains as part of a larger engineering workflow.

For those who have learned this in the context of C or C++ development, you might recall Makefile boilerplate files which look like:

Makefile

# Source: https://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/
CC=gcc
CFLAGS=-I.
DEPS = hellomake.h
OBJ = hellomake.o hellofunc.o

# pattern rule: compile any .o from its matching .c file, and re-run if any header in DEPS changes
%.o: %.c $(DEPS)
	$(CC) -c -o $@ $< $(CFLAGS)

# link all object files into the final binary
hellomake: $(OBJ)
	$(CC) -o $@ $^ $(CFLAGS)

Simple Make example for C code (source). For the opinionated Make users, I apologize for my sins.

The named hellomake target expresses both a set of dependencies that need to be built before it and a list of shell commands that need to run in order to actually build it. With the top-level target that the user intends to make, we are able to easily express a declarative directed acyclic graph (DAG) of tasks which all contribute to a common end product (e.g. binary executables, container images, software artifacts). Its design will feel natural to anyone steeped in Unix's philosophy of composition, and its pattern-action style will feel familiar to those who have spent time with older Unix text-processing tools like awk or sed (which makes sense given that the original version is from 1976!).

I tend to think of build systems for large projects as workflow compilers. Instead of optimizing the compilation of source files into executables, they orchestrate the execution of language-specific compilers as part of a higher-level software development flow. This becomes more necessary as individual shell scripts or Makefiles fail to form the cohesive glue connecting polyglot components to each other.

If your project lives in a single language with its own toolchain (e.g. Cargo for Rust, uv for Python, npm for JavaScript), this might seem like a solved problem. However, as projects grow in size, scope, and heterogeneity, they become hard to manage both organizationally and at scale for a few reasons.

1. Large Polyglot Projects

Contrived example of complex dependencies across different projects in an organization. A strong build system serves to provide the orchestration layer of complex workflows across dependent projects.

I think the polyglot scenario is often underrated when thinking about build systems. If you have a large software project composed of many smaller projects, you end up needing an intermediate layer of "this is how you consume project B from project A". You could choose to do this at the network layer (e.g. each project is a microservice that talks to the others), the version control layer ("I pinned this vendored submodule at commit xyz"), or the artifact layer ("my OCI container sits on top of this versioned release of another image"). In practice, you will often find these mechanisms break down as not all components use the same interface to express versioned, testable dependencies.

Build systems such as Make express these dependencies at the level of an individual directory of code, but they don't prescribe a systematic recipe for:

2. The Reason We Typically Think of

Build system enthusiasts are always eager to discuss how Makefiles break down at scale, but I think the history of how we got to the point of improvement is interesting. In a talk on the history of Pants and Bazel, Benjy Weinberger mentions that at Google, Make was the default build tool until it was effectively unusable. This makes sense as the size of the graph starts to grow — it's harder to build all of the dependencies, let alone calculate the changes to the dependency graph, on a single machine. You can imagine a monorepo with thousands of source files, each being stat-ed at make-time to determine what needed to be rebuilt, only to slow the developer's workstation to a grinding halt.

This assumes that you even have the storage to fit all of the relevant source files on the developer's workstation. For hardware development (and as Google later developed with their virtual filesystem), this expectation breaks down due to the cumulative volume of the sources in the project.

But I want to take this a step further: distributed execution of the graph is not the only problem that would need to be solved. In fact, there are numerous implementations of "distributed Make" concepts that attempt to resolve this issue. The problem with this approach is that yes, you are able to get your build system to actually handle the size of your graph. However, you haven't solved the problem of caching in a meaningful way. You will still have long cold-build times and frustrating incremental builds. And, if someone else has already built the same exact content before, you shouldn't have to re-pay the cost of doing it again. A true and meaningful improvement over what Make provides is one that supports distributed execution and content-addressable caching of build artifacts and actions.

Where We Are Now: REv2 and the Modern Build Ecosystem

Today there are several build systems that solve these problems systematically. After Google open-sourced their internal build system, Blaze, as Bazel in 2015, one of the most significant contributions wasn't the tool itself but the protocol it brought with it: the Remote Build Execution API (REv2, also known as REAPI). This API standardizes the way in which any build system client can integrate with a remote backend that can execute build actions on distributed workers and cache actions and build artifacts. Now, nearly every modern build system integrates with REv2. Besides Google's Bazel, we also have Meta's Buck2 and the community-driven Pants. The theoretical underpinning of what distinguishes these systems from each other is laid out well in Build Systems à la Carte (Mokhov et al., ICFP 2018), if you want to go deeper.

Architecture of REv2 service BuildBarn (source), illustrating how a build client interacts with a REv2-compliant backend.

What REv2 actually decouples is the build client from the execution backend. A client submits an Action — a content-addressed description of inputs and the command to run — to a remote service. The service checks whether that exact action has been executed before, using the content-addressable store (CAS) as a lookup key. If it has, the cached result is returned immediately. If not, the action is dispatched to a worker, run in an isolated sandbox, and the outputs are stored in the CAS before being returned to the client. The result is that any two builds sharing the same inputs and commands, regardless of who ran them or when, automatically share their outputs.

Distributed make just gives us parallelism, but this gives us memoization at the action level, allowing us to share incremental build results across all users and CI jobs simultaneously!

Builds for Hardware

So REv2 solves the build-at-scale problem for software, but there are some interesting challenges that you have to solve for separately for hardware flows (see this post for a background of EDA flows and tooling).

Scheduling Requirements

For the vast majority of pure-software projects, the resources required to build, run, and test an individual component are not typically extreme. With EDA tools, there is a wide range of resource requirements you could have (e.g. running an RTL linter such as Spyglass might be much "cheaper" than running a simulator such as VCS).

This is what drove the adoption of batch schedulers for hardware development (and other resource-intensive domains like HPC), especially LSF and Slurm. These schedulers treat resource requirements (including EDA tool licenses!) as first-class citizens in a job's definition, which lets you manage compute pools effectively. In pure software, this kind of job with complex resource requirements has only come up in the last few years with the rise of LLM training — which seems to be one of the core reasons Kubernetes introduced Kueue for workload-based batch scheduling in late 2022.

What this means for hardware workflows is that most builds which involve orchestrating the invocation of EDA tools rely on an HPC batch scheduler as the mechanism to do the scheduling. Attempting to map this onto RBE is possible, but it's not provided out-of-the-box with most REv2 services. BuildBarn, for example, has its own scheduler which relies on tags to distribute the jobs to independent worker pools (where workers in a given pool share the same platform properties). These workers are expected to be long-lived daemon processes — more like persistent K8s pods than pre-scheduled batch executors.

Large Binary Sizes

Artifacts produced by running EDA tools such as the simv executables produced by VCS are notoriously large, especially when working with a complex hardware design.

If you don't have CAS, this problem doesn't exist in the sense that you never had to worry about storing large quantities of iterative output. You could simply have a basic storage architecture with golden outputs being released, and something even as simple as NFS could store these (even though it pains me deeply to promote NFS).

But with CAS, in a hardware context you need to think much more carefully about storage consumption and cache management in order to only keep content that is likely to be re-used across someone else's build. These large binaries are a forcing function for ensuring data locality. If you have one action dispatched to a node that produces a simv target and another action dispatched to a separate node to run it, you have to pay the cost of waiting for the first executor to upload the content to the CAS and then the second executor downloading it from the CAS.

For software builds, this is still an issue but it is mitigated by the fact that incremental artifacts tend to be manageable in size. For hardware, this becomes a hard constraint if your goal is to minimize end-to-end build latency — the data movement cost alone can dominate the savings from remote execution.

Absolute Paths + Multiple Layers of Caching

Complex EDA tools are typically vertically-integrated, meaning that they attempt to provide many features to make the development workflow of a hardware engineer more efficient (and presumably to justify their high cost). One of these features is caching for the purpose of incremental compilation. This is somewhat analogous to a Python development workspace where each execution leaves around .pyc files that speed up subsequent runs.

The problem is that this tool-internal state is fundamentally at odds with the CAS model. REv2 actions are supposed to be hermetic: they run in a clean sandbox with only their declared inputs, and the same inputs always produce the same outputs. But EDA tool caches encode the history of previous runs on a particular machine — they're stateful, often embed absolute paths, and aren't portable across workers. If you try to include the tool cache as part of an action's declared inputs (making it part of the content hash), you still have to upload and download it from the CAS on every invocation, which can easily cost more than the compilation time it was meant to save. And if you leave it out, the tool starts cold on every action, defeating the purpose of the cache entirely. The two layers of caching — CAS-level action memoization and tool-level incremental state — operate on incompatible assumptions about what it means to "reuse prior work."

Debuggability of Remote Builds

When you are going to make a change which impacts the developer flow for engineers (especially those whose primary expertise is in hardware rather than software), you have to think carefully about the UI/UX for end-users. Bazel provides a Build Event Protocol (BEP) that streams structured build events (e.g. actions dispatched, cache hits, test results) to a result store, which helps significantly for CI observability. However, this doesn't solve the deeper problem of interactive debugging.

Hardware engineers are accustomed to running EDA tools interactively: SSHing into the same machine the tool ran on, opening the tool's own GUI (DVE for VCS waveform debugging, for instance), and inspecting intermediate files left on disk. Remote builds are somewhat incompatible with this concept since the hermetic sandboxes are intended to be ephemeral in nature. This of course, is not a nonstarter for adoption for RBE (especially if you customize your RBE or build on top of the BEP). It's more that concepts such as "hermeticity" need to demonstrate clear benefits in terms of cache reuse to sell the vision of adopting a system with such a model.

Worth Exploring

This is an area I work in and plan to keep writing about — the build ecosystem is moving fast, and there are a lot of interesting problems still being worked out in the open. If any of this was interesting, here are some things worth poking around in: