"But LLMs are not deterministic!"
The most common retort to my “LLMs are compilers” framing is:
“… but compilers are deterministic, LLMs just generate probabilistic output!”
I think that retort is not even wrong, and I’ll explain why.
The analogy’s central idea was not that LLMs replicate every characteristic of compilers, but that using natural-language-driven LLMs is another large step up in abstraction. Traditional compilers enabled programmers to express themselves in high-level languages instead of assembly. Similarly, LLMs empower programmers to express their intent directly in natural language, translating it into code.
Let’s examine the “but they’re not deterministic!” argument closely:
The entire field of engineering revolves around combining unreliable, unpredictable components into reliable, predictable systems. This principle is foundational in computer systems as well, best described in the classic paper “End-to-end Arguments in System Design”. All modern computers are built on this philosophy, achieving reliability through layered abstractions and error-handling.
Building robust systems with LLMs follows the same approach. Developers can introduce reliability and predictability through techniques such as:
- Writing clear, detailed specifications
- Utilizing tests and functional checks to validate outputs
- Employing feedback loops (including human-in-the-loop validation) to enforce correctness
In fact, many components of contemporary computer systems already exhibit fundamental non-determinism at both hardware and software levels. Some examples:
- Library dependencies: External libraries your code relies upon may update unexpectedly, introducing changes outside your control.
- Garbage collection: Runtime memory management is inherently unpredictable—triggering at different times depending on complex internal states.
- Multi-threading and concurrency: Thread execution order, timing, and context-switching are unpredictable, yet managed through synchronization mechanisms to create deterministic outcomes.
- JIT compilation: Modern runtimes dynamically decide whether and how aggressively to compile bytecode based on real-time data and usage patterns.
- Caching: Cache behavior (hits, misses, eviction policies) is non-deterministic due to dependence on system state and usage patterns.
Modern computing is built around turning non-deterministic components into deterministic experiences for users.
We can—and should—apply precisely the same strategy to LLMs. This is especially straightforward in programming, where we have the luxury of a clear and quick reward function: run the generated code and validate it against intended outcomes.
Rather than viewing non-determinism as a disqualifier for LLMs, we should embrace it as simply another characteristic to manage—just as we’ve always done in computing.