A modern GPU driver is not one program. It is a treaty between an application API, a shader compiler, memory management, command submission, a kernel scheduler, proprietary firmware, and hardware that may have no public programming manual. If any field in a shared structure is wrong, the machine may acknowledge the work and quietly do nothing. If synchronization is wrong, an apparently successful frame can corrupt unrelated memory.

That makes the recent M4 driver experiment by Cody Ho and Niklas Sheth unusually instructive. The pair report that they built a Linux kernel driver and a Mesa user-space driver for newer Apple AGX GPUs in roughly a month. Their current code runs desktop compositing, WebGL, and games; their test run passes the OpenGL ES 3.0 conformance suite, excluding optional extensions. They also published the reverse-engineering experiments and driver forks behind the result.

The important word is current. This is a working research prototype, not an upstream Linux driver or a supported end-user release. The authors say the code still needs broad testing, human review, decomposition into reviewable changes, and coordination with the older M1/M2 driver work. Reported frame rates and compliance results are project claims rather than an independent product certification. Even with those boundaries, the project exposes a powerful engineering method: reduce an opaque system to controlled experiments, preserve known-good traces, and replace captured state with understood code one piece at a time.

A GPU driver is split across a trust boundary

On Linux, most knowledge of graphics APIs and GPU commands lives in user space. An application calls OpenGL. Mesa validates that state and translates it through shared infrastructure. Gallium provides a common driver interface, while NIR acts as the optimizing intermediate representation used by most Mesa shader compilers. A hardware backend lowers NIR into the target GPU’s instruction set and builds command buffers and descriptors.

The kernel side does a different job. It manages GPU-visible memory, accepts command submissions, orders work, tracks completion, and recovers from failures. Linux represents GPU completion with DMA fences; a fence signals when the associated rendering or compute activity finishes. The DRM scheduler can hold a job until its dependencies resolve, submit it to hardware, and signal a finished fence when the lower-level completion object fires.

Apple Silicon inserts another major layer. The Linux kernel driver does not simply write every command to AGX registers. It communicates with GPU firmware through Apple’s RTKit framework and shared memory. The firmware owns some fields, the host owns others, and the two sides exchange events. Asahi Linux’s hardware documentation describes AGX as a bespoke, PowerVR-influenced design and notes that firmware communication uses RTKit. In practice, that firmware ABI becomes part of the device interface that a Linux driver must reproduce.

This split explains why “a GPU driver” can be both surprisingly small and impossibly broad. Mesa, Gallium, NIR, DRM, buffer objects, fences, and scheduling are reusable infrastructure. The new driver does not need to reinvent OpenGL. It does need to define exactly how a new AGX generation encodes shaders, textures, descriptors, command streams, shared memory, and completion events.

The existing Asahi work on M1 and M2 supplied the shape of a solution. In 2022, the project explained that Mesa’s common code reduced OpenGL into Gallium, NIR provided a reusable compiler layer, and DRM removed kernel boilerplate. The M4 effort could reuse those architectural choices and the existing user/kernel interface. It still had to discover where the newer hardware and firmware changed the contract.

Observe, replay, subtract, reconstruct

The project’s central reverse-engineering loop began with macOS running inside a purpose-built hypervisor. A small Metal workload produced a known-good GPU transaction. The hypervisor captured GPU-visible memory and firmware events around that transaction. The team could then reboot, replay state, and ask a simple question: did the same output pages change in the same way?

The first replay was deliberately crude. It restored a large captured memory image, issued the firmware-visible event—often called a kick—and observed the result. A successful replay proved that the capture contained enough state. It did not explain which bytes mattered.

The next step was subtraction. Remove copied pages and replay again. When the result still worked, those pages were not necessary for that experiment. When it failed, the removed state contained a dependency. Once the surviving state was small enough, the team followed pointers, identified object relationships, inferred field ownership, and replaced captured pages with structures constructed by code.

This is more rigorous than asking a model to guess a driver from a memory dump. Every hypothesis meets a physical oracle: the real device. A field description is useful only if changing it causes a predicted behavior. A reconstructed object is correct enough only when it can replace captured state. The experiments, traces, and provenance notes in the published agx-re repositories make that chain inspectable.

The authors describe this as a clean-room process based on live probing, self-written shaders, and opaque treatment of required Apple blobs. That is their documented methodology, not a legal conclusion. Clean-room provenance matters for any reverse-engineered driver, and model-assisted work adds a newer question: maintainers must be able to audit both the source material and how generated code was derived. Publishing experiments is therefore not a side artifact. It is part of the technical case for the code.

Small traces beat heroic analysis

The largest delays came when the experiments violated one principle: isolate the smallest transaction that answers one question.

An early render capture came from late in the GPU’s lifetime, after firmware had accumulated state from many earlier events. Replaying that snapshot appeared plausible, but newly submitted work was acknowledged and retired without executing. Moving the capture to the first transaction after firmware startup exposed a missing one-byte descriptor. The problem was not a profound scheduling mystery; the trace was contaminated by history.

Compute work was harder. In the normal graphical boot path, macOS had already submitted substantial render activity before the first useful compute transaction. One capture grew to 336 MB. That was too much dynamic state to replay or reason about effectively, and the attempt consumed more than a week.

The successful experiment changed the environment instead of adding more analysis. Boot into single-user mode. Launch a tiny service at the earliest moment Metal becomes available. Run one purpose-built compute program. Capture that nearly pure transaction. With the surrounding noise removed, the project reports that the trace was decoded within hours and working compute followed within days.

Partial rendering required the same discipline. Apple GPUs use tile-based deferred rendering. Geometry for a tile is accumulated in a Tiled Vertex Buffer. When that storage is insufficient, the driver must either grow it or perform a partial render, preserving progress and resuming with the remaining geometry. That is effectively save-and-resume inside a render operation, with more state than an ordinary submission.

Instead of waiting for a complex application to trigger the condition unpredictably, the team wrote a shader that hammered one tile with enough triangles to force partial renders. One known-good transaction became several, exposing the repeated state transitions. The lesson is broader than GPU work: when an opaque system is stateful, make the environment boring before making the analysis clever.

User space was a compiler problem

Firmware discovery made it possible to submit work, but useful graphics required understanding a new user-space programming model. The M4 and A18 Pro generation brought changed descriptors and a changed instruction set relative to M1/M2. The team generated small Metal programs, varied one property at a time, captured the resulting shaders and command streams, and changed bits to test candidate meanings.

Disassembly was only the first milestone. A decoder can assign names to bit patterns without knowing enough to generate arbitrary correct programs. The real goal was a backend that accepted Mesa’s NIR and emitted AGX instructions with correct register behavior, control flow, texture operations, and descriptors.

NIR made this tractable. Mesa describes NIR as an optimizing compiler stack at the core of most Mesa driver compilers. Frontends translate shading languages into NIR; shared passes optimize and lower it; a hardware backend performs the final lowering into the GPU’s proprietary ISA. That architecture confines much of the new work to the last mile.

Two development strategies emerged. One path tried to map hardware features comprehensively before integrating them. The other implemented Mesa features incrementally and performed reverse engineering only when a concrete test required it. The Mesa-first path advanced faster because every investigation had a product-level acceptance test. Interesting but unused instructions could wait; a failing texture test could not.

That distinction is important for agent-assisted engineering. Exhaustiveness feels safe, but it can become an unbounded research program. A vertical slice—a real API feature compiled, submitted, rendered, and tested—creates a hard definition of done. It also reveals the next missing fact in context.

The project reports discovering several capabilities not emitted through Metal in its experiments, including a native 64-bit add, higher anisotropy settings, another matrix mode, and a larger immediate form. Those findings are useful clues, but they should be treated as reverse-engineered observations until independent testing and review establish their exact semantics across chips.

Kernel bring-up was a synchronization problem

The first driver was a synchronous Python prototype. That is a sensible starting point because it minimizes concurrency while the ABI is still uncertain. A production-shaped kernel driver needs asynchronous submission: multiple clients create work, dependencies must be honored, completion must wake the right waiters, and one job must not overwrite memory still in use by another.

The team reports moving from its prototype into a Rust Linux driver in stages. First, reproduce the synchronous drm-shim behavior. Next, make the front end asynchronous while keeping the actual device submission serialized. Then associate submissions with firmware events and fences instead of polling. Finally, batch work and remove obvious bottlenecks.

This sequence separates two kinds of correctness. The firmware ABI answers “what bytes make the device act?” The kernel model answers “who owns those bytes, when may they change, and how does completion become visible?” Linux’s DRM and DMA-fence infrastructure supplies the vocabulary, but the driver still has to connect each firmware completion event to the right job and fence.

The published result is impressive precisely because it crosses the entire stack. A shader compiler alone could render offline test vectors. A kernel driver alone could submit opaque buffers. Running desktop compositing and WebGL requires the user-space and kernel halves to agree on memory, commands, and synchronization continuously.

Conformance is the grounding mechanism

Graphics development has an advantage that many agentic coding projects lack: an enormous pre-existing test oracle. Khronos conformance tests exercise required API behavior, edge cases, state interactions, formats, and shader semantics. Passing thousands of cases does not prove maintainability, security, or freedom from intermittent hangs, but it is far stronger evidence than a single spinning cube.

The authors report a complete OpenGL ES 3.0 CTS pass for the supported scope, with skipped cases corresponding to optional extensions. Their repository also shows real workloads, including browsers and Minecraft. These are complementary signals. CTS provides breadth and precise regressions; applications expose integration behavior, long-running state, compositing, and performance.

Tests also changed how the agents were used. The model could implement or alter one feature, run a narrow group of tests, inspect the failure, and generate a smaller hardware experiment. The loop had an objective result at every turn. Where the project stalled, it was usually because the experiment or trace was poorly scoped, not because the model lacked another paragraph of instruction.

This suggests a practical rule for ambitious systems work: build the oracle before scaling the agent. The best prompt in the project was often a controlled workload, a trace diff, or a conformance failure.

Working is not the same as upstream

The one-month result compresses exploration and implementation, but it does not compress trust. Upstream graphics code must survive adversarial inputs, multiple processes, suspend and resume, memory pressure, different firmware releases, and years of refactoring. Kernel code adds stricter expectations around locking, lifetime, error recovery, and ABI stability.

Reviewability is its own deliverable. A large body of generated code can work and still be impossible to merge if maintainers cannot establish why each structure, bit, and synchronization rule exists. The next phase must turn experiments into concise documentation, separate mechanical code from inferred behavior, remove dead ends, and create patches that humans can reason about independently.

There is also a sequencing constraint. The project’s kernel work builds on the existing Asahi user/kernel interface, while the M1/M2 kernel driver has its own upstream path. The authors expect the older foundation to land first. Their Mesa work can potentially move on a different schedule, but it too requires testing, review, and a maintainable series rather than a research dump.

Vulkan 1.4, desktop OpenGL 4.6, OpenGL ES 3.2, OpenCL, compatibility layers, and ray tracing remain goals, not delivered features. The published code is valuable because it establishes a working vertical slice and a repeatable discovery process. It should not be mistaken for completion of that roadmap.

The real acceleration was experimental bandwidth

It is tempting to describe this project as an agent writing a GPU driver. That misses the more transferable achievement. The agents increased the number of experiments that could be written, run, compared, and discarded. The humans chose targets, noticed when the trace was polluted, changed the boot environment, prioritized Mesa integration, and defined what evidence counted.

Three practices made the acceleration useful:

1. Anchor every inference to a known-good transaction. Replay before reconstruction; preserve the oracle while replacing the mystery.
2. Reduce the environment, not just the code. Early boot, tiny shaders, forced edge cases, and single-purpose traces beat giant captures.
3. Drive toward a vertical test. A Mesa feature passing CTS and rendering in an application is more valuable than a complete catalogue disconnected from a driver.

The result does not make undocumented hardware easy. It changes the economics of disciplined probing. A small team can explore more hypotheses without lowering the standard of evidence—provided it keeps the device, the tests, and human review in charge.

That is the durable lesson of the M4 driver: opaque systems become understandable when each layer is forced to answer a narrow, falsifiable question. The speed came from running that loop many more times, not from skipping it.

Sources

• Cody Ho: Building a GPU Driver From Scratch in One Month
• ADevWithAnIdea/agx-re experiments and provenance
• GravityLinux Mesa fork
• GravityLinux M4 kernel branch
• Asahi Linux AGX hardware documentation
• Asahi Linux: Apple GPU drivers now in Asahi Linux
• Mesa NIR documentation
• Mesa Gallium documentation
• Linux kernel DMA fence documentation
• Linux DRM scheduler documentation
• Khronos OpenGL and OpenGL ES conformant products