White-Collar Bench: Benchmarking Long-Running Project Work
By Vivek Haldar
(This blog post is derived from this video on my YouTube channel.)
I’m releasing a benchmark I’ve been working on for a while: white-collar-bench.
As the name implies, it benchmarks an agent’s ability to keep track of long-running, cross-team project work–the kind of knowledge work most of the economy actually runs on. Modern knowledge work consists of collaborating with dozens of people across multiple projects, and a huge part of the job is simply keeping a complete and consistent picture of where everything stands: who is working on what, who owes what action items to whom, by when, and how the state of the world changed. And you have to do this on a daily basis over weeks and months.
Pebbles and projects
I’ve wanted a solution to this problem pretty much since I started working. Over a decade ago, back in 2014, I wrote about how poorly our patchwork of email, messaging, and docs fits the way work actually happens:
Imagine these emails/docs as pebbles, and each project as a collection of pebbles. This project is red pebbles. That one is blue pebbles. Yet another one is yellow pebbles. Then imagine throwing all of them on the floor. Then imagine being asked “can you construct for me a coherent state of all the yellow pebbles?” That’s how it feels trying to get a high-level view of a project from all these pebbles flying around.
More recently, I wrote about how (and why) to be an effective meat proxy. I want to hand off all this grungy project tracking and bookkeeping to agents so I can spend my time on strategy, judgment, and the work that actually needs my brain.
In my own day-to-day setup, an agent maintains a plain Markdown store of projects, people, and open loops. It works well enough that I wanted a real way to measure it—to compare models, test different agent harnesses, and see where they actually break down.
Why this benchmark
white-collar-bench is a composite benchmark. Keeping a consistent and complete model of evolving knowledge work exercises many capabilities all together:
- long-term memory
- knowledge ingestion
- temporal reasoning
- maintaining a coherent world model
- language understanding and synthesis
- disambiguation (nicknames, handles, shared first names, indirect references)
There are many benchmarks that test each of those in isolation, or some small subset of them. Workplace-agent benchmarks like TheAgentCompany or WorkBench pose independent tasks, so a mistake never carries forward. Enterprise search benchmarks like HERB query a fixed, read-only corpus. Memory benchmarks like LongMemEval or MEMTRACK treat memory as an internal black box and grade it only through downstream Q&A.
What I couldn’t find was a benchmark that tests the whole composite activity of long-running knowledge work and project tracking as its main focus.
Daily ingestion and synthesis
Think about how your actual workday unfolds. You come into work, to a constant incoming stream of emails, chat messages, meeting transcripts, documents, and calendar invites. You ingest all of it and update your internal world model: how a project evolved, who handed work off to whom, which deadline moved, which decision got changed in the fine-print of a meeting’s notes.
Then you draw on that world model to answer questions and move work forward. Those questions fall into two broad buckets:
- Factual questions. Who owns this work item? What do I owe John? What action items does Mary owe, and by when? These look like simple facts—except they evolve over time. In chat, someone says they’ll have an API ready “by Mar 5”; a week later, in a meeting transcript, it slips to “the 10th.” You have to track the current truth, not the first thing you saw. (And some questions have no answer in the inputs at all, in which case the right move is to say so.)
- Synthesis questions. Tell me the overall status of Project X. List what got done, what’s still open, what the blockers and risks are, and what the highest-priority items to tackle next are.
The central pain point today is that answering questions like these means manually sifting through Slack, email, docs, and meeting notes and stitching the state together in your head. An entire job family exists to tackle this task: project manager. That is what I’m trying to automate, and that is what white-collar-bench captures.
How the benchmark works
Under the hood, every world in white-collar-bench is generated from a hidden, event-sourced world model. This is a ground-truth log of people, organizations, projects, commitments (“open loops”), and decisions, each with validity intervals.
Because the inputs are rendered from that log and the gold answers are computed directly from it, two nice things fall out for free: every checkpoint has exact, auditable ground truth, and we can generate fresh synthetic worlds at will. There’s no confidential real-world data to leak, and no static test set to memorize.
A run replays one to four simulated weeks for a single knowledge worker (around 300 artifacts per week across a dozen projects and dozens of people), alternating two phases:
- Daily ingestion. Each simulated day, a fresh agent process receives that day’s inbox—emails (
.eml), chat threads, meeting transcripts (.vtt), calendar invites (.ics), office documents (.xlsx,.docx,.pdf), and the worker’s own shorthand notes. The agent updates its store on disk. Then the day’s inbox is deleted. The store is the agent’s only memory. Getting this right is crucial, because errors in ingestion today will compound into the future. - Checkpoint questions. At set checkpoints, a fresh agent process answers questions from the store alone and cites its evidence, across eight equally weighted question types: point facts, lists, status summaries, briefs, changes, dates, priorities, and unanswerable questions.
The inputs are messy in the ways real work is messy. People show up under nicknames, initials, handles, and job titles; two colleagues can share a first name. Evidence can be weak or partial. For example, an unsent draft or a calendar acceptance doesn’t signal that something got done. Newsletters and chatter mix with real commitments, and generated worlds even slip in an instruction-like line the agent is supposed to ignore.
The benchmark has two tracks:
- Track A (specified contract): The agent starts from a correct Markdown store built from prior weeks and must maintain it under a published store contract (people, organizations, projects, loops, and daily logs). We grade both the answers and the store itself, and also run the same questions against an oracle store so you can separate synthesis skill from ingestion cost.
- Track B (open representation): The agent starts with an empty folder plus the prior weeks’ raw inputs and no schema. It isn’t told who the people are or what the projects are—it has to infer all of that from the incoming stream, organize its memory however it likes, and is graded solely on its answers.
Results so far (and why I haven’t run Opus)
The repo ships with 11 public worlds (one custom 1-week pilot world plus 10 procedurally generated dev worlds spanning 1 to 4 weeks, 745 questions total). I ran two models on it so far using the Pi harness, scored with the built-in heuristic judge:
- DeepSeek V4.1 Flash (Pi harness): 84.6 on Track A, 67.5 on Track B
- Qwen3.8-27B (Pi harness): 72.3 on Track A, 55.0 on Track B
Two things jump out even from these early runs:
First, the benchmark is far from saturated. Even on Track A, there is plenty of headroom—especially on tracking what changed across days and distinguishing actual commitments from unaccepted suggestions.
Second, both models drop roughly 17 points on Track B when they have to invent and maintain their own memory layout from scratch instead of working inside a structured contract.
In my real job, I run my personal setup on much stronger frontier models like Opus. So why didn’t I benchmark Opus, Fable, or Astra here? Purely the token bill. Just running Qwen and DeepSeek across all 11 worlds and both tracks cost a couple hundred dollars in tokens (about $119 for the DeepSeek run alone). Extrapolating that out, a full run on Opus or Fable would probably run a couple thousand dollars, and I personally do not want to pay that bill.
Try it—and help improve it
white-collar-bench is at v0.1 and not perfect by any means. I wanted to get it out there early and get feedback on the idea rather than keep it private and polish it forever.
So I have two requests:
-
Run it on your models and harnesses, and share the scores. The repo includes adapters for Claude Code, Codex, raw Anthropic or OpenAI-compatible API loops, Docker containers, and custom scripts. You can get a smoke test running in a few lines.
If someone out there has the token budget to run this against the stronger, more expensive frontier models—or wants to pit their own memory harness against Track B—please go wild and report back what you get.
-
Contribute to the benchmark. The repo is public under the Apache 2.0 license. Open issues, fix bugs, help calibrate the judge against human labels, or submit PRs to improve the benchmark itself.
Go take a look at the GitHub repo. I’d love to hear what you think—and see what scores you get.