Anthropic ran a 201-person AI trading floor. The model mattered far more than the instructions

Anthropic turned its own offices into a trading floor. In Project Swap, 201 employees across six locations -- 115 in San Francisco, 57 in New York, 12 in London, 8 in Seattle, 6 in Washington DC, 3 in Dublin -- each brought a book they wanted to give away, had a five-minute conversation with Claude about what they actually like to read, and then sent a Claude-powered agent onto an open trading floor to pitch, haggle, and close deals with everyone else's agents. No human touched a single negotiation after that first intake chat.

The point of the exercise wasn't to see whether the agents would trade -- Anthropic's earlier Project Deal, a $100-per-person gift-card marketplace that closed 186 trades worth just over $4,000 among 69 employees, already showed they would. It was to measure, for the first time with a real yardstick, how well they traded. Project Deal used idiosyncratic goods -- ping-pong balls, snowboards, free-form haggling -- where there was no clean way to say how good an outcome *could* have been. Project Swap fixed that: books can be ranked, rankings can be scored against a theoretical best-case allocation, and a market's efficiency can be stated as a single number.

The first number is the one worth sitting with: after a five-minute onboarding conversation, Claude's read of a participant's book preferences matched that participant's own real rankings on 61% of pairs. That beats a coin flip (50%), beats ranking by simple popularity (53%), and beats collaborative filtering (55%) -- the kind of recommendation-engine approach that usually requires far more data than one short chat provides.

But 61% accuracy on preference pairs is not the same as a well-run market, and Anthropic didn't let the two get conflated. Measured against participants' *actual* preferences, the overall marketplace hit an efficiency score of 0.55 -- against a theoretical optimum of 0.89 for that same pool of books and people. That's a real gap, and Anthropic's own researchers decomposed it rather than waving it away: roughly 85% of the shortfall traced to Claude's imperfect read of what people wanted, the same limitation the 61%-not-100% preference-matching number already flagged. Only about 15% came from the agents actually negotiating badly once they had a preference model to work from.

What actually moved the number

Anthropic also tested what would improve results, and here the finding cuts against a lot of agent-building folk wisdom. Swapping the underlying model mattered enormously: agents running on Claude Haiku hit 0.75 efficiency measured against Claude's own rankings, while agents on Claude Opus hit 0.88 -- a substantial gap driven purely by model capability. Swapping *instructions* -- telling an agent to negotiate "ruthlessly" versus "prosocially" -- moved the same efficiency measure by just 0.02, a difference small enough to be noise. The lesson Anthropic draws explicitly: for a task like this, which model you deploy matters far more than how cleverly you prompt it.

That last line matters because Anthropic is explicit about what this study can't tell you. Every participant was an Anthropic employee, which the researchers themselves flag as likely inflating trust in Claude relative to a general population. Nobody had real money or meaningful stakes riding on the outcome. Only well-behaved Claude variants were tested -- nothing adversarial, nothing from a rival lab. And the marketplace's own rules stayed fixed throughout, so the study says nothing about how these numbers would move if the rules themselves were being gamed. Anthropic also reported a quieter failure mode: some participants simply never brought their promised book, leaving trades unfulfilled in a way the researchers couldn't systematically track.

The timing lands in the middle of a real agentic-commerce rollout, not a hypothetical one. This is the same week Amazon opened its Seller Central tools to outside agents through a Claude plugin while blocking a rival shopping agent from checking out on its own storefront -- a live example of exactly the marketplace-governance question Project Swap's own recommendations are aimed at: who gets to run an agent in someone else's market, and on what terms. Anthropic's proposed answer -- certify agents on sample decisions before they act alone, log everything so a human can audit it after the fact -- is a design pattern platforms are already being forced to invent policy for in production, not in a research paper.

What Project Swap actually establishes, stripped of the marketplace framing, is narrower and more useful than "AI agents can trade for you": a strong model reading a stranger's stated preferences for five minutes already beats a recommendation algorithm trained on real behavioral data, and the single biggest lever for improving an agent's real-world negotiating isn't a better prompt -- it's a better model underneath it. For a beat that spends most of its time evaluating whether a benchmark number means what a lab says it means, this is a rare case of a lab publishing the number that complicates its own pitch: Claude is good at this, and still gets most of the credit for the *remaining* gap, not the part that's already closed.