[Experiment 12] Can Claude Fable 5.2 beat my DeepSeek agent as a retail category manager?
Introduction
In experiment 9, a Hermes agent running a cheap DeepSeek model learned to manage a three-product tomato category. It did well. Best score $5,509.55 over 14 runs, and strategy summaries that were better than what most humans would write.
But one question stayed with me after that. How much of the result came from the agent setup and the documentation, and how much from the model behind it?
So for experiment 12 I took the exact same simulation and gave it to Claude Fable 5.2. Same game. Same 14 runs. Same kind of documentation. Different brain.
In short: Fable won, and it was not close. The best run ended at $6,586.74. The average over 14 runs was $5,331 against $4,373 for DeepSeek. And the leaderboard now shows fable52bot on 8 of the top 10 places.
The Problem
The business problem is the same as in experiment 9. You run the tomato category of a supermarket for 42 days. Every morning you set the shelf price of three products and place orders with five suppliers. Buy too much and it rots. Buy too little and the customer walks out. And a customer who walks out is VERY expensive.
The second problem is one I created myself in experiment 9. My main conclusion back then was that the agent changed its strategy too quickly and without enough evidence. I wanted to see if a stronger model, with a stricter documentation structure, handles that better.
The Hypothesis
- H1: Claude Fable 5.2 beats the DeepSeek-based Hermes agent on the same game, measured in profit per run.
- H2: One strategy document with clear decision rules, where every old version is kept, makes the strategy changes traceable and better founded.
As usual, I am the only judge. Profit per game is the one hard number. The quality of the strategy work is my own assessment.
The Technical Setup
The Game
The game is Retailer Tomatoes v2, the same simulation as in experiment 9. A short reminder of what is in it:
- Three products: Value (imported), French and Organic French
- Five suppliers: two importers that deliver in two days and three farms that deliver the next morning
- Shelf life of 5, 4 and 3 days depending on the product
- A different price sensitivity per product
- Monday volume forecasts from the suppliers, with a discount if you commit and a penalty if you do not deliver on it
- Strikes, storms, hail, bad batches and short deliveries
- Every customer who walks out costs about $5 in lost basket
The score is revenue minus supplier payments, penalties and lost baskets. One season is 42 days, so 42 decisions.
The AI Agent
This time there was no Hermes, no VPS and no Telegram bot. I used the Claude app with Claude Fable 5.2. It played the game through the built-in browser and wrote its documentation into a Google Drive folder.
The whole instruction was one message. Sign up as fable52bot, play the game 14 times and go for the highest possible score by following the game rules. The documentation I asked for:
- Technical guidance (how to navigate, how to save) kept separate from business strategy
- One strategy guidance document with the decision rules, updated after every game, with hypotheses at the bottom and every old version kept
- One document per run: the strategy at the top, a record of the decision in each of the 42 turns, then a summary, what went well, what could be better and new hypotheses
- Any tools, code or calculations documented as well
I did not give it anything from experiment 9. No learnings, no strategy file. It started from the published game rules only.
What came back was 14 run documents, the strategy guidance in 15 versions (v0 to v14) with 62 numbered hypotheses, a technical guide, a summary and a tools folder. Looking at the timestamps in my Drive, the first run document was created at 10:12 and the final summary at 11:47. So about an hour and a half for 14 seasons and all the paperwork.
The Outcome
Short Summary
Overall, a success on both hypotheses, with a reservation on H2 that I come back to further down.
What impressed me most was not the score. It was run 1. The agent played the first season by hand, at fixed prices, as a measurement run. From that it worked out how the shoppers in the game think. They compare each shelf price with 1.6 times the cheapest supplier quote of that morning, and demand drops in a straight line above it. Saturday sells 1.5 times a Monday. Sunday is the weakest day.
From run 2 it reset the three prices every morning, each one at the point where the profit is highest. That came out at about 1.66x the cheapest quote for Value, 1.82x for French and 1.98x for Organic.
Then it did something I did not ask for. Seasons in this game are random, and the profit you can make in one season ranged from about $4,000 to $6,400. So the agent built its own yardstick. For every season it calculated what a player with perfect pricing would earn when buying every kilo at the cheapest quote of the day it is sold, with no waste and no empty shelf. Its score as a share of that number it called efficiency.
Efficiency was 97 to 99% in runs 1 to 4 and 100.6% on average over runs 5 to 14. Above 100% is possible because the agent learned to buy several days of stock when a quote is low.
Detailed Outcome
The top 5 scoreboard:
- Run 8: $6,586.74
- Run 5: $6,068.83
- Run 6: $5,906.13
- Run 14: $5,599.00
- Run 7: $5,576.65
And all 14 runs, with the efficiency in brackets:
- Run 1: $4,765.18 (98.5%). Measurement run with fixed prices, played by hand. The demand model was found here.
- Run 2: $3,988.95 (98.7%). First run played with the new pricing and the agent's own script. A small season.
- Run 3: $4,367.37 (96.8%). Tried one main supplier per product for reliability. Purchase cost went up 2 to 6%, so it was reversed.
- Run 4: $5,515.14 (98.5%). Back to buying from the cheapest supplier. 43 kg left on the last day.
- Run 5: $6,068.83 (103.8%). Bigger safety stock for French and Organic, and more days of stock bought when quotes are low.
- Run 6: $5,906.13 (99.7%). First forecast penalty ($51.84).
- Run 7: $5,576.65 (98.0%). Splitting orders between two suppliers cost about $70. A hail-damaged load of 55 kg expired.
- Run 8: $6,586.74 (103.4%). BEST. 0.3 kg binned in the whole season and Value bought at 92% of the selling-day quote.
- Run 9: $5,558.61 (100.5%). No walkouts, and no manual decision by the agent.
- Run 10: $5,486.31 (96.7%). The worst run by efficiency. Three storms, $73.20 in penalties and 60 kg of French binned after a hail alert.
- Run 11: $4,800.09 (99.7%). A small season. A strike without notice emptied the Value shelf for two days.
- Run 12: $5,166.78 (100.7%). A quiet season. 9 kg of Organic expired in the first days.
- Run 13: $5,243.41 (103.2%). No walkouts, Value bought at 94% of the selling-day quote.
- Run 14: $5,599.00 (100.0%). Two rules for rare events misfired and cost about $72.
A few things it figured out along the way that I find really good:
- The shelf sells in order of arrival, not in order of expiry. So a damaged load with 2 days of life that arrives behind older stock will rot.
- A supplier you have not ordered from for about a week forgets you and delivers short, or nothing at all.
- A Monday forecast commits you for the following week, and the penalty per missing kilo is about eight times the discount per kilo. So only commit to a supplier that is the cheapest over the week and on the day, and only for a discount of 6% or more.
- Daily demand moves about 11% around the forecast whatever you do. That sets the minimum safety stock.
A technical side note. Like Hermes in experiment 9, Fable did not click through the game for long. After run 1 it wrote a script that runs inside the browser page, reads the day, applies the rules of the strategy document and submits the game's own form. From run 5 a full season took three or four calls. I accept it for the same reason as last time. It made play and learning much faster, and the rules in the script are the rules in the strategy document.
Experiment 9 vs Experiment 12: DeepSeek vs Claude Fable 5.2
Same game, same number of runs, same leaderboard. Here are the numbers side by side, DeepSeek Flash 0731 (in Hermes) first and Claude Fable 5.2 second:
- Best run: $5,509.55 vs $6,586.74 (+20%)
- Average of the 14 runs: $4,373 vs $5,331 (+22%)
- Median run: $4,215 vs $5,501
- Worst run: $3,295.15 vs $3,988.95
- Runs above $5,000: 2 vs 10
- Fable runs that beat the best DeepSeek run: 7 of 14
- Fable runs that beat the average DeepSeek run: 12 of 14
- Leaderboard today: places 1 to 7 and 9 are fable52bot. Places 8 and 10 are the old bot from experiment 9.
The scores are the result. The more interesting part is why.
- Understanding against rules of thumb. DeepSeek priced everything at about 2x cost, and when it tried a different price per product in run 12, the Organic demand collapsed. Fable first measured how the shoppers react to price and then set each price from that. Its prices came out below 2x for all three products.
- Customers walking out. DeepSeek lost $738 in baskets in its first run and still had 38 and 54 walkouts in runs 11 and 14. Fable had 26 walkouts in total over 14 seasons, and four seasons with zero.
- Waste. DeepSeek binned 415 kg in run 4 and fought the Value volume against rot trade-off through the whole experiment. The worst Fable season was 74.5 kg binned and the best was 0.3 kg.
- Recovery after a bad run. DeepSeek peaked in run 7, got hit by a supplier disruption in run 8 and never came back to that level. Fable also had its bad run (run 10, three storms), and the four runs after it came in at 99.7%, 100.7%, 103.2% and 100.0% efficiency.
- Evidence before change. This was my main complaint in experiment 9. Fable was not perfect here, as you will see below. But from run 10 it tested new ideas against the recorded data of all earlier seasons before changing a rule, and it rejected four of its own ideas that way.
Now, to be fair to DeepSeek:
- Seasons are random, so the two agents did not play the same 14 seasons.
- The agent setup is different. Hermes on a VPS with markdown files against the Claude app with a browser and Google Drive. So this compares two complete setups, not two models in a lab.
- Fable 5.2 is a much more expensive model than DeepSeek Flash. I am not comparing cost per run here, only the quality of the decisions.
- In experiment 9, runs 11 to 14 were experiments that I asked for, which may have cost some score.
Still. The spread between runs was about the same for both agents, roughly $680. A gap of $958 in the average over 14 runs each is too large for me to explain with lucky seasons. For this simulation, DeepSeek is clearly the weaker category manager.
What Did Not Go So Well
This is my reservation on H2. The documentation made every change traceable. It did not stop bad changes from happening.
- Three strategy changes made the result worse before they were reversed: the main supplier rule (run 3), the order splitting (run 7) and the hail rule.
- The hail rule went wrong twice. First the agent ordered 25% more after a hail alert, and 60 kg rotted in run 10. Then it avoided the affected farm at any price, and paid $57 too much in run 14. The fix is in the final version v14, which was written after the last game. So it is untested.
- After run 5 there was no real upward trend. Each change was worth $10 to $50 per season, which is less than the difference between two seasons. One run cannot confirm a change of that size.
- Efficiency is the agent's own yardstick. The game does not report it. I like the idea a lot, but the agent is grading its own homework there.
What I did appreciate is that the agent reported its own mistakes without me asking. One figure in strategy version 10 was calculated against the wrong week and is corrected in version 11. The message dates in the logs of runs 1 to 9 may be one day early. And the plan in the run 9 document was written after that run had started. All three are written in the documents.
Further Improvements
- Run both models in the same agent setup and on the same fixed seasons, to make the comparison clean
- Compare cost per run, and not only the score
- Give the agent the strategy document from this experiment and see if it starts at the run 5 level right away
- Add random demand to the agent's own simulator, so that safety stock can be set per product and weekday (its own suggestion in the last hypothesis)
- Let a few humans play the same game, which is still on my list from experiment 9
Closing Thoughts
So, can Claude Fable 5.2 beat my DeepSeek agent as a retail category manager?
YES. By about 20% on the best run and 22% on the average, with far fewer lost customers and far less waste.
But the score is not what I will remember from this one. In experiment 9 the agent learned good rules of thumb. In experiment 12 the agent first worked out how the game behaves and then calculated its rules from that. That is the difference between a good operator and a good analyst, and here I got both in one.
My idea from experiment 9 still stands, though. Strategy changes need proof, and one run is not proof. Fable started to do this by itself halfway through, but the two rules that cost money in run 14 were both written from one or two cases. For a real store I would keep the same rollout as before: the agent proposes, the category manager validates, and control is handed over step by step.
What feels particularly good is that I can open the strategy document and read why every single rule is there, which run it came from and what it cost when it was wrong. I have worked with supply chain teams for 20 years, and I have not seen many decision logs like that.
Cheers,
Andreas