Can an AI on Our Own Server Match Claude? How We Tested It

Our busiest free tool is the birth chart calculator on Explore Astrology. Our own code works out where the planets were when you were born, and an AI turns that into a personal reading. In October 2026 every reading was written by Anthropic's Claude Haiku 4.5, and every one cost money. Downstairs in our office there is a computer with two graphics cards that we already owned. So I wanted to know: could an AI running on our own machine write those readings well enough to replace Claude? We tested it properly. The answer was no, and then, halfway through the test, the question changed.

Why it seemed worth trying

An open model runs on your own hardware, so there is no charge per reading, no dependence on one supplier and nothing leaves the building. OpenAI's open model gpt-oss, released under the Apache 2.0 licence, comes in a 20 billion parameter version that fits on our machine, a Dell Precision tower with two NVIDIA RTX 4000 Ada cards and 40 GB of graphics memory between them. I wrote about the machine itself in We Spent £5,000 on Our Own AI Server.

How we tested it

A test like this is easy to get wrong, so we set it up carefully.

  • Real requests. We took 20 real birth chart requests from the live tool, with any repeats removed.
  • Identical instructions. Both models received exactly the same instructions and the same calculated chart, so they were only being tested on how well they read and explained it, not on the astronomy.
  • Blind judging. Each pair of readings was marked out of 10 by an AI judge on accuracy, following the instructions, insight, writing and an overall verdict. The judge was never told which model wrote which reading, and it listed every factual error it found against the chart.
  • Order swapped. AI judges tend to favour whichever answer they see first, so every pair was judged twice, once in each order. If the two verdicts disagreed, it counted as a tie.
  • Two judges from different companies. AI judges also tend to favour writing that sounds like their own. Our main judge was Anthropic's Claude Opus, the same family as Haiku, so every pair was judged again by OpenAI's GPT. It is the approach I describe in Using Two AIs to Check Each Other.

The results

Our own model lost clearly. Claude Opus scored Haiku 4.5's readings 7.2 out of 10 on average and our model's 4.7, and preferred Haiku's reading on 18 of the 20 charts. GPT marked everything lower but agreed: 5.8 against 4.4. Our model was also slow, taking about 33 seconds per reading against Haiku's 16.

We tried two ways to help it:

  • Less thinking. Turning its reasoning down made it as fast as Haiku, but its score fell to 3.8. Speed cost about a point of quality.
  • Simpler input. We gave it the chart already cross-referenced, so it did not have to work out which planet sat in which house. That cut the errors GPT found by about a quarter and narrowed the gap a little, but nowhere near enough.

Then we tried bigger open models, too large for our machine, through a hosting service. Size mattered more than anything we did to the instructions. The 120 billion parameter version of gpt-oss closed about half the gap to Haiku, DeepSeek V3.2 came closer still, and the largest model we tried, Alibaba's Qwen3 at 235 billion parameters, came within a fraction of a point. But it needs far more memory than we have, and even hosted it took 23 to 30 seconds per reading, too slow for a visitor waiting on a page.

Chart of each model's score compared with Claude Haiku 4.5 under two judges: our own gpt-oss 20B scored 2 to 3 points lower, larger hosted open models came close, and Claude Haiku 5.5 scored about 2.4 points higher
How each model scored against Claude Haiku 4.5 on the same 20 birth charts, out of 10, under each judge. Below zero means worse than Haiku 4.5.

The judges did not always agree

Across 120 pairs, the two judges gave the same verdict 92 times and opposite verdicts only 11 times, and they ranked every model in the same order. But Claude Opus consistently gave Haiku a bigger lead than GPT did, by 30 to 50 per cent, mostly on the quality of the writing. That fits with a judge favouring its own family's style, although GPT also marked more harshly overall, so we cannot be sure. The lesson is simple: if you use an AI to judge AI, do not let a model judge its own relatives alone.

Your situation may be slightly different. ask a question below ↓ and our editorial team will reply with our advice.

Then the supplier moved

On 7 October 2026, in the middle of the test, Anthropic released Claude Haiku 5.5, which reasons through a problem before it answers, at a tenth of the price of Haiku 4.5. We ran it through the same test and found a trap straight away. Haiku 5.5 does its thinking inside the same allowance as its answer, and at our limit of 2,500 tokens every reading came back cut short, some before a single word of the reading itself. Had we simply swapped the model name, every tool on our sites would have broken while appearing to work.

With room to think, it was the best result of the whole test. Claude Opus scored it 7.95 against 5.58 for Haiku 4.5 on the same charts, and GPT 8.05 against 5.62. It won on all 20 charts under Opus and 19 of 20 under GPT, made far fewer errors, and the biggest gains were on accuracy, exactly where every open model had fallen short. It took 17.5 seconds against 15.7, and cost $0.0020 per reading against $0.0074. A spot check of our ten most used tools found both judges preferring it on all ten.

You may notice Haiku 4.5 scoring 5.58 here and 7.2 earlier. That is not a mistake. Haiku 4.5 wrote its readings afresh for every test, and the judges mark each pair side by side, so its score moves from one test to the next: between 6.35 and 7.2 across the open model tests, and 5.58 when it sat beside Haiku 5.5. That is why we always compare each model with Haiku 4.5 within the same test, never across tests.

On 8 October 2026 we moved all 441 tools on our sites to Haiku 5.5, each with its thinking switched on or off to suit the job.

What I learned

  • Test on your own real work. Benchmarks tell you little. Twenty real requests, judged blind, told us more than any league table.
  • In our tests, bigger models won. No amount of tidying the instructions made our small model write like the larger ones. Size is not the only thing that matters, since how a model is built and trained counts too, but on this task it made the biggest difference.
  • Use judges from two companies. One judge can carry a bias you cannot see.
  • Check for hidden limits when you change model. A newer, better model can break a working system in ways that do not show as an error.
  • Making your own is a moving target. Within a single day the bought option became better, cheaper and about as fast as anything we could build. A home-made model has to improve faster than the market to stay worth running.

We did look at one more route: training our small model on readings written by a far better one. Claude was the obvious teacher, and Anthropic's terms do not allow it. That story is in Why We Cannot Train Our Own AI on Claude or ChatGPT Answers. The machine downstairs has not gone to waste: it now turns around 300 audio recordings a month into text, a job open models do very well, and the recordings never leave the building.