We Spent £5,000 on Our Own AI Server: What It Does and What We Learned

In 2024 I spent around £5,000 on a computer to run AI in our own office. The idea was simple: if the AI ran on our own machine, we could give it our own documents without sending anything to another company. It did not work out the way I expected. The models we could run ourselves were not good enough, and since then the big cloud AIs have changed so much that most of the reason for having it had gone. It has ended up earning its keep in a way I did not plan. This is what we bought, what went wrong, and what I would tell anyone thinking of doing the same.

What we bought

It is a Dell Precision 7875 tower, a workstation rather than a server in a rack. The parts that matter for AI are the graphics cards, because that is where AI models run.

• Two NVIDIA RTX 4000 Ada graphics cards, each with 20 GB of memory, so 40 GB in total
• An AMD Threadripper PRO processor with 12 cores
• 64 GB of memory and a 1 TB solid state drive
• Ubuntu Linux, with Ollama to run the AI models

It sits in our office. We reach it over a private Tailscale connection rather than the open internet, so nothing on it is exposed to the outside world.

The models were not good enough

You cannot run Claude or ChatGPT on your own machine. What you can run are open models that companies publish for anyone to download, such as OpenAI's gpt-oss. The ones that fit in 40 GB of graphics memory are impressive for their size, but they are well behind the big cloud models. For the work we wanted, such as reading our documents properly and writing something we would put our name to, they were not good enough.

We tried RAG, and then it stopped mattering

In 2024 the AI models could only take in a limited amount of text at once, so you could not simply hand one a pile of documents. The standard answer was RAG, short for retrieval augmented generation. Instead of reading everything, the system searches your documents for the passages that look relevant and gives the AI just those. I set that up on the server so it could answer questions about our own material.

That was the best option at the time, but then the cloud models changed. The amount of text they can take in at once, called the context window, grew so large that you can now give them huge amounts of text: whole contracts, long email threads, sets of documents, all in one go. There are still limits, and very long inputs can make them less precise, but for our documents they are more than enough. That was not possible in 2024. Once you can hand over the whole document, we no longer need the careful searching that RAG does, and the cloud models do a better job with it. RAG still has its place for very large collections of documents, far more than any model can read at once, but that is not our situation.

The other half of the problem was privacy, and we solved that differently. We now take the confidential details out before anything is sent, replacing them with tokens and putting them back afterwards, as we explain in how we use Claude and ChatGPT safely. With that in place, most work can go to the better cloud models without giving anything away.

Your situation may be slightly different. ask a question below ↓ and our editorial team will reply with our advice.

What it does now

The server found a different job. Its busiest work now is speech to text: it runs Whisper, an open model that turns audio recordings into written text, and it handles around 300 recordings a month, mostly on weekdays. Speech to text is something the open models do very well, it runs comfortably on one of the two graphics cards, and the recordings never leave the building.

It is also a fallback for the occasional document that cannot sensibly be anonymised, so we can still use AI on it without sending it anywhere. Its graphics cards use about 21 watts between them when idle, so leaving it running costs little. But it is no longer the centre of how we use AI for writing and documents, and that is not what I expected when I bought it.

What I would do now

For most small businesses I would not start with your own AI server. Use the cloud models through their API or a business plan that does not train on your data, take out confidential details before you send anything, and spend the money elsewhere.

It can still make sense if you have information that must never leave your building, a large and steady amount of routine AI work, or you want to learn how these systems work from the inside. If that is you, I would not buy another tower like ours. I would buy an NVIDIA DGX Spark: a small desktop box, about the size of a hardback book, with 128 GB of memory shared between the processor and the graphics. Our two cards have 40 GB between them, and a model has to be split across both to use it all. The Spark has more than three times that in one pool, and memory decides how large a model you can run, which was exactly the limit we hit. It launched in the UK at around £3,700 including VAT, but prices have risen sharply since because of memory shortages and now vary a lot between suppliers, so check current prices before you buy. Asus, Dell, HP and MSI sell their own versions built on the same chip.

Whatever you buy, test the open models on your own real work first. The gap between them and the best cloud models is what caught us out, and AI is moving fast enough that the reason you buy hardware today may not last.