Two Thirds of Our Website Searches Found Nothing: How We Fixed It

Most websites have a search box, and most owners never look at what people type into it. We hadn't either, until today. When we did, we found that about two thirds of the real visitors who used the search box on our websites were told there was nothing to find. Many of them were looking for things we had written about. This is how we found out, how we fixed it in a day, and what it costs to run.

What we found

We run a network of over 300 information websites from one codebase. Every site has the same search box. We had never kept a record of what people searched for, so the first job was to build a report that does.

The web server logs go back far enough to rebuild the last month. In the month to 16 November, the search pages were requested more than 25,000 times. Almost all of those requests came from bots, mainly search engine crawlers trying out search addresses they had found. Our page view tracker already separates real visitors from bots, so we matched each search against it and kept only the searches made by real people. That left 189 searches, 175 of them different.

121 of those 175 came back with nothing. Reading through them showed why. Our search only looked for the exact phrase that was typed, so anything other than a short, correctly spelt phrase failed:

  • Questions: "How many people over 100 in uk?" found nothing on a site about living to 100, which has an article called How Many People Live to 100 Across the Globe?
  • Spelling mistakes: "concentratrion", "zodiac compatability" and "segetarians" all failed.
  • Longer phrases: "Cut up jeans" found nothing on a recycling site with an article called How to Make a Bag from Old Jeans, because those three words never appear together.
  • Plurals and wording: "Rosy loach" found nothing on a tropical fish site with an article called Loaches.
  • The answer was on another of our sites: someone searched "pest control" on our affordable housing site. We have a whole site about pests, but the search only offered it at the bottom of the page, under four housing articles that happened to mention pests.

Every one of those visitors left thinking we had nothing for them, when we did.

The fix, in three steps

Step one: match each word. If the exact phrase finds fewer than three articles, the search now splits the search into its meaningful words, ignoring small words like "how", "the" and "in", and looks for each one. It also trims word endings, so "freeze" finds "Freezing". Articles are ranked by how many of the words they contain, with matches in the title counting most. This needs no AI and costs nothing to run. "Cut up jeans" now finds the jeans bag article first, "Rosy loach" finds Loaches, and "willow tree" on our neighbour disputes site finds the tree dispute tool and four articles about neighbours' trees.

We soon found that matching words is not the same as finding the answer. A question about whether a bank can freeze your account brought up "Simplest Way of Keeping Accounts" on a working from home site, because it shares the word "account". So the search now only treats a result as a good match if its title holds at least two of the words, or the only word in a one word search. Anything weaker is shown lower down, and the next two steps take over.

Step two: look at our other sites. When a site has only weak matches and another of our sites has articles with every word of the search in the title, those go first. "pest control" on the housing site now opens with ten articles from our pest site, and the housing articles appear underneath under "Also mentioned on this site".

Step three: ask AI for the best matches. If there is still nothing strong, the page asks Claude Haiku 5.5, Anthropic's smallest and cheapest model. It is given the search, every article and tool title on that site, and the 150 titles on our other sites that share the most words with the search. It is asked to work out what the visitor meant, allowing for spelling mistakes, and pick up to five items that would help most. They appear at the top of the page under "Best matches". The search about frozen bank accounts now leads with "When Your Bank Closes Your Account Without Telling You Why" from our banking site, followed by an article on court orders that freeze a debtor's bank account. "segetarians" now shows "Did you mean vegetarians?", and "concentratrion" on a learning disabilities site finds the article on ADHD.

The AI only chooses from articles that exist. It never writes an answer or invents a page, so the worst it can do is pick a poor match.

Did it work?

We ran all 175 different searches from the last month back through the finished search. 171 now show results, and 117 of the 121 that found nothing before now find something.

  • 100 are answered by the site itself, from the exact phrase or the word match.
  • 10 lead with stronger matches from another of our sites.
  • 65 go to the AI step, and 61 of those get best matches. 11 of the 61 include articles from our other sites.
  • 4 still find nothing. Three are the same spam web address pasted into three different sites, and the fourth is a smart home brand name searched on a writing site.

Not every match is a perfect one. When a site has nothing on the subject, the AI picks what comes nearest, and nearest is sometimes not very near. "rats in the loft" on the housing site now finds our articles on rats and mice, but the list also includes loft storage. That is still better than a blank page, and the searches themselves show us where articles are missing.

Problems we hit on the way

The AI ran out of room. In testing, a few searches came back empty for no obvious reason. The logs showed that, given a long list of titles, Haiku sometimes started thinking out loud about what the visitor might mean and reached its length limit before it wrote the answer. Moving the instructions to the end of the request, after the list, and telling it not to explain itself fixed every case we had.

American spelling. Haiku "corrected" cheque to check. It is now told these are UK websites and to correct only genuine mistakes, in British English.

The same picture over and over. Searching "cheque" showed five articles from three sites with the same photo, because years ago they were all given the same stock picture. The search page now shows each picture only once and leaves repeats without one.

It looked bare on a phone. The search page was a heading and a box, with no button to press. It now has a Search button, a line explaining that you can type a question, the site's free tools and its most read articles of the month, so there is something useful before anyone types.

Your situation may be slightly different. ask a question below ↓ and our editorial team will reply with our advice.

What it costs

Only searches without a strong match reach the AI, which was 65 a month on last month's figures. Each request sends at most a few hundred titles, a few thousand tokens (the pieces of words AI models are charged by). Haiku charges 10 cents per million tokens sent to it, so each search costs well under a tenth of a penny, and the whole month would have cost a few pence.

The bots were the real cost risk, because 25,000 bot searches a month would add up. Three things keep them out:

  • The AI step is started by JavaScript in the visitor's browser after the page loads. Most bots do not run JavaScript, so they never reach it.
  • Each answer is saved for 30 days, so the same search on the same site never goes to the AI twice.
  • Each visitor can trigger it at most 30 times a day.

What we learned

The biggest lesson was not about AI. We had a search box on hundreds of sites for years and never checked whether it worked. Recording what people search for took an hour, and it showed us a problem we had no idea about. The searches are also a free list of what readers want: names, places and subjects we have not written about yet.

If your website has a search box, it is worth checking what people type into it. Search for a few things a customer might, with a spelling mistake or as a question, and see what comes back. If your site keeps no record of searches, ask whoever built it to add one. And if the results are poor, start with word matching. In our case the simple fixes answered 110 of the 175 searches, and the AI only handled what was left.