It’s not as valuable as you’d think.
For years, people working on and evaluating search engines have talked about optimizing for search relevance. This is exactly wrong, and it’s also not how companies like Google and Amazon rate their search. What companies should really decide on and optimize for is the business metrics they want search to drive for them. “Search relevance” is at best a means to an end.
The decline of search relevance
This is a controversial statement, so some examples might help.
If users on an electronics retailer search for laptops, the retailer could display any of their thousands of laptops on the first page and those would be relevant results. But, if the retailer displays old laptops made in 2009, or only the most expensive laptops that are rarely purchased, most of the users would be turned off and leave. The results are relevant— they’re definitely laptops— but they aren’t attractive to most users, and they’re likely to turn users off and hurt the retailer’s brand. All of a retailer’s search results can be both perfectly relevant and incredibly unhelpful to users’ buying journeys and the retailer’s reputation.
A more extreme example is a search for “chips” on a grocery website. Should the grocer show Cheetos? It’s not strictly relevant to the search because Cheetos are not chips, but what if users who search for chips are highly likely to buy Cheetos? Should the retailer pedantically disagree and only show potato chips, or should it show the user Cheetos, even if they aren’t strictly relevant?
It’s worth pausing here to describe how most companies determine search relevance: there will be a person, or a group of people who eyeball results and decide which are relevant and which are not. This is both easy to do and actively harmful to a retailer’s revenue targets and customer loyalty. Are laptops being returned for the “laptops” query? Sure. Are Cheetos chips? Nope. Will these decisions hurt the end user experience? You bet.
The metrics to focus on
So, given how important site search is to a retailer’s bottom line, how should they build and evaluate it?
The answer is to focus on the business metrics the company cares about: revenue, purchases, profit, or whatever else most needs to be increased. Then both evaluate the search, and drive up the probability that each customer has a successful buying journey using the clickstream data that happens after the search. In short, show each customer products for each search in the order that will most likely lead to an increase in the business metric.
To achieve this, retailers should use three key attributes balanced together: relevance, attractiveness, and personalized attractiveness. Relevance should prevent something like shoes showing up for that search for laptop. Attractiveness should ensure that the most attractive laptops (the ones most likely to be purchased from that search) should show up at the top, and personalized attractiveness should use what we know about each user, their individual clickstream data, to show the particular results most attractive to them. If a user tends to buy Apple products, show them more MacBooks. If a user tends to buys a lot of video games, show them premium gaming laptops.
The most important lessons are two:
1. When evaluating search engines, retailers should focus on business metrics, not relevance.
2. Retailers should use clickstream data as the main driver to power search and to evaluate it. Doing anything else does a disservice to both retailers’ customers and their bottom line.
How CCPA can affect your eCommerce site — in search, browse and beyond
The California Consumer Privacy Act went into effect on January 1st. Join the Krestor team on January 23rd at 10 AM PST to learn why the CCPA can put you at greater risk than Europe’s GDPR and the steps you need to take now.
