Because It Just Seems Better
Some random bits scribbled by Jeremy Zawodny
When I wrote When Better Isn't Good Enough. I had both features and design foremost in mind. I wasn't thinking as much about relevancy because I've seen enough data from blind comparisons to know that they're often too close to call. Tim and I talked a bit about relevancy in the comments and in a follow-up IM conversation.
The Search Engine Experiment is public blind test that's trying to sort out who is better than who. Seeing the results prompted Seth to write Can more than 60% be wrong?
About 65% of those tested said that Yahoo or MSN was the most relevant.
I won't go into the flaws with this method, since that's really not the point. He goes on to say:
Which reinforces my point that Google isn't "better" for most people if "better" means more relevant or deeper. Google is better because it feels better and quicker and leaner and easier to use. The story we tell ourselves about Google is very different, and we use it differently as a result. Think about that the next time you insist you need a "better" formula or a faster server or a stronger first baseman. Music sounds better through an iPod because we think it does. Design matters. Stories matter most of all.
That's really what I had in mind in the "better isn't good enough" post. But Seth said it far better than I did.
Posted by jzawodn at November 25, 2005 07:15 PM
Hmm, I feel mixed about this one.
I agree that what people _feel_ when they use a service is paramount. It's some mixture of a bunch of ineffable factors, and the users aren't responsible for sorting them out --- they just know what they like.
On the other hand, the cool thing about blind tests is that it subtracts out the influence of the brand. If you've never done this, try the blind-taste test thing with a (willing) friend who's a (beer, scotch, cola) snob. I've been a beer snob in my day, and I was able to make some crude distinctions once when I tried it: stout vs. lager, sure, but subvarieties of Czech pilsners? Not so much.
Anyway, the interesting thing about the blind test is that, if you can't tell your favorite thing from the others, it means either that you're under the power of the brand, _or_ you're under that power of the intangibles of the user experience (the color of the label, the way the cap comes off, the usual serving temperature).
Seth's post munges the brand and the user experience together. Whether this matters depends on whether you believe that (in the long run) brand loyalty is correlated with how good the underlying thing is. Myself, I believe that brands are some sort of combo of current marketing and past product quality, and that when the product quality equation shifts, the brand will too, even if really slowly.
on November 25, 2005 08:14 PM
Whoa - holy dubious stats batman. I certainly agree with your key points - that perception is important, Google is overrated vis a vis other search, and that Yahoo should shoot for "much better" and not "a bit better" results to win the search war. But statistically Seth seems to be making a very dubious interpretation of these results, which suggest Google is still the top engine by a significant margin. Applying his faulty logic the headline could also have read "Yahoo only serves the most relevant result one third of the time".
These results should be interpreted as indicating Google still wins the search wars, not the other way around.
on November 25, 2005 09:23 PM