The Search Engine Crossroads

If you stay in touch with what the search engines such as Google are doing, then you will know we’ve all reached a crossroads. The incredible mass of parked domain websites with automatically generated content and backlinks is causing major problems for the Google PageRank concept. Google has recently introduced a new Big Daddy infrastructure for its search databases and algorithm. However it is not yet clear from their keyword search results that they have the answer. Recent news is that perhaps they are being stretched by computer network problems in trying to stay ahead of the veritable flood of Web-based information, partly fuelled by the world’s response to the PageRank concept. As Eric Schmidt of Google is quoted as saying, “Those machines are full. We have a huge machine crisis.”

For Google and the other search engines with contextual advertising, such as Google Adwords, there is a dilemma. One observer this week suggested that it is not in Google’s iinterest to give highly relevant results to keyword searches. “100% relevant, on-topic, high quality search results are not good for the ad model.” If they had 100% relevance, then a searcher would just need to click on #1 knowing that this was the most relevant result. There would be no need to try one of the Google Adwords sponsored links. This would potentially cut their ad revenue, which is a big part of their economic drive. If they deliver 70% relevance, then there is need to click on a few items to find the most appropriate. In some ways, if the searcher clicks on what turns out to be an “ugly” automatically generated website, then this may be in Google’s interest. The potential purchaser will probably click on one of the Adsense ads as the best way out, and ker-ching Google makes its Adwords revenue.

I believe that argument is fallacious. Search engines are supposed to deliver relevant results. The search engine that does that will beat the rest. As it happens, William Slawski yesterday pointed out a recent Yahoo! patent that deals with the concept of TrustRank rather than PageRank. This approach where human judgement is factored into the algorithm seems very powerful. Human edited directories such as DMOZ are another way but they suffer from all the difficulties of getting large enough armies of dedicated volunteers. The TrustRank concept seems to have no such obvious limitations.

We are passing through frustrating times with the debris created by PageRank but hopefully the best search engine will find the right solution.

Tags: , , ,