JUST ANOTHER A SHORT HISTORY OF SEARCH ENGINES

The Usefulness Algorithm did not always exist.

In the beginning, search engines were well-calibrated to report accurately on the subject matter of each web page. It was simple. Authors of web pages would tell the search engines what the pages were about by what they included in their meta tags. In a fairly academic environment with little competition, this made sense. Who better to define what a web page is about than its own author, right? Search engines could easily rank the relevancy of web pages to a particular query based on the honest declaration of the pages’ authors.

But competition inevitably arose, partly because there were more documents being added to the Web (no longer was there just a single page about neo-thermal induction theory) and partly because commercial interests with a stronger motivation to draw “traffic” started creating web pages. With competition, web page authors started trying to increase their traffic by doctoring the meta tags.

Search engines could no longer trust web page authors to accurately describe their content. The poor search engines had to start reading the pages for themselves to determine what each one was really about.

It wasn’t long before people began doctoring their pages to give them the edge in the rankings. Keyword stuffing, invisible text and a number of other shady tactics still around today, as well as the more acceptable SEO copywriting, were born. Relevance still ruled the rankings; it was simply determined in a different manner.

As the number of web pages that were all extremely relevant for any particular search continued to grow, the search engines needed to distinguish between those that were highly relevant and highly important on the one hand, and those that were highly relevant and pretty unimportant on the other hand. Thus were born two distinct but related approaches that are still very much part of modern search engine algorithms: link popularity and PageRank. (By the time you finish reading this book, these will sound archaic, but they will likely remain part of the algorithms for some time yet.)

Link popularity is a very straightforward accounting of how many links are pointing toward a specific web page. The premise is that the more links pointing to the page, the more important it is. After all, not many people would point links to unimportant websites, but lots of people would point links to important websites. Each link would be like a vote for the webpage it points to. Democracy in action…gotta love the Web.

PageRank, a Google invention, works a lot like link popularity, except that it tries to determine which links are more important. It works on the premise that two doctors recommending a headache medication are probably more reliable than two
lemon pickers. It knows this to be true, because more people have recommended the doctors to their friends and fewer people have recommended the lemon pickers. Less democratic than link-popularity, more accurate a measure of importance (like the U.S. electoral college), not encumbered by any constitutional protection of universal suffrage or mediocrity.

These days, importance and relevance share a somewhat balanced role in search engine rankings (a topic that fuels countless debates amongst SEO specialists), and link-building has become an increasingly important aspect of SEO. A new dimension of relevance was added as the anchor text of a link and the topic of a linking page was considered, something along the premise that a pharmacist saying “Take this headache medication.” carries more weight for buying headache medication than a “This is a purple pill.” from that same pharmacist.

In addition to relevance and importance, the search engines do try to determine credibility or trustworthiness of a web page through a number of means, which may include age of the domain, date of most recent update, TLD (top level domain – for instance, they might consider a .edu or .gov domain more credible), etc.