Our life is made infinitely easier by the wide availability of search engines. You need only seconds to take your mobile phone out of your pocket and type in your search terms to get a result. In fact, search engine giant Google has lent its name to the whole process – we are now googling the answer to questions instead of simply looking it up.
There are many search engines out there and listing all their names would take a long time – but it would also confuse people. We may be familiar with Google, Yahoo!, AOL and Bing, and some of us may remember Altavista, but how many know Alibaba, Baidu, and Yandex or even heard of Hotbot, Lycos, Fetch or Goto?
The search engines we know today are quite different from their former selves from the nineties. They now use complex algorithms to determine which results are best to answer your query, taking into account more human factors in determining the relevance of a page such as reputation and content quality.
The history of search engines can be traced back to 1990 when Archie was rolled out to the public. Archie was a directory listing, and would be considered very primitive nowadays as it couldn’t display the content of each site due to limited space. But as every first player in a big game, it paved the way for better and more comprehensive directories. One of the most notable search engines from this period was Aliweb, which was created by Martijn Koster and made available online in 1993. Aliweb also allowed webmasters to submit their own site to the index but it’s worth noting that many were unsure of how to do that.
Web search was developing at a constant pace and in 1993 the internet got three platforms that were edging closer and closer to a modern search engine. JumpStation and the World Wide Web worm indexed websites and displayed them in the order they were found without taking any rankings into account. The RBSE Spider however did have a ranking system, but it made it difficult to get results unless the search query was an exact match to a page’s title.
The following year saw many big names enter the picture, some of which we still remember and even use today. AltaVista came online in January 1994 and had unlimited bandwidth and an advanced search technique. It also offered its users tips on how to word their queries in order to get better results and gave people the option to submit their own site to the engine. While AltaVista was pushed to the side by the arrival of Google and other rival search engines, it still managed to get a pretty good offer in 2003, when Overture wanted to acquire it for $80 million in stock and $60 million in cash.
Yahoo! Directory also surfaced in 1994 and started out as a simple list of quality web pages chosen by its founders, Jerry Yang and David Filo. It soon turned into a searchable directory, and its listings included a description of the site and featured both commercial webpages as well as purely informative ones. Until 2002, Yahoo! outsourced their search services; a change in direction meant that after this year they started buying out search directories and incorporating them into their own services. WebCrawler came out at the same time as Yahoo! Directory, but did something altogether different. It was the first crawler that actually indexed entire webpages, and just a year later it was bought by internet giant AOL.
Google’s founders revolutionised the way search engines evaluated sites when PageRank was released in 1996. Its novel concept meant that pages were ranked based on how many other websites linked to them, which was seen as an endorsement. The quality of the websites linking to a certain page also had a say in determining the final rank of the latter. They managed to get venture capital funding for Google and the most famous search engine began working towards the platform we know today in 1998. In its first few years, Google partnered with many other search providers like Yahoo! or AOL, but it quickly found its niche in AdSense and AdWords advertising services. Its search algorithm has gone through many changes since its launch and it’s currently intent in trying to think as a human would do when assessing a page’s quality and deciding what rank to give it.
The next big name to come into the market after PageRank/Google was Ask.com, which had human editors acting as a liaison between the person using their service and the results generated by the search query. Ask.com prided itself in being a natural language search engine, where you could get accurate results even if your search was not all that computer friendly.
MSN search launched in 1998 and worked on the back of existing search engines until 2005 when it swapped Yahoo!’s technology for MSN’s own creation. Microsoft also launched Windows Live Search a little while after but both search services were rebranded in 2009 as Bing, which offers a related websites section on its search engine results page.
When looking at the big players in the search market, it’s important not to forget about search engines that work primarily in languages other than English. Yandex is Russia’s biggest search engine. Launched in 1997, it now has a majority share of the market in the country, around 60 per cent. It’s one of the few national search engines that are significant competitors for Google in their own markets. One other such search engine is Chinese Baidu, founded in 2000. It offers over 50 search related services and its market share in China is a huge 76 per cent.
The future of the search engine looks ever so promising. Google is currently working on a project that’s meant to revolutionise the way we interact with the web and introduce the concept of a conversational search – where a search engine doesn’t just answer our questions, but anticipates them. The sentient web is closer than we thought.