Chapter 1: How Search
Search engines essentially do two things.
They crawl using spiders and they index what they find. The index is then used to provide searchers with the information they are looking for in a ranked style I.E with what the search engine considers to be the most relevant website to the search term used at the top of page 1 and so on down.
Below we will also talk about how a search engine determines what is relevant and popular.
Crawling And Indexing
To help you to understand how crawling and indexing work you should think of the internet as a bus system in a large city with many stops. The stops are like each different document on the internet and the engine needs a way to “crawl” to find each unique stop. Think of the links they use as the roads they take between stops. The links help to pull all of the pages on the internet together, like a road map. The search engines’ robots are called “crawlers” or “spiders” and the links they follow allow them to reach all the different documents on the web.
By establishing huge data centres all over the world, search engine companies are able to hold metadata on all of the billions of pages the crawlers find, in an index. When they find these pages, they decipher the code and store certain data on the pages in these datacentres so that it can be easily accessed almost instantaneously. The speed with which the engines do this is extremely important, because when a person does a search they want their results as quickly as possible. Even a delay of a second or two can cause them to be upset.
Search engines are also there to provide answers to the person performing a search. The search engine will go through all of the data available to it and return results to the searcher that is relevant to their query. The next thing a search engine will do is to rank the answers in terms of the most relevant and popular results.
Have No Fear, Fellow Search Marketer!
In the beginning, search engines didn’t do much more than find pages with text matching the search performed. Over the years, engineers have made the way search engines work much more efficient and smart. There are many things that influence relevance nowadays. A search engine will typically go with a site that is popular, making the assumption that it contains more valuable information. An engine will employ mathematical algorithms to determine the relevance and then sort the results in terms of popularity. These algorithms often have hundreds of variables referred to as “ranking factors,” all of which influence the relevance considerations.
Search engine marketers have found different ways over the last fifteen or more years to influence how their sites are ranked on search engines. They use this information to help their clients figure out ways to achieve a better ranking.