Hi and welcome to today’s article about how search engines work. We have talked about so many things concerning SEO but we have never talked about search engines. Yet the entire SEO is all about search engines. So in today’s article, we are going to talk about how the search engines work. This will help you do your SEO like a pro. You cannot assume to master SEO without understanding how the engines that you are targeting work.
When we talk about search engines, the most common and most popular is google. Google has at least 90% of all search traffic on the internet.
So without wasting any more time, let us go straight into it. First of all you need to understand that its not very complex. The process by which the search engine operate is very easy. It can be broken down into 3 steps below.
1. Crawling.
The very first and most important step of the search engines is the crawling. Crawling is the process by which search engine bots read your website. The bots visit your website pages, posts, images and every other content on your website in order to read it.
The crawling is done by special software called crawlers (or colloquially, spiders or bots). For example, the google crawlers identify as googlebot. If you look at your website server for all the user-agents visiting your website, you will notice one called googlebot.
The way these crawlers discover pages/posts and crawl them is by following URLs. Say for example the crawler comes to your homepage. Then on the homepage you have a menu-bar that contains all the important links of your website. The crawlers will follow all those links on your menu and read them. By doing so, that’s how the crawlers are able to index almost the entire internet.
The main goal of crawling the internet is to find new pages and update the existing ones.
Hence you as an SEO expert, if you want to get all your website and web pages indexed, you need to make them available to the crawlers. You need to create a sitemap.xml file with all your links and submit it to the search engine.
You cannot rank on search engines like google without first getting your website and webpages indexed.
2. Indexing.
Once the search engines crawl the internet, the next step is to index the URLs that they have crawled. You know when we talk about the internet, we are talking about tens of trillions of URLs. The search engines surely need a system to organize all these trillions of URLs. The search engines need to know how to organize the links based on value. Moreover, the search engines need to know which link is which. The engines need to know what link goes where. What each link is all about. Is it a link for a page, or a blog-post or an image or a file or what? The search engine systems need to do all this data analysis.
All this data from all the links that the search engine has crawled is analyzed and organized into a massive database called index. Hence the term search engine indexing.
But now i know you are asking what is stored? The search engines do not store everything. The reason for this is because there is a lot of junk on the internet. Hence the search engines have to filter a lot of this junk and leave it out. Some of the content that is stored is information like keywords and freshness(how recent is your content).
In addition to that, the search engines also store the page structure of things like headings and link structure. Moreover, the engines also store things like media, you know images, videos and files.
Now i know you have a new question, why is indexing necessary? The indexing is necessary so that the search engines can quickly retrieve relevant results when a user searches. If it weren’t for indexing, search engines would have to go and crawl the page every time one searches anything.
3. Ranking (Retrieval).
Now this is the final step of how search engines work. This is the step that we all get to interact with the search engines. Every time you search on any search engine, you are on this step of the search engines.
When you type any keywords on the search engine, the search engine checks its index (from step 2 above) to find the most relevant results. The engine checks its index to find the most relevant pages based on hundreds of factors called ranking signals. In this article we are not going to talk about the ranking signals. If you want to learn more about the search engines ranking signals we can make a separate article for that.
Just to give you a slight insights of the ranking signals, here is a small list.
- Relevance to the search query (does the page match the keywords?).
- Page authority (how trustworthy is the site?).
- User experience (is the page fast and mobile-friendly?).
- Freshness of content.
- Number and quality of backlinks.
Based on the hundreds of the ranking signals, the search engine shows you a set of ranked pages called SERPs. SERPs is Search Engine Results Pages. These are the pages on the search engine whereby they show you ranked lists of pages from across the internet.
Now you as an SEO expert, this is where you want to be. This is where you want your website/webpage to be. The moment that you get on these SERPs, then you have made it. That’s what you hear about ranking on search engines like ranking on google. When you get top of these SERPs, like the top 3 results of the first page, then you have made it. You will be getting thousands or millions of free traffic from these SERPs when people search.
Conclusion.
Now you understand how search engines work. We have kept it as simple as possible. We have tried to not complicate the article so that anyone with no basics can easily understand. If you are an SEO expert or you want to start SEO, now you understand how it works. The first and most important step is first understanding the basics. From MediaGeneous, we wish you all the best in your SEO career.
