Pavel Zaněk PavelZanek.com
Select language
Account

What is page indexing in search engines

Since I still use the phrase "page indexing" in the context of SEO and some of you ask me what page indexing means, I decided to write an article that will explain this term to you. You can have the best websites, but if you don't have them properly indexed, you can lose visitors (your website doesn't appear in search results).

Published at 2017-11-30 by Pavel Zaněk

Estimated Reading Time: 7 minutes

Page Indexing in Search Engines

Table of Contents

What is page indexing

Website indexing is a process where a search engine bot visits you, tries to go through all the content, and stores it in its database (indexes it). When you search for "something" using a search engine, it doesn't start searching the entire internet. The search engine looks in its database and offers you relevant results for your query - so the pages in the search result must be indexed by it. If the search engine doesn't have them stored in its database, it's clear that it won't display them. You can think of the database as a kind of library of websites from which the search engine finds the best answer to the searched query. Therefore, it is important to focus on website indexing in search engines for every project.

Poor website indexing

As usual, with websites, it is possible that search engines don't know about all the pages you want to display in search results. Then you can lose visitors and may not cover a segment that is beneficial for your business. Imagine, for example, that you might not have your main page, e-shop category, blog articles, etc., in the search engine index. And if the search engine doesn't have them stored in its database, it can't display them in search results.

Many websites commonly use non-indexing of pages - you don't want to display pages in search results. It can be, for example, thank you pages, an e-shop cart, user sections, and much more.

However, sometimes you want to index a page, you send the page URL to search engines for their crawling bot to evaluate. You might have it in a sitemap or even manually send it for indexing - "Add Page" on Seznam and "Submit URL" on Google. However, it can happen that the search engine may decide not to index it anyway. Then it is important to find out what might be the reason why the search engine doesn't want to display it and what prevents it from indexing.

Analysis of website indexing

There are many ways to check the status of indexed pages. Look at the basic steps I follow.

Checking page indexing using the site operator

I tried to generally write a guide on how you can also check the status of indexing your websites. Since this question was asked quite often, I created an answer to it in the SEO clinic, where you can also find other SEO tips. It's mainly about comparing the number of URLs you want to display in search results with the number of pages returned using the site operator.

Checking page indexing using sitemap and Marketing Miner

If you have created a sitemap with URLs that you want to display in search results, we can specifically find out which pages are being indexed and possibly those that are not displayed in the search. First, before checking the display of pages, see if your sitemap really only contains URLs that you want to send to search results.

For the actual analysis of the pages sent through the sitemap, I use Marketing Miner, which contains a useful Fulltext Index Checker function.

  1. Go to the Marketing Miner website and log in
  2. Choose that you want to create a "New report"
  3. To the question of what query you want to insert, choose "URL"
    Page Indexing - URL input - Marketing Miner
  4. Choose "Sitemap / RSS" and enter the sitemap URL, you can rename the report and choose "Go to miners"
    Page Indexing - Insert Sitemap to Marketing Miner
  5. Choose the Fulltext Index Checker among the miners
    Page Indexing - Fulltext Index Checker in Marketing Miner

TIP: You can also choose URL Indexability - you will find out which URLs are indexable, which are canonicalized, or which are prohibited in robots.txt

Marketing Miner will show you which URLs are indexed and which are not. In my opinion, this is the fastest and simplest way to create a quality analysis of page indexing.

Checking page indexing, for example, using Google Analytics or Google Data Studio

In general, but not exactly, you can also determine page indexing through landing pages - from organic search. You can filter landing pages directly in the Google Analytics analytical tool, or you can create your own overview using Google Data Studio. Just download the reports for a certain period and compare them with the URLs sent in the sitemap.

If there was traffic on the landing pages from search engines, then they logically appear in search results. And if the traffic is zero, then it's worth checking why.

Why might pages not appear in search?

There can be several reasons why web pages do not appear in search results. Sometimes the cause can be visible at first glance, but sometimes you have to focus more deeply on the issue of poorly indexing pages.

  1. Check if the page contains a meta tag with the "noindex" parameter - used to prohibit page indexing.
  2. Check if the page is blocked through robots.txt (even via robots.txt it is possible to disable indexing of pages, but this file should not be used for this purpose).
  3. Check if the page has a simple URL that is not obstructing indexing for search engines (hashtags in URL, parametric URLs)
  4. Is the page internally linked and leads to a working link?
  5. Are there any redirects on the target URL? Or are there many of them?
  6. Is the page listed in the sitemap?
  7. Isn't the page duplicate to an already created page on your website? Does the search engine have a reason to also display it in search results?
  8. Page quality or penalty (see below)

TIP: Support the page by linking it with backlinks. There can be many more reasons, but these are the most common causes I encounter.

Page usefulness - is the page beneficial for users?

If the page is not indexed, even though it has all the prerequisites to be indexed, then another option is to check the content of the page. In addition to not being duplicate with another page, see if the page content is useful for users. If so, then check if you are linking to relevant websites (outgoing links), if you don't have a lot of generated texts on the page, if you don't display different content to search engines than to users, and other possible shortcomings that could bother search engines.

Once I also encountered that websites used article ratings, but this rating was generated - so it was not added by the user himself. Google then penalized the website for these pages and stopped displaying them.

Penalties

The worst case that can happen is a penalty. You can find out about a possible manual penalty directly in Google Search Console, then in the "Manual Actions" section.

It can also happen that you have malware on your pages. You can then find it in GSC in the "Security Issues" section.

How to quickly index newly created pages

If you want to quickly index the content of your websites, there are procedures to speed up indexing. If you create a page that you want to display in search results, then:

In Google Webmaster Tools (Google Search Console), you can also use another function - Fetch as Google. Through the tool, you request a rendering of the web page, and if no problem is found, you can immediately request page indexing. For example, you can find out which resources Googlebot couldn't download.

Page Indexing - Fetch as Google in GSC

In conclusion

There can be several reasons why a page may not be indexed correctly. However, make pages primarily for users and not for search engines. Then pay attention to other technical prerequisites that search engines expect and you won't cause them problems. If you decide to use the noindex parameter somewhere, then check the changes made. There is nothing worse than deindexing pages and then losing traffic.

Share:
You have not rated yet.
Pavel Zaněk

Full-stack developer & SEO consultant

Pavel Zaněk is an experienced full-stack developer with expertise in SEO and programming in Laravel. His skills include website optimization, implementing effective strategies to increase traffic and improve search engine rankings. Pavel is an expert in Laravel and its related technologies, including Livewire, Vue.js, MariaDB, Redis, TailwindCSS/Bootstrap and much more. In addition to his programming skills, he also has a strong background in VPS management, enabling him to handle complex server-side challenges. Pavel is a highly motivated and dedicated professional who is committed to delivering exceptional results. His goal is to help clients achieve success in the online space and achieve their goals with the help of the best web technologies and SEO strategies.

Suggested Articles

Google changes visibility of HowTo and FAQ results

Published at 2023-08-09 by Pavel Zaněk

Estimated Reading Time: 2 minutes

seo

Google has introduced new changes to the visibility of two forms of rich results, namely HowTo and FAQ. These changes may have a significant impact on publishers and webmasters who have relied on these rich results to gain higher visibility in search.

Continue reading

Your experience on this site will be improved by allowing cookies. - Cookies Policy