Indexing is a significant event in your websites search engine journey.
If your website is not indexed in a search engine, it can not be found in that search engine.
If you create a post or page on your website, it will not be immediately available in Google or any other search engine. The search engine’s bots or crawlers have to find the page and analyze it and then index it. Then only it can be found on google.
This means – each and every page of your website that you want to be found in Google must be indexed. So, each time you create a new web page or blog article, you want to get them indexed faster.
But nowadays, Google takes a long time for new sites’ pages and posts getting indexed on Google.
So, how to get your website indexed on Google?
- Crawl accessibility: Make your website index ready.
- Google Index checker
- 5 Ways get google to index your website
- How to make Google index your site faster
- FAQs about Indexing Website
Crawl accessibility: Make your website index ready.
Here is “Mozlow’s hierarchy of SEO needs“, which explains the way people should go about doing SEO.
As you can see, the foundation of SEO begins with ensuring crawl accessibility and moves up from there.
So, here are a few things to take care of to make your website accessible to search engines.
- Search Engines bots are not blocked from crawling your website in the robots.txt file.
- Make sure the crawlers can find your website’s import content easily.
- Make sure important texts are not hidden in non-text content like images videos. Make sure you are not displaying important text using images only; robots can’t read it. If you are using such images, make sure to mention them using text format as well. If you are using video, you can have the transcript written in the post. An excellent Example will be Moz’s Whiteboard Fridays.
- Your site navigation is essential too. As crawlers find new pages/posts by navigating through links, crawlers will have a hard time finding important pages or posts if your site navigation is not good. Have a good website structure that is easy to follow.
- Ensure your website is well-coded so that crawlers do not get an error while trying to crawl your site. In Google Search Console, the “crawl Error” report will show you if Google bots have encountered any error while crawling. But ideally, you should be able to find and fix it before google crawlers find it.
Learn more about Making your website Search Engine Friendly.
Google Index checker
Before you begin other SEO efforts, you need to ask yourself – “Can Google index my website”? “Is it accessible to the Search Engines”?
There are several ways to check about it.
‘site:’ search operator
Using the ‘site:’ search operator, you can check whether your website or part of your website indexed.
- site: domainname.com – this will show you all the indexed pages of your website. It even shows you the number of pages.
- site: domainname.com/blog/ – it will show all the indexed pages with ‘domainname.com/blog/’ at the beginning of their URLs. Show it will show all blog articles that are Google has indexed.
- site: domainname.com/blog/slug/ – This search will show if this particular page or post is indexed or not.
Coverage report in Google Search Console
Navigate to your Google Search Console dashboard. To the left, find the Coverage report under the Index section. There you can see every crucial detail about your websites crawlability, divided into four categories –
- Error: Here, you will see all crawl error details that are holding that pages to get crawled and indexed
- Valid with warning: These pages have minor issues, but they are crawlable and indexable
- Valid: These pages Google found are absolutely OK, and these are indexed. So, you want every critical page of your website to be available here.
RankSider – Website Cralability Checker
Although Google Search Console Coverage reports provide all the crucial details about your website’s crawlability, it is better to find any issue yourself before Google sees it. So, you can use third-party tools like RankSider.
RankSider provides a tool that checks crawlability of individual pages for Google bot, Firefox, Bing bot, Yandex bot, etc. All you have to do is put the URL and choose which user agent (like Google bot, Bing bot) you want to check crawlability. It will let you know if the URL is crawlable and indexable by that user agent or not.
You can also use screaming frog to crawl your website and find each page’s status.
5 Ways get google to index your website
1. Share your content in other places
Once you create a new website or publish new content, you should share it everywhere you think your target audience will be hanging.
For example, sharing on Social Media platforms, forums or getting backlinks from other websites.
Doing that will create many links from other websites to your website.
And as we have discussed, Google uses links to discover new websites or pages; hence, this activity will increase the chance of your pages getting found and indexed on Google.
2. XML Sitemap
Not every page of your website will be linked from prominent places that can be found easily by search engines. That’s why we need something called XML Sitemap.
XML Sitemap is like a Table of Content, which lists all the URLs of your website in an excellent structure. It makes sure even the orphan pages in your website is found by the search engine crawlers.
It makes search engine crawlers job easy. Whenever a crawler visits your website, it looks for the sitemap first.
Along with URLs, it can also tell Google the priority of each URL and Change Frequency and Last Updated. You can find all the protocols here. And here is how it looks like –
Please don’t get scared by how it looks. Creating an XML sitemap takes only a breeze.
Create XML Sitemap:
It can be created manually created. But why to waste time!
You are very likely using WordPress, or if not, you can search for how to create a sitemap for the platform you use.
In WordPress, you can install any SEO plugin like Yoast, Rank Math, All in one SEO plugin. I am going to show it for Rank Math, as I am using the same.
Install the Plugin and Activate. The Sitemap is already created.
Go to Rankmath > Sitemap settings. You will see your sitemap is ready, and the URL is displayed like this –
You can change the settings if you know what you are doing or copy the part of the URL – sitemap_index.xml and move on to the next step.
Note: This sitemap URL can be different depending on the method you are creating it.
Submit XML sitemaps to Google Search Engine
- Open your Google Search Console dashboard > Index > Sitemaps > Paste the text copied from the previous step like this –
- Hit Submit. Once submitted, you can see a success message like this –
3. Request indexing – submit URL to google
You can request individual indexing URLs to Google Using the GSC Inspect URL box above on your GSC dashboard. There you can paste the URL.
It will inspect and tell you if the URL is on Google or not. If it is not on Google, you can Request Indexing.
Generally, within an hour, the page will be indexed if everything is fine with your website. But if not, check out this section.
4. Index your blog posts and new pages faster using the API method.
Did you know that you can automate this Request Indexing to Google?
Yes, there is this Google API Console method.
Each time you publish a new page or post on a WordPress website, it will automatically submit a request to Google.
Refer to this video to learn exactly how to set this thing up
Linking pages and posts internally works like a charm.
When crawlers land on one page of your website using one of the above methods, it crawls other pages using the link on this page. That way, crawlers discover new pages and posts on your website.
But for this to happen faster, follow the next section.
How to make Google index your site faster
You have requested indexing to Google using one of the above methods, doesn’t mean Google will immediately index your site or webpage. It is up to Google if it wants to index and when it will index the content.
Sometimes, Google takes even more than a month to index a new piece of content.
So, how to ensure Google Indexes your website faster?
Here are a few things
Utilize Crawl Budget Effectively
Crawl budget is the average number of pages Google bots crawl per day on your website before leaving. So, every website has a limited crawl budget. That is why it is crucial not to waste it.
- Make sure your website architecture is such that your important pages are easily found.
- Your website should not return 4xx error and 5xx error. Your website should only contain 200 (OK) and 301 (Redirection) status codes. Also, don’t have unnecessarily too many redirections.
- Your website should not have duplicate pages or low-quality pages with no value for search traffic. Things like unnecessary tags pages, the author pages in a single author website, site search results pages and so on should be avoided.
- Block the part of the website which are not needed to be found on search engines. For example, the website’s admin section, internal use parts that only employees should access, and so on. You can do it using the Robots.txt file.
Every website has a limited crawl budget. But the budget can be increased.
Improve Crawl Frequency and Crawl Budget
Every aspect of SEO is dynamic. It is the same for the crawl budget of your website. You can improve your website’s crawl frequency and crawl budget.
Well, the frequency can be set by the sitemap that you submit on Google Search Console or other search engines. But let’s say, in your XML Sitemap, you have specified your website’s change frequency is daily, i.e. <changefreq>daily</changefreq>, and you truly update your website like once a month. Google is going to ignore your sitemap.
Publish Google quality article consistently. It will increase the crawl frequency and also crawl the budget automatically. That is why you can see news websites, get their content indexed in a few minutes after publishing their content. It because they have created a reputation for publishing content frequently.
But at the same time, maintain quality.
FAQs about Indexing Website
Why my website is not indexed?
There can be several reasons why your site is not indexed. I would categorize it in two –
Accessibility issue: Somehow, you have blocked crawlers from crawling your pages. Or maybe your page is crawlable, but it isn’t easy to find.
Few things to do:
- Check for the accessibility of your website or page.
- If you think it should be indexable, try indexing using one of these methods. Try the 3rd one for individual URL.
Your website is not worthy enough in Google’s eye: Your website is new, and you have not enough high-quality content to build a reputation. Google may ignore these sites. So, create a reputation by publishing more content. Check out this section for more.
But above all, have patience. I see many people do get mad seeing their sites not getting indexed. It is usual for new websites not getting indexed properly for even months. Check the next FAQ for more.
How often does google index my site?
Here hear the answer from the Head of Google Web Spam Team, himself.
The key point here is that Google crawlers have a limit on how many pages per day they will crawl, and the limit is that it is different from website to website. Learn how to increase this limit for your website and get your website index faster.