.Google's John Mueller answered an inquiry concerning why Google.com indexes webpages that are disallowed from crawling through robots.txt and why the it is actually risk-free to neglect the relevant Look Console files regarding those creeps.Crawler Visitor Traffic To Inquiry Specification URLs.The person talking to the concern recorded that robots were actually generating web links to non-existent query guideline URLs (? q= xyz) to pages along with noindex meta tags that are actually likewise shut out in robots.txt. What motivated the concern is actually that Google is actually creeping the links to those pages, getting shut out by robots.txt (without seeing a noindex robots meta tag) at that point receiving shown up in Google Browse Console as "Indexed, though blocked by robots.txt.".The person inquired the following concern:." But here is actually the large inquiry: why would Google.com mark webpages when they can not also see the web content? What is actually the perk in that?".Google's John Mueller confirmed that if they can't creep the webpage they can not find the noindex meta tag. He additionally produces an interesting reference of the site: search driver, recommending to dismiss the end results since the "common" users won't view those end results.He composed:." Yes, you're appropriate: if we can't creep the web page, we can't view the noindex. That pointed out, if our company can not creep the pages, then there's certainly not a lot for our team to index. Therefore while you may see several of those pages along with a targeted site:- query, the normal user will not observe them, so I wouldn't fuss over it. Noindex is also great (without robots.txt disallow), it only suggests the Links will end up being crept (as well as wind up in the Browse Console file for crawled/not recorded-- neither of these statuses induce problems to the rest of the web site). The fundamental part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limitations in using the Site: search progressed hunt driver for analysis causes. One of those reasons is due to the fact that it is actually certainly not hooked up to the frequent search mark, it's a separate trait completely.Google.com's John Mueller talked about the internet site search operator in 2021:." The short solution is that an internet site: inquiry is actually certainly not suggested to become total, neither made use of for diagnostics functions.A web site inquiry is actually a specific sort of hunt that confines the results to a particular website. It's generally merely the word web site, a colon, and afterwards the internet site's domain.This query limits the results to a certain internet site. It's certainly not implied to become a thorough selection of all the pages from that internet site.".2. Noindex tag without making use of a robots.txt is actually great for these sort of circumstances where a crawler is actually linking to non-existent webpages that are actually acquiring uncovered through Googlebot.3. Links with the noindex tag will definitely produce a "crawled/not catalogued" entry in Search Console and that those will not have an unfavorable impact on the remainder of the internet site.Go through the concern and address on LinkedIn:.Why would certainly Google.com index web pages when they can't even see the material?Included Picture through Shutterstock/Krakenimages. com.