Google’s Gary Illyes says artificial intelligence has made bot impersonation a bigger problem for website owners. In a new interview, Google’s search advocate said rogue crawlers increasingly pretend to be Googlebot to slip past blocks and harvest content for training AI models. Anyone can fake a user-agent string in about thirty seconds, he warned, so a claim to be Googlebot proves nothing on its own. Website owners, he said, should verify crawlers by their IP addresses and reverse DNS records instead.
Why AI made it worse
Illyes gave the interview to Aneta Samkoff, host of the Odys Podcast, in a conversation devoted to bots and crawlers. Asked why impersonation has grown, he answered simply: “I think it is AI.” More companies want text and data to train their own models, he said, and the internet looks like a large farm where content can be harvested freely.
Firewalls and content delivery networks have improved at spotting fake crawlers, but they are not perfect, and some unwanted traffic still gets through. The motive is straightforward: a smaller AI engine, or anyone wanting to scrape a site, can claim to be Googlebot and be waved through gates that would otherwise block it.
Google has already withdrawn features that scrapers abused, including the option to view 100 results per page, which it dropped in September 2025 as an anti-scraping measure. Illyes said he has run online honeypots that attract crawlers and found requests claiming to be Googlebot arriving from IP addresses with no connection to Google.
How to verify Googlebot
Google publishes its crawler IP ranges and documentation on verifying Googlebot requests, and Illyes urged owners to use them. Checking the IP address and reverse DNS record of a visitor claiming to be Googlebot will show whether it genuinely comes from Google. A mismatch means the visitor is impersonating the crawler, whatever its user-agent string says.
Server logs are the other basic tool. Owners should watch for suspicious patterns, such as unexpected IP addresses or unusually frequent requests from a single source. Those signals often reveal a fake crawler before it causes damage, and they cost nothing to check.
For sites that need stronger protection, a content delivery network with bot management, or a web application firewall, is the practical next step. Hosted platforms such as Shopify already provide protections, Illyes noted, while larger or more popular sites may eventually need more sophisticated controls.
When bots cause real damage
Fake crawlers are not a security concern alone; they can take sites offline. Google has spent years tuning its crawling infrastructure so it does not exhaust a site’s resources, Illyes said, but other crawlers may lack the same safeguards. A poorly configured bot can request pages repeatedly, consume bandwidth and server capacity, and leave visitors unable to reach the site.
Much of that damage, he added, may come from ignorance or poor implementation rather than deliberate attack. That distinction matters for how owners respond: the first step is to find out whether bot traffic is actually causing a problem, rather than buying protection against a threat that does not exist.
Robots.txt is not security
Illyes was blunt about robots.txt, describing it as a suggestion rather than an enforceable restriction. Legitimate crawlers generally respect it, but any bot can ignore it, so owners who need to block unwanted traffic should use network-level protections instead. The file remains useful for expressing preferences to well-behaved crawlers, but it cannot force a malicious bot to comply.
Google separates search crawling from AI uses. Owners can allow Googlebot to crawl pages for Search while using the Google-Extended robots.txt token to restrict eligible uses of content for training or improving Google AI models and services. Illyes noted that Apple offers a similar Applebot-Extended token, and he expects access controls for automated clients to become more granular over time.
What comes next
One longer-term idea is Web Bot Auth, an effort involving cryptographically signed HTTP requests. Instead of relying on IP addresses or user-agent strings, a website could verify a signature against a public key to check whether a request genuinely comes from the claimed organization. Illyes said that could simplify authentication as IP addresses become harder and more expensive to manage.
He also raised the idea of reputation scores for crawlers, aggregating feedback from websites and security providers about whether bots behave responsibly. The idea remains early-stage, he cautioned, and a system controlled by one company could introduce bias. For now, the practical advice stands: verify before trusting, and act only when bot traffic causes a real problem.
Why It Matters
Bot impersonation sits at the intersection of search, security, and the AI training debate. Publishers face AI summaries that reduce their traffic, and fake crawlers add a pressure: content harvested without permission, sometimes at a cost to site performance. Illyes speaking about the problem signals that Google sees it as a structural issue, not a niche complaint from site owners.
The guidance also fits a wider pattern of Google explaining how its crawling works, from warnings about AI spam tactics to the quiet removal of features used to block scrapers. As AI companies compete for training data, the ability to tell a real crawler from a fake one will matter to every publisher deciding who gets access to their content.
Background
Gary Illyes is a search advocate at Google who regularly discusses crawling, indexing, and robots.txt with the search community. His comments in the Odys Podcast interview add to a series of recent Google explanations about how bots interact with websites, as AI training demand puts new pressure on crawling infrastructure built for search.
0 Comments