In August 2023, OpenAI documented its GPTBot web crawler and told website owners they could block it in robots.txt. On September 28, 2023, Google announced Google-Extended, a robots.txt token that lets publishers manage whether their content helps improve Google's generative AI models. Google says Google-Extended does not affect inclusion or ranking in Google Search. For private practices, which generally want AI assistants to describe them accurately, blanket blocking of AI crawlers was rarely the right default.
Before 2023, robots.txt was mostly a tool for search engines. Website owners used it to keep crawlers out of admin areas or duplicate pages. In August and September 2023, it became something new: the main way websites could signal whether their content should be used to train AI models.
OpenAI documented its GPTBot crawler and explained how to block it. Google followed with Google-Extended. Publishers responded quickly, and many large sites blocked AI crawlers outright. For private medical practices, the question was different. Most practices do not sell content. They want prospective patients to find them, and increasingly those patients ask AI assistants for recommendations.
This retrospective covers what happened in 2023 and how practices should think about AI crawlers today. It connects to our coverage of Cloudflare’s 2025 AI crawler changes, the ChatGPT search launch in 2024 and the 2023 launches of Bing Chat and Bard.
What actually happened in 2023
August 7: GPTBot. As The Verge reported, OpenAI published documentation for its GPTBot web crawler and told site owners they could disallow it in robots.txt or block its IP addresses. OpenAI said pages crawled with GPTBot may be used to improve future models, and that crawled pages are filtered to remove paywalled sources, sources known to gather personally identifiable information and text that violates its policies.
No retroactive removal. The Verge noted that blocking GPTBot did not remove content that had already been collected for earlier training.
September 28: Google-Extended. Google’s announcement introduced Google-Extended, a control publishers could use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future models.
Separate from Search. Google’s crawler documentation now describes Google-Extended as a standalone product token, used through robots.txt rather than a separate crawler, that does not impact a site’s inclusion in Google Search and is not used as a ranking signal.
Controls evolved later. OpenAI’s current crawler documentation describes separate settings for GPTBot and OAI-SearchBot, so a site can allow search visibility while disallowing training.
Why it mattered for private practices
For news publishers, blocking AI crawlers was a business decision about protecting valuable content. For a private practice, the calculation is reversed. A practice’s website exists to inform prospective patients and earn trust. If an AI assistant is asked about the best options for a treatment in a city, the practice wants that assistant to know accurate, current information about it.
That does not make every crawler equal. Training crawlers, search crawlers and user-triggered fetchers serve different purposes, and the controls introduced in 2023 were the first step toward separating them. But a blanket block, copied from a publisher’s robots.txt or switched on by a plugin, could reduce a practice’s presence in AI tools without any clear benefit.
What typical agencies did
Many agencies did nothing, which was often fine. Others added sweeping AI blocks to client robots.txt files after reading publisher coverage, without asking whether it fit a practice’s goals. Some hosting and security tools later added AI blocking options that site owners turned on without understanding the tradeoffs.
The better response was a deliberate policy: know which crawlers access the site, decide what each should be allowed to do and document the decision. That takes an hour, not a project, and it prevents a practice from disappearing from AI tools by accident because of a setting nobody chose on purpose.
What it means for a practice today
AI assistants now play a growing role in how patients research care. These steps keep a practice’s crawler settings aligned with its goals.
1. Read your current robots.txt
Check what your site currently allows and blocks. Many practices discover rules added years ago by an agency, plugin or hosting provider that nobody remembers approving. Some of those rules block far more than intended.
2. Check hosting and security settings
Firewalls and content delivery networks can block AI crawlers independently of robots.txt. Review those settings too, since they often override what robots.txt says. A single checkbox in a security dashboard can block crawlers the practice actually wants.
3. Separate training from search
Decide whether you are comfortable with content being used for training, and separately whether you want to appear in AI search results. Current controls allow different answers, so the two decisions do not have to be the same.
4. Keep sensitive content off public pages
Robots.txt is a request, not security. Patient portals, forms and anything containing personal information must be protected by proper access controls.
5. Make your public content accurate
If AI tools read your site, make sure services, providers, locations and hours are current and consistent across the web. Inconsistent details are a common source of wrong answers.
6. Revisit the policy yearly
Crawler names and controls keep changing. Review your settings at least once a year and after any website, hosting or security vendor change.
Training versus retrieval
The most important distinction to understand is between training and retrieval. Training is when content is used to build or improve an AI model. Retrieval is when an AI tool looks up current web content to answer a specific question, often citing the source. The controls introduced in 2023 focused on training. Later controls, such as OpenAI’s OAI-SearchBot, address search and retrieval separately.
For a practice, retrieval matters most. When a patient asks an AI assistant about a treatment or a local provider, the assistant’s ability to retrieve accurate, current information about the practice can influence whether it is mentioned. Blocking retrieval crawlers removes that possibility. Blocking training crawlers has a less direct and less predictable effect.
A reasonable approach for many practices is to allow search and retrieval crawlers, make a considered decision about training crawlers and keep all of it documented. Our guide to getting your practice recommended by AI explains the content side of that strategy.
What robots.txt can and cannot do
Robots.txt is a public file that tells well-behaved crawlers what the site owner prefers. Reputable companies, including OpenAI and Google, say they honor it. It is not enforcement. A crawler that ignores robots.txt is not technically prevented from accessing public pages.
That makes robots.txt the wrong tool for protecting anything sensitive. For practices, the rule is simple: if information should not be public, it should not be on a public page. Patient portals, intake forms and documents should sit behind proper authentication, regardless of crawler settings.
The content AI tools see
If AI tools are going to read a practice’s website, the content they find matters. Thin service pages, outdated provider lists, old pricing or inconsistent addresses can all lead to inaccurate answers. Clear, current pages for each treatment, accurate provider bios with credentials and consistent business information across Google, Apple, Bing and directories give AI tools a better chance of describing the practice correctly.
That is the same content that serves patients and traditional search. There is no separate AI website. There is one website that should be accurate and helpful to every reader, human or machine.
Why some practices may still block training
There are reasonable reasons a practice might block training crawlers. A practice that has invested heavily in original educational content may not want it used to train models without attribution. A practice with strong views on AI may simply prefer not to participate. Those are legitimate choices for a practice to make.
The key is making the choice deliberately, understanding that training and search controls are separate today, and avoiding unintended blocks of the crawlers that help patients find the practice. A blanket rule copied from a news publisher is rarely the right fit for a practice whose goal is visibility.
A sensible default policy
For most private practices, a sensible starting policy looks like this. Allow major search engine crawlers, including Googlebot and Bingbot, without restriction on public pages. Allow AI search and retrieval crawlers so that AI assistants can find and cite current information. Make an explicit decision about AI training tokens such as GPTBot and Google-Extended, based on how the practice feels about its content being used for model training. Block crawlers from admin areas, internal search results and any utility pages that add nothing for patients.
Then document the policy in plain language: which crawlers are allowed, which are blocked and why. Share it with whoever manages the website, hosting and security tools, so a well-meaning change in one place does not quietly undo the decision made in another.
That written policy also makes the next review easier. When a new crawler appears or a vendor changes how its controls work, the practice can compare the change against its stated goals instead of starting from scratch.
How to check what is actually happening
Settings on paper and behavior in practice can differ. Server logs and some hosting dashboards show which crawlers visit a site and whether they receive successful responses or blocks. Reviewing them occasionally confirms that the intended crawlers are getting through and the blocked ones are not.
It is also worth testing the result from the patient side. Ask a few AI assistants about the practice and its main services and see whether the answers are accurate and current. Wrong hours, outdated providers or missing services usually point to inconsistent information across the web rather than crawler settings alone, but both are worth checking.
The broader 2023 context
GPTBot and Google-Extended arrived in a year dominated by AI search experiments. Bing Chat and Bard launched early in the year, and Google opened its Search Generative Experience to testers in May. Publishers were negotiating licensing deals, filing lawsuits and blocking crawlers. Website owners everywhere were trying to understand what AI meant for their traffic.
For practices, the year established a pattern that has continued: AI tools increasingly shape how patients discover care, and practices that keep their information accurate and accessible are best positioned. We explored the search side of that shift in our retrospective on the 2023 SGE launch.
What 2023 does not tell us
We do not have reliable data on how many medical practices blocked GPTBot or Google-Extended in 2023, or on how those choices affected their later presence in AI answers. OpenAI and Google did not publish practice-level data. What 2023 does show is the documented purpose of each control and Google’s statement that Google-Extended does not affect Search.
The premium practice advantage
Premium practices benefit when AI tools describe them accurately and favorably. See our services, who we help, our custom websites and our approach to Google Maps and AI search visibility. Our team works from Austin with practices nationwide.
The bottom line
GPTBot and Google-Extended gave website owners their first simple, practical AI training controls in 2023. For practices that want patients to find them, the right answer was a deliberate crawler policy, not a reflexive block.
Check if your market is open. We take one practice per specialty per market. Request a market check or call (716) 400-0084.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI's web crawler. OpenAI said in August 2023 that pages crawled by GPTBot may be used to improve future models, and that site owners could disallow it in robots.txt.
What is Google-Extended?
A robots.txt product token Google announced on September 28, 2023. It lets publishers manage whether content Google crawls may be used for its generative AI models and certain grounding uses.
Does blocking Google-Extended affect Google Search rankings?
No. Google's crawler documentation says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.
Should a medical practice block AI crawlers?
Usually not by default. Most practices want AI tools to have accurate information about their services. Practices should make a deliberate choice for each crawler based on their goals.
Is blocking GPTBot the same as blocking ChatGPT search?
No. OpenAI now documents separate controls, including OAI-SearchBot for search, so a site can allow search visibility while disallowing training.
Does robots.txt protect patient information?
No. Robots.txt is a public request to crawlers, not a security control. Patient information should never be on publicly accessible pages in the first place.