Labrador: ChatGPT's own index and what it demands from your website
ChatGPT no longer gets its web search only from partners. Behind it runs its own retrieval index, internally called Labrador: OpenAI's own store of web pages that the system pulls results from instead of buying them in. And it is not one index but a family of vertical indexes: web, PDF, YouTube, news, arXiv, Wikipedia, local, finance, legal, medical, shopping, images.
That is the difference from the earlier logic. Until now, ChatGPT Search was essentially a layer on top of third-party search results, Bing class, partly via scraped Google results. Testimony under oath in the Google antitrust case showed that, from OpenAI's point of view, the search data from its non-Google partners had significant quality problems, and that the company had therefore been working on its own index since 2023, with the stated goal of answering a large share of queries from it. Bing has not disappeared, it still comes in through Microsoft's Web IQ and Deep Research. But the question is a new one now: does OpenAI's own index know you?
Check whether OAI-SearchBot can reach your pages at all, in robots.txt and in your firewall or CDN. If access stays blocked, your chances of appearing in ChatGPT's own index or in its search answers drop, no matter how good your content is.
How the index crawls, evaluates and selects content
How much this in-house index actually carries can be measured. Between May 21 and July 21, 2026, ChatGPT's own data stream included a field for every result that named its source: labrador for the in-house index, plus three external scraper pipelines. Since July 21, 2026 the field has been gone, and attribution now relies on format features such as title shape and snippet length.
RESONEO measured in this phase that in the free Think mode 74.7 percent of results came from OpenAI's own index, while in the paid Thinking mode 75.3 percent came from scraped Google results. So the mix depends on which version your readers use.
The path into the index runs through three stages: crawled, stored, served in an answer. And each stage can be the end of the road. A bot visit does not guarantee inclusion, OpenAI apparently filters, and its criteria have not been published.
That content is stored is now established: a parameter in OpenAI's web search API allows a mode in which ChatGPT answers exclusively from its own cache, and content that had been sitting there for more than 30 days was still retrievable. In April 2026, OpenAI's help center confirmed the index indirectly by describing an offline web search over indexed and cached web content for eligible workspace accounts. That OpenAI keeps testing its own index against scraped search results is most visible in shopping.
Technical requirements so your site gets captured at all
This mechanism leads to hard technical requirements. According to OpenAI’s own description, OAI-SearchBot is responsible for search. GPTBot collects training data, ChatGPT-User fetches pages live while an answer is being generated, triggered by a specific user request. The three can be controlled separately in robots.txt. So you can block training and allow search. Blocking all AI bots across the board lumps three different business decisions together.
Changes to robots.txt take about 24 hours before OpenAI's systems take them into account. A revised version of the documentation also suggests that ChatGPT-User may no longer be strictly bound by robots.txt elsewhere, because user actions can bypass the rules, and when both are allowed OpenAI may use a single crawl for search and training.
The more common cause of invisibility is not in robots.txt anyway, but in the firewall and CDN. Cloudflare, Akamai, AWS WAF, Imperva and Fastly ship default rules that aggressively throttle unknown user agents or answer them with challenges. Crawlers do not solve CAPTCHAs, they get a 403 or 429 and are out, even though robots.txt allows them. So check status codes in your logs, verify genuine requests against OpenAI's published IP ranges, and exempt those IPs from JavaScript and CAPTCHA challenges.
According to observations from earlier tests, the crawler does not render JavaScript and probably reads only what is delivered server-side, so content loaded later apparently does not arrive. A cap has also been described for very large pages, above which a page is apparently not read at all rather than just truncated. How reliably embedded scripts, iframes or structured data in JSON-LD format arrive during ingestion is not clearly established, but several sources report that in practice such markup gets lost in the conversion to plain text. Directives such as Cache-Control: no-store or a noindex tag also do not seem to reliably stop retrieval, based on observations so far.
Concrete steps for more visibility in the new index
The first step is trivial and still gets skipped: an explicit allow rule for OAI-SearchBot instead of assuming a wildcard covers it, then a look at the logs to see whether requests are answered with 200. How often it crawls also tells you how deeply the index knows your domain at all.
The second lever is the page content itself. Because the fetch tool reads pages in text chunks, according to the reconstruction by Dan Petrovic in a kind of jumping window that discards design and scripts and pulls short passages of plain text, there is a good case for delivering the answer server-side and making every section understandable on its own. Observed, not official, but consistent with what gets through technically. Keep pages below the size cap, cut out rendering dependencies, and do not rely on markup to carry your message: write it in the text.
As a map, you can also offer an llms.txt, a file in the root directory that summarizes your most important content in plain text and gives AI systems orientation. It does not control access and blocks nothing, it is an offer.
And because Bing data still comes in through Web IQ, clean Bing indexing including a sitemap can still pay off. It is just no longer a substitute for the question of whether OpenAI's own index knows you.
Is my existing visibility in ChatGPT answers losing importance now?
+
No, it is shifting. Visibility in ChatGPT answers still comes from two paths: the index and live retrieval during a user question. The index path depends on its own rules, which OpenAI sets and apparently moves from time to time. If you block OAI-SearchBot, you drop out of search answers according to OpenAI's documentation, but can still appear as a navigational link. How firmly that promise still holds in the revised documentation is hard to judge from the outside. In practice, existing visibility is no guarantee, and you have to actively keep index access open.
Is this change permanent or is OpenAI planning further adjustments?
+
This is clearly movement, not an end state. The source field in the data stream was visible for two months and then disappeared again, a sign that OpenAI is still working internally on the balance between its own index and scraped search results. The crawler documentation has also recently been adjusted on individual points. And the antitrust case showed that OpenAI itself, even with extensive access to third-party index data, expects a multi-year path before it can reliably say how many queries its own index can carry alone. So expect further adjustments and check your crawler settings regularly rather than once.
What does this mean for my Google search traffic and my SEO strategy?
+
It stays your bread and butter. Good Google indexing is a precondition for appearing in the scraped result sets at all, and in German website logs AI chatbot requests still make up only a tiny share for most projects, while leads keep coming from classic search systems. Classic SEO, meaning clean indexing, internal linking and structured content, is the foundation that the index work builds on.
Do I need to consider data protection when OpenAI crawls my site?
+
For your own publicly served pages, crawling is first of all a question of access, nothing more. What matters is where robots.txt stops: it controls OAI-SearchBot and GPTBot, while ChatGPT-User apparently is not bound by it to the same degree for user-triggered actions. Content that should not end up in other hands therefore needs server-side protection, not a directive. On the user side, OpenAI says it passes search queries to external search providers in rewritten form, along with the approximate location derived from the IP, but not the IP address itself or account information.
