Short answer: structure a website for AI search by writing self-contained passages of 40 to 120 words that name their own subject, shipping content in the server response rather than client-side JavaScript, marking up Organization, Person and Article schema consistently, and allowing the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) separately from the training crawlers in robots.txt.
Most websites are built for two readers: a person who needs the page to look right, and a search crawler that needs the markup to make sense. A third reader has arrived, and structuring a website for AI search is a separate job from either one. Large language models do not rank pages. They select passages, and they only select passages they can lift cleanly out of the page around them.
For two decades that split did not exist. Designers learned to satisfy the visitor and the crawler at once without much conflict, because both wanted roughly the same things: fast load times, clean markup, sensible headings, a sitemap that did not lie.
The third reader behaves differently. It does not browse a page, it parses it. It does not return ten links, it synthesizes an answer from fragments of many pages and credits a handful of them. Research published by SE Ranking in mid-2026, tracking 101,574 websites across 16 months, found that traffic from AI search engines grew roughly sixteenfold between 2024 and 2026. That is still a small share of total visits. No other channel is growing at that rate.
This is why answer engine optimization, usually shortened to AEO, stopped being a content marketing tactic and became a front-end concern. It is also why a site that ranks well in Google can be effectively invisible inside ChatGPT, Gemini, Claude or Perplexity. The gap is rarely content quality. It is almost always structure: how the content is chunked, what the server actually returns, and whether the page tells a model what kind of thing it is.
Table of contents
- What this guide covers
- Key Takeaways
- Why extractability has become a design problem
- How long should a passage be for an AI assistant to cite it
- What schema markup actually buys you now
- The technical hygiene that decides whether you get read at all
- Which AI crawlers to allow, and the setting most sites get wrong
- Do you need an llms.txt file? The 2026 evidence says probably not
- Start with the pages that make money
- How to test whether an AI assistant can read your page
- What Google actually says about optimizing for AI
- What this means for the next site you build
- Frequently Asked Questions About Structuring a Website for AI Search
What this guide covers
This is the implementation layer. It covers how to write extractable passages, which schema types carry weight, which AI crawlers to allow in robots.txt, and how to test whether an assistant can actually read a page. It is written for designers and developers who own templates, not for marketers choosing a budget.
It deliberately does not cover AEO strategy, pricing or agency selection. If that is what you need, BestFirms published a complete 2026 playbook for getting cited by AI that covers the strategic layer this article assumes you have already thought through.
Key Takeaways
- AI models select passages to cite, they do not rank whole pages.
- A citable passage names its own subject and stands alone in roughly 40 to 120 words.
- Blocking GPTBot does not block OAI-SearchBot. Each AI crawler needs its own robots.txt directive.
- Schema markup works as a disambiguation layer, telling models what kind of thing a page describes.
- Google states there are no special requirements for AI Overviews. That applies to Google, not to ChatGPT, Claude or Perplexity.
- Ahrefs found 97% of llms.txt files received zero traffic across 137,000 sites in May 2026.
Why extractability has become a design problem
The practical unit of AI search is not the page. It is the passage. When a model assembles an answer, it pulls discrete chunks of text that can stand on their own, attributes them, and moves on. A beautifully written 2,000-word essay whose key claim only makes sense after reading the preceding four paragraphs is, from the model’s perspective, unusable.
This has a direct consequence for layout and information architecture. Content that survives extraction tends to share a few traits. Each section opens by restating its subject rather than leaning on a pronoun that refers back to an earlier heading. Claims arrive early rather than after a windup. Comparative information sits in tables rather than in prose. Procedural information sits in numbered lists.
None of this is new advice in the abstract. What is new is the penalty for ignoring it. Previously, a poorly structured page could still rank on the strength of its backlinks and earn the click anyway. Now the page either gets quoted or it does not, and the decision is made by a system that never sees the visual design at all.
How long should a passage be for an AI assistant to cite it
There is no published word count that guarantees a citation. There is a practical range that shows up consistently in citation analysis: a passage that answers one question in roughly 40 to 120 words, opens by naming its own subject, and does not depend on the paragraph above it.
The test is mechanical. Copy any paragraph out of your page, paste it into an empty document, and read it cold. If a stranger cannot tell what product, company or concept it describes, a model cannot either, and it will quote whoever wrote a version that stands alone.
| Pattern in the draft | Why a model drops it | What to write instead |
|---|---|---|
| “This makes it much faster.” | No subject. “This” resolves to a sentence that was not retrieved. | “Server side rendering makes first paint faster because the HTML arrives complete.” |
| “As we saw above, the second option is usually better.” | Depends on a passage the model never selected. | “Static generation usually beats client side rendering for content pages, because the crawler receives finished HTML.” |
| A 400-word section with the conclusion in the last sentence | The answer sits past the point where the chunk was cut. | Lead with the conclusion, then support it. |
| A comparison written as flowing prose across three paragraphs | Nothing aligns. Models rarely reconstruct comparisons from narrative. | A table with one row per option and identical columns. |
| “Pricing starts at a competitive rate.” | Unverifiable, so unquotable. | “Plans start at $29 per month for three sites.” |
This is the difference between ranking and selection, and it is why AEO advice often contradicts habits built for traditional SEO. A ranked page competes as a whole document. A selected passage competes on its own, stripped of the layout, the navigation and the paragraphs on either side. We covered how that selection dynamic plays out at the brand level in our guide to generative engine optimization for startups, where the same principle decides which companies models can recommend confidently and which ones they hedge on.
What schema markup actually buys you now
Structured data has spent years being sold to developers as a route to rich snippets, which made it feel optional for anyone not running a recipe site or an ecommerce catalogue. That framing has aged badly.
Schema now functions as a disambiguation layer. It tells a model what kind of thing a page describes, who wrote it, what organization stands behind it, and how that organization relates to its products, people, and locations. Models use those signals to decide whether a source is a credible authority on a subject or a page that merely mentions the subject in passing.
Getting the syntax right matters less than getting the entity right, but it still has to validate. Our structured data and schema SEO usage guide walks through the JSON-LD implementation and the validation tools, if the markup itself is where you are stuck.
The common implementation failure is inconsistency. A company that calls itself one thing in its schema, something slightly different in its footer, and a third variant on LinkedIn has fragmented its own entity. Models resolve that ambiguity by picking whichever version has the most corroboration elsewhere, which may not be the version the business prefers.
The technical hygiene that decides whether you get read at all
Before extraction can happen, retrieval has to happen. A surprising volume of AEO failure traces back to plumbing rather than strategy.
The recurring culprits are familiar to any front-end developer:
- Client-side rendering that leaves critical content invisible to bots without JavaScript execution
- Slow server response times that cause crawler timeouts on deep pages
- Blanket robots.txt rules that block AI crawlers such as GPTBot, PerplexityBot, and ClaudeBot
- Content locked behind interstitials, cookie walls, or accordion components that never render in the initial payload
- Heading structures that skip levels or use headings purely for visual sizing
Server response time is the one on that list most teams underestimate. A page that takes four seconds to return HTML gets crawled less often and can time out entirely on deep URLs, which is a slow bleed rather than a visible failure. If your stack is WordPress, our walkthrough on improving Google PageSpeed Insights scores covers the server-side fixes that move time to first byte rather than just the cosmetic Lighthouse opportunities.
That last one deserves emphasis. When a designer uses an H3 because it looks right at that size, the semantic map of the document breaks. A model reading the outline sees a section nested under nothing in particular and treats it accordingly.
Which AI crawlers to allow, and the setting most sites get wrong
The single most common self-inflicted AEO failure is a robots.txt file copied from a 2023 blog post. Back then the advice was simple: block the AI bots. Since then every major provider has split its crawlers by purpose, and a blanket block now removes you from AI search results while doing nothing you actually intended.
| Crawler | Operator | Purpose | What blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Your content stays out of training data |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search | ChatGPT search cannot cite you |
| ChatGPT-User | OpenAI | Fetching a page a user asked about | ChatGPT cannot open your link on request |
| ClaudeBot | Anthropic | Model training | Your content stays out of training data |
| Claude-SearchBot | Anthropic | Indexing for Claude search results | Claude search cannot cite you |
| Claude-User | Anthropic | Fetching a page a user asked about | Claude cannot open your link on request |
| PerplexityBot | Perplexity | Crawling for the Perplexity index | Perplexity cannot cite you |
| Google-Extended | Gemini training and grounding control | Nothing in Google Search or AI Overviews |
Three points that trip teams up. First, these directives operate independently: blocking ClaudeBot does not block Claude-SearchBot, and blocking GPTBot does not block OAI-SearchBot. Each user agent needs its own entry. Second, Google-Extended does not control AI Overviews. Overviews are served through Googlebot, so the only way to opt out of them is to opt out of Google Search. Third, Claude-Web and anthropic-ai are deprecated. They do no harm in an old file, but they are not doing the job people think they are.
If your position is to stay out of training data while remaining citable in AI search, this is the shape of it:
# Training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Search and retrieval crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xmlOne caveat worth stating plainly: robots.txt is a request, not a control. Perplexity has publicly argued that its user-initiated fetcher is an agent rather than a bot and therefore is not obliged to honor it, and Bytespider has been documented ignoring directives outright. If you need enforcement rather than a polite signal, that is a server-level problem, not a robots.txt one.
Do you need an llms.txt file? The 2026 evidence says probably not
llms.txt is a proposed Markdown file that tells AI systems which pages on your site matter most. It has been recommended widely enough that plugins now generate it with one click. The measured results are close to zero.
Ahrefs analyzed 137,000 sites and found that 97% of llms.txt files received no traffic at all in May 2026. AI bots were not requesting the file on domains where it existed, and never probed for it on domains where it did not, which is what you would expect if any crawler wanted it. SE Ranking analyzed roughly 300,000 domains and found no statistically significant correlation between having the file and how often a domain was cited in AI answers. Google’s John Mueller compared it to the keywords meta tag, which is the canonical example of self-declared metadata that no search system reads. No major provider, OpenAI, Google, Anthropic, Meta or Perplexity, has publicly committed to reading it in production.
It is not worthless. Its real constituency is developer tooling: AI coding assistants pulling current API documentation into a session, and internal retrieval pipelines you control yourself. If you ship developer docs, an llms.txt is a reasonable convenience for your users. If you run a marketing site and your goal is citations, it delivers nothing measurable.
One implementation to avoid outright: generating an indexable Markdown mirror of every page on the site. That creates duplicate content at scale, dilutes crawl budget, and can suppress the original pages. The cost of llms.txt is rarely the file. It is the hours spent on it instead of on the passage structure that actually correlates with citations.
Start with the pages that make money
There is a natural instinct, inherited from a decade of content marketing, to answer a new search channel by publishing more blog posts. In AI search that instinct is close to backwards.
Informational queries are the ones assistants answer completely. Someone who asks what answer engine optimization means gets a clean paragraph and stops reading. The citation is real. The visit often is not. Commercial queries behave differently, because a person choosing between two vendors has to leave the conversation to sign up, book a demo or buy. Those are the answers where a citation still leads somewhere. They are also the answers with the fewest credible sources competing for them, because most companies never wrote the page in the first place.
So the sequencing runs opposite to the usual content calendar. Restructure the pages that already sit near a buying decision before commissioning a single new top-of-funnel article.
| Page type | The question it catches | The structural fix it usually needs |
|---|---|---|
| Comparison pages | “what is the difference between X and Y” | A real table with identical rows for both options, not three paragraphs of narrative |
| Alternatives pages | “what are the best alternatives to X” | Name every competitor explicitly, including the ones you lose to. A model will not infer a list you did not write |
| Pricing pages | “how much does X cost” | Actual numbers in text, not inside an image and not behind a currency toggle. “Contact us” is not an extractable answer |
| Use case pages | “best tool for [specific job]” | One page per job, each opening with a sentence naming the job and who does it |
| Case studies | “does X actually work for [industry]” | The result stated as a number in the first fifty words, not saved for the closing paragraph |
| Integration pages | “does X work with Y” | A dedicated page or table row per integration. A logo grid is invisible to a text model |
| Documentation | “how do I do [task] in X” | Each task on its own URL, with a heading that matches how someone would phrase the task |
Choosing which of those to fix first is not a judgment call. It is a twenty-minute exercise. Write down the fifteen or twenty questions a buyer asks in the week before they pick a vendor in your category. Run each one through ChatGPT, Claude and Perplexity without naming your brand. Note who gets cited and what kind of page the citation points to. The gap between those questions and the pages you actually own is your queue, already in priority order.
Two findings tend to surprise teams doing this for the first time. Competitors are usually cited from thin pages that simply exist rather than from good ones, which means the bar is lower than it feels. And a meaningful share of citations point at third-party sources, review sites, directories and forums rather than at any vendor’s own domain, which means part of the queue is not a page you write at all.
The conversion case for this sequencing is real, and softer than it is usually presented. Several 2026 analyses report AI-referred visitors converting at multiple times the rate of traditional organic traffic, with longer sessions and higher return rates. A Semrush study of 54 websites found no statistically significant conversion difference at all. The honest summary is that AI referral volume is small, the intent behind it is sharper, and the conversion premium is contested enough that it should not be the only reason you reorder a roadmap.
The stronger reason is simpler and does not depend on any of those numbers. A comparison page answers a question someone asks once, at the moment they are deciding, and there is usually one obvious answer worth citing. A definition page answers a question thousands of people ask idly, and there are five hundred acceptable sources. Restructuring the first kind is a small amount of work aimed at a decision. Publishing more of the second kind is a large amount of work aimed at a summary.
How to test whether an AI assistant can read your page
Most AEO diagnosis is guesswork because teams never check the thing they are theorizing about. This takes about twenty minutes per template.
- Read the raw HTML. Right click, View Page Source, and search for a sentence you can see on screen. If it is not in the source, it was injected by JavaScript and any crawler that does not execute scripts will never see it. Command line equivalent:
curl -A "OAI-SearchBot" https://yoursite.com/page/ | grep "your sentence". - Check the response time on a deep page, not the homepage. Homepages are cached and warm. Category pages five levels down are where crawler timeouts actually happen.
- Read your own robots.txt line by line against the user agents in the table above. Look specifically for a blanket block written before the search crawlers were split out.
- Validate your structured data with Google’s Rich Results Test and the Schema.org validator. Then check that the organization name in your JSON-LD matches your footer, your LinkedIn and your Google Business Profile exactly.
- Paste the URL into ChatGPT, Claude and Perplexity and ask each one to summarize the page’s main claim in a single sentence. A wrong or vague summary is not a model failure. It means the claim is not sitting in a passage that stands alone.
- Ask the same assistants a question your page should answer, without naming your brand, and see who gets cited instead. That tells you what a winning passage looks like for that query.
- Check your server logs for the AI user agents. If GPTBot and OAI-SearchBot have never appeared, the problem is access, not content.
We ran this across visualmodo.com in August of 2026 and found nothing broken, which is the outcome you want and also the reason most teams never bother. The audit is cheap. The failure it catches is silent, so nobody notices they needed it.
Steps five and six are worth repeating on a schedule, because citations churn faster than rankings do. Doing it by hand across a handful of prompts is fine to start. Once you are tracking dozens, it becomes a tooling problem, and we compared the options in our roundup of the best AI visibility platforms.
What Google actually says about optimizing for AI
It is worth being honest about the strongest argument against everything above. Google’s own documentation on AI features and your website states that the established SEO best practices remain relevant for AI Overviews and AI Mode, that there are no additional requirements to appear in them, and that no special optimization is necessary. Google’s 2026 generative AI guidance goes further and lists machine-readable files like llms.txt among the tactics that are not needed.
Two things reconcile that with the practical experience of teams doing this work.
First, Google is talking about Google. AI Overviews and AI Mode are grounded in Google’s own index, built by Googlebot. ChatGPT, Claude and Perplexity operate their own retrieval stacks with their own crawlers and their own selection logic. A robots.txt that satisfies Googlebot says nothing about whether OAI-SearchBot can reach you.
Second, “no additional requirements” describes eligibility, not selection. Being eligible for a citation and being the passage that gets chosen are different outcomes, and the gap between them is where structure does its work. Google is correct that there is no secret AEO ranking factor. It does not follow that how you chunk a page has no effect on which chunk gets quoted.
What this means for the next site you build
The uncomfortable conclusion for design teams is that AEO is not a marketing task that arrives after launch. Extractability is baked in at the template level, and retrofitting it is more expensive than building it correctly.
A few practices are worth adopting as defaults. Give every page a short, self-contained summary near the top that answers the page’s implied question in under sixty words. Build heading components that enforce semantic order rather than leaving it to editorial discretion. Ship schema as part of the template rather than as a plugin afterthought. Make sure that anything a model needs to read is present in the server response.
None of these choices harm the human experience. Clear summaries help skimmers. Semantic headings help screen readers. Fast server rendering helps everyone. The happy accident of answer engine optimization is that most of what makes a page legible to a language model also makes it legible to a distracted person on a phone.
The web did not stop being a design medium when AI assistants started reading it. It simply acquired a reader with no eyes, no patience, and an enormous amount of influence over what gets recommended. Building for that reader is now part of the job.
Frequently Asked Questions About Structuring a Website for AI Search
Roughly 40 to 120 words, answering one question, opening by naming its subject rather than referring back to a previous heading. There is no guaranteed length. The reliable test is whether the passage still makes sense when read in isolation.
That depends entirely on whether you object to training use. Blocking GPTBot keeps your content out of OpenAI’s training data and has no effect on whether ChatGPT can cite you in search, because that is handled by a separate crawler. Blocking both removes you from ChatGPT results.
GPTBot collects content for model training. OAI-SearchBot builds the index behind ChatGPT’s search feature. They are separate user agents with separate robots.txt directives, and blocking one does not block the other. Anthropic makes the same split between ClaudeBot and Claude-SearchBot.
Not as a direct ranking lever. Schema works as a disambiguation layer: it tells a model what kind of thing a page describes, who wrote it and what organization stands behind it. That matters most when a model is deciding whether you are an authority on a subject or a page that merely mentions it.
Sometimes, and unreliably. Some crawlers execute JavaScript, others take the raw HTML response and nothing else. If your primary content is client-side rendered, you are betting your citations on which crawler arrives. Server-side rendering or static generation removes the bet.
It is a proposed Markdown file listing your most important pages for AI systems. As of 2026 no major provider has committed to reading it, Ahrefs found 97% of them received zero traffic, and SE Ranking found no correlation with citations. It has genuine value for developer documentation consumed by AI coding assistants. For a marketing site, it is optional at best.
View the raw page source and confirm your key sentences are actually in the HTML, check that OAI-SearchBot is not blocked in robots.txt, then paste the URL into ChatGPT and ask it to summarize the page’s main claim. A wrong summary usually means the claim is buried in a passage that cannot stand alone.
