To extract your website’s structure for ChatGPT, give the model two things together: a clean outline of every page and its headings, and each page exported as Markdown. A list of URLs alone is structure without substance, and a dump of raw HTML is substance buried in noise. The pairing is what the model actually uses. Feed the outline as an index so it knows the shape of the site, and the relevant Markdown files as detail so it knows what each page says. WPBuildAI produces both a full structure map and a clean per-page Markdown export, which is exactly the context a model reasons over well.
Weigh how much context you actually need against how the assistants fetch, since ChatGPT dominates the traffic that comes back: SE Ranking’s analysis of 101,574 sites put ChatGPT at 74.78 percent of AI referrals with Claude at 2.62 percent, so an extraction tuned for one tool covers most of the practical ground.
Why structure and content go together
The outline is the map and the Markdown is the territory, and a model needs both. With only a sitemap, ChatGPT knows what pages exist but not what they say. With only raw HTML, it has the words but cannot tell a navigation menu from a heading or one page from the next. The Generative Engine Optimization study found that models work best with content that has clear structure and defined topics, which is precisely what an outline plus clean Markdown provides: the shape and the substance, each in a form the model can parse. Give it both and it can answer questions about the site as a whole, not just the page in front of it.
What the structure map should contain
A useful outline is more than a flat list of URLs. For each page it records the title, the headings in order, where the page sits in the hierarchy, and the URL. That lets the model see how the site is organised, which pages relate to which, and where a topic is covered. Built well, the outline reads like a table of contents for the whole site, and a model handed a table of contents can navigate to the right page far better than one handed a pile of unlabelled text. The map turns a heap of pages into a structure the model can reason about.
Keep the context clean and small
Raw HTML is expensive. The 2024 Web Almanac documents how much of a page is markup and script rather than text, and every byte of that eats the model’s context window. Clean Markdown strips it away, so each page uses a fraction of the tokens and you can feed only the pages a task needs. This is what makes a whole-site context practical: a clean outline plus clean pages fits where the raw site never would. The token savings are not a nicety; they are the difference between fitting your content in the window and overflowing it.
Why a live URL fetch is the wrong input
It is tempting to skip the export and just give ChatGPT your URLs, but that depends on a fetch that often fails. Pages that build their content with JavaScript can hand a fetcher an almost empty shell, the gap Google describes in its JavaScript SEO basics, and a slow page can return only part of itself. Across a whole site, a few of those partial captures are easy to miss, and the model then reasons from incomplete context without telling you. Feeding files you control is dependable where a fetch is a gamble, which is why the export beats pointing the model at live URLs.
How to do it, step by step
- Crawl the site into an outline of pages and headings, using internal links and the sitemap together so nothing is missed.
- Export one Markdown file per page, dropping HTML and scripts, with the URL noted at the top.
- Feed the outline as an index, so the model knows the shape of the site.
- Provide the relevant Markdown files for the task at hand, rather than the whole site at once.
- Summarise pages for very large sites, keeping the outline as the map and pulling full detail only when needed.
The crawl mechanics are the same as in mapping a site’s structure before migrating, and the per-page export is the same as turning a site into ChatGPT-ready Markdown.
A worked example: answering across a 150-page site
Picture a 150-page site and a question that spans several pages, like how a policy is described across help, terms, and product pages. Pasting URLs would risk partial fetches and give the model no sense of how the pages relate. The structure-plus-content approach handles it: the outline shows the model that the policy is covered in three places, and the three Markdown files give it the exact wording. The model can then reconcile them and cite the right page for each detail. Without the map it would not know to look in all three; without the clean files it would not have the wording. Together they let it answer a cross-site question accurately, which is the whole point of giving it structure as well as content.
The token budget in practice
On any real site you will not fit everything at once, so the outline does double duty as a budgeting tool. Feed the full outline, which is small, so the model always has the map, then spend your token budget on the specific pages a task needs. For a broad question, summaries of many pages plus the outline may fit; for a deep question, a few full pages plus the outline is better. Clean Markdown stretches the budget because it is compact, and the per-page structure lets you swap detail in and out without re-uploading the whole site. Thinking in terms of “outline always, pages as needed” keeps even a large site usable.
The llms.txt connection
There is a recognised convention for the clean, machine-readable form this relies on. The llms.txt standard points models at the plain-text version of a site precisely because stripped Markdown is what they parse best, and an outline that links to per-page Markdown is essentially that idea applied to your whole site. If you publish your site with this structure in place, an AI tool can read it in context without any manual export at all, which is the live-site version of what you are assembling by hand here.
When you do not need the full structure
If your question is about a single page, you do not need the whole site’s structure; the one page’s Markdown is enough. The full outline-plus-content approach earns its effort when a task spans many pages, when you are building an assistant over the site, or when you need the model to understand how the site is organised. For a one-off question about one page, skip the map and paste the page. Matching the effort to the task keeps you from exporting a whole site when a single file would do.
Common mistakes that waste the context
A few habits make the context less useful than it should be. Pasting raw HTML instead of Markdown fills the window with markup and leaves the model less room for real content. Giving only the URL list, with no page content, lets the model see the shape of the site but not answer questions about it. Dropping the outline and pasting pages at random leaves the model unable to see how pages relate. And exporting once and never refreshing means the model answers from a stale snapshot. Each mistake comes from treating the job as a quick dump rather than assembling a structured map plus clean content, which is what actually makes a whole site usable as context.
Keeping the context current
A site changes, and the context is a snapshot, so plan to refresh it. Because each page is its own Markdown file and the outline is a separate index, you update only what moved: re-export the changed pages, refresh the outline if the structure changed, and leave the rest. Noting the update date in each file makes it obvious which version the model is working from, which matters when a question touches a page that has since changed. Treating the export as something you maintain, rather than build once, keeps the model from confidently answering out of date.
Structure helps the model reason, not just retrieve
There is a subtle benefit to giving the model the outline as well as the pages. With the map in hand, the model can reason about coverage and relationships: it can tell you which pages discuss a topic, notice when two pages disagree, and point you to the right section rather than guessing. A model handed only loose page content can answer about a page; a model handed the structure can answer about the site. For tasks like auditing your own content, finding gaps, or reconciling overlapping pages, that site-level view is the whole point, and it is why the outline is worth assembling even though the pages alone would technically fit.
A note on doing this for a large site
On a big site, assembling the outline and exporting every page by hand is real work, and it is the same work a migration requires. That overlap is worth exploiting: if you are going to crawl, outline, and clean-export the site for a model, you have most of what a rebuild needs, and vice versa. Rather than do it twice for two purposes, do it once and reuse the output. That is the practical case for treating the extraction as groundwork rather than a throwaway task, especially when the site is large enough that the crawl and export are not trivial.
The extraction doubles as migration groundwork
The structure map and the clean per-page content you assemble for ChatGPT context are the same inputs a clean migration runs on, so the work serves two ends. Rebuild from the exported content, keep the slugs, and map every old URL to a 301, and your live site and your model context stay sourced from the same clean material. The structure that helps a model reason about your site is also what helps public AI engines understand and cite it, the link drawn in getting your website cited by AI search, and the RAG version of the same dataset is in extracting content for Gemini or RAG. If assembling the outline and exporting a whole site by hand is more than you want to take on, WPBuildAI produces both from one crawl; send your site URL for a fixed quote.
Building the outline so a model reads it well
The outline is most useful when it is itself clean and structured, not just a dumped sitemap. Group pages by section, indent or nest them to show hierarchy, and include each page’s title and a one-line note of what it covers, with the URL. A model handed that reads it as a table of contents and can decide which page to ask for. A flat, unlabelled list of URLs is far weaker, because it shows existence without meaning. The same clarity that helps a person skim a contents page helps a model navigate the site, so spend a little effort making the outline readable rather than treating it as raw output.
Key points to remember
Give ChatGPT a clean outline of pages and headings plus a per-page Markdown export, because the map alone is too thin and raw HTML too noisy. Feed the outline as an index and the relevant pages as detail, summarising for large sites and pulling full content only when a task needs it. Skip the live-URL fetch, which returns partial content, and feed files you control. The same structure map and clean content are the inputs a clean migration needs, so the work is worth doing once and reusing. If assembling the outline and exporting a whole site by hand is more than you want to take on, WPBuildAI produces both from a single crawl and uses them to feed a model and to rebuild the site; send your site URL for a fixed quote.
Not affiliated with OpenAI, Google, WordPress, or Lovable.