To turn a website into a knowledge base for a custom GPT, export the site as clean Markdown, one file per page, keep the original URL on each file, and upload that set as the bot’s knowledge. The point worth internalising is that a custom GPT does not understand your business; it retrieves passages from the files you give it and writes around them. So the export is the whole game. Feed it clean, well-structured files and it answers accurately and cites the right page; feed it a live URL or raw HTML and it retrieves a mess. WPBuildAI produces exactly that clean, per-page export as part of a migration.
Build the knowledge base from your own export rather than hoping a crawler assembled it correctly, because Cloudflare’s data shows roughly 80 percent of AI crawling exists to gather training data rather than to answer a specific question, and training crawls are not organised around your site’s structure.
Why the export quality sets the bot quality
A custom GPT answers from its attached files and nothing else, which means the files are the product. A live page works against that in two ways. First, its content is buried: the 2024 Web Almanac shows how much of a page is theme markup, script, and styling rather than text, and all of that noise either drowns the real content or pushes it out of the model’s reach. Second, it is unreliable to fetch: pages that depend on JavaScript to render often come back partial, the gap Google describes in its JavaScript SEO basics. Strip each page to Markdown and the bot has a clean, quotable chunk to work from instead.
Structure decides retrieval as much as cleanliness
A clean file is necessary but not sufficient; the structure of that file decides how well the bot retrieves from it. The Generative Engine Optimization study found that content with clear headings, defined entities, and concrete facts is selected and quoted far more reliably than loose prose, and the same property that helps a public AI engine cite a page helps your private GPT pull the right passage. So keep the headings, keep one page per file, and note the URL at the top so the bot can attribute its answer. A well-structured file is the difference between a precise quote and a vague summary of the wrong page.
How to build the knowledge base
- Crawl the site to find every page, using the sitemap and analytics so nothing is missed.
- Export one Markdown file per page, dropping the HTML, scripts, and CSS, with the URL noted at the top.
- Keep the headings, so each file has internal structure the model can navigate.
- Split very long pages into sections so no single chunk overflows or blurs topics.
- Upload the set as the GPT’s knowledge and test it against real questions, watching which page it cites.
The convention behind this clean, machine-readable form is captured in the llms.txt standard, which points models at the plain-text version of a site precisely because stripped Markdown is what they retrieve from best. Building a knowledge base is the same idea applied across every page at once.
Custom GPT or a retrieval setup
There are two ways to use the same export, and the choice is about scale. A custom GPT with attached files is the simple version: you upload the per-page set and it searches within it, which works well for a site of modest size. A retrieval setup indexes the files in a vector store and fetches only the most relevant chunks for each question, which scales to far larger content without overflowing the context window. Both run on identical input, so you do not redo the export to switch between them. The RAG version is covered in extracting website content for Gemini or RAG, and the single-blog case in uploading a WordPress blog into Claude or GPT.
Keeping the knowledge base accurate over time
A knowledge base is a snapshot, so plan to refresh it. When you change pages, re-export the ones that moved and replace those files rather than rebuilding the whole set; because each page is its own file, the maintenance stays small. Noting the publish or update date in each file makes it obvious which version the bot is answering from, which matters when someone asks about a price or a policy that has since changed. A stale knowledge base answers confidently from old content, which is worse than admitting it does not know, so treat freshness as part of the build, not an afterthought.
A worked example: a support knowledge base
Picture a company that wants a custom GPT to answer customer questions from its help pages, product pages, and policies, around 120 pages in all. Pointing the bot at the live site would be unreliable, since some pages render late and the bot cannot tell which page an answer came from. The export approach gives it 120 clean Markdown files, each a single page with its title, headings, body, and URL, grouped into folders for help, products, and policies.
Uploaded as the GPT’s knowledge, that set lets the bot answer “what is your return window” by retrieving the returns policy file and citing its URL, rather than guessing from a blurred view of the whole site. When the company changes its shipping policy, it re-exports that one page and replaces the file, and the bot is current again. The structure that makes this work, one focused file per page with its URL, is the same structure that makes a public AI engine quote the page accurately, which is why the export serves both.
Common retrieval problems and how to fix them
When a custom GPT answers poorly, the cause is usually the files, not the model. If it blends two topics into one muddled answer, the pages are probably too long or cover several subjects, so split them into focused files by section. If it cannot find an answer that is clearly on the site, check that the page was actually exported and that its headings name the topic in the words people use, since retrieval leans on those headings. If it answers from old content, the export is stale and needs refreshing. And if it cites the wrong page, the files likely lack the URL note that lets it attribute correctly.
The pattern behind these fixes is that retrieval rewards clean, focused, well-labelled chunks. A page that tries to cover everything retrieves vaguely; a set of focused pages, each clearly about one thing, retrieves precisely. Improving the bot is mostly improving the files it reads.
How large can the knowledge base be
A custom GPT with attached files handles a modest site comfortably, but there is a practical ceiling: as the set grows, searching within it gets slower and less precise, and very large sets exceed what the tool can hold. The signal to move to a retrieval setup is when the bot starts missing answers that are present, or when the content runs to thousands of pages. A vector index fetches only the most relevant chunks per question, so it scales where a flat file set does not. Either way the input is the same per-page Markdown, so growing from one to the other is a change in plumbing, not a re-export.
When a knowledge base is not the right tool
A custom GPT is the right tool when the content is fairly stable and you want answers grounded strictly in your own material. It is the wrong tool when the underlying data changes constantly, like live stock or pricing, where the bot would need a real-time connection rather than a static file set, or when a simple search of the site would serve users better than a chat layer. Match the tool to the job: a knowledge base for explaining stable content, a live integration for data that moves, and ordinary site search where a bot adds nothing.
Writing the GPT’s instructions around the files
Clean files are half the build; the GPT’s instructions are the other half. The knowledge base supplies the facts, and the instructions tell the bot how to use them. Good instructions tell it to answer only from the uploaded files, to say plainly when the files do not cover a question rather than inventing an answer, and to cite the page it drew from. They also set the tone and the boundaries, what it should and should not advise on, so it stays in its lane.
This pairing is what keeps a custom GPT trustworthy. Without the instruction to stick to the files, a model will happily fill gaps with general knowledge that may not match your business, which is how a support bot ends up stating a policy you do not have. With clean files and tight instructions together, the bot becomes a reliable front door to your own content rather than a confident guesser. The files decide what it knows; the instructions decide how it behaves.
Testing the bot before you rely on it
A knowledge base needs testing the way any product does, against the questions real people ask. Collect the actual questions your customers or readers raise, run them past the bot, and check three things: it found the right page, it answered accurately, and it admitted when something was not covered. The misses are the useful part, because they point at a page that is missing, mislabelled, or too long to retrieve cleanly, each of which is a fixable file problem.
Re-test after every refresh, since replacing files can change what the bot retrieves. A short, repeatable set of test questions turns “it seems to work” into evidence that it does, and it catches the case where a confident answer is quietly wrong. A bot you have tested against real questions is one you can put in front of customers; one you have only spot-checked is a risk.
The export is also migration groundwork
The clean, per-page set that powers a custom GPT is the same inventory a clean migration runs on, so the work serves two ends. Rebuild from the exported content, keep the slugs, and map every old URL to a 301, and your live site and your knowledge base stay sourced from the same clean content while the site loads faster and ranks the same. The structure that makes a page easy for your GPT to retrieve is also what makes it easy for public AI engines to cite, the link drawn in getting your website cited by AI search. If exporting and cleaning a whole site by hand is more than you want to take on, WPBuildAI produces the per-page export and the redirect map together; send your URL for a fixed quote.
Why citations matter for a knowledge-base bot
A custom GPT that cites its source is more useful and more trustworthy than one that does not, which is why keeping the URL in each file pays off at answer time. When the bot says where an answer came from, the reader can open the page and confirm it, and your team can spot when an answer traces to a page that needs updating. A bot that answers without attribution gives you no way to check it and no way to debug it when it is wrong. Citations turn the knowledge base from a black box into something you can audit.
This is the same property that helps your content in public AI search: a page clear enough for your own bot to quote and attribute is clear enough for an external engine to do the same. Building the knowledge base well, focused files, clean structure, URLs intact, is not a separate skill from being citable on the open web; it is the same discipline pointed inward.
Key points to remember
A custom GPT is only as accurate as its knowledge files, so the work is the export: clean Markdown, one file per page, headings kept, the URL noted, long pages split. Skip the live URL, which retrieves poorly, and upload files you control. Refresh the set when pages change, and match the tool to the job rather than forcing a bot onto data that moves. The same export feeds a custom GPT, a retrieval setup, and a clean migration, so it is worth doing once and reusing.
Not affiliated with OpenAI, WordPress, or Lovable.