How do I clone a website's structure with a crawler?

You clone a website’s structure with a crawler by letting it follow every internal link, then turning the resulting map of URLs and links into an editable blueprint of the site’s information architecture. The crawler does the tedious part: it records what pages exist, how they are organized into sections, and how they link to one another. That reverse-engineers even a messy, decade-old WordPress site, the kind nobody fully remembers, into something concrete a designer or SEO can plan from. The point is not to copy the old design, but to redesign from the real structure rather than from memory: keep the parts that work, fix the parts that do not, and carry the URLs across. WPBuildAI crawls the site and outputs that structure map, so a redesign starts from an accurate picture instead of a guess.

Crawl each host variant separately rather than trusting one seed, since Google’s June 2026 site move update treats every subdomain and both the www and non-www variants as things you have to account for explicitly, and structure that exists only on a legacy variant is invisible to a single-origin crawl.

Structure versus content

It helps to separate two things a crawl can capture. Content is the words, images, and media on each page. Structure is the architecture: which pages exist, how they nest into sections and categories, and how they link together into a navigable whole. Cloning the structure is about the second, producing a map of the skeleton rather than the flesh. You often want both, but for planning a redesign the structure is what you reason about first, because it decides navigation, URL design, and how authority flows through the site. A crawler is the tool that makes the structure visible.

What a crawler reveals about your architecture

A crawler follows links the way a search engine does, and in doing so it exposes the real architecture, not the intended one. It shows the full set of URLs, how deep each page sits from the homepage, which pages are heavily linked and which are barely reachable, and where navigation dead-ends. On an old WordPress site, this almost always surfaces surprises: forgotten sections, duplicate paths, and orphan pages with no links in, the kind covered in finding orphan pages. The 2024 Web Almanac is a reminder that sites accrete more structure than anyone designs, which is exactly what the crawl brings to light.

Turning a crawl into an editable blueprint

A raw crawl is a list; a blueprint is something you can plan with. The step that adds value is organizing the crawl output into an editable map: a hierarchy of sections and pages, with the URL and key signals for each, that a person can rearrange. That might be a spreadsheet, a tree diagram, or a sitemap document, the formats discussed in the best tool to scrape a website’s structure. The aim is a document a UX designer can mark up, deciding what stays, what merges, and what goes, rather than a static dump nobody can act on.

Why map the old structure before a redesign

Redesigning without mapping the old structure is how sections and URLs get lost. Teams rebuild what they remember, which is never the whole site, so pages and even whole categories quietly fail to make the new plan. Mapping first means the redesign decisions are made against the real inventory: every section is accounted for, and nothing is dropped by accident. It also feeds the migration, because the same map of old URLs is what you redirect from, per Google’s site move guidance. Planning from a blueprint turns a redesign from a guess into a deliberate restructuring.

Spotting the problems worth fixing

A structure map is also a diagnosis. It reveals pages buried too many clicks deep, thin sections that should merge, orphan pages that should be relinked or retired, and important pages that lack internal links, which matters because links are a ranking signal, as Backlinko’s analysis found. Traffic is concentrated too, per the Ahrefs study, so the map helps you check that your high-value pages sit prominently in the architecture, not lost in the depths. The redesign then becomes a chance to fix these, guided by evidence rather than opinion.

From blueprint to new information architecture

With the old structure mapped and its problems flagged, you design the new architecture deliberately. Keep the sections and paths that work, merge or cut the ones that do not, and decide the new URL structure, ideally reusing old paths where possible so most pages do not move. Where URLs do change, the old-to-new mapping from the blueprint becomes your redirect map. This is the bridge from analysis to a clean migration, the same discipline as migrating without losing SEO. The blueprint is what lets the new IA be an improvement on the old, not a random reshuffle.

A worked example

A team faced a ten-year-old WordPress site nobody fully understood, ahead of a headless rebuild. A crawl produced a map of 600 URLs, revealing three forgotten subsections, dozens of orphan posts, and a category tree five levels deep in places. They turned the crawl into a tree document, marked up what to keep, merge, and retire, and designed a flatter new architecture from it. Crucially, they reused the old URL paths wherever the new structure allowed, so most pages kept their addresses, and built a redirect map for the rest. The redesign started from the real site, so nothing important was dropped and the URLs were handled deliberately.

Common mistakes

• Redesigning from memory instead of a crawl of the real structure.
• Treating the crawl as a static dump rather than an editable blueprint.
• Ignoring the orphans and dead ends the crawl surfaces.
• Designing a new URL structure without reusing old paths where possible.
• Forgetting that the old structure map is also the redirect map.

Key points to remember

You clone a website’s structure with a crawler by following every internal link and turning the URL-and-link map into an editable blueprint of the information architecture. It reverse-engineers even a messy old site so you can redesign from the real structure, not memory, keeping what works and fixing what does not. The blueprint doubles as a diagnosis (orphans, dead ends, buried pages) and as the redirect map for the migration, since old URLs map to new. Reuse old paths where you can so most pages do not move. WPBuildAI crawls the site and outputs that structure map. Send your site URL for a free structure blueprint.

Not affiliated with WordPress, Lovable, Webflow, Shopify, Wix, or Squarespace.