# What is the best tool to map a website's structure and hierarchy?

> The best approach is a crawl that follows internal links and the XML sitemap, then exports a flat list of every URL with its depth, parent, and internal links, reconciled against your analytics so orphaned pages are caught. That inventory is what you plan a rebuild and a redirect map from, and its value is completeness, not the tool. WPBuildAI runs that crawl as the first step of a migration, producing a complete URL and structure map so nothing is left behind when the site moves.

Source: https://wpbuildai.com/best-tool-to-scrape-website-structure-hierarchy/
By lawrence-arya · 2026-06-06

---
The best way to map a website's structure is a crawl that follows every internal link and the XML sitemap, then exports a flat list of each URL with its depth, its parent, and its internal links, reconciled against your analytics so orphaned pages are not lost. That inventory is the thing you actually plan a rebuild and a redirect map from, and no migration should start without it. The right answer is less about which tool you use and more about whether the map is complete, because the pages you miss are the ones that break the move. WPBuildAI runs exactly that crawl as the first step of a migration, so the structure is fully mapped before a single page is rebuilt.

Map what earns visibility, not just what exists: since [Google added Search Generative AI performance reports to Search Console on 3 June 2026](https://ppc.land/google-finally-gives-search-console-its-own-generative-ai-visibility-reports/) you can see which specific URLs surface inside AI Overviews and AI Mode, which is a sharper guide to what must survive the move than the sitemap alone.

## Why the inventory comes first

You cannot redirect or rebuild a page you did not know existed, and most sites have more pages than their owners think. Ahrefs analyzed a large sample of indexed pages and found in its [search traffic study](https://ahrefs.com/blog/search-traffic-study/) that the majority get little or no organic traffic, while a small set quietly carries most of it. A complete map is how you tell those two groups apart before you move, so the pages earning traffic are preserved and the dead weight is pruned on purpose rather than by accident. Starting the rebuild before the inventory is the single most common way a migration loses traffic, because the decisions get made by omission.

## What a good structure map captures

A useful map records more than a list of addresses. For each page it should hold:

- The **URL**, discovered from links and the sitemap together.
- The page's **depth and parent**, so the hierarchy is visible.
- The **internal links** in and out, so you can see what holds the site together.
- A flag for **orphans**, the pages nothing links to, surfaced from the sitemap and analytics.
- A measure of **value**, the traffic or revenue, so you know which pages matter most.

With those columns, the map stops being a spreadsheet of URLs and becomes a plan: you can see the shape of the site, which pages carry the business, and what would break if a given page disappeared.

## Why a crawl alone misses pages

A link crawl only finds what something links to, so it is blind to orphans by design. The 2024 [Web Almanac](https://almanac.httparchive.org/en/2024/) shows how large and tangled real sites become over time, with deep templates, tag and archive pages, and generated URLs that owners rarely see and the menu never shows. Those are exactly the pages a crawl-only approach drops. Combining the link crawl with the XML sitemap and an analytics or Search Console export is what catches them, because a page can be invisible to the navigation yet still indexed and still earning visits. Completeness comes from the union of sources, not from any single crawl.

## Prioritise by value before you plan

Not every page deserves equal attention. Sorting the map by traffic and revenue tells you which redirects to verify by hand and which pages to rebuild first. Because a small share of pages usually carries most of the value, getting those exactly right protects most of the business, while the long tail can be handled in bulk. This prioritisation also informs pruning: a page that earns nothing and holds no backlinks may be better consolidated into a stronger page than carried forward, which reduces clutter and the number of redirects you have to manage.

## From map to redirect plan

The structure map is not just documentation, it is the redirect plan in waiting. Google's [site move guidance](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes) treats a complete old-to-new URL mapping as the core of a safe move, and that mapping is simply your structure map with a new-URL column added. Map each old URL to its closest new page, keep slugs where you can, and mark genuinely retired pages for a real 404 or 410. The spreadsheet form of this is the [URL mapping template](/website-url-mapping-template-excel-automated/), and the discovery step is detailed in [finding all URLs for 301 redirects](/find-extract-all-urls-website-301-redirects/).

## The internal-link graph is part of the structure

A site's structure is not only its pages; it is also how they link to each other. The internal-link graph routes authority and helps engines discover and rank pages, and Backlinko's analysis of [11.8 million search results](https://backlinko.com/search-engine-ranking) found internal linking and structure correlate with ranking. A migration that preserves the pages but flattens the link graph weakens the site even though every page still exists. So a good structure map records the links, not just the URLs, and the rebuild rebuilds the graph, pointing internal links at the new addresses rather than relying on redirects to catch them.

## Tools versus a managed crawl

Dedicated crawler tools are genuinely useful and will produce a solid URL list, but a tool reports only what it can reach, so the completeness still depends on you feeding it the sitemap and reconciling against analytics. For a migration there is a second requirement: the map has to flow into a redirect plan and a rebuild, not sit as a static export. That is the difference between a raw crawl and a managed one. WPBuildAI runs the crawl, reconciles the sources, and carries the map straight into the redirect plan and the rebuild, so the inventory does the job it was made for rather than becoming a file nobody acts on.

## A worked example

Picture a ten-year-old WordPress site the owner believes has about eighty pages. A link crawl finds ninety. Adding the sitemap surfaces another forty tag and archive pages. The analytics export then reveals a dozen old landing pages, linked from nowhere, that still pull steady traffic from old campaigns and backlinks. The real number is closer to a hundred and forty, and the forty the owner never knew about include some of the most valuable. Migrating from the owner's mental list of eighty would have silently dropped them; migrating from the complete map preserves them. That gap, between what you remember and what exists, is the entire reason the inventory comes first.

## Common mistakes that leave gaps

A few errors recur. Trusting the menu or a mental list instead of a crawl misses everything not linked from navigation. Running a crawl without the sitemap misses orphans. Skipping the analytics reconciliation misses pages that are linked from nowhere but still earning traffic. And treating the map as a one-time export, rather than the live basis for the redirect plan, means the careful inventory never actually shapes the move. Each mistake has the same effect: a page that mattered is absent from the plan, so it is absent from the redirects, so it 404s after launch.

## What to do with the orphans you find

Finding orphaned pages is only useful if you decide what each one deserves. Sort them by value first. An orphan that still earns traffic or holds backlinks should be preserved and redirected, and ideally re-linked from the navigation on the new site so it is no longer an orphan. An orphan that earns nothing and serves no purpose is a candidate to retire with a clean 404 or 410, which trims clutter and reduces the redirect list. The mistake is to treat every orphan the same: blanket-redirecting them all to the homepage wastes their value and trips soft-404 handling, while deleting them without checking can drop a quietly valuable page. The value column in the map is what turns this from guesswork into a quick, defensible decision.

## How long the mapping takes

For most sites the crawl itself is quick, a matter of minutes to hours depending on size, and the reconciliation against the sitemap and analytics adds a little more. The real time goes into reviewing the result: confirming the orphans, deciding what to keep, and prioritising the high-value pages. That review is worth doing properly, because it is cheaper than discovering a missed page after launch. A small site is an afternoon; a large, old site with thousands of URLs is longer, not because crawling is hard but because each valuable page deserves a deliberate decision.

## Keeping the map accurate during a long migration

On a larger project, the site keeps changing while you work, so the map can drift. New posts get published, pages get edited, and URLs get added between the crawl and the launch. The fix is to re-crawl close to cutover and diff the result against your working map, so anything added since the first crawl is caught. Treating the map as a living document rather than a one-time snapshot prevents the awkward case where a page published during the project launches with no redirect because it was never in the inventory.

## The map is reusable beyond the migration

A complete structure map and the clean content behind it are not single-use. The same inventory that powers a redirect plan can feed a clean content export for AI tools, a knowledge base, or a retrieval dataset, because all of them need the same thing: a complete, clean view of what the site contains. It also gives you an honest picture of your own site, which pages exist, which earn traffic, and how they link together, that is useful for content planning long after the move. Building the map once and reusing it is far less work than rebuilding it each time a new need appears.

## Verifying the map is complete

Before you trust the map, sanity-check it. Compare the page count against your own expectation and against the sitemap, and investigate any large gap. Spot-check a few sections to confirm deep pages were reached. Cross-reference the top pages in analytics against the map to make sure none of your most valuable URLs is missing. A map that matches the sitemap, the analytics, and a manual spot-check is one you can plan from; a map that is suspiciously smaller than the sitemap is a warning that the crawl did not reach everything. The point of verifying is to catch the gap now, while it is a line in a spreadsheet, rather than later, when it is a 404.

## When a full map is more than you need

If you are not migrating and only want to understand a small site, a quick crawl is enough and the full reconciliation is overkill. The complete map earns its effort when the cost of missing a page is high, which is exactly the case for a migration, a replatform, or a large site where orphaned but valuable URLs are likely. For a five-page brochure site you will not move, the lightweight version is fine; for anything you are rebuilding, completeness is worth the extra step.

## Map first, then choose the destination

A useful side effect of mapping before you decide where to move is that the map informs the choice. Seeing how many pages you really have, how they are organised, and which carry the traffic tells you whether the new platform needs a deep content structure, a simple flat one, or something in between. Owners who pick a destination first and map later often discover the new platform fits the site they imagined, not the site they have. The inventory keeps the decision grounded in what actually exists, which is the same reason it keeps the migration grounded: you plan from the real site, not the remembered one.

## Key points to remember

Map a site by crawling its links and sitemap into a flat inventory of every URL, its depth, and its connections, then reconcile against analytics so orphans are not lost. The value is completeness, not the tool, because the pages you miss are the ones that 404 after launch. Record the internal-link graph and each page's value, prioritise the important pages, and turn the map into the redirect plan. WPBuildAI produces that complete map as step one of a migration and carries it into the rebuild, so the move is built on a full picture instead of a guess; send your site URL for a fixed quote.

Not affiliated with WordPress or Lovable.