# What Is a Crawl-First Migration? The Method Explained

> By Lawrence Arya, Founder, WPBuildAI. Published 2026-08-12. 10 min read. Guides.
> Source: https://wpbuildai.com/blog/what-is-a-crawl-first-migration/
> Language: en

You cannot preserve what you never measured, and exports do not measure the site.

**TL;DR.** A crawl-first migration starts by crawling the live site and recording everything reachable, URLs, content, metadata, images, links, schema, before the new platform is touched, then treats that inventory as the contract: the rebuild carries it, the redirect map is generated from it, launch is graded against it. It exists because export-and-memory migrations lose the long tail silently, discovered by Google weeks later, while crawl-first failures are loud, pre-launch checklist items. Scan before build is the whole method, and every WPBuildAI migration runs in that order.

A crawl-first migration is a platform move that begins by crawling the live site and recording everything a visitor or search engine can reach, every URL, page, image, title, meta description, heading, internal link, and schema block, before anyone touches the new platform. That inventory becomes the contract for the whole project: the new site is built to satisfy it, the redirect map is generated from it, and launch is graded against it. The method exists because the alternative, migrating from exports and memory, reliably loses the things nobody remembered, and it is the method WPBuildAI is built around: our migrations start with the scan, not the design, because you cannot preserve what you never measured. What the crawl actually captures, why export-first migrations lose rankings, and how the inventory drives each later stage.

## The problem crawl-first solves

Every migration is secretly two projects: building the new site, which everyone plans, and preserving the old site's accumulated equity, which everyone assumes. The equity is larger and stranger than assumed: years of URLs that earned links and rankings, titles and descriptions tuned by iteration, images embedded across the web, internal link patterns that tell crawlers what matters, and schema feeding rich results. None of this lives in one exportable place. A WordPress database export carries posts and pages but not the rendered reality, not the plugin-generated URLs, not the redirect history, not what the site actually serves; [the export tool itself](https://wordpress.org/documentation/article/tools-export-screen/) is explicit about carrying content, not the site. Memory carries even less, and it carries it with survivor bias, the pages the team remembers are the ones that were never at risk.

So the migrations that go wrong go wrong invisibly: the new site launches beautiful and complete-looking, and the losses surface weeks later as [Search Console](https://search.google.com/search-console/about) 404s and sliding queries on pages nobody thought about. Crawl-first is the discipline of making the invisible inventory visible before it is at stake, which converts preservation from an assumption into a checklist.

## What the crawl actually records

The scan walks the live site the way a search engine does, from the outside, following links, and records per URL: the full address and its status; the title, meta description, and canonical; the heading structure; the body content; every image with its address and alt text; the outbound and internal links; and any structured data the page emits. Alongside the per-page rows, it derives the site-level facts that matter later: the complete URL tree, the internal-link graph showing which pages the site itself treats as important, the redirect rules already in flight, and the orphaned corners reachable only from old links. Because it reads the rendered site rather than the database, it captures what plugins and themes actually output, which is regularly different from what the CMS thinks it contains, and it works even when admin access is missing, the [same front-door approach that rescues locked-out and hacked sites](https://developer.wordpress.org/rest-api/) applies, since the public site is the one source that cannot be withheld.

The output is unglamorous and decisive: a structured inventory, rows and columns, that the rest of the migration consumes as its source of truth, and that survives the project as the site's permanent baseline document.

## How the inventory drives every later stage

| Stage | What the inventory provides | What goes wrong without it | Verdict |
| --- | --- | --- | --- |
| Scoping and quote | Real page, image, and template counts | Estimates from memory, surprises | Honesty starts here |
| Rebuild | Content, metadata, structure per page | Retyped titles, thinner pages | Parity by construction |
| URL strategy | The full tree to preserve or map | URLs invented fresh, equity orphaned | Preserve by default |
| Redirect map | Old-to-new rows generated, not guessed | Wildcards, missing long tail | The load-bearing file |
| Launch check | A test case per recorded URL | Spot-checks that miss failures | Mechanical pass-fail |
| Monitoring | The baseline to diff against | 404s discovered by Google first | Diffs beat vigilance |

Two rows carry most of the value. The redirect map is generated by joining the inventory to the new structure: every old URL either keeps its address or gets an explicit row to its successor, and the long tail, parameter URLs, images, PDFs, is in the map because it was in the crawl, which is precisely the tail [that bulk redirect testing exists to verify](/blog/how-to-test-301-redirects-after-a-migration/). And the launch check inverts the usual quality process: instead of asking "does the new site look right," the grade is "does every inventory row resolve correctly," a mechanical pass-fail that [Google's site-move guidance](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes) essentially describes, run before the switch rather than diagnosed after it.

## Crawl-first versus export-first, honestly

Export-first migration, database dump in, importer out, is faster to start and fine for one narrow case: a like-for-like move between identical platforms with URLs unchanged, where the database genuinely is the site. It fails proportionally to how much the platforms differ, which is why it fails hardest on exactly the moves people most want, WordPress to a modern stack: the export drops plugin-rendered content and shortcode output, carries none of the metadata plugins managed, knows nothing about URLs beyond permalinks, and imports into a system whose structure differs enough that everything lands subtly displaced. The crawl-first cost is the scan and mapping stage up front, hours, not weeks, and the payoff is that the rebuild starts from the site's rendered truth rather than its internal storage, with the preservation contract already written.

The two methods also fail differently, which matters for risk: export-first failures are silent and discovered by Google, crawl-first failures are loud and discovered by the checklist, because an unmet inventory row is visible before launch. Silent versus loud is the entire risk argument, and it is why the [migration checklist](/blog/website-migration-checklist-before-you-switch-platforms/) we publish puts the crawl at step one rather than as a verification afterthought.

## What crawl-first looks like in a real project

In a WPBuildAI engagement the sequence is fixed. The scan runs first, before design conversations, producing the inventory and an honest scope: real page counts, template patterns, problem areas like orphaned content or redirect chains already in flight. The rebuild then consumes the inventory, every page's content, metadata, and structure carried into the new build, which is [how a WordPress site moves to a modern platform](/blog/how-to-migrate-a-wordpress-site-to-lovable/) without the retyping stage where equity historically evaporated. The URL strategy defaults to preservation, identical addresses wherever the new platform allows, explicit mapped rows where it does not, and the redirect file is generated from that join rather than written by hand. Launch runs the mechanical check, every inventory URL requested and graded, and the first weeks run on diffs: Search Console watched against the baseline, new 404s converted to redirect rows the week they appear.

The same discipline scales down to a do-it-yourself move: a desktop crawler, a spreadsheet, and the sequence above cover a small site in a weekend. The method is not proprietary; the value we add is running it at production grade with the rebuild attached. What is non-negotiable in either version is the order, scan before build, because the inventory taken after the old site is gone is a reconstruction, and reconstructions are exactly as complete as the memory they came from.

## The limits, stated plainly

Crawl-first records the public site, which defines both its power and its edges. Content behind logins, unpublished drafts, and form-handling logic live in the application, not the pages, so they need the database export as a supplement, the two methods compose rather than compete, with the crawl as source of truth for everything public and the export filling the private gaps. Dynamic functionality, search, carts, member areas, is rebuilt, not crawled, and scoping that rebuild honestly is part of the project rather than a surprise. And a crawl records the site as it is, including its problems: thin pages, duplicate titles, and orphaned sections come along unless the inventory review deliberately retires them, which is the right moment for that pruning, with each retirement becoming a deliberate 410 or a mapped consolidation instead of an accident discovered later in a ranking report.

None of these limits argues for skipping the crawl; they argue for knowing what it is: the complete record of everything the outside world can see and rank, which is precisely the everything a migration is trying not to lose, and the one record no export, backup, or memory can substitute for.

## Key takeaways: crawl-first migration

Crawl-first means the migration starts by recording the live site's rendered reality, every URL, page, image, metadata field, link, and schema block, and treating that inventory as the contract the new site must satisfy. It exists because export-first moves lose what exports do not carry and memory does not hold, silently, discovered by Google weeks later; the crawl converts those losses into loud, pre-launch checklist failures instead. The inventory then drives everything: honest scoping, a rebuild with metadata parity, a generated redirect map covering the long tail, a mechanical launch check, and a monitoring baseline. That order, scan before build, is the whole method, and it is the order every WPBuildAI migration runs in, because you cannot preserve what you never measured.

## Quick answers

### What is a crawl-first migration?

A platform move that begins by crawling the live site and recording everything reachable, every URL, page, image, title, meta description, heading, internal link, and schema block, before the new site is touched. The inventory becomes the project's contract: the rebuild carries its content and metadata, the redirect map is generated from it, launch is graded against it, and monitoring diffs against it. It is the method WPBuildAI runs on every migration, because preservation only works as a checklist, never as an assumption.

### Why not just export the WordPress database instead?

Because the export carries the content, not the site: plugin-rendered output, shortcode results, managed metadata, the real URL tree, and redirect history are all outside it, and they are where rankings live. Export-first failures are also silent, discovered as Search Console 404s weeks after launch, while crawl-first failures are loud, visible as unmet inventory rows before the switch. The methods compose: the crawl is the source of truth for everything public, the export supplements it for drafts and private data.

### What does the pre-migration crawl actually capture?

Per URL: address, status, title, meta description, canonical, headings, body content, images with alt text, internal and outbound links, and emitted structured data. Site-wide: the complete URL tree, the internal-link graph, existing redirects, and orphaned corners. Because it reads the rendered site from the outside, it records what plugins and themes actually serve rather than what the database contains, and it works even without admin access, which is why the same approach rescues locked-out and hacked sites.

### Can I run a crawl-first migration myself?

Yes, at small scale: a desktop crawler builds the inventory, a spreadsheet joins it to the new structure for the redirect map, and the launch check reruns the crawler against the list, a weekend's work for a modest site, with the non-negotiable being the order, scan before build. What a done-for-you service like WPBuildAI adds is production grade: the same method with the rebuild attached, metadata parity handled in the build rather than by hand, and the mechanical checks run as a matter of process.

### When is crawl-first overkill?

Almost never for a site with any search equity, but the honest edge cases exist: a brand-new site with nothing indexed has no inventory worth preserving, and a like-for-like host move with URLs and platform unchanged needs only the standard backup discipline. The moment URLs, structure, or platform change on a site that ranks for anything, the crawl pays for itself in the first prevented 404 cluster, and skipping it converts known, preventable losses into surprises with your traffic attached.

## Sources

- [Tools Export Screen (WordPress documentation)](https://wordpress.org/documentation/article/tools-export-screen/)
- [Site moves with URL changes (Google Search Central)](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes)
- [WordPress REST API Handbook](https://developer.wordpress.org/rest-api/)
- [Google Search Console](https://search.google.com/search-console/about)

---
*Published by [WPBuildAI](https://wpbuildai.com/). Free to read, index, quote and cite with attribution and a link.*