Your robots.txt file does not carry over on its own when you migrate, because it lives at the domain root and is served by the platform, so the new site needs its own copy, and the single biggest risk is launching with a leftover staging rule that blocks everything. A staging or preview site usually carries Disallow: / to keep it out of search, and if that file ships to production, you tell every crawler to stay out and your site can drop from the index within days. So the job is to publish a correct production robots.txt on the new site: allow crawling of the pages that should be public, keep any genuinely private paths disallowed, and point the Sitemap line at your new sitemap URL. Old WordPress-specific rules like Disallow: /wp-admin/ simply become harmless once WordPress is gone. WPBuildAI ships a correct production robots.txt that allows crawling and references your sitemap, and never carries a staging block to launch.

robots.txt is a file on the domain, not part of your content

The thing to understand is where robots.txt lives and what serves it. It is a plain text file at the root of your domain, at yourdomain.com/robots.txt, and it is produced by whatever platform is running the site, whether that was WordPress or the new build. That means it is not part of the content that a migration moves; it is configuration that the new platform is responsible for. So you do not inherit it, you author it, and treating it as something to write deliberately for the new site is what keeps it correct. Google’s own introduction to robots.txt is the reference for what the directives mean.

The one mistake that deindexes a site

If there is a single robots.txt error to fear in a migration, it is shipping a staging block to production. While the new site is being built, it usually lives on a staging or preview URL, and the correct thing there is Disallow: / so search engines do not index the half-finished site. The danger is that this file, or that mindset, follows the site to launch, so the production robots.txt says Disallow: / and quietly tells every crawler to stay out. The pages then start dropping from the index over the following days, which looks like a catastrophic ranking loss but is really one line in one file. This is a leading cause of a site going deindexed after a migration, and it is entirely avoidable by checking the production file at launch.

What a correct production robots.txt looks like

A good robots.txt after migration is short and intentional. It allows crawling of the pages that are meant to be public, which for most sites means not disallowing anything important. It disallows only the paths that genuinely should not be crawled, such as private or duplicate areas. And it includes a Sitemap line pointing at your new sitemap, so crawlers can find the full URL list quickly, which matters because submitting a fresh sitemap speeds rediscovery of the new structure. Long legacy files full of WordPress-specific rules are not worth copying; once WordPress is gone, a rule like Disallow: /wp-admin/ refers to a path that no longer exists, so it is clutter rather than protection.

Point the Sitemap line at the new sitemap

The Sitemap directive in robots.txt is easy to overlook and worth getting right. It tells crawlers where your sitemap lives, and after a migration the sitemap URL or its contents usually change, so the reference has to be updated to match. A robots.txt that still points at the old WordPress sitemap path sends crawlers to a file that may no longer exist, which slows discovery of your new URLs. Since a clean migration is fundamentally a careful site move, and traffic concentrates on a minority of pages that Google needs to find quickly, as the Ahrefs study shows, pointing robots.txt at the current sitemap helps those pages get recrawled sooner.

Do not silently change AI crawler access

There is a deliberate decision hiding in robots.txt: whether to allow the crawlers used by AI search engines. This is a choice about visibility, not a migration setting, so the move itself should not flip it by accident in either direction. If being cited by AI answers matters to you, make sure the new robots.txt does not block those crawlers; if you want them out, add the rules on purpose. The point is to decide it consciously, as covered in whether to block AI crawlers, rather than discover after launch that a default file quietly changed who can read your site.

Steps to handle robots.txt in a migration

  1. Treat robots.txt as a file to author for the new platform, not inherit.
  2. Never ship the staging Disallow: / to production; that alone can deindex you.
  3. Allow crawling of public pages; disallow only genuinely private paths.
  4. Update the Sitemap line to point at the new sitemap URL.
  5. Set AI crawler access deliberately to match your visibility intent.
  6. Check yourdomain.com/robots.txt at launch and confirm it allows crawling.

Worked example: catching a block before it cost rankings

Consider a business that rebuilt its site on a staging URL, where robots.txt correctly said Disallow: / so the unfinished site stayed out of search. At launch, the team followed a checklist: the very first post-launch check was to open yourdomain.com/robots.txt on the live domain. It still said Disallow: /, carried over from staging. Because they caught it in the first hour rather than the first month, they replaced it with a production file that allowed crawling and pointed at the new sitemap, then requested recrawling. No meaningful deindexing happened, and the rankings held. The whole incident was a single line, found because someone deliberately looked.

Limitation: robots.txt controls crawling, not indexing directly

It is honest to bound this. robots.txt tells crawlers what they may fetch, but it is not a precise indexing control: a URL that is blocked can still appear in results without content if it is linked elsewhere, and a page you truly want out of the index is better handled with a noindex directive on the page than with a Disallow. So use robots.txt to manage crawl access and point to your sitemap, and use page-level noindex for pages that must not be indexed. Getting robots.txt right prevents the catastrophic accidental block; it does not replace deliberate indexing decisions on individual pages.

Common mistakes

  • Shipping the staging Disallow: / to production and quietly deindexing the site.
  • Assuming robots.txt migrates with the content, so the new site launches without one.
  • Leaving the Sitemap line pointing at the old, now-missing WordPress sitemap.
  • Copying a long legacy file full of WordPress paths that no longer exist.
  • Accidentally blocking or unblocking AI crawlers instead of deciding it on purpose.

Key points

Your robots.txt does not carry over in a migration, because it lives at the domain root and is served by the platform, so you author a fresh one for the new site. The one mistake that can deindex you is shipping a staging Disallow: / to production, so checking yourdomain.com/robots.txt at launch is the single most important step. A correct file allows crawling of public pages, disallows only genuinely private paths, and points its Sitemap line at your new sitemap; old WordPress-specific rules just fall away. Decide AI crawler access deliberately rather than letting the move change it, and remember that robots.txt controls crawling, not indexing, so use page-level noindex where you truly need pages out. WPBuildAI ships a correct production robots.txt that allows crawling and references your sitemap, and never carries a staging block to launch. Send your web address for a free analysis.

Not affiliated with WordPress, Lovable, Webflow, Shopify, Wix, or Squarespace.