If your entire site vanished from search after a migration, the cause is almost always that search engines were explicitly told not to index it, and that is good news, because instructions are easy to change. There are two usual culprits. The first is a leftover noindex: staging and development sites are normally set to noindex so they stay private, and that setting gets carried to the live site, or a sitewide noindex is switched on during the move, so every page tells search to drop it. The second is a robots.txt that disallows everything: a new platform can ship a default robots.txt that blocks all crawling, or a staging rule that blocks the whole site gets pushed live. Both produce the same dramatic symptom, a near total disappearance, and both are quick to confirm and quick to fix. Once you remove the noindex or correct the robots.txt and ask search to recrawl, indexing usually returns, because the content was never the problem. WPBuildAI scans your site and returns each page with its index status and crawl rules, so you can find the block fast.
Total disappearance has a small list of causes
The shape of the symptom narrows the diagnosis. A partial drop, where pages still show up but rank lower, has many possible causes. A total disappearance, where the site is simply gone from search, has very few, and they nearly all amount to one thing: search was told not to index the site. That is reassuring, because it means you are looking for an instruction, not investigating a mystery. Content does not vanish from search on its own after a migration; it vanishes when a noindex or a blocking robots.txt tells search to remove or never read it. Recognizing that total deindexing points to an explicit block is the insight that makes the fix fast.
Culprit one: a leftover noindex
The most common cause is a noindex that should not be there. A noindex is a small instruction, usually a robots meta tag in the page head, that tells search engines not to keep the page in their index, as described in the guide to blocking indexing. Staging sites use it deliberately so the unfinished site stays out of search. The danger is that the staging configuration, including its noindex, gets promoted to production, or that a sitewide discourage search engines setting is left on after launch. The result is every page quietly carrying a do not index instruction. Check a few pages source for a noindex tag; if it is there on the live site, that is very likely your answer.
Culprit two: a blocking robots.txt
The second cause sits one layer earlier, at crawling rather than indexing. A robots.txt file tells crawlers what they may fetch, and a rule that disallows everything stops search from reading any page at all, as the introduction to robots.txt explains. New platforms sometimes ship a default robots.txt that blocks crawling until you change it, and staging environments often block everything to stay private. If that file or rule is pushed live, crawlers are turned away at the door. Check your robots.txt directly: if it disallows the whole site, that is the block. Note that a blocking robots.txt and a noindex are different mechanisms, and a site can suffer from both at once, so check each.
Why both come from staging
It is worth seeing the common root, because it tells you where to look. Both culprits usually originate in the staging or development environment, which is intentionally hidden from search using exactly these tools. The migration then copies the environment to production without flipping the switches back, so the privacy settings meant for staging end up protecting your live site from being found at all. This is why pushing staging to live is the classic moment for total deindexing. The fix is not just to remove the block now, but to make turning it off part of your launch routine, so the next migration does not repeat it.
How to confirm which it is
Diagnosis is fast and worth doing precisely before you change anything. Look at the live page source for a robots meta noindex tag, and open your robots.txt to see whether it disallows everything. Search Console will also report pages excluded by noindex or blocked by robots.txt, which confirms the cause at scale rather than page by page. Because Web Almanac 2024 (HTTP Archive) is a reminder that sites have many URLs, a report that shows the same exclusion reason across all of them is a strong signal you have found a sitewide block. Confirm the mechanism first, so you remove the right thing rather than guessing.
How to fix it and recover
The repair matches the cause. If it is a noindex, remove the noindex tag or turn off the discourage search engines setting so pages are indexable again. If it is robots.txt, replace the blocking file with one that allows crawling of your real content. Then request recrawling and resubmit your sitemap so search revisits the pages and restores them. Recovery follows the recrawl schedule rather than a fixed timeline, but it is usually full, because the content and URLs were intact the whole time; only the instruction was wrong. Keep watching Search Console until the excluded count falls and pages return. The relief here is justified: this is one of the few migration problems where the content was never actually damaged.
Steps to diagnose and fix total deindexing
- Confirm the scale: is the whole site gone, or just ranking lower?
- Check page source for a robots meta noindex on the live site.
- Check robots.txt for a rule that disallows everything.
- Use Search Console to see the exclusion reason across pages.
- Remove the block: delete the noindex or fix the robots.txt.
- Request recrawling and resubmit the sitemap, then monitor recovery.
Worked example: a launch that hid the whole site
Imagine a company that migrates to a new platform on a Friday and, by the next week, finds it has vanished from search entirely. The team first confirms the scale: it is not a dip, the site is simply gone. They check a page source and find a robots meta noindex on every page, inherited from the staging configuration that was promoted to production. They also check robots.txt and find it fine, so noindex is the sole cause. They remove the noindex sitewide, request recrawling, and resubmit the sitemap. Over the following days, as search recrawls, the pages return with their previous rankings, because nothing about the content or URLs had changed. The only damage was the time spent disappeared, which is why catching this early matters, unlike the slower diagnosis of a partial SEO drop.
Limitation: this explains total loss, not partial drops
It is important to apply this diagnosis only to the right symptom. A noindex or blocking robots.txt explains a total disappearance, but it does not explain a partial drop where pages still rank, just lower. If your pages are still indexed and merely weaker, the cause lies elsewhere, in missing redirects, lost metadata, thinner content, or broken internal links, and removing a block you do not have will not help. So match the fix to the symptom: total loss points to an indexing block, partial loss points to the migration fundamentals covered in the risks of migrating to a new CMS. Confirming the scale first keeps you from chasing the wrong cause.
Common mistakes
- Promoting the staging configuration, including its noindex, to production.
- Shipping a platform default robots.txt that blocks all crawling.
- Assuming a total disappearance is a penalty rather than a self inflicted block.
- Checking only one mechanism when both noindex and robots.txt can be wrong.
- Removing the block but never requesting a recrawl, so recovery stalls.
Key points
When a whole site is deindexed after a migration, the cause is almost always that search was told not to index it, through a noindex left on from staging or switched on sitewide, or a robots.txt that disallows everything. Both usually originate in the staging environment and get promoted to production, both produce the same total disappearance, and both are quick to confirm in the page source, in robots.txt, and in Search Console. Remove the block, request recrawling, and resubmit the sitemap, and indexing usually returns in full, because the content was never the problem. Match the fix to the symptom: total loss means an indexing block, while a partial drop points to redirects or content, and starts with checking 404 errors in Search Console. WPBuildAI scans your site and returns each page with its index status and crawl rules, so you can find the block fast. Send your web address for a free analysis.
Not affiliated with WordPress, Lovable, Webflow, Shopify, Wix, or Squarespace.