You can extract data from a WordPress site with Python and BeautifulSoup, and for some jobs that is the right tool. But for most migrations, writing a scraping script is solving a problem that already has a no-code answer. WordPress exposes its content as structured data through its REST API, and a crawler or hosted scan can pull every page, its metadata, and its media into JSON, Markdown, or CSV without a line of code. Python earns its place for unusual, repeated, or heavily customised extractions, not for a standard one-time content export. Whatever route you take, the goal is the same: capture the URLs, content, metadata, and media completely. WPBuildAI scans the live site and returns the content as structured files, so you skip the scripting and still get a clean, complete export.

Whichever language you reach for, pin the WordPress version you are reading from: 7.0 shipped on 20 May 2026 with 7.0.1 and a 7.0.2 security release following in July, and REST responses can shift between releases in ways a script written against the old shape will not survive.

The short answer: Python works, but is rarely necessary

The honest answer to “how do I extract WordPress data with Python” is that you can, but you usually should not have to. A one-time content export for a migration is a solved problem with simpler tools, and reaching for Python first often means more setup, more debugging, and more chances to miss fields than the job needs. Python becomes the right choice when the extraction is unusual, when you will run it many times, or when you need custom transformations no off-the-shelf tool offers. For the common case, a standard export of pages and posts, the API or a crawler is faster and less error-prone. Knowing which case you are in saves a lot of wasted effort.

Route 1: the WordPress REST API, no scraping

The cleanest route needs no scraping at all. Most WordPress sites expose content through the WordPress REST API, which returns posts, pages, media, categories, and tags as structured JSON at predictable endpoints. Because the data is already structured, you get clean fields, title, content, slug, date, author, rather than having to parse them out of HTML. You can pull it with any HTTP client, including a browser or a no-code tool, not just a script. The one caveat is that some sites restrict or disable the API, in which case you fall back to crawling. When it is available, it is the most reliable way to get content out without scripting.

Route 2: a crawler or hosted scan, no code

When the API is off or you want the rendered pages, a crawler reads the site the way a browser does and captures what is actually published. A hosted scan takes this further: you give it a URL and it returns the content already converted to Markdown, HTML, JSON, or CSV, with the metadata and media included, the same idea as scraping a WordPress blog to JSON or exporting an entire website to one file. This route needs no installation and no code, and it captures pages regardless of how WordPress built them. For most people migrating a site, this is the path of least resistance, because it turns the whole site into import-ready files in one step.

Route 3: Python with BeautifulSoup, when it makes sense

Python is genuinely useful in specific situations, so it is worth knowing when to choose it. With Requests to fetch pages and BeautifulSoup to parse the HTML, you can extract exactly the elements you specify, which suits unusual page structures, custom fields a generic tool misses, or transformations you want applied during extraction. It also makes sense when the extraction is recurring, where a script you run repeatedly pays back its setup. The trade-off is that you own the code: handling pagination, rate limits, encoding, and the inevitable irregularities in a real site’s markup. For a single migration of standard content, that overhead usually is not worth it; for a repeated or bespoke job, it is.

What to extract, whatever the route

The route changes; the target does not. A complete extraction captures the page titles and body text, the author and publish dates, the categories and tags, the internal links, the featured and inline images with alt text, and the metadata, titles, descriptions, and canonicals. Missing any of these thins the migrated pages. The reason completeness matters so much is that traffic is concentrated: the Ahrefs search traffic study found a small share of pages earns most visits, so a partial extraction that drops the wrong pages or fields costs disproportionately. Treat the extraction as a checklist of fields, not just “get the text.”

Output formats and what they are for

The format you want depends on where the content is going. Markdown suits static site generators and many headless setups, and is readable and portable. JSON suits programmatic imports and framework content collections, where structured fields map directly. CSV suits CMS importers like those in no-code builders, which ingest collections row by row. And content chunked into sections suits feeding a site to an LLM for retrieval or rewriting, the use behind uploading a WordPress blog to an LLM. A good export offers more than one of these, because the same content often needs to go to more than one destination. This is also why the WXR XML export falls short, as why a WordPress XML export fails explains: it is one rigid format that only another WordPress site reads.

Why this matters for a migration

Extraction is the first real step of any migration, and its completeness sets the ceiling for everything after. You cannot redirect a URL you did not capture, and you cannot rebuild a page whose content you missed. The 2024 Web Almanac reflects how much content sits on WordPress, and a typical site has accumulated more URLs and pages than its owner remembers. So the extraction has to be exhaustive, feeding the URL inventory that the redirect map depends on, the map Google’s site move guidance treats as the core of a safe move. A clean, complete export is what makes the rest of the migration possible.

A worked example: a 500-post blog

A 500-post WordPress blog needed extracting for a move to a static framework. The owner had considered writing a BeautifulSoup script, then checked the simpler routes first. The REST API was enabled, so a single pass pulled all 500 posts as JSON with their titles, bodies, dates, categories, and image references, plus the pages. The content was converted to Markdown with front matter, ready for the new framework, and the images were downloaded with alt text. A crawl confirmed the post count matched and caught 30 tag and archive URLs the API list did not foreground, which were added to the redirect plan. No script was written, the export was complete in an afternoon, and the migration proceeded with every post and URL accounted for.

Common mistakes extracting WordPress data

  • Reaching for Python first when the REST API or a crawler would do the job faster.
  • Scraping only body text and dropping titles, descriptions, and alt text.
  • Trusting one source for the URL list and missing tags, archives, and old pages.
  • Exporting to a single rigid format when the content needs to go to several places.
  • Treating the extraction as finished without checking the count against a crawl.

Key points to remember

You can extract WordPress data with Python and BeautifulSoup, but most migrations need no code: the REST API returns structured JSON, and a crawler or hosted scan returns import-ready Markdown, JSON, or CSV without scripting. Reserve Python for unusual, repeated, or customised jobs. Whatever the route, capture every URL, the content, metadata, and media in full, because the extraction sets the ceiling for the whole migration. Offer the output in more than one format so it can go where it needs to. WPBuildAI scans the live site and returns the content as structured files, so you get a clean, complete export without writing or maintaining a script. Send your site URL for a free migration assessment.

Not affiliated with WordPress, Lovable, Webflow, Shopify, Wix, or Squarespace.