Web Scraping API

Turn any website into clean, LLM-ready Markdown, and its YouTube, TikTok and Instagram videos into transcripts, with one API key.

✔️ One GET request returns Markdown, title and links

✔️ Map a whole site or crawl up to 5,000 pages per job

✔️ 1 credit per page. Python & JS SDKs, MCP, n8n

Get 100 Free RequestsNo Credit Card Required

View Documentation

Trusted by top teams

curl 'https://api.supadata.ai/v1/web/scrape?url=https://supadata.ai/blog/how-to-get-youtube-transcript' \
  -H 'x-api-key: {your_api_key}'

{
  "url": "https://supadata.ai/blog/how-to-get-youtube-transcript",
  "content": "# How to Get a YouTube Transcript (5 Easy Methods in 2026)\n\nWhether you're building an AI app, doing research, creating blog content from videos, or just want to read along — knowing how to get a YouTube transcript is one of those skills that saves hours.\n\n## Why You Might Need a YouTube Transcript\n\n* **Content repurposing** — Turn a YouTube video into a blog post, newsletter, or social thread\n* **AI apps** — Feed video content into LLMs for summarization, Q&A, or analysis\n\n...",
  "name": "How to Get a YouTube Transcript (5 Easy Methods in 2026)",
  "description": "Learn 5 ways to get a YouTube transcript: built-in YouTube UI, free online tools, Python, browser extensions, and the Supadata API.",
  "ogUrl": "https://supadata.ai/opengraph-image.png",
  "countCharacters": 9693,
  "urls": [
    "https://supadata.ai/youtube-transcript",
    "https://supadata.ai/youtube-transcript-api",
    "https://supadata.ai/pricing",
    "..."
  ]
}

curl 'https://api.supadata.ai/v1/web/scrape?url=<PAGE_URL>' \
  -H 'x-api-key: YOUR_KEY'

Scrape, Map, Crawl: Three Endpoints, One Key

Supadata’s web scraping API has three endpoints: scrape for one page, map for every URL on a site, and crawl for the content of a whole site. All of them return clean Markdown or plain URL lists, never raw HTML.

Web + Video: The Part Other Scrapers Skip

Web scrapers stop at the text. A product page with a demo video, a docs site with YouTube tutorials or a creator’s landing page with TikToks holds most of its information in the videos, and an HTML scraper can’t read them.

With Supadata, the same API key that scrapes the page also gets the videos:

• YouTube transcripts, with AI transcription when a video has no captions
• TikTok, Instagram, X and Facebook video transcripts
• Structured data from any video via a prompt or JSON schema

One account, one bill, one SDK for both the web and video sides of your AI pipeline. Get your free API key →

Built for Developers, by Developers

Nobody likes bloated APIs and unnecessary complexity. That’s why we’ve built our service to be:

Use Cases

Supadata vs. Other Web Scraping APIs

The best web scraping API depends on what you scrape. Here is how Supadata compares with popular options for LLM-ready output (as of October 2026):

”—” means not offered as a built-in endpoint on the vendor’s site.

Honest take: if you scrape only web pages at very high volume, Firecrawl is cheaper per page on its larger plans. If you fight heavy anti-bot protection, ScrapingBee is built around proxies. For converting the odd single URL with no sign-up, Jina Reader is fine. Choose Supadata when your app needs both web pages and video content, with one key and one bill. See the full Firecrawl alternative comparison.

Try it free → — 100 requests/month, no credit card.

Simple, Credit-Based Pricing

1 page = 1 credit, whether you scrape it on its own or as part of a crawl. A site map is 1 credit. Start free with 100 requests a month; paid plans scale to 1,000,000 credits.

View Pricing

Code Examples

Scrape a page or map a site in one call. No browser to run, no proxies to manage, no HTML to clean up.

# Scrape one page to Markdown
curl 'https://api.supadata.ai/v1/web/scrape?url=https://supadata.ai/pricing' \
  -H 'x-api-key: YOUR_API_KEY'
 
# List every URL on a site
curl 'https://api.supadata.ai/v1/web/map?url=https://supadata.ai' \
  -H 'x-api-key: YOUR_API_KEY'

from supadata import Supadata
 
supadata = Supadata(api_key="YOUR_API_KEY")
 
# Scrape one page to Markdown
page = supadata.web.scrape("https://supadata.ai/pricing")
print(page.name)
print(page.content)
 
# List every URL on a site
site_map = supadata.web.map("https://supadata.ai")
print(f"Found {len(site_map.urls)} URLs")

import { Supadata } from '@supadata/js';
 
const supadata = new Supadata({ apiKey: 'YOUR_API_KEY' });
 
// Scrape one page to Markdown
const page = await supadata.web.scrape('https://supadata.ai/pricing');
console.log(page.content);
 
// List every URL on a site
const siteMap = await supadata.web.map('https://supadata.ai');
console.log(`Found ${siteMap.urls.length} URLs`);

Scrape response:

{
  "url": "https://supadata.ai/pricing",
  "content": "# Supadata Pricing\n\n| Plan | Credits/month | Price | ...",
  "name": "Supadata Pricing — Free Tier, Plans from $5/mo",
  "description": "Supadata pricing: 100 free credits a month, paid plans from $5/month...",
  "ogUrl": "https://supadata.ai/opengraph-image.png",
  "countCharacters": 5520,
  "urls": ["https://supadata.ai/", "https://dash.supadata.ai/", "..."]
}

Crawl a whole site. Crawling runs as a job: start it, then poll until status is completed. Polling is free.

# 1. Start the crawl — returns a jobId
curl -X POST 'https://api.supadata.ai/v1/web/crawl' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://supadata.ai/blog", "limit": 100 }'
 
# 2. Poll for results; follow "next" while it is present
curl 'https://api.supadata.ai/v1/web/crawl/JOB_ID' \
  -H 'x-api-key: YOUR_API_KEY'

{
  "status": "completed",
  "pages": [
    { "url": "https://supadata.ai/blog/...", "name": "...", "content": "# ...", "countCharacters": 9693 }
  ]
}

FAQs

What does the Web Scraping API return?

Clean Markdown of the page plus its title, description, Open Graph image, character count and the list of links found on the page. Pass noLinks=true to strip Markdown links and keep only the link text.

Does it work on JavaScript-heavy websites?

Yes. Pages are rendered before extraction, so content that single-page apps load with JavaScript ends up in the Markdown.

How does crawling work and what does it cost?

POST /v1/web/crawl with a URL and an optional page limit (default 100, max 5,000) returns a job ID. Polling the job is free and every crawled page costs 1 credit. The crawler follows links below the starting URL. Crawl is available on the Pro plan and above.

How do I get every URL on a website?

Call GET /v1/web/map with the site URL. It returns the URLs found on the site for 1 credit, so you can pick which pages to scrape.

Can it get the content of videos embedded on a page?

Yes, with the same API key. Send a YouTube, TikTok, Instagram, X or Facebook video URL to /v1/transcript for its transcript, or to /v1/extract for structured data from a prompt.

Can AI agents use it?

Yes. The Supadata MCP server gives Claude, Cursor and other MCP clients scrape, map and crawl tools. The same endpoints are available in Make, Zapier, n8n and Active Pieces.

Is web scraping legal?

Scraping publicly available pages is generally allowed, but it depends on your jurisdiction, the site’s terms and how you use the data. Respect robots.txt and each site’s terms of service, and don’t collect personal data without a legal basis.

Web Scraping API for AI: A 2026 Guide

A web scraping API fetches a web page for you and returns its content in a format your code can use. For AI applications, that format is increasingly Markdown rather than HTML. This guide covers how scraping APIs work, why Markdown matters for LLMs, and when to scrape, map or crawl.

What Is a Web Scraping API?

A web scraping API is a hosted service that loads a URL, renders it like a browser, and returns the useful content through a simple HTTP request. It replaces the stack you would otherwise run yourself: headless browsers, retry logic, HTML parsers and the cleanup code that strips navigation, ads and cookie banners. You send a URL and an API key; you get structured content back.

Why Markdown Instead of HTML for LLMs?

Markdown keeps the structure an LLM needs (headings, lists, tables, links) and drops everything it doesn’t (scripts, styles, layout divs and tracking markup). A typical page is several times smaller as Markdown than as raw HTML, so it costs fewer tokens, fits more context into a prompt and splits cleanly into chunks for retrieval-augmented generation (RAG). That’s why AI-focused scrapers like Supadata, Firecrawl and Jina Reader all return Markdown by default.

Scrape vs. Map vs. Crawl

• Scrape when you already know the URL: a link a user pasted, a product page, an article. One request, one page, 1 credit.
• Map when you need to know what exists on a site before reading it. It returns the URL list for 1 credit, so you can filter by path (/docs/, /blog/) and scrape only what matters.
• Crawl when you want the content of a whole section or site, for example to index a documentation site into a vector store. The crawler follows links below the starting URL and returns Markdown for every page.

Do You Need JavaScript Rendering?

Many modern sites are single-page apps built with React, Vue or similar frameworks, and their HTML is nearly empty until JavaScript runs. A plain HTTP fetch returns that empty shell. Supadata renders the page before extracting content, so these sites return the same Markdown a visitor would see.

Scraping Pages That Contain Video

Product demos, tutorials, webinars and creator content increasingly live in embedded videos. A page scraper returns the surrounding text and the video link, but not what’s said in the video. To capture it, send the video URL to a transcript API. With Supadata the same key covers both, so a pipeline can scrape a page, find the YouTube or TikTok links in urls, and fetch their transcripts in the next step.

Web Scraping Best Practices

• Respect robots.txt and each site’s terms of service.
• Scrape only the pages you need; map first, then scrape selectively.
• Cache results for pages that rarely change.
• Avoid collecting personal data unless you have a legal basis for it.

Other Supadata APIs

The Web Scraping API shares its key, credits and SDKs with every other Supadata endpoint:

• YouTube Transcript API — timestamped transcripts for any YouTube video
• TikTok Transcript API — captions and AI transcripts from TikTok
• Instagram Transcript API — transcribe Reels and video posts
• AI video analysis API — structured JSON from any video
• Free Website to Markdown converter — try a single page in the browser
• Free Website to Text converter — plain text from any page

Ready to build?

Get your API key in 30 seconds. No credit card required. Start Building Now →