A web scraping API should turn a URL into a usable result with a failure mode you can understand. The right choice depends on the source pages, required output, freshness, and what a failed request costs. Test those conditions with the same page set before comparing headline feature lists.
url2llm is built around a synchronous, one-page scrape request. It also offers a separate map request for discovering site URLs. That separation makes it clear when you are finding pages and when you are paying to read one.
What should a web scraping API return?
Start with the output your application consumes. A retrieval index may need clean Markdown and source URLs. A monitoring job may need HTML or public product data. A design tool may need screenshots and brand information. Asking for every format on every page increases payload and may force a browser even when text alone was enough.
url2llm currently supports markdown, html, rawHtml, links, screenshot, screenshot@fullPage, assets, branding, and shopify. JSON extraction and summaries are not available yet. Check the API's response against your own schema before committing to a workflow.
The response wraps the page in data. The page object includes metadata.source_url, final_url, status_code, from_cache, and credits_used. Check that the requested output field is present and useful, then retain the metadata beside it. A 200 from the API alone is too weak a quality check.
How do you test page coverage?
Build a small set of pages that represents your real workload:
- A static HTML page with useful content in the response.
- A JavaScript-rendered page where content appears after load.
- A page with a table, links, or product fields you need to preserve.
- A blocked or unavailable page so you can inspect the error and charge.
For each page, record whether the returned content is complete, the time taken, and whether the result is easy to trace back to the source. A successful HTTP status alone does not mean the extracted text is useful.
Use a pass condition tied to your application. For a documentation index, require the page title, one known section heading, and its source URL. For a product monitor, require the correct variant and currency. For a visual audit, compare the screenshot at the device and locale you actually use.
How should the API handle failures and freshness?
Look for a clear distinction between a missing page, a site block, a timeout, and an extraction failure. A 403 may come from the source site's access rules or bot controls. More retries from the same address may only repeat the block. A cached response can be useful for stable documentation and wrong for a changing price.
url2llm accepts max_age for acceptable cache freshness and max_age: 0 for a forced fresh fetch. It supports an Idempotency-Key header on scrape and map calls to prevent duplicate work and billing from repeated accepted requests. A repeated accepted key returns a conflict instead of replaying the old result.
What belongs in the vendor comparison?
| Requirement | Test |
|---|---|
| Content quality | Compare returned text with the visible page |
| JavaScript support | Fetch a page whose content is client-rendered |
| Discovery | Find URLs without scraping each discovered page |
| Billing | Quote the actual formats, retries, and refresh pattern |
| Traceability | Keep URL, status, and fetch time with output |
The pricing guide shows how to normalize costs. If you are evaluating a specific alternative, see the Firecrawl alternatives guide. For an AI ingestion use case, start with web scraping for RAG.
Try the same three public pages in the url2llm playground and inspect the output before choosing a production format and refresh schedule.