Web scraping for RAG has four jobs: discover allowed pages, extract useful text, keep source identity, and refresh changed content. A scraper only solves the fetch step. A reliable retrieval system also needs versioning, permissions, deletion, and tests that show whether the right passage reached the model.
url2llm supports the first two jobs with /v1/map and /v1/scrape. Map returns URLs from a site's declared sitemaps and page links. Scrape returns one page in a requested format such as Markdown.
How do you build a web scraping pipeline for RAG?
- Define which public hosts and paths belong in the collection. Exclude login pages, search result pages, and duplicate parameter variants.
- Call
POST /v1/mapwith a starting URL. It returnslinks[], including the discovered URL and source. - Select the URLs you actually need, then call
POST /v1/scrapeonce per page withformats: ["markdown"]. - Store each passage with its source URL, title when known, fetch time, and content version.
- Re-fetch changed pages and remove passages for deleted or excluded pages.
- Test retrieval with questions that require a specific page, an exception, or a recent update.
Do not hand every discovered URL to an embedding job. A map result is a candidate list, not a quality judgment. Filter it before spending fetch and index capacity.
For a first pass, use a small allowlist such as /docs/ or /help/. Deduplicate redirected URLs before chunking. When a fetch changes, replace that page's old chunks as one unit so old and new instructions do not appear together in retrieval results.
Why start with sitemaps?
A sitemap is a site's published list of URLs. Google's sitemap guidance describes it as a way to tell crawlers about pages and other files. It can include pages that a home-page link crawl misses. It can also include pages you do not want in a retrieval index, so selection still matters.
url2llm's map endpoint checks sitemap locations declared in robots.txt, tries conventional sitemap locations, and also reads links from the requested page. A map call discovers URLs. It does not scrape every URL it returns. This keeps ingestion under your control.
What should you store with each chunk?
Keep the source URL and a version or fetch timestamp with every chunk. Preserve the heading that explains the passage. If two pages give conflicting rules, retrieval needs enough metadata to prefer the current and authorized source.
A minimal record is {source_url, final_url, fetched_at, heading, text, version}. The scrape metadata supplies source_url and final_url. Your ingestion job supplies fetched_at and version; the API does not give a permanent content version. Store an access scope in your index when documents have different readers.
| Field | What it helps diagnose |
|---|---|
| Source URL | Which page supports the answer |
| Fetch time | Whether the answer may be stale |
| Section heading | Whether a chunk lost its context |
| Content version | Which chunks to replace on refresh |
| Access scope | Which user may receive the passage |
For private data, apply access checks before material reaches the model. OWASP's authorization guidance calls for checking permissions on relevant requests. A public-page scraper does not make a private corpus safe by itself.
How do you know ingestion worked?
Start with a small set of questions whose supporting pages you can name. Check whether retrieval returned the right passage before judging the model's final answer. If the evidence was absent, change discovery, extraction, chunking, or ranking. If it was present and the answer was wrong, investigate answer generation.
The website-to-Markdown guide covers the page format. The JavaScript rendering guide covers pages whose content arrives after the initial HTML.
Create an API key, map one documentation site, choose ten useful URLs, and inspect the Markdown and source metadata before indexing the whole site.