A website-to-Markdown workflow fetches a public page, extracts its useful content, and converts it into readable Markdown. For AI applications, keep the source URL and fetch time beside that text. Markdown helps a model read headings, lists, and links, but it does not make a page accurate, current, or safe to trust.
With url2llm, request markdown from POST /v1/scrape. The API reads one URL per call. Use /v1/map first when you need a list of pages.
How do you turn a website into Markdown?
Send a public URL and ask for the Markdown format. The default scrape format is already markdown, but naming it makes the request clear:
curl -X POST https://api.url2llm.com/v1/scrape \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/docs/start","formats":["markdown"]}'
Read data.markdown from the response. Keep the requested URL with it so you can trace an answer back to a page. A scrape is one page, even if that page links to a thousand more.
The response also includes data.metadata.source_url, final_url, status_code, from_cache. Use final_url as the fetch destination when a redirect changed the page, while retaining the original URL your application requested. Check the API's success field before reading data.
| Input | Keep in your record | Why |
|---|---|---|
| Page URL | Original URL and fetch time | Find and refresh the source |
| Markdown | Text and heading structure | Search or provide model context |
| Links | Outbound or internal URLs when requested | Find related pages without guessing |
What gets lost in conversion?
Markdown represents text structure, not every browser behavior. A tab that never opened, content behind a login, or a price changed after the fetch may be absent. Images may remain as references without the visual information inside them. A table can also lose layout details that mattered in the original page.
CommonMark defines the core text syntax, not browser interactions or the truth of a source. For a comparison table, inspect the converted values against the original page. If column relationships are essential and conversion flattened them, keep HTML for that section.
For a decision that depends on a visual state, request a screenshot alongside Markdown and compare them. For exact product fields, check whether a structured source is available. url2llm can return Shopify data for supported public product pages; it does not currently produce an arbitrary JSON extraction from a prompt.
When is a browser needed?
An HTTP response may already contain the useful text. In that case a browser adds work without adding information. If JavaScript fills the page after load, the rendered DOM may be necessary. url2llm handles rendering automatically when the page requires it. A screenshot or branding request requires browser capabilities by design.
For a JavaScript-heavy source, read the rendering decision guide. If a page returns 403, use the block diagnosis guide before retrying.
How should Markdown enter an AI pipeline?
First remove navigation and repeated page furniture. Then split text at useful section boundaries, retain the page URL for every segment, and store a fetch timestamp. Retrieval can then return the segment and its source together. Re-fetch pages on a schedule that matches how often the source changes, and remove stale segments when a page disappears.
Markdown is a useful intermediate format, not a source of truth. Verify extracted text against the original page when the answer affects a user or a business process. The RAG ingestion workflow covers discovery, refresh, and source tracking in more detail.
Try one public documentation URL in the url2llm playground, inspect the Markdown, then decide whether that source needs links, HTML, or a screenshot too.