A 403 Forbidden response means the origin refused the request. In web scraping, that can reflect access rules, bot controls, geography, credentials, or an address the site distrusts. Treat the response as a diagnosis signal. Repeating the same request without changing the cause rarely makes the content available.
url2llm does not bill a scrape when the source responds with 403. It also does not automatically turn a WAF block into repeated browser attempts. The API reads public, logged-out pages and does not bypass authentication.
Why does a scraper get 403 when a browser works?
Your local browser and a cloud scraper may arrive from different networks and carry different cookies, headers, and browser signals. A site can allow one and reject the other before looking at the page path. Cloudflare describes bot management signals that help sites evaluate traffic. The exact reason for a specific block belongs to the site, so inspect the response and any published access policy.
Do not assume JavaScript is the cause. A browser from the same blocked datacenter address may still receive 403. Also distinguish 403 from 401, which often signals authentication, and 429, which signals rate limiting.
What should you check first?
- Confirm the URL is public when logged out and does not require an account.
- Check the response status, headers, and page body for a block or rate-limit message.
- Read the site's robots and published API or data access terms.
- Compare a request from the deployment network with one from a normal browser.
- Reduce unnecessary request volume and add backoff for transient limits.
If an official API or export exists, use it for exact fields. A login-only page is outside url2llm's public-page scope. If the source blocks datacenter addresses, a different network path may be required, subject to the source's rules. Changing extraction from Markdown to HTML does not change the origin's decision.
When should you retry?
Retry a timeout or temporary server error with a bounded policy. A 429 should respect Retry-After when the server supplies it, as described by MDN's 429 reference. A stable 403 needs a change in access conditions or a different approved source, not a tighter retry loop.
Keep the origin response separate from your application's HTTP response. A scraping API may return its own error envelope even when the origin status is 403. Record both where available, along with the page URL and attempt time. For url2llm, an origin 401, 403, or 429 is unbilled; a source 404 is billable because it is a fetched page result.
Record status, URL, time, and network path. Those details make a later failure explainable and keep blocked attempts out of your content index. The JavaScript rendering guide explains when a browser helps. The API pricing guide shows how to account for failures when comparing vendors.
Use an API key to try the exact public URL from your deployment environment. If the origin blocks it, stop the job and identify the access path before scheduling more requests.