How to Scrape a JavaScript-Rendered Website

You wrote a scraper, ran it, and got back an almost-empty page with no products, no prices, no listings. The data is clearly there in your browser, but not in your script’s response. That gap almost always means one thing: the site renders its content with JavaScript, and your request never ran it.

Here’s how to handle those sites, step by step.

Step 1: Confirm the content is JavaScript-rendered

Before reaching for heavier tools, verify the diagnosis. Two quick checks:

• View source vs. inspect. Open the page, then view the raw page source (not the live DOM). If the data shows up when you inspect the element but is missing from view-source, it was added after load by JavaScript.
• Fetch it plainly. Request the URL with a basic HTTP client and search the response for a value you can see on the page. If it isn’t in the raw response, you need a render.

If the data is in the raw HTML, stop here: a plain request plus an HTML parser is faster and cheaper than anything below.

Step 2: Render the page with a real engine

To get JavaScript-built content, you need something that actually executes it: a real browser engine. It loads the page, runs the scripts, applies the network responses, and produces the finished DOM, the same page a visitor sees. This is exactly what a headless browser does.

The key mindset shift: you’re no longer downloading a document, you’re running an application and reading its output once it’s ready.

Step 3: Wait for the right signal, not a timer

This is where most JavaScript scrapers become flaky. A fixed sleep(5) is a guess: sometimes too short (you grab an empty page), sometimes wastefully long. Instead, wait for a concrete signal that the content exists:

• Wait for a specific selector that only appears once data has loaded (a product card, a table row, a results container).
• Wait for the network to go quiet after the page’s data requests finish.
• For infinite scroll, scroll and wait for the item count to stop increasing.

Waiting on a real condition is the single biggest reliability win you can make.

Step 4: Extract structured data, not raw HTML

Once the page is rendered, don’t ship brittle string-parsing logic. Target stable structures:

• Prefer semantic selectors and data attributes over deep, position-based CSS paths that break on any layout change.
• Where a page hydrates from an internal JSON payload, read that structured object directly rather than re-parsing the rendered text.
• Normalize as you go (dates, currencies, and whitespace) so downstream code gets clean fields.

Step 5: Make it survive at scale

A script that works once on your laptop is not the same as a job that runs reliably a million times:

• Handle detection. Sites distinguish real visitors from automation. An engine that renders with an authentic, coherent fingerprint gets the real page; one that leaks automation signals gets blocked or fed decoy content. (More in our guide to browser fingerprinting.)
• Reach the target reliably. High-volume requests from a single origin get rate-limited fast. Routing that spreads and paces requests keeps you from tripping defenses.
• Fail loudly, retry sanely. Treat an empty or challenge page as a failure to retry, not a success to store.

Skipping the plumbing

Steps 2 through 5 are most of the work, and they’re the same work on every JavaScript site. GhostCrawl handles them for you: you send a single API request for a URL, and it renders the page with a real Chrome, Firefox, or WebKit engine, waits for the content to settle, manages the routing to reach the target, and returns clean, structured output. You focus on what to do with the data, not on operating a browser fleet.

If a specific site has been giving you empty responses, the fastest test is to point a real render at it and compare the result to what you see in your own browser.