A scraper that runs for weeks — and can show you where every row came from.
Point a recorded flow at a list of targets or an interval and it keeps going. Every row it produces carries the screenshot and trace of the page it was read from. When it cannot read something, it says so, instead of writing a plausible blank.
How a durable scrape is built
1 · Record one target
Drive one example all the way through — search, open the record, read the fields, download the document. Extraction steps mark what you want kept.
2 · Parameterise the target
The id, the date range, the county, the page number — whatever varies becomes an input the flow declares.
3 · Put it on a list or a clock
Run it across a target list, or hand it an interval and let it wake up on its own. Runs queue rather than stampede.
4 · Collect rows with evidence
Structured output per run, plus a verdict, a screenshot and a trace. Rows that could not be read are visible as gaps, not invisible as absences.
A dataset nobody can audit is a rumour
Six months from now, a number in your table looks wrong. There is exactly one useful question: did the source change, or did our code change?
Every row keeps the page it was read from and the flow_version that read it. That version changes when a step's meaning changes and not when a screenshot filename does — so the two causes are actually distinguishable.
What travels with a row
- The extracted values, as declared outputs
- PASS / FAIL / UNMEASURED for that row
- A screenshot of the page it was read from
- A Playwright trace of the run
- The
flow_versionthat produced it decided_byon anything a model chose
What this will not do
- No CAPTCHA circumvention. Not on a site you do not control, not ever. A challenge stops the run, keeps the screen, and hands the decision to a person.
- Robots and terms are the operator's responsibility. QA Reef records-and-stops at walls instead of pushing through them, but it does not read a site's terms for you. We build this for targets you are authorised to operate.
- Polite by default. One warm browser per site, calls to the same site serialised through a queue, a deep queue answering
429rather than piling on, and a wall-clock budget per run. - No login you do not have. It can drive a sign-in you hold valid credentials for, exactly as a person would. It does not guess, stuff or reuse credentials.
- Never a plausible blank. A page that never settled is UNMEASURED. An empty result and an unread page are different facts and stay different in the data.
The same discipline applied to a decades-old records portal: making an API out of a 2004 web portal.
What runs today, and what is next
Live now
- Recording a flow with extraction steps that declare structured outputs
- Parameterised inputs with an origin allowlist, so a parameter cannot steer the browser off the recorded site
- A per-flow schedule — hourly or daily — that wakes the flow on its own
- A shared run queue so scheduled, manual and batch runs execute one at a time
- Verdict, screenshot, trace and
flow_versionstored per run
Roadmap
- Resumable cursors across a long target list, so a restart picks up where it stopped
- Cross-run dedupe and an accumulating dataset store with export and query
- Finer schedule intervals set from the CLI rather than the app
Today each run's output is stored with its own run record and evidence; assembling those into one table is still your join.
Questions about long-running scraping
What makes a scraper long-running rather than a one-off script?
Three things: it survives restarts, it does not re-collect what it already has, and it can tell you why a row is missing. A recorded flow gives you the first, because the flow and its schedule live server-side rather than in a terminal session; the verdict model gives you the third, because a page it could not read is recorded as UNMEASURED instead of silently skipped.
Why does every row need a screenshot?
A dataset nobody can audit is a rumour. Each row keeps the screenshot and trace of the page it was read from and the flow version that read it, so months later you can settle the only question that matters about a scraped number: did the source change, or did our code change?
Does QA Reef respect robots.txt and terms of service?
It records-and-stops at walls rather than pushing through them, and it is deliberately a polite, single-lane client. But respecting a target's terms and robots rules is the operator's responsibility, the same as with any browser-automation tool. We build infrastructure for sites you are authorised to operate.
What happens when a site challenges the scraper?
The run stops there. QA Reef does not solve CAPTCHAs and does not attempt to defeat access controls on sites you do not control. The screen is kept as evidence, the row is marked UNMEASURED, and a person is handed the decision.
Can it scrape a site with no useful HTML — a canvas, a PDF viewer, an old frameset?
That is the case it exists for. QA Reef can read the rendered screen with OCR and move a real cursor to a visible label, so a surface with no selectors is still operable. The standing rule is that numbers come from the DOM wherever a DOM exists; screen reading is for clicking and for surfaces that offer nothing else. See legacy system automation.
Tell us what you need collected.
Bring a target and the fields you want. We will record the flow once and show you a row with its evidence attached.
Talk to the team