- 2,478permit records delivered
- 30columns per record
- 0duplicate record numbers
- 0records with a detail error
What the client needed
A construction lead-generation client wanted every roofing, siding, and window permit issued by a US county between January and late September 2026. For each permit they needed the list fields (date, record number, type, address, status) plus details that only appear on the record’s own page: the project’s cost, roof slope, window count, and the licensed contractor’s name, licence, and phone.
The deliverable was a workbook and the script, so they could rerun it later for new dates. The final file had 2,478 rows. Its Run Info sheet records the search, the date range, and the record count, so the client could see exactly what was asked of the portal.
Why the obvious approach doesn’t work
The portal runs on Accela, an ASP.NET WebForms application. Results pages have no page number in the URL. Each “Next” click is a form postback that carries hidden state (__VIEWSTATE, __EVENTVALIDATION) from the page before it. A browser handles that for you, but driving a browser through hundreds of list pages and 2,478 detail pages is slow and breaks easily.
The second problem is length. A run takes hours. If it dies at record 1,300, starting over wastes the morning, and resuming carelessly can skip or duplicate records.
Decisions
Requests instead of a browser. I used an HTTP client with a Chrome-like TLS fingerprint and rebuilt each postback by hand: read every live form control from the current page, find the Next link’s __doPostBack target, and post it back with the right __EVENTTARGET. The run ends when there is no Next link.
Record number as the identity. Every row is keyed on the permit’s record number. Duplicates seen across pages are dropped during the run, so a permit that shows up twice in the portal’s results appears once in the workbook.
Checkpoint completed details, not page positions. This is the decision that matters most, and the one I’d make again. Saving “I was on page 132” is fragile, because ViewState from an old session is useless after a restart. So the checkpoint stores finished records instead, each with a _detail_complete flag. On restart the script re-walks the list pages, which is cheap, and skips the detail fetch for any record already complete, which is the expensive part.
[
{"Date": "03/02/2026", "Record Number": "SYN-2026-000101",
"Record Type": "Roofing Siding Window", "Status": "Issued",
"Details URL": "https://example.invalid/CapDetail.aspx?id=SYN101",
"Application Information - ROOFING - Roof Slope": "4/12",
"Licensed Professional - Name": "Example Roofing LLC",
"Detail Error": "", "_detail_complete": true},
{"Record Number": "SYN-2026-000102", "...": "...",
"Detail Error": "Application Information section not found",
"_detail_complete": false}
]
What an interruption looks like
Here is the restart path, reconstructed from the script’s own log messages. The first run is stopped during page 132. The rerun reloads 1,311 finished records, walks the list again, and only fetches details for what’s missing.
List page 132: 10 records; processing details now
Details 1/10: SYN-2026-001311
Details 2/10: SYN-2026-001312
^C
Interrupted; latest checkpoint retained. # exit code 130
$ python permits_scraper.py ... # same command
Checkpoint loaded: 1311 completed records available for resume.
List page 1: 10 records; processing details now
Checkpoint: 10 completed records on this page will be skipped.
...
List page 132: 10 records; processing details now
Checkpoint: 1 completed records on this page will be skipped.
Details 2/10: SYN-2026-001312
...
Collected and processed 2478 unique records
Saved permits_roofing_siding_window_2026.xlsx # checkpoint deleted, exit 0
The checkpoint is written to a temporary file and then renamed over the old one, so a crash during the write leaves the previous good checkpoint in place, never half-written JSON. The script also exits with a non-zero code if any record ended incomplete, and keeps the checkpoint in that case, so a scheduled run can’t report success with gaps in it.
The output
| Date | Record Number | Status | Roof Slope | Cost of Project | Contractor | Detail Error |
|---|---|---|---|---|---|---|
| 03/02/2026 | SYN-2026-000101 | Issued | 4/12 | 12000 | Example Roofing LLC | |
| 03/03/2026 | SYN-2026-000102 | Finaled | 8500 | Sample Exteriors Inc |
Every row links back to its record page on the portal, so any value can be checked at the source. The detail-error column was empty for all 2,478 rows in the delivered file.
Proof
Delivered client work
- Delivered export with a Run Info sheet recording the search, date range and record count.
- Client and data are confidential.
Good to know
- Built around the portal's current flow; if the portal is redesigned, the navigation steps are updated.
- Paced to what the portal handles comfortably.
Try the resumable job runner →