CyberScraper: Cross-Platform Job Intelligence Platform
Project Overview
CyberScraper is a local-first job intelligence and application-tracking platform built for a focused Canadian technical job search. It collects postings from employer career sites, normalizes inconsistent ATS data, classifies relevant roles, applies a configurable local search profile, and presents the results through a browser-based dashboard backed by SQLite.
The project began as a scraper and evolved into a cross-platform operational system with source-health monitoring, persistent application state, market-intelligence views, automated tests, CI, dependency auditing, and native launch workflows for Windows and macOS.
Problem
Employer career sites expose job data through many different platforms and formats. The same field can appear as structured JSON, rendered HTML, embedded JavaScript, or a browser-only API response. Location, posting date, salary, and experience fields are inconsistent, and broad keyword searches often mix relevant technical roles with unrelated results.
The project needed to solve five problems:
- Collect data reliably from heterogeneous employer career systems.
- Normalize source-specific fields into a consistent schema without discarding raw evidence.
- Separate objective role classification from personal location and experience preferences.
- Preserve application and dismissal state across repeated scrapes and two operating systems.
- Make source failures, stale data, and classification decisions visible rather than silently hiding them.
Verified Scope
The current configuration contains:
- 59 employer career sources
- 20 ATS and site-specific adapter types
- 3 role-classification lanes: Cyber Defense, IT Operations, and Technical Delivery
- 62 configured target/profile acquisition paths, including narrow supplemental searches for selected sources
- Native launch workflows for Windows and macOS
The adapters cover major platforms including Workday, SuccessFactors, Oracle HCM, BambooHR, Greenhouse, Lever, Dayforce, Jobvite, Ashby, Rippling, ADP Workforce Now, Phenom, ApplyToJobs, and several guarded site-specific integrations.
Architecture
Acquisition Layer
Each configured target identifies the employer, career-site URL, adapter, and any source-specific parameters. Some sources also define additional acquisition profiles for narrow queries that are merged with the default result set and deduplicated by canonical application URL.
Browser-backed modules use Playwright only when a stable HTTP or public API path is not available. Network controls validate redirect destinations, bound response sizes, and restrict browser egress.
Normalization and Evidence
Source-provided fields are normalized before scoring and storage:
- Location
- Posting date
- Salary
- Experience requirements
- Canonical application URL
The database preserves raw location evidence alongside the normalized value. Missing and failed fields are represented explicitly through states such as unavailable, could not verify, stale, and verification pending rather than being replaced with assumptions.
Classification and Search Policy
Role classification is handled independently from the user’s local search preferences. Every posting can be evaluated against the three canonical lanes, while the saved local profile controls which role types, locations, remote conditions, experience levels, default view, and sort order appear in the personal queue.
This separation prevents a personal preference change from rewriting collected evidence or narrowing the market-intelligence dataset.
Persistence and Interface
SQLite stores normalized jobs, source-health history, search preferences, application state, and lifecycle information. The local web interface provides:
- A daily-action dashboard
- A complete job queue
- Applied and dismissed views
- Source-health status
- A lane-selectable market-intelligence view
- Direct application-state updates with failed-save rollback
Application and dismissal state can be synchronized between Windows and macOS through a locked merge-on-write JSON workflow.
Reliability and Quality Controls
The project includes:
- Canonical URL deduplication
- Atomic per-source database updates
- Source idle and hard deadlines
- Persistent Source Health records
- Closed-job lifecycle handling without deleting history
- Automated unit testing
- GitHub Actions CI on Linux and Windows
- Exact dependency pinning and dependency auditing
- Blocking desktop UI audit viewports
- Read-only live-source audits with retained evidence
Historical audits have been used to identify and remediate issues involving pagination completeness, malformed or stale dates, browser-route cleanup, closed postings, source-health consistency, and cross-run repeatability. The codebase avoids using a static test-count claim because the suite continues to evolve.
Operational Decisions
Local-First Deployment
The application binds to the local machine and stores its primary database locally. This keeps personal application activity and search preferences out of a public service while still providing a browser-based interface.
Evidence Over Guessing
A successful source read, a failed verification, and an unavailable source field are treated as different states. This reduces the risk of presenting inferred values as employer-published facts.
Historical Retention
Jobs that disappear from successful source runs can be closed and removed from the active queue without deleting their record or associated application state. This allows the system to remain operationally current while preserving history for analysis and tracking.
Cross-Platform Consistency
Windows PowerShell and macOS shell launchers follow the same lifecycle: prepare the environment, run or refresh the data pipeline, start the local server, expose diagnostics, and open the dashboard.
Skills Demonstrated
- Python application architecture
- SQLite data modeling and lifecycle management
- Browser automation with Playwright
- HTTP and browser network controls
- ATS and site-specific integration work
- Data normalization and evidence-state design
- Rule-based classification and scoring
- Cross-platform scripting
- Automated testing and CI
- Dependency and source-health auditing
- Technical documentation and runbook development
Current Status
CyberScraper remains an actively developed private project. The repository contains personal search configuration and operational state, so this portfolio write-up documents the architecture, verified scope, design decisions, and technical work without publishing private data.