Architecting CyberScraper: Engineering a Production System with AI Coding Agents

Project Overview

CyberScraper is a local-first job discovery, classification, and tracking platform built to monitor early-career Cybersecurity and IT Support & Operations opportunities across the Canadian technology and public-sector markets.

The system automates the full lifecycle of career intelligence: collecting postings from 150+ direct employer career endpoints, normalizing heterogeneous Applicant Tracking System (ATS) data across 26 distinct ATS platforms, evaluating postings against deterministic scoring policies, and delivering an instant desktop triage interface and market analytics engine.

I architected and engineered CyberScraper alongside AI coding agents. I owned the system architecture, product behavior, technical specifications, data contracts, acceptance criteria, debugging direction, verification, and release decisions. AI coding agents, including Codex, performed most of the source-code implementation under those constraints. I reviewed their output, tested behavior, rejected weak or bloated designs, refined the specifications, and iterated with the agents until the system met the required contracts.

This is the development model described throughout this case study. It is not a claim that I manually typed the implementation line by line. My engineering contribution was to define what the system should do, design how its parts should fit together, direct implementation, find failures that automated checks missed, and decide when the evidence was strong enough to accept a change.

---

Engineering Model: Human Architecture, Agent Implementation

The project used an explicit division of responsibility:

• My engineering ownership: problem framing, architecture, data models, system boundaries, invariants, requirements, acceptance criteria, debugging direction, quality standards, test expectations, and release decisions.
• AI coding-agent implementation: source-code changes, refactors, test implementation, repository audits, implementation proposals, and repetitive code generation under the project constraints.
• Shared iteration loop: the agent proposed or implemented a change; I inspected the result, exercised the real system, identified failures or architectural drift, refined the specification, and accepted or rejected the change based on evidence.

The goal was not to hide AI involvement. The goal was to use AI coding agents as the implementation layer while retaining human ownership of architecture, engineering judgment, and verification.

---

The Operational Problem

Discovering relevant early-career cybersecurity and technical operations roles through commercial aggregators presents persistent operational challenges:

1. Heterogeneous & Opaque ATS Architectures: Employers publish roles across dozens of distinct platforms (Workday, SuccessFactors, Oracle HCM, Taleo, Greenhouse, Lever, Ashby, BambooHR, Dayforce, Jobvite, Rippling, ADP Workforce Now, Phenom, ApplyToJobs, Workable, Pinpoint, and custom static sites). Schemas vary widely. Some expose clean JSON APIs, while others require headless browser DOM traversal.
2. Inconsistent & Hallucinated Evidence: Posting dates, salary brackets, remote eligibility, and years of experience (YOE) are frequently omitted or formatted inconsistently. Many commercial platforms hallucinate or assume missing fields.
3. Geographic & Seniority Noise: Keyword searches for “security” frequently return senior management, sales engineering, or non-Canadian postings. Personal location preferences can corrupt underlying market data if filtering is applied too early in the ingestion pipeline.
4. State Fragility: Tracking applied, dismissed, and active roles across repeated acquisition runs requires robust deduplication and persistent state management that survives employer URL restructuring.

---

System Architecture

CyberScraper is engineered as a zero-bloat, modular Python pipeline backed by SQLite with Write-Ahead Logging (WAL) and native cross-platform launchers for Windows and macOS.

┌─────────────────────────────────────────────────────────────┐
│                  Employer Career Endpoints                  │
│  (Workday · SuccessFactors · Oracle · Greenhouse · Lever)   │
└──────────────────────────────┬──────────────────────────────┘
                               │ HTTP / Playwright (Guarded)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                 Acquisition & Security Layer                │
│    • Egress validation        • Resource bounds             │
│    • Redirect loop guards     • Source health telemetry     │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│               Evidence & Normalization Engine               │
│    • Canonical URL hashing    • Strict field evidence       │
│    • Canada remote detection  • Canadian municipality dict  │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│         Deterministic Role Classification & Scoring         │
│    • Cybersecurity (SOC/DFIR) • IT Support & Operations     │
│    • Policy v17 (GIAC, CompTIA, Sysmon, Zeek, KQL, SPL)     │
│    • PRAGMA user_version cache invalidation marker          │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                SQLite WAL Persistence Layer                 │
│    • jobs (deduplicated)      • job_state (triage history)  │
│    • source_health (metrics)  • schema_migrations (atomic)  │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│             Desktop Interface & Market Analytics            │
│    • Custom SVG dropdown architecture (6 filter controls)   │
│    • Official vector flag icons (Canada & US SVG assets)    │
│    • Sub-millisecond pre-warmed market queries (< 0.4ms)    │
│    • Instant client-side SPA navigation (zero-flicker)      │
└─────────────────────────────────────────────────────────────┘

---

Non-Negotiable Architectural Contracts

To prevent autonomous AI agents from introducing silent drift, hallucinated data, or architectural bloat, I established strict foundational contracts in AGENTS.md:

1. Defensive Data Invariants & Truthful Evidence: Evidence is never fabricated or inferred. Source fields are strictly classified as present, not_published, unverified, or unavailable. If an employer does not state compensation or experience requirements, the system marks it explicitly as Unavailable.
2. Raw Evidence Preservation: Original location strings are preserved in raw_location, while canonical normalizations are stored in location using a dictionary of 60+ Canadian municipal patterns.
3. Decoupled Classification: Canonical role classification and scoring are completely geography-independent. User search profiles filter only their local queue view, preventing personal preferences from corrupting historical market data.
4. Deterministic Identity & SQLite WAL Persistence: Postings are identified by a SHA-256 url_hash computed from the stripped canonical URL. Triage state (applied, dismissed) lives in a decoupled job_state table.
5. Cache Invalidation via SQLite PRAGMA user_version: The scoring policy version is tied directly to the SQLite PRAGMA user_version header. When classification regexes change (Policy v17), the background server detects the version bump and safely re-scores stored postings in atomic batches.

---

The Agentic Engineering Methodology in Practice

I was not acting as a traditional line-by-line programmer on this project. I used AI coding agents as active implementation partners while I retained engineering ownership. The important work was specifying the system precisely enough that an agent could implement it, detecting when the implementation violated the intended behavior, and forcing the architecture back toward a simpler and more reliable design when necessary.

Below are four real incidents that show that engineering loop in practice.

┌──────────────────────────────────────────────────────────────────────────────┐
│                        THE AGENTIC ENGINEERING CYCLE                         │
│                                                                              │
│    1. Precise Architectural Specifications & Non-Negotiable Contracts        │
│                                      │                                       │
│                                      ▼                                       │
│    2. Coding-Agent Implementation & Repository Work                          │
│                                      │                                       │
│                                      ▼                                       │
│    3. Human Inspection & Edge-Case Discovery                                 │
│                                      │                                       │
│                                      ▼                                       │
│    4. Architectural Correction & Anti-Bloat Gatekeeping                      │
│                                      │                                       │
│                                      ▼                                       │
│    5. Test Verification, Acceptance, and Release Decisions                   │
└──────────────────────────────────────────────────────────────────────────────┘

Case Study 1: Rejecting “Duct-Tape” UI in Favor of Scalable Architecture

• The Problem: When implementing filter dropdowns, the agent defaulted to a patchwork solution: browser-native <select> pop-ups with emoji flag icons that rendered inconsistently across operating systems and popped up old emoji states on reset.
• My Architectural Directive:

  “We are not duct-taping this UI together. Build a proper, scalable custom dropdown architecture across all controls. Use official vector SVGs for the US and Canada flags with clean rounded corners, keep the Canadian province dropdown completely free of emojis, and refine cursor states so static containers never trigger text-selection I-beams.”

• The Engineering Outcome: The coding agent implemented a unified, accessible custom dropdown component architecture (.custom-select-wrap, .custom-select-trigger, .custom-select-menu) backed by official W3C flag geometries, clean provincial typography, and CSS user-select: none; container guards while keeping hidden native <select> elements synchronized for full automated test compatibility. I evaluated the resulting behavior and accepted the architecture only after it matched the intended interaction model.

Case Study 2: Forensic Quality Control & Data Pipeline Auditing

• The Problem: While all unit tests passed green, manual visual inspection of live job cards revealed that certain Workday postings (e.g. CDW Canada) were missing location and salary metadata pills despite having complete data on the employer’s live portal.
• My Architectural Directive:

  “You need to look deeply into this. Trace the entire data pipeline across all ATS modules. Why are location and salary fields dropping out on Workday cards when the data exists in the detail payload?”

• The Engineering Outcome: I directed the investigation and isolated the behavioral gap. The coding agent then implemented facility-code stripping in normalization.py, added URL slug path extraction as an immediate fallback, and expanded regex extractors in scorer.py (Policy v17) to capture CAD currency variations and hourly compensation. I verified the repaired behavior against live examples rather than treating the passing unit tests as sufficient evidence.

Case Study 3: Relentless Anti-Bloat Gatekeeping

• The Problem: Over weeks of rapid iteration, the application transitioned from an experimental multi-card overview dashboard to a streamlined two-view architecture (Job Board at / and Market Intelligence at /market). However, the legacy dashboard builder and obsolete templates remained orphaned in the background.
• My Architectural Directive:

  “We used to have an overview dashboard, but I no longer use it. Run a full-scale dead and stale code audit across the entire repository. Be 100% sure everything can be removed safely before touching anything.”

• The Engineering Outcome: I specified the audit boundary and required proof before deletion. The coding agent performed the AST-based analysis and removed dashboard.py, dashboard.html, test_dashboard.py, unused module imports, and 258 lines of duplicated CSS, pruning 930+ lines of dead code while preserving a 100% test-suite pass rate.

Case Study 4: Release Governance & Zero-Defect Gatekeeping

• The Problem: Preparing the codebase for its official initial milestone release.
• My Architectural Directive:

  “Before we cut 1.0.0, run a full pre-version audit across security, contracts, UI themes, and tests. Prune all stray branches and worktrees, verify CI compilation, and ensure the test suite is completely green.”

• The Engineering Outcome: I defined the release gate and acceptance conditions. The coding agent executed the repository-wide checks and implementation work. I reviewed the evidence: zero tracked credentials, verified light and dark theme behavior, clean Git state, and a fully passing 496 / 496 automated test suite. Only after those checks passed did I accept the release and cut v1.0.0.

---

Verification & Operational Metrics

The following metrics describe the v1.0.0 milestone documented in this case study:

• Unit Test Suite: 496 passing automated tests covering all 48 test modules (Ran 496 tests in 24.6s. OK (0 failures)).
• Target Coverage: 150+ monitored employer targets across 26 dedicated ATS adapter modules (workday, greenhouse, ashby, lever, dayforce, oracle_hcm, successfactors, workable, pinpoint, rippling, bamboohr, etc.).
• Ingestion & Throughput: Benchmark-audited ingestion rate of 7,877.7 rows/sec and concurrent target scheduling of 1,090 targets/sec at concurrency=20.
• Page Load Latency: Pre-warmed market intelligence queries execute in < 0.4ms.
• Scoring Engine: Monotonically versioned rescoring engine (Policy v17) mapping SOC, DFIR, SIEM (KQL/SPL), and industry certifications (GCIH, GSEC, Security+, CySA+).
• Release Milestone: Formally tagged v1.0.0 release.

---

Key Takeaways: Engineering with AI Coding Agents

CyberScraper reinforced three principles for me:

1. Specifications are an engineering artifact: When AI agents can implement code quickly, architecture, contracts, invariants, acceptance criteria, and system boundaries become even more important. The quality of the implementation depends heavily on the quality of those constraints.
2. Human engineering ownership still matters: Coding agents can implement quickly, but they can also introduce technical debt, local fixes, and architectural drift. I treated the agent as an implementation partner, not as the source of product judgment.
3. Verification cannot be delegated blindly: Automated tests are necessary, but they are not sufficient. Several important failures only appeared through manual inspection of the live system. My role was to identify those mismatches, direct root-cause work, and require evidence before accepting a fix.

The result was not “AI wrote an app for me.” It was a system I architected and engineered through an agentic development process, with AI coding agents doing much of the implementation and me retaining responsibility for what the system was supposed to become and whether it actually worked.