At NUS Technology, we specialize in building Data & Intelligence Platforms and Workflow Automation systems. When a business relies on large-scale data aggregation, off-the-shelf scraping tools often fail under the weight of real-world volume and complexity.

Recently, our engineering team tackled a massive data pipeline challenge for ReviewPiper, a platform designed to track and match job postings across 18+ different Applicant Tracking Systems (ATS) for recruiters. What started as a straightforward crawling task quickly evolved into a high-stakes engineering challenge involving memory leaks, rate limits, and the unpredictable nature of unstructured data.

Here is how we redesigned the architecture to process hundreds of thousands of records reliably, using intelligent queuing, SHA-256 hashing, and AI-powered fuzzy matching.

The Challenge: Scaling Without Crashing

The scale of the operation was immense. The system needed to monitor a seed list of ~30,000 companies, running extraction cycles three times a week. After deduplication and URL validation, each crawl batch actively processed between 15,000 to 20,000 companies, yielding hundreds of thousands of job postings.

Initially, the architecture processed all companies in a single, long-running process. As the HTML bodies and parsed object graphs accumulated in memory, the system experienced severe memory leaks. The RAM usage spiraled out of control, easily exceeding the Heroku dyno limits (512MB–1GB), resulting in constant R14 and R15 Out-Of-Memory (OOM) crashes.

We needed a resilient pipeline that could handle this scale without burning through expensive infrastructure or crashing during critical operations.

The Solution: Fan-Out Architecture and Smart Queuing

To stabilize the system, we completely overhauled the background processing model and introduced aggressive resource management.

1. The "One Job Per Company" Fan-Out Model

We discarded the monolithic processing loop. Instead, we adopted a fan-out architecture backed by Solid Queue (a PostgreSQL-backed queuing backend that ensures jobs are durable and never lost upon restarts).

• A master job now simply enqueues tasks in batches of 200 companies, preventing the system from loading the entire dataset into RAM at once.
• Each sub-job is responsible for exactly one company. Once that company is processed, all references are dropped, and we enforce proactive Garbage Collection (GC).
• The Result: Memory usage stabilized perfectly at ~250–400MB per worker, completely eliminating the OOM crashes regardless of how many thousands of companies were processed in a row.

2. SHA-256 Change Detection

Crawling 15,000 companies three times a week is incredibly resource-intensive, but we realized that most job boards don't change daily. To prevent wasting API calls and database writes, we implemented a content-based change detection mechanism.

• We hash the entire fetched content using SHA-256.
• The system compares this hash with the last successful snapshot. If the hash matches, the system skips processing entirely.
• The Impact: Only 10-30% of the companies (around 2,000–6,000) actually require heavy processing during each cycle, drastically reducing system load and third-party API costs.

3. Strict Concurrency and Rate Limiting

To prevent "database bombing" and hitting external rate limits, we configured the crawl concurrency to exactly 1 thread per process. By processing sequentially at the worker level, the system naturally rate-limits itself. External IP rotation and outbound rate-limiting were offloaded to ScrapingBee, while database writes were optimized using upsert_all to execute 1-2 bulk queries per company instead of querying row-by-row.

Architecture Flow

The 2-Tier AI Fuzzy Matching Engine

Extracting the data was only the first half of the battle. The business value lay in matching these raw job postings with the specific criteria of 31 recruiters.

Titles on job boards are notoriously inconsistent. A recruiter might look for a "VP of Sales", but the job board lists it as "VP Sales" or "Sales Vice President". Doing this via naive SQL queries would result in a massive N+1 database querying nightmare and poor accuracy.

We built a highly accurate, configurable matching engine using a two-tier approach:

1. Tier 1: Generative AI Enrichment (GPT-4o)
   Before matching, we use GPT-4o to automatically generate up to 200 variations, synonyms, and common abbreviations for every target job title defined by the recruiter.
2. Tier 2: Levenshtein Fuzzy Matching
   We run a Levenshtein similarity algorithm (scaled 0–1) to catch any remaining typographical or formatting differences. A score of $\ge 0.95$ is flagged as an exact match, while lower scores are categorized as fuzzy matches.

To optimize performance, we pre-filter the datasets by geography (standardized to ISO 3166 alpha-2), company size, and industry to eliminate irrelevant postings early. The similarity threshold is completely configurable per recruiter via an Admin UI (defaulting to 0.8), allowing manual tuning based on real-world observation.

The Impact

By combining architectural best practices with AI, we transformed a brittle, memory-leaking script into a robust Data Intelligence Platform.

• Zero Downtime: The fan-out architecture and explicit garbage collection ensured the system could scale infinitely without ever crashing Heroku dynos again.
• Massive Cost Savings: SHA-256 change detection eliminated 70-90% of unnecessary workload per run.
• High-Fidelity Intelligence: The two-tier AI matching engine successfully bridged the gap between messy real-world data and precise business requirements.

Are Your Operations Outgrowing Your Software?

This case study demonstrates our approach to building Operations Backbone Platforms. We don't just write scripts; we design resilient systems that anticipate real-world failures, manage infrastructure resources efficiently, and deliver actionable intelligence.

If your business is struggling with scaling data pipelines, integrating AI into your workflows, or managing systems that constantly break under load, Schedule a Strategy Session with NUS Technology today. Let us build the reliable operational backbone your enterprise needs.