Real Elite CRM — Real Estate Lead Scraping Pipeline

Overview

Real Elite CRM is the data engine of a real estate lead-generation platform. It collects public property-distress signals, such as trustee sales, pre-foreclosures, probates, tax defaults and code violations, and turns them into clean, de-duplicated leads that a CRM dashboard can use.

How it works

The project is a pnpm monorepo split into apps and shared packages:

• API (Fastify): serves /leads with filtering and cursor pagination, plus a lead-detail endpoint.
• Worker and scheduler (BullMQ on Redis): run scrapers on a schedule.
• Healer: a service reserved for automated scraper repair.
• Scrapers package: sources are organised by type (county, listings, aggregator) and registered in a central registry.

Every scraper snapshots the raw HTML to S3-compatible storage before parsing, so records can be re-parsed and audited later. The first production scraper collects statewide trustee-sale notices with fetch and Cheerio. Browser automation is loaded lazily, only for sources that need it.

Engineering notes

A normaliser built on carefully gated regular expressions extracts parcel numbers, sale numbers, deed instruments, principal amounts, filing dates, street addresses and trustor names. It then computes a SHA-256 dedup key, and 17 unit tests cover it. The PostgreSQL schema (7 tables, managed with Drizzle) uses PostGIS geography columns with a GiST index for location queries. Raw-record hashing keeps runs idempotent, and every run is logged.

The project follows a milestone plan. Foundation and the first end-to-end scraper are complete. Queue orchestration, address and owner enrichment, and an AI-assisted scraper healer are the next milestones.