Scrapers

Context

A website often publishes exactly what I need without offering any way to retrieve it. Contact lists, documents, links, catalogues, product pages, information scattered across dozens of separate sites. Since 2022 I have been building scrapers and crawlers to automate that collection, during my apprenticeship, for my own projects and for people close to me.

The need always comes in the same shape. Collecting by hand would take hours, and it would all have to be done again at the next update.

How it works

Each source calls for its own method. When an API exists, I go through it, with Node.js and Express. Otherwise I write Python scrapers, Scrapy for static pages and Selenium when the content depends on JavaScript or on multi-step navigation. Crawlers follow links from page to page to walk a whole site before extracting what matters. Requests are throttled so as not to weigh on the sources.

Raw data is cleaned before being kept. Label normalisation, slugified titles to match the same entity seen across several sources, merged duplicates, discrepancies kept for manual arbitration and a history of changes.

The output fits the use. MongoDB when the data feeds an application, an Excel or CSV file when it will be read and reworked by hand, a text file for a plain list, or a Discord message directly when only the result matters.

Runs are scheduled. Cron jobs on Linux, or the Windows Task Scheduler when the workstation is the only machine available, launch the jobs every day. The result goes out by email or to Discord through a webhook. On failure, the same notification carries the error.

Outcome

These tools replace hours of searching and copy-pasting with a job that runs on its own and reports its own failures. Supervision comes down to reading the notifications.

Above all, the exercise taught me to work against a source I do not control. Structure that changes without warning, incomplete pages, anti-bot protections, volumes that force the collection to be split and resumed where it stopped.