Scrapers
Automated web data collection, from the source to the database.
Context
At Dreamension, data fed everything, the JV360 catalog, the recommendations, the trend analyses. Collecting it by hand would have taken up a full-time person. So I built an automated collection pipeline, from scraping to consolidation.
How it works
Each source has its own method. For Steam, collection goes through the official API with Node.js and Express, with a weekly pass for new releases and changes of release dates. Xbox, PlayStation and Nintendo do not offer a complete public API. I developed Python scrapers with Scrapy and Selenium, which also cover the libraries of the Game Pass and PlayStation Plus subscriptions.
The data converges in MongoDB. A slugification system unifies the titles of a same game across platforms, Assassin's Creed Valhalla becomes assassins-creed-valhalla. Duplicates are merged, the discrepancies between sources are kept for manual arbitration and each profile keeps the history of its changes.
The Twitch API completes the setup, retrieval of game identifiers, analysis of streamed titles and of daily, weekly and monthly trends.
The whole thing runs on its own. Cron jobs launch the scrapers every day. In case of failure, a Discord webhook notifies the team with the explanation of the error.
Outcome
The Game Pass and PlayStation Plus catalogs, about 450 games each, are up to date every morning in five minutes. Supervision comes down to reading the Discord notifications.
I apply the same techniques to personal needs or for people close to me, monitoring of websites, export to a CSV file or sending the results straight to Discord.