Scheduled scraping pipelines

The seed information was collected by hand. The owner's team went to each source themselves, extracted the data and sent it to us as a CSV file. It was tedious and slow, because for every new product and every update they had to go back and update the document.

The idea was to give scrapers a try, because they save a lot of time and can run automatically on a schedule. On a schedule, the server visits a fixed set of external sources with Puppeteer and Cheerio and checks for changes. It compares each record's source ID and name with the database and ignores what it already has. Only new records move on, through a Cloud Tasks queue into the next step, which is the AI that writes the descriptions. The queue lets the pipeline handle plenty of new data, and it retries a failed step instead of losing it. After the AI step, a person on the client team approves the result.

The manual process it replaced. For the tools there was no big comparison: Puppeteer and Cheerio were popular at the time, so I used them.

Developed a web-scraping system to periodically collect and update seed information from multiple external sources.

Built automated scraping and scheduled-processing workflows using Node.js, Puppeteer, Cheerio, and cron jobs to retrieve seed information and keep existing platform data refreshed as source information changed.

The pipeline reduced reliance on manual collection of seed information and provided the underlying data-ingestion mechanism required to maintain the platform's seed catalog.

• Scrapers break when a site changes its interface. The scraper itself works perfectly with the data, but whenever a source updates its UI, the scraper has to be modified too, so we have to be very careful.
• Alerts, so breakage is noticed quickly. When a scraper keeps breaking, the system sends a notification to the team automatically, and they work on the scraper right away.
• Legality. Scraping other sites raises a legal question, which is why the sources aren't named here.
• Rate limits. The sites we scraped don't have strict ones, so I didn't focus on it. If they had, the plan was to use proxies.

The catalog stays current without the client team collecting data by hand.