Built the data pipeline behind the catalog value reports, about 2 billion records
To measure how much of the catalog came from automated feed parsing, I built a tool that pulled catalog search results, split them and loaded them into the data warehouse, back to 1970 across 12 marketplaces.
~2B
catalog records backfilled
930M vendor-contribution plus 1.1B create-by-date responses, back to 1970
12
marketplaces
~30,000
items processed per day
ongoing daily job
- Without it
- Value-metrics reports on automated feed parsing would have run on incomplete history, because the warehouse could not ingest hundreds of gigabytes of single-line responses.
- The call
- Automate the pull, split the data in code and backfill all history, then keep it current with daily jobs.
- What I did
- Designed and wrote the query generator, the splitter and uploader, and set up the daily jobs that keep the data current.
- What changed
- About 2 billion records were backfilled and about 30,000 new items flow in each day, so the reports cover the full history.