Datta Vellal
Back

16 Jun 2016Amazon IndiaProject

Built the data pipeline behind the catalog value reports, about 2 billion records

To measure how much of the catalog came from automated feed parsing, I built a tool that pulled catalog search results, split them and loaded them into the data warehouse, back to 1970 across 12 marketplaces.

  • Catalog
  • Delivery

~2B

catalog records backfilled

930M vendor-contribution plus 1.1B create-by-date responses, back to 1970

12

marketplaces

~30,000

items processed per day

ongoing daily job

Without it
Value-metrics reports on automated feed parsing would have run on incomplete history, because the warehouse could not ingest hundreds of gigabytes of single-line responses.
The call
Automate the pull, split the data in code and backfill all history, then keep it current with daily jobs.
What I did
Designed and wrote the query generator, the splitter and uploader, and set up the daily jobs that keep the data current.
What changed
About 2 billion records were backfilled and about 30,000 new items flow in each day, so the reports cover the full history.