History

  • Internal project Dremel, built in 2006 to run SQL over files (CSV etc.) sitting in GFS (32:45).
  • Originally shared-nothing: data had to be copied from the object store into the compute node before querying. Rewritten in 2010 to be shared-disk on top of GFS / Colossus (33:12).
  • Google was publicly the NoSQL company, but Dremel is credited with making SQL popular again internally; Spanner and others followed (33:34).
  • Commercialized as BigQuery in 2012. It predates Snowflake as an internal project. The papers still call it Dremel; the product name changed because Dremel is a real power-tool company (33:58, 34:17).

Architecture at a glance

  • Shared-disk / disaggregated storage, vectorized query processing, explicit shuffle operator (34:42).
  • Internal file format, believed to be called Capacitor; not public, but described as "smells like Parquet/ORC" — PAX columnar storage (34:42).
  • Zone maps, Bloom filters, dictionary and RLE compression (35:12).
  • Only extra index type: inverted indexes, for finding keys inside JSON/text fields (35:12).
  • Hash joins only — no sort-merge joins (35:12).
  • Optimizer is rule/heuristic-based, plus runtime adaptivity enabled by the shuffle phase (35:12).

The shuffle service — BigQuery's differentiator

This is the key thing BigQuery does that Snowflake does not (35:44).

  • There is an explicit shuffle between every pipeline, run on dedicated hardware with custom silicon (FPGA-like) (35:44).
  • Shuffles are pipeline breakers by definition — data has to move elsewhere to be processed (36:15).

What it buys:

  1. Fault tolerance — results always live in the in-memory shuffle service, so a dead worker is replaced and the new one resumes from the shuffle (36:31).
  2. Straggler mitigation — a slow worker is simply killed and its work reassigned; nobody is blocked on it because everyone pulls from the shuffle service (36:56, 37:22).
  3. Dynamic scaling — statistics kept on partition fullness let the coordinator decide whether the next stage needs more or fewer workers; worker count need not be the same between stages (37:22, 37:48).
  4. Dynamic partition load balancing — if one partition gets too full mid-shuffle, the coordinator broadcasts a re-partition instruction and workers apply another hash round to just that partition, exactly like recursive partitioning in a grace hash join; the overflow is later fed back through a worker to redistribute (38:10, 38:26, 38:48).
  • Side benefit: individual operator implementations never need to know how many nodes they run on — the shuffle phase hides all scale-out/scale-down complexity (39:05, 39:27).