Managed storage (the Snowflake model)

  • Snowflake disaggregated compute and storage, but Snowflake still managed the storage layer: S3 held the bits, Snowflake tracked what was in them 1:09:08 1:09:21.
  • Data goes in through the DBMS: CREATE TABLE → compute node sets up a catalog in S3 and inserts the data in a proprietary format 1:09:21 1:09:46.
  • Proprietary formats are the norm — BusTub's page byte layout is unreadable by any other system, just as BusTub can't read SQLite, MySQL, or Postgres data 1:09:46 1:10:10.
  • This is called managed storage 1:10:34.

Data lakes

  • Premise: not everything wants to write through the database. Python jobs, Lambda functions, and pipelines dump Parquet/JSON/CSV directly into S3 1:10:34 1:11:04.
  • You still need a catalog listing available files — but it is no longer perfectly in sync. With managed storage the system observed every insert and could build zone maps and Bloom filters on the way in; a lake catalog effectively says "here are some files, I don't really know what's in them" 1:11:04 1:11:26 1:11:43.
flowchart TB
  subgraph Managed["Managed storage"]
    APP1[App] --> CN1[Compute node] --> ST1[(S3: proprietary format)]
    CN1 --> CAT1[Catalog: exact, zone maps + Bloom filters]
  end
  subgraph Lake["Data lake / lakehouse"]
    APP2[App] --> CN2[Compute node]
    PY[Python job / Lambda / pipeline] --> ST2[(S3: Parquet / JSON / CSV)]
    CN2 --> ST2
    CAT2[Catalog: Unity / Iceberg / HCatalog<br/>approximate, notification-driven] -.-> ST2
    CN2 --> CAT2
  end

Lakehouse

  • Databricks branded the "lakehouse": anyone can dump files into your blob storage, but a Databricks-managed layer keeps track of which files map to which tables and knows how to read them and produce results 1:11:43 1:12:23.
  • Snowflake still defaults to managed storage but supports this model too; Redshift went shared-nothing → shared-disk → lakehouse-capable, consistently lagging Databricks and Snowflake 1:12:23 1:12:55.

Why not just transform data on the way in? (student question)

Three arguments 1:12:55 1:13:21 1:13:43 1:13:57:

  1. You may not control the producer — a data scientist without DB permissions is just dropping files in S3.
  2. Same problem as partitioning: you'd be guessing what future queries need; a year or two later the application's demands change and every transformation assumption is wrong. "That's very common."
  3. S3 storage is cheap relative to EBS/direct-attached, so you let people dump and clean up later. It lowers the barrier to storing data at all — force people to transform first and they'll just keep the dataset on their laptop.
  • The instructor doesn't take a side: it depends on organizational protocols and application demands. The failure mode has a name — a data swamp 1:13:57.

Concurrency control over data you don't own (student question)

  • S3 is append-only: you can append and delete, but you cannot overwrite an existing file — so at the physical/file level there is no concurrency control problem 1:15:10.
  • The problem lives at the logical level: a table is a bag of files and you want updates and deletes over it. That is what Iceberg does — a CRUD interface over logical datasets that maps down to S3 files 1:15:41.

Composable building blocks

  • The big trend (and the theme of the semester's seminar series): no more monolithic vendors building storage + engine + optimizer + UI. Instead, reusable components you assemble into a distributed OLAP system quickly 1:16:15.
  • Reality check: it isn't easy. One semester speaker started with DataFusion, found it didn't fit, and is now rewriting everything from scratch 1:16:51.

Catalogs

  • Seems unsexy (metadata about metadata) but is intensely active because everyone needs it 1:17:07.
  • HCatalog came out of Hive (which compiled SQL into MapReduce jobs and needed to track HDFS files); cloud vendors and Databricks each have their own 1:17:32.
  • Apache Iceberg is the hottest: Databricks paid $1 billion for ~40 people this year, outbidding an unnamed cloud vendor that was already trying to buy them 1:17:52.
  • Iceberg does more than cataloging: an insert/update/delete interface over S3 files — writes land as JSON, then background compaction converts them to Parquet 1:18:15.

Query optimizers

  • Little activity, because it is extremely hard; CMU is attempting it and it's "going okay" 1:18:34.
  • Apache Calcite is the most famous — a standalone, extensible transformation-and-optimization service that doesn't force one SQL dialect 1:18:34 1:19:03.

File formats

  • Goal: an open binary format many engines can read, so the query engine doesn't care which library produced the data 1:19:03.
  • Parquet and ORC dominate — but a recent paper by the instructor with a former student (now at Tsinghua) and the creator of pandas / Apache Arrow showed Parquet and ORC are 10-year-old designs that make no sense for modern hardware; they are working on a new file format 1:19:32.

Execution engines

  • Velox and DataFusion are the main ones; Intel's own attempt did not pan out 1:19:55.