The NoSQL movement, ten years on

  • About a decade ago the NoSQL movement declared transactions a bad idea — slow, not "web scale" (1:04:44).
  • The biggest proponent was Google, with BigTable — no transactions, faster, more web scale (1:04:56).
  • Then Google published the Spanner paper (~10 years ago), which contains the reversal: it is better to let application programmers hit performance problems from overusing transactions than to have everyone write code to cope with the lack of them (1:04:56).
  • Instructor's gloss: rather than every random JavaScript programmer reimplementing logic to handle inconsistent data, put smart people in one room and make transactions fast — far more productive for the organization (1:05:37, 1:06:02).
  • Outcome: essentially all the NoSQL systems that bragged about not supporting transactions — Cassandra, Couchbase and others — added them within ten years (1:06:16).

The dirty secret coming next class

  • Next class covers two-phase locking, and then isolation levels — the dirty secret being that most systems don't support serializability, and don't give it by default. The default is a weaker level where the anomalies from 07 can actually occur (1:06:42, 1:07:10).
  • Does it matter? "Nobody knows." The class will open PostgreSQL and look (1:07:10, 1:07:22).

Student Q&A

  • Do OLAP systems need transactions? Yes — mainly for bulk updates: you load a batch of data, then apply corrections, and you want that to happen atomically (1:07:22).
  • Are there systems with no transaction support? Plenty, on both the OLAP and operational sides, but pretty much all the major OLAP systems and most key-value stores support them now (1:08:01, 1:08:41).
  • DynamoDB's unusual approach: Amazon long refused transactions, then added them with a twist — the transaction runs first without updating anything, recording its reads and writes, then submits the whole operation set so the system can see the schedule ahead of time. FoundationDB works similarly. Most systems do not (1:08:16, 1:08:41).