The tempting shortcut

  • What the buffer pool does — map page IDs to arbitrary memory locations and fake having more memory than you have — is exactly virtual memory (19:46, 20:08).
  • mmap maps a file's contents into your process's address space lazily: you get a start and end region, and touching an unmapped page triggers a page fault that the OS services by reading the page in (20:08, 20:37, 21:01).
  • When physical memory runs out, the OS picks a victim itself and stalls / deschedules your thread while it swaps — deliberately transparent, so you cannot see it coming (21:18, 22:01).

Four reasons it "murders your database system"

1. Uncontrolled flushing of dirty pages

  • The OS can flush any dirty page at any time (23:01).
  • Breaks atomicity: an update touching several pages needs all changes written out atomically, but hardware can only write ~4 KB atomically. If the OS starts flushing mid-operation and you crash, you cannot tell what it actually wrote (23:01, 23:56).

2. Unpredictable stalls

  • You cannot inspect the OS page table, so you never know what is resident; any memory touch may stall (23:56).
  • The workaround — a separate thread that "tickles" pages ahead of time — requires building most of a buffer pool manager anyway, and you cannot know which pages to tickle before running the query (23:56, 24:42).

3. Error handling becomes a SIGBUS interrupt, everywhere

  • Failures arrive as interrupts, not return codes, so the interrupt handler must be duplicated across every place in the system that touches mmap-backed memory (24:53, 25:11, 26:28).
  • With read/write through libc/POSIX you get an errno at one place — the disk scheduler — so the "blast radius" of an I/O error is confined to the code doing I/O (25:26, 26:59, 27:20).
  • (The lecture flags that errno itself "will lie to you sometimes" — see the fsync story in topic 06.)

4. Performance

  • The OS maintains its own page table protected by mutexes — the libc futex, when better implementations exist (25:51).
  • Every contention reduction and smart eviction decision you could make is thrown away because the OS does everything for you (25:51, 26:28).

Partial escape hatches (and why they do not help)

CallWhat it doesWhy it is not enough
madviseHints the OS about sequential vs random access (27:38)Still a hint; core I/O decisions stay with the OS
mlockPins memory ranges so they cannot be paged out (27:38)Does not prevent the OS from flushing dirty pages
msyncFlushes memory ranges to disk, the fsync equivalent (28:12)Contorting mmap this far just rebuilds a worse buffer pool manager (28:12)

Evidence from real systems

  • The migration pattern is common enough that the group wrote papers about it — systems split into "full mmap users" and "started full, went partial, then dropped it" (28:40).
  • MongoDB is the cautionary tale: originally built entirely on mmap, with funding, customers and top engineers — and still threw away the mmap engine for WiredTiger, a buffer-pool-manager-based engine. "If MongoDB couldn't figure it out, what hope does a random startup have?" (29:38).
  • Weaviate started on mmap, realized it was a bad idea, and removed it (confirmed to the lecturer that same day); some remnants remain (28:40, 29:19, 30:31).
  • SQLite has mmap for portability reasons, off by default (29:38).
  • Elasticsearch and read-mostly systems may get away with it, since the write-safety issues do not bite (30:31).
  • LMDB is the polar opposite position — its author advocates mmap for everything and emails other DBMS teams telling them they are wrong (30:31).
  • Cloud angle: mmap hides resource usage, hurting observability of performance/resource consumption (29:38).
  • Reference material: the paper "Are You Sure You Want to Use MMAP in Your Database Management System?" plus a short companion YouTube video where someone buys an mmap buffer pool manager from a shady guy on the street and his database falls apart (31:35).

The thesis

The DBMS always knows more than the OS — what queries want, what has been modified, what is coming next — so it can always make the better decision. The OS is needed to survive, but should not be relied on for anything, especially memory management (31:08, 31:35).