CMU Database Group · 1:12:55 · https://www.youtube.com/watch?v=yVpzHdAP0TY
0:05 official members bounce bust off the Chrome realize it's real hell what we call Home official members bounce bust off the Chrome realize is real hell where we from the ins side one I Vision fat lcks and M Clips the Chrome will make the average imitator do back flips you like a [Music]
0:27 pistol all right so today's lecture is is going to be the third hardest one of the semester because obviously losing data is bad we don't want people to lose data and you'll quickly realize this is again why you don't want to be writing if you're random job SCP programmer don't want to be writing your your own recovery stuff or recovery components in your in in your application you want to be using a database system that know knows what that you know is is is Battle Harden and tested and can do this corre do this correctly all right so quickly for the administrative stuff um project three or so project three was du last night uh as well as homework four uh or homework five which one and then so project four that's not that's way wrong
1:09 sorry that should be whatever it is in December I think it's the date it's the Sunday I think the 6th or 7th right so ignore that date but the recitation is tomorrow night at 8m and there's a post on pza for this all right so so this week I got to go to Chicago deal with with mushu's court date stuff uh try to get them out so I won't be giving the lecture next class it it'll be uh my number one pH student we Shen Lim uh so he's going to kick off the discussion on distributed databases okay he's smarter than I am uh so please show them the same respect you would show me okay all right so let's jump into this so last class was talking about the the the logging protocol right when when transactions are running
1:51 uh as they make changes to Pages we're going to put entries in this right ahead log that's going to record what those changes were right the be a transaction ID the object we're getting modifi we're modifying and then before and after uh values right so that was all the things we were doing while again during normal processing so that if there's a crash or there's a there's a unclean failure unclean uh shutdown that we can come back now and and pick up a pieces and put the database back into the correct state so that's what today is about today is about saying okay given all the things we were doing while we were running normally uh now after a crash when we come back how we use that wrer head log to put us back into the state we should have been so then that we maintain the adicity consistency and durability guarantees that we want our transactional data system to provide for
2:40 us right so I'm going to go very slowly uh you know throughout today's class there's a lot of stuff to cover um and there'll be some obvious optimizations we can do especially when we start talking about the how we'll do the recovery um but I'll sort of do the the sort of basic the basic version of it and then should there be some cases where we say okay if we did did this instead of this things will go faster
3:02 but again we recovery be super super careful so we want to be very methodical and deliberate and very conservative in how we're going to apply these changes and then once we make sure that that's correct then we can go back and optimize it further so recall last class we were talking about buffer pool policies and we said that we're going to use steel no Force steel says we're allowed to flush out dirty pages from the buffer pool
3:23 that been modified by transactions that have not committed yet only if the the rad log records that correspond to the changes been written and no Force says that we don't have to we're not required to flush out all the dirty pages from the buffer pool when a transaction commits we do have to write out the the rad log changes but not not the buffer P changes and so the basic setup is is what we're trying to achieve is that say we had three transactions T1 T2 T3 T1 does does a does a does a write and commits T2 does a write and then aborts and then T3 does a read and then right and then now there's a crash so now after recovery when the data set comes back online it says okay I can start running transactions again for you
4:02 all right we want to be the case that T1 is changes are durable right because we told the outside world when when they committed that yes your transaction is committed so if they come back and read it again they should be able to see those changes but T2 and T3 should be aborted meaning when we come back we should not see any partial updates from
4:20 uh from T3 and obviously we should not see any changes made by T2 right because T2 aborted we told it it aborted uh and so we we want to clean that thing clean that that up so the core algorithm we're going to use today is called Aries actually quick show at hands who has heard of Aries before no all right 10% 15% that's that's different that's good okay so
4:46 I'll say that like the I I'm going to describe what sort go goes along with the textbook this is at a high level what the original Aries paper talks about um and I would say not every single datab system is going to implement exactly what we'll talk about here today because again there'll be some some some optimizations and things we can do differently uh like you know most systems are not going to flush out all the pages after you finish Aries
5:08 right they'll say all right it's good enough and just keep it running um but this this paper is basically the the the canonical implementation or definition of how to make a database system safely recoverable right prior to this paper it came out in like 1992 it's like 70 pages long if you read it like it's it's not like you sit down read one one one go it's very very uh uh methodical in describing all the different scenarios that can occur so there were recovery protocols before this paper came out but they were all trying to optimize sort of
5:38 given certain things and and most of them were missing uh not handling all the different Corner cases that occur and one of the big problems that we're going to see as we go along is that we'll have the ability to also recover the D system if we crash during recovery
5:53 which can occur right and areas protocol will be able to handle that for us so again I'm not so post is not going to do exactly what this does but at a high level it's going to be good enough or it's going to be it's going to be close
6:07 enough so there's three key ideas in Aries the first is that we've already covered from last class that there's a right Ahad log right that we're going to uh when transaction made change in the pages we'll make log entries that says here's that what that change was and then when a transaction commits or before we're allowed the DAT allow to flush out the dirty page uh from you know from uncommitted transaction that we have to flush up the log records for that right and this this is just steel no force that we talked about
6:34 before another key idea is that they're always going to under Aries you're always going to repeat history during redo so that means that when you crash and come back you're going to be very careful about figuring out at what point in the right Ahad log I'm going to jump back in time and I'm going to replay all the entries redo all of them until I get to my end of the log right even though some of those changes may have already been uh successfully flushed a dis we're going to be super careful we we're not going to we're not going to try to guess whether oh yeah we made a change and that got flush out the dis because there again there's no right ahead log entry that says page flush the dis so rather than trying to guess about whether uh we got the the the page out the disc before the crash we're just going to apply the change all over
7:22 again right and that that way we can guarantee we know we're putting the D in exactly the same state it should be the other big idea is now is that we're going to log the changes when we undo an aborted
7:35 transaction right we're going to introduce new log records just as if we were doing when a transaction is running normally when we do an update we put a log entry in that when we now abort a transaction and roll back is changes we're also going to add add log entries for what those changes
7:49 are so so again you could not do this and try to optim you know maybe run faster but since we want to be very careful we're going to be very B and what we're putting in the rad log so we we can guarantee we're putting the Dat Back in the correct
8:05 state so first thing we need to do is talk about extending our uh log records now includes log log sequence numbers that's going to give us an ordering of when these uh events occur in the log then we'll talk about what uh how transaction would normally execute uh or abort uh when when we're running sort of extension what we talk about last class then we introduce an optimized version of checkpointing that is uh allow transaction to keep running stopping everything and then we'll talk at the end of like okay now we we put everything together and we do the three phase three phases in Aries
8:39 okay all right so the first thing we got to do is now we need a way to to record the order in which uh transaction uh log entries are going to appear in the log and obviously you could to say okay well every every line or every
8:52 every every delimiter character in our make sure I was recording sorry every delimiter character in the in the log is enough to figure out what's going on but when now when we do recovery the log can be really big and we want a way to identify you know each log log record individually and also now keep track of what's the sequence of of log records I would have for a given transaction so these log sequence numbers are we're going to add are just a monotonically increasing counter that says here's log entry whatever and it gets log record one and the next one gets a log sequence number this right and just we're adding one over and over again sort of like the time stamp stuff we talked about when uh currency control uh where it's it's the synthetic number the system is maintaining but we're going to use this for a different purposes again to keep track of the order of these
9:38 events and then now we're going to use these log sequence numbers all throughout the rest inside the database system to keep track of who cares about what log sequence number to tell what what's been written a dis or where the next thing we need to is where the next in the system we need to
9:54 modify so again there's log sequence numbers that can maintain for every entry we have now but now in every page in our database system we now need to extend our format to now include the page LSM and the page LSN is just going to be what was the last log sequence number of the entry uh for the last update or modification to that page and this is basically going to give you a way to tell you where is the where is that log entry is it in memory or is it out in disk because if it's out in disk then I know it's okay to flush that dirty page because the log entry has been been written a
10:31 dis then we add this this flush LSM and that's just keeping track of the the the max the the largest LSN of the last flush we did to from the redhead log from dis into memory so then now again when I'm trying to decide can I flush a page for my buffer pool at the disc either because the eviction policy says we need to make space and you got to go or the uh the background writer comes along and says you know we want to write this out now you could look at this flush LSM and say for a given page is the page LSN less than or equal to my flush LSN if yes then I know that the log has been flushed out to a later Point than than than the entry for this page the modification for this
11:14 page so that's how we're going to be able to keep track of like what's what's in dis and needs to stay on disk in our pages and what's out what's actually out on the in in the log these LSN going to be used for a bunch of other things we as as we go along uh we already talk about flush P flush LSN and Page LSN there'll be this W LSN we need for the dirty page table which will come later on when we do recovery and this is keeping track of what was the what was the oldest update we made to a page uh since it was last flushed um like what was the first entry
11:47 first modification we made to it and that just tells us if we got since we got to replay everything where to jump to in the right ahead log so we can put put the page back in the correct State last LSN keeps track of what's the the the last recorded for given transaction so I know how to jump to its starting point in the log and the master record is a a global LSN that keeps track of what's the the last checkpoint that I took every time I take a checkpoint it's an entry in the log so that's least going to be a starting point for us to go jump into that location in the log and then figure out what was going on in the system at the time of that checkpoint to decide how far back further we need to go in time
12:25 and that's a shortcut shortcut from uh prevent us from having to scan the entire log which again could be several gigabytes or terabytes depending on how long you keep the log around for and how how how many transactions you're running per second so the master record is just to LSN to go find things so don't worry about the rec LSN L LSN we'll see this later when we do recovery uh but the page LSN flush LSN that's the key the key thing we need to
12:46 understand so it looks like this so now we send our log record uh our entries now we include this additional column in the beginning it's just a log sequence number just think a global counter that we increment by one every time we want to hang hand out a new log record
13:00 then we have now in our our our pages but in memory and on disk we'll have this page LSN that this corresponds to the last log record that made it made a change to it and then the flush LSN is just saying what's the what's the LSN of the last entry that was written out from memory to dis in the
13:18 walk and the master record just says where my checkpoint is all right so say in this example here our page LSN is pointing to record 12 or LSN 12 out on disk can we flush this page yes right because the log entry 12 is on dis so it's safe to flush disc because the page LSN is less than less than or equal to the flush LSN that's good but if the page LSN points to Something in the in the inmemory portion of the rad log you can't flush it right
13:52 so again in in the EV in the buffle manager when it runs the eviction policy if it sees a page that where the page LSN is is less than the flush LSN it knows you can't flush it uh sorry if the page Ln is greater than the flush LSN you know you can't flush it so either you have to immediately flush flush that the log record then you flush the page or you got to go go find another page to evict
14:13 to reclaim space you guys didn't have to do that in your first implementation because and in Project one because there wasn't a notional log sequence numbers but there's extra checks you'd have to do to make sure whether uh something can be
14:26 evicted all right so this basically summarizes what we we do talked about all Lo log recers have an LSN we update the page LSN every time a transaction modifies the page uh and then we update the the flush LSN every time you flush out the the the buffer the right head log buffer in memory out out the
14:44 dis all right so now when Insurance action normally executes as we said before the the it's going do a bunch of reads and writes uh to to to data records we we don't put read read operations in the rad log because it's not making any modifications that doesn't make sense we only put the rights and then now there be makes a begin or commit or sorry makes a uh commit or roll
15:08 back and then we got to decide okay how are we actually going to then say what's the protocol for deciding okay what we're actually put in the rad log to truly say that this transaction has committed and What needs the system needs to do in the back end uh to write out all the dirty Pages uh NE that that it needs to for a given transaction so we'll make some quick assumptions about for the rest of this lecture to make our lives easier so we're going to assume that all our log records will fit in a single page the way you handle that if log records need to be spam multiple Pages like if you have a really big field uh and your your page size is 8 kilobytes and you're updating you know
15:45 160 kilobyte record you basically have SE segment numbers so within a log record you would say I'm one of two two of two or you know two or three and then that way on recovery you would know how many log log records you expect to see or sort of partial log record you expect to see for a given One log record and you would know that if you don't have all of it then it's actually not truly written flush to
16:06 dis we'll assume our dis writes are automic even though again may be a larger page Size Doesn't Matter we'll assume that we're doing single version uh transactions or single version commercial with strong stri topl so we don't worry about running things uh you rolling back changes from transactions that may may have read something we can ignore all that and obviously as I said before we assume we're doing steel no Force
16:30 okay and we'll ignore B plus Street logging again there's a bunch more going on I want you to be aware of but to get through today's class we'll just understand the basic of areas and you can then see how you would extend that to support uh you know more complex objects in datab or other concurrent show
16:46 protocols all right so last class we said when a transaction commits that we're going to add a commit log record to or commit record to the log and then we just guarantee that before we tell the outside world that your transactions committed that we flush all the log records for that transaction
17:03 right but now we're we're going to add this new kind of log record called transaction end and this is just for internal bookkeeping that we're going to use for ourselves to say at this point when we see a transaction end we know that we'll see no more entries for this transaction ever again and therefore we can drop it from our internal uh data structures at that point in time so think if I'm scanning through the log uh going from
17:28 from older to to newer records once I see transaction end I know after that I won't see this anymore I won't see this transaction anymore so when we successfully flush out the commit record we can then tell the outside world your transaction has committed but internally we may keep some additional metadata around for it until we till we get the transaction end and then when we add the transaction end record to the log we don't need to flush it out immediately it'll just get flushed out eventually as part of the flush for another transaction because again this is only internal bookkeeping for ourselves
17:59 it's not necessary for to tell the outside where the transaction is committed so sort of looks like this so I have in my in my red head log in memory I'm have transaction T4 doesn't begin doesn't update on a a does update on B then it calls commit so at this point here when it calls commit we have to flush the the redhead log buffer uh in memory out the dis when that's successful uh we we update the flush LSN and out point to this the the log record
18:30 and then we can tell the outside where your transtion action is committed but we'll maintain some internal metadata about like this transaction had successfully committed and maybe it it modified these objects and so forth right similar to what we saw in OCC where we got to kind of keep keep the rewrite sets around for a little bit
18:46 longer then at some later Point there's another once once we know that we're finally completely done with this transaction we add this transaction in record here again that's just internal bookkeeping for ourselves to say we don't need to uh wear anymore and then we we can reclaim the memory and use it for other stuff right that'll make more sense in in a second when we talk about aborts which is next
19:07 slide so in Aries aborting is basically the special case of uh of a transaction where you just kind of reversing the changes you would make as if if it was the the transaction itself was calling update to reverse changes for you so we're going to add log records for for those reversal of changes but we're going to keep track of that they're they're sort of special for us to know that this is being used as part of un the undue process rather than the normal execution of a transaction that we'll eventually
19:40 commit and then now we're going to add this additional field called the previous LSM and this is just going to be for for to improve the efficiency of of the protocol of the algorithm so that when we're looking through the log the previous LSM will point to the next uh the next log sequence number the next log record for this aborting transaction that we need we need to then uh
20:03 undo and so think of this it's just a link list to say okay as I do the scan try to figure out here's all the things I need to reverse for this transaction that AB boarded the previous LSN is now you're going back in time you want you if you execute the the queries in in in one order when you do Undo You got to replay them in the reverse order so the previous LS just tells you where need to jump to the log to say what's the next thing I I need to undo right you don't need this for correctness you need this for efficiency
20:30 all right so say we have this transaction here well that sucks all right whatever let me go back sorry sorry sorry all right so now you see we have the log sequence number and then IM next to we have the previous LSN so the first record the previous LSN is null or nil right because there isn't one but as I go down and keeping track of the next one I have the update on a and its previous LSN is going to be 12 because that's that's the one that came right before and again these are contiguous so it's pretty obvious what they are but imagine something where like I have multiple transaction running at the same time and they're they're putting much entries in or lead with each other and as I want to go back and reverse them I have a easy jump off to find figure out where they are right so now the transaction aborts right and then then later on we we do some cleanup and then we put the transaction end in so the thing we
21:22 okay right make sure that we undo all the changes for this aborted transaction so that if we crash back we don't see any of it partial rights so to do this we're going to add a new type of log record called the compensation log record the CLR and this is just again the reversal of a normal update operation to go back to the previous version that we had and this is why in our log records we were recording both the before and after value so that when we have to generate
21:50 the the the CLR to reverse an AB boarded transaction we know what the previous value was right and that's why you're storing absolute values in the log records right this is what physiological logging or physical logging is doing for us rather than logical logging we're storing with the exact value it is that we're placing in the in the go back to previous version rather than something like your value equals Value Plus One and the reversal is value equals value minus one
22:16 right we need to know exactly what the previous value should be so that we can install that back into it so now unlike in a regular transaction when we apply updates and then when when it commits and finishes then we have to flush uh those log records at the moment we know transactions aborted we can just tell them they've
22:37 aborted right they don't need to wait for anything so when we add these these clrs into uh into the redhead log the data system knows it doesn't need to immediately flush them to write them out the disc they'll just get written out eventually like like other things that we talked about
22:53 before and then just like the previous LSN for these clrs we're also going to maintain the do next LSN which is again for efficiency reasons it's just a point to say what's the next LSN we we need to reverse to undo
23:05 things all right so let's now expand out uh what the log record is looking like so instead of showing it in that sort of single column thing I'm going to break it up into a table because now we're adding a lot more our metadata we're running out of space on PowerPoint so we're put it in table form here all right so we have this transaction T1 it goes ahead and calls abort and then at some some later point it says okay as soon as you see the abort message you tell the outside world yes you you boarded and then now we're going through and adding new the clrs to say here's how to reverse back the change that you made right so we have now a type of what the the entry is going to be so now we have this is CR
23:40 002 right just the number doesn't mean anything so this is going to correspond to well the 02 means it corresponds to the the undo of of log sequence number 02 uh but you could store that as a separate field rather than as part of the type so you see here now again the when we first did the update the before hour was 30 and the after hour was 40 now the crr is just doing the reverse of that
24:05 just swapping that in there and then we also have this undo next LSN which is keeping track of here's the next thing that we need to then uh undo and it's different than the previous LSN because in this case here the previous LSN is is taking you to 11 and that's the abort
24:22 entry right so we could look at that jump to that and then this this is previous Al tell us how to get up to the the previous ones right but it's really we want to know what's the next thing we actually need to undo and since this transaction only did one update the next thing we need to do is basically take us back to the begin and then therefore we know we're
24:41 done right so the previous Ln takes you to the in to the in log sequence order here's the changes that the transaction made the undo next LSN is is sort of like here's the here's the changes you need to reverse and you can skip over you know metadata things like aborts and and so forth
24:59 and then now once we have once we reach the end of by following undo next LSN once we get back to the begin we know we undo all the changes of this trans transaction made when it ran you know normally then we can go ahead and now add the transaction end record and this is saying that again we won't see any more clrs any other log records for this transaction anywhere anymore in in the
25:25 log is this clear explain again what's the difference between commit and transaction this question is what's the difference between commit and transaction end so a commit is when you tell the outside world like your transaction successfully completed and at that point You' flushed all the log log records for that transaction out to dis so everything's durable on dis for the log records and that's enough that's all you need that's enough for you to be able to replay the transaction if there's a crash so it's safe to tell the outside where you've committed but at some later point you uh once you know you've cleaned up everything about this transaction we'll see that in that these additional metadata tables are going to maintain in a second once we once we know we we're never going to see this transaction ever again we can go ahead and remove it from that so could there be any other L between the commit L and the
26:14 transaction this question could there be other other commit log records between a a transaction commit and an end uh in this Protocol no for abort yes because you had the clrs between between the abort and and the end okay
26:35 yes her question is if there's two updates what would happen to the CLR mean like the update object a update object B you would have separate clrs for those because they would be separate update log entries right and so you you would so I first update a and then update B then upon aort I would first reverse B with a CLR then reverse a the CLR and then you're
27:00 done assume a a and b are are tups they don't have to be but assume they are so even though like on a I may update three three fields and then B might update Seven Fields each of those are still be individual rad log log entries is it always going to be like two level or going to be like other granularities for cl his question is uh you say for the class or for in in in real systems system so the question is like
27:30 uh I just said that oh assume A and B are tupal in real systems would they would it always be would a log entry always be you know a single Tuple no there are typic so typically yes but there sometimes there there's sometimes special cases where uh like in postgress I think in
27:52 postgress when you bring a page into memory the first time you update it the log entry will be the entire contents of the page and then if there's subsequent updates to it while the page is in memory then there'll be sort of the more fine grain log entries because again the reason why they're doing because they don't want to lose data they're super paranoid about this so let me just make another copy of the page make sure I have that always as the starting point
28:16 but the original Aries protocol you didn't have to do that but typically typically yes it'll always be the same granularity of the content okay so how to do board again we've already covered this you first write the board record to the log for the transaction then you look at the the re the the right set of the transaction of all the updates that that have occurred and you're going going to uh unapply them in reverse order that they originally
28:49 applied and then you every time you undo a change you you create a CLR record right to the log and then restore the old value and then once you to reverse everything back you can go ahead and write the last transaction and and
29:04 record the clrs never need need to be undone meaning like if I'm undoing uh a transaction and I uh and I write some um say it updates five tupal and I add clrs for the first two tupal updates then I crash and come back I'm not going to add more clrs for the ones i' I've already applied I just replay them redo them
29:33 right any any more questions about this this is the core Bedrock of how we're going to be able to do recovery now the question is like how do we do recovery in the right order so now we got to talk about how we do checkpoints uh in an efficient manner because again the right head log grows forever we don't have to replay the entire log because think about like I have transaction from 30 days a ago I and that I aborted I'm going to you know replay the clrs over and over again I want to avoid that so we'll first look at sort of two bad ways to take checkpoints and then that'll be a motivation to see how we do fuzzy
30:08 checkpoints uh as in Aries so last class we we sort of had a straw man proposal where we say anytime we want to take a checkpoint we just stop the world we haul all transactions uh we any new transaction from starting and any active transaction that is currently running we let them run to completion then we go ahead and take a checkpoint because at this point we know that we have a consistent snapshot of the database uh of the the pages in The buffer pool because we're not seeing partial updates from any transaction at that point there's no active transactions Direction
30:40 running right so this makes recovery really easy because now when I come back I have my checkpoint it's a consistent you know snapshot of the database there's no partial updates so now I just need to replay all the log entries that occurred after I took that checkpoint
30:55 it's obviously bad for performance because say I have one transaction I I have I have two transactions that are running so I block any new transaction from starting the first transaction finishes in like 10 milliseconds but the next transaction runs for like three days or something crazy like that now my system has to wait for those three days before it can take a checkpoint and and it's blocking everything right so obviously that's that's that's a nonstarter for
31:20 us all right so a better way to do this is that that we want to pause transactions that are that are that are running in the system go take a snapshot of you know writing out all dirty Pages
31:36 uh and then and then you know let them sort of keep on running again so even if they have active queries instead of saying wait to completion we'll just stop the queries from running uh you no matter what where what stage they're in then we go ahead and take the checkpoint and then once that's done then they can start resuming
31:56 again is this a good idea a bad idea I've already said it's a bad idea why is a bad idea partial updates absolutely right so again say say I have a really simple buffer pool I only has three pages right
32:14 and you think of checkpoint is basically a sequential scan of all the pages that are in memory but you you know you're writing them out the disk right so say I this transaction here I wants to update a bunch of pages so transaction starts at the bottom my checkpoint uh and make modifies page three my checkpoint is going to start at the top scan through all the pages and write them out so the transaction already modified page three
32:35 so now when I scan through I see that see that modification I write those out to disk then I unpause the transaction who then goes ahead and updates page one but now I'm going to miss that change because when I when I did when I paused and did my scan it had made a modification to page one so now if I crash come back I have half the changes that that this transaction made
33:03 so this is essentially what we're going to want to do but we need to maintain additional metadata to keep track of what transactions are running what pages has been modified since we started and now we're we're going to keep a demarcation in the log that says here's when the checkpoint started here's when the checkpoint finished and then we just know that things may have gotten modified in between them and this additional metadata will help us keep track of
33:30 this so the actual transaction table is just an internal hashtable we're going to maintain the system that's say for every single currently actual transaction what is what is it transaction ID what is its current status of of or or code or mode of the system like is it actually running is it in the process of committing or or is it something we're going to want to undo so under normal execution unless we see a a an abort or a commit every transaction is just running upon recovery though what we're going to do we're going to assume every transaction is going to get
34:05 aborted and then it's up for the log to tell us whether we should we should put it to be committed or committing so again we cheek track the status and then we'll keep track of the last LSN of the the most recent log record that that they created so that we know again what changes that they made was that written out the dis uh in between our our checkpoint uh interval
34:30 and this is the sorry this is the um sorry this is this table we we will maintain uh we maintain ENT any transaction until we get that transaction end message at that point we can drop it out of this because we know we're not going to see any more changes from
34:47 it again now when if I'm boarding a transaction I might be putting a bunch of clrs reverse Pages while I'm taking the checkpoint and I want to know what was going on during that checkpoint so that's why there's an abort message and an end message because in the case of aborting transactions there might be some more stuff that you're doing in
35:07 there the dirty page table is just keeping track of all the pages in the buff pool uh that are dirty at the point when the checkpoint starts that have not been flushed out to disk and then now we're just going to keep track of and and for every single dirty page in the page table an LSN of the log record that first caused the page to be dirty so a transaction maybe reads a record and a page is brought in but as soon as a transaction modifies
35:33 that page we keep track of that that that that LSM so that we know at the time in which the the checkpoint started and then the page was modified did that occur before after the the checkpoint began the checkpoint
35:49 ended so let's look at something like this so first the checkpoint uh the first checkpoint we're taking here right we'll assume that we've modified uh sorry assume that page 11 has been has been flushed up above right because it's not in our dirty page table and that we are aware of a of a transaction two right so in the checkpoint record we're maintaining now just assume like this is when the checkpoint occurs we're we're going to keep track of this metadata in there right so we know about transaction T2 and then our dirty page aable says that there's a dirty page
36:21 22 then now at the second checkpoint we see that we aware of two transactions T2 and T3 right because T3 started after the first checkpoint and t uh two started before the first checkpoint right and then we now have two dirty Pages 11 and
36:39 33 right so this this gives us additional meod to be able to solve that problem we solved before of like for that partial update but in this still in this example here we're still stalling the transactions we're stalling the queries while we're taking the checkpoint so that's not great so that'll handle the case where like the query is running it updates that first page then we take the checkpoint we pause it we record in our dirty page table what we saw uh at the moment we we we took the checkpoint so then afterwards we would know here's the log records that maybe modify things we missed in our checkpoint but again we're still stalling transactions while we're doing
37:19 this so the fuzzy checkpoints solves this problem by just taking the adding an entry to say this is when the checkpoint starts in the log keep track of the what the dirty page table is and what the ACT transaction table looks like then you let the checkpoint
37:37 run and then when the when when when the checkpoint completes then you add another entry into the uh into the log for the checkpoint end and that's where you stuff in the ACT transaction table and the in the dirty page
37:51 table to to tell you okay if you see this checkpoint end here's what I saw at the beginning and then you figure out all the log records between the beginning and the end what potentially got modified right just because there's a log entry that say a page got modified it made was made dirty doesn't guarantee that it's flush out the dis but the dirty page table at least tell you what what was not what was modified beforehand and whether it got flushed out and then now between the checkpoint
38:17 what was modified and check see whether that that has been uh successfully flushed out all right so go back to our example here now you see again we have the we at LSN 7 and LSS 10 LSN 10 now we have the the begin checkpoint log record and then the endpoint check checkpoint log
38:38 record and so we have this transaction up here T2 right we it started before the checkpoint started so that's going to be an actual transaction table and then before the checkpoint started T2 modified page 22 so we have an entry for that in our dirty page table so this is telling us is that when the transaction started there was a transaction T2 and again it's a hint to the recovery uh protocol say hey there's a transaction T2 was doing something before the checkpoint started go find out what it did and make sure you apply all it changes or reverse it if it if it aborts
39:11 and the dirty page table says before I took the checkpoint there was a dirty page 22 that would have got should have gotten flushed between the checkpoint start and the checkpoint end uh but it also may have been modified by another transaction before the transac you know during that checkpoint after the checkpoint flush it out the disc so go make sure you understand what's going on yes so during recovery that just means that anything in the dirty page table doesn't have to be undone because it was never committed
39:41 this question is is the dirty page table telling us that um if it's in the P do we never have to update it because you may may have to redo or undo I'll see in a second is am I never have to update it because because I can guarantee that it was flushed out during the checkpoint it wasn't flushed right because it was dirty it was flushed but you don't know whether another transaction modified it after it was
40:06 flushed right right because because we'll see this in a second but like you're basically going to say I I see the checkpoint begin checkpoint end figure out what's in between that and that's going to tell us what I need to do but I again there's no log record it says this is when the page was flushed so I don't know uh say that say page table can 11 right so at at that trans up there it updated page
40:32 11 this thing transaction also updates page 11 but I don't know whether my flush got this change or the change up above so go go make sure you make make sure you find out what happened so again this is additional metad for us again keeping track of like what was what was going on the state of the system at the moment of the crash to help us be able to reconstruct reconstruct the the system understand what going
40:59 on when we successfully flush out the I think the checkpoint end that's when you go ahead and update the uh uh the master record to say okay if you're now coming to me after a crash here's where to jump into the right ahead log for the beginning of a checkpoint and then use that as the starting point to figure out what was going on in the system at the time of the crash or before the crash
41:22 again to to reinstantiate all the active transactions and put you back in the correct state all right so now using the log sequence numbers the composition log records the transaction end uh log record types and then now these fuzzy checkpoints with checkpoint begin checkpoint end now that's enough for us to be able to recover the database and so under Aries it's a three-phase protocol and the first one it's called analyze basically you go back look at the right ahead log at some point in time based on the Master record
41:58 and figure out what transactions were actually running at this given time and what was dirty in my page table again using that that checkpoint information so then you scan forward in the log and figure out here's all the things I need to undo and and and
42:12 redo in the second phase after analyze then you're GNA you're going to go back in time sorry you're going to go forward in time where you're going to jump to some point in the log and then to replay all the changes that that you're going to see
42:28 even for transactions that are going to abort then in the undue phase when after the redo phase you're going say okay my actual transaction table contains a bunch of these transactions and I never saw a commit message for them therefore I know they aborted so now I'm going to go back in reverse order and undo any of those changes for un uncompleted
42:47 transactions and add the compensation log records to to record that I undone their changes so analyze is going to go forward in time uh redo is going to go forward in time and then the uh the um undo is going to go reverse in time so say this is our red head log again going from time beginning to the end the first phase analyze you basically want again figure out what was going on in the system since the last since after the last checkpoint or during the last checkpoint to to the end of the log and go figure out you know who needs to get aborted who need get who needs to get uh reapplied so you start the last checkpoint you and then scan for it in time and then look at all the
43:29 information yes um sorry can you go back a couple slides how far back sorry um to the the one right before this yes I think like you said we're assuming that we're doing a strong strict topl and so we have a conflict serializable
43:54 schedule but case that a transaction which starts after another transaction like the the schedule might make it so that this is conflict equivalent to a schedule where um that transaction happened before the other transaction uh your question is uh if I'm assuming strong strong St
44:18 2pl is it possible that a transaction will uh start before another transaction well is it is it possible that if we take a checkpoint where we just say okay we're only going to keep track of like these ones which began before I started my checkpoint is it possible that the schedule we generate might or like the the equivalent serial schedule might still treat some of the the transactions before this checkpoint as occurring
44:49 after ones which happened after the checkpoint started so is the it's it's the concern that wrad log might actually write entries out such that when you replay them it would it would change the order of the transactions yes uh actions would be out
45:11 of um so no so I don't think it's an issue because in order for a transaction to update a record update update an object in the database I got to get the lock on it right so now I have the lock on it I do the update or I add the log entry there so the strong tripl in this this example the order in which the log sequence numbers are showing up is the order in which that the transaction are planing
45:39 changes if we if we assume serializable ordering then this makes this a lot easier once you start getting into like multiv versioning or uh or lower isolation levels this complicates a lot of this we can ignore that for
45:54 now right again so so I'm showing this rad log you still have to think about all the stuff we talked about before we talked about concurrency control there's still you know Cas a strict 2pl there's still a lock manager there's still a protocol running above to decides who's allowed to update what
46:09 at this level we assume that something else somebody else has already figured out for us who's allowed to update things and we're just dealing with the entries that we're seeing right so going back here again the analyze you stump start to the last checkpoint scan forward in time figure out what was going on in the system system then now on redo you you look at the smallest Rec LSN in the dirty page table so what was the earliest log record that modified a page that was in our dirty page table when the checkpoint
46:43 started jump that back in time now scan forward and get all those changes because again I have a I have a page on my dirty page table that's dirty I got to know who modified it so what's the earliest log entry since that page is brought into memory
46:57 that that corresponds to the modification of that page there may be subsequent modifications after that page was brought into memory but I only care about the first one and that's the W LSN gives me so this is going to redo everything
47:11 so if you have a uh transaction that got aborted you would apply the changes see the abort message and then apply the clrs right and then the trans transaction and you know you know it's done right so you can replay all the changes you're going to see in in the redo phase
47:27 then now you got to go look at say for all the transactions that were in my active transaction Table after the redo phase completed getting the end of the log which ones didn't commit which ones have the status code as a candidate for being uh for
47:42 undue then I got to go find the the oldest log record that they have for any transaction that I got to roll back and now you got to go back in reverse order and unapply all those changes you skip over the transactions that that successfully committed
47:57 you're just undoing the ones that that got aborted and you're creating the C new clrs to the right at log as you're doing that so re so analyze jump forward some point in time scan forward figure out what's going on then with the redo phase jump to another portion higher up in the log potentially scan for reply everything then look at your act active transaction table figure out what's not running or what shouldn't have not committed and then rever reverse back in time to undo
48:25 them that's that's that's the core idea of Aries all right so this just repeats everything I've said so again analyze scan for time to less successful checkpoint uh and you're you're adding entries into the actual transaction table based on whether uh you know as soon as you see it you add it but assume it's going going to uh uh abort it and once you see the commit message then you know you can change the status to commit
48:52 right and then if you see for any any update log records if the page is not the dirty dirty page table again we'll be maintaining this as we go along so then we keep track of it and to know that this is the modification we solve for this page uh for for this log record
49:08 uh so that we know that if we never need to uh undo it we know how to find back to the transaction that created that uh that that change so the anal analysis phase finishes uh the actual transaction table is going to tell us everything that was all the transactions that were actual time that the the crash and the DPT is going to tell us all the pages that may have not been successfully flushed dis
49:32 since since uh at the moment of the crash so let's look look another example so we had the checkpoint begin right so we add that entry into to our our log record right this is now now we're just showing what's in the at and the DPT then as again I'm scanning for in time now I see this transaction
49:53 t96 well I don't have it my acual transaction table doesn't have anything in it yet I didn't see a begin message for this transaction 96 since my begin message so I know that somewhere in the log up above this begin trans checkpoint begin is is this transaction 96 starting and maybe doing some stuff so I'm going to add entry into my uh ATT it says I know about a transaction 96 because I'm seeing it in the log and I don't know what's going to happen to it at this point as I'm doing my analysis so there I'm going to assume it's going to bort
50:22 so I said it's sadus to uh to to make sure I undo it then we see that this transaction modified page 33 so I I'm going to add entry into my uh my DPT to says here's page 33 that was modified and here's the here's the the rec LSN the oldest LSN for the transaction that modified
50:42 it now I get to my transaction begin transaction end right and now I have additional information about these transactions well Lo beh hold yes I saw 96 that's good uh but there's also this 97 up above I didn't see any log messages for that so there's another transaction 97 I got to figure out what was going on with them and then now I know I have another dirty page that was my B pole for page 20 that was modified by Rec you know an LSN 8 before the checkpoint
51:09 began then now I get I see the commit Mage from 96 fantastic I update my ET and say oh t96 you made it good job said it set it status to be commit I still don't know anything about 97 so it's still uh Still Still uh canid to be undone
51:26 then at this point now I see the transaction end and now you see why you want have this transaction end because now I say okay when this transaction is done at this this point here I can go ahead and remove it from my from my LS my at because I don't care about it anymore after
51:42 this right yes is it not enough to remove it it's question is it not enough to move it at at at the commit phase um that's an opposition you could do that yes okay so again now the reading phase the point is okay so now at the end of analysis phase we have this at says here are the transactions that we knew about
52:10 here's the and here's the dirty page table so now we got to go figure out what was in what pages were modified uh sorry question yes quick question about that if you were to um remove it at the commit then because when when you write the message that doesn't mean changes were made in yes yes he's right yes so the reason why you still need it is because it could have modified something in a dirty page table and I need to know whether it Comm thank you yes yes so that's why you got to keep it
52:41 around thank you yeah because yeah implicitly through the re LSN so this guy modified what 96 modifi 33 so implicitly this LS of 20 corresponds to this transaction t96 so implicitly I I had I'd have to I want to know you know what what transaction generated this what generated this this log record 20 did it commit so if it's not in the at as he pointed out then I don't know so that's why you got to keep it
53:17 around all right so the redo phase again this is where we're going to jump back into to to history before the checkpoint usually uh before the last checkpoint began and then we're going to scan for a time and replay all the changes even if we know the transaction is going to abort because we see it in our at as a candidate for abort at the end we're still going to replay everything because again we wanted to put the put the Davis in exact same state that it was at the moment of the crash even through all the transitions to to the end of it there's obvious some optimizations you can do
53:47 like if I see a bunch of changes at the same page could I could I Collapse them to do one update yes we're not doing that we're going to be we're going to very very careful right so it's going to seem very wasteful but again if it's your data you don't want to lose
54:04 it all right so we're going to scan forward in the log containing the smallest recos n of a transaction or of something in the dirty page table and at that point in time we know that this P there's the earliest log record that modified a page that we saw during our our during during our analysis phase and we're just going to scan through every single one and and replay all the changes so the logic basically says that
54:32 if we for each log record C or CLR then if the page is not in the dirty page table then we got to bring into memory and modify if the page is in the dirty page table but the last log record that modified the page uh is less than the than the Rex LSM then we know that
54:54 the that it's a previous version of the page right again this is we're bringing in Pages after the crash that we know this the previous version of the page and we we want to apply then we know that the the change that we would try to apply has already been applied so therefore we don't need to reapply it so we this is how we know that the page was successfully written out the dis uh
55:16 since since since the last checkpoint or since at least the log record that that we're analyzing and therefore we don't need to reapply or change if it's not already page table then it wasn't in memory so we don't care this is basically saying if I bring it back into memory you see that the it's basic version number is in the future then the rec the the LSN I'm trying to reapply then I don't I know I don't need to reapply it because all the changes made out there right otherwise we we we have we have to go ahead and update
55:43 it so to redo an action you just replay the entry whether it's an update as as a from a regular transaction or a CLR it always works the same thing same way and then now we just set the page LSN to be our LSN
55:56 there's no additional logging we need maintain as we reapply changes because we already have the right ahead log that tells us what we should do we don't need to maintain a redundant one and as we're also in this redo phase as we're scanning along we don't need to flush the pages as we're uh you know when we see a commit message for a transaction because again we're in this recovery phase we're not allowed to run regular new transactions anyway so we don't have to make sure that we're flushing things when when we see ends ends or
56:23 commits then now when we hit the redo phase with the end of it for any transaction that was successfully committed and are still in the ATT because we didn't see the transaction end meage for it we go ahead and add the the transaction end message for it and we go ahead and remove it from at because at this point here when we finish redo the only entries that should still be in the actual transaction table or transactions that did not commit
56:47 during normal operations and then now in the last phase we need to undo them this is basic what it says again any transactions that are still in the actual transaction table means we didn't see a commit message form or or the abort in the trans or the abort in transaction end therefore we need to go clean them up so we're going to process them in reverse order and this is why you have that undo next LSN uh entry in the clrs so we can jump through in in the right ahead log to find the entries that we need to undo a transaction and not may have to scan everything because if we only have one transaction that that that got aborted in this really log
57:27 uh you log sequence we got we got to replay we don't want to scan all that back back over again we just want to jump to the entries that we want and this is what the uh this is what this this undo NEX give us gives us so we're going to reverse them in the same order and just basically undo the changes that that that we we make and then for every modification every change that we're reversing we add a CLR for it again we don't need to make clrs for clrs because when we did the the redo phase we would apply the changes
57:57 CLR and then if we crash during the redo phase well we just come back and just do it all over again so let's look full example here so we have a transaction or sorry we have an a right log we call begin trans checkpoint begin checkpoint end then we have uh some you know T T1 makes a change T2 makes a change T1 aborts right so then now once in normal operation when we see that T1 abort we want to start undoing the changes for T1 so the first thing we're going to add is the CLR for it and then now since it's the only change we have to undo we can go ahead and add the transaction end I'm not showing this but implicitly there's also the previous lsns we're just keeping track of all that and again undo next LSN as as
58:39 well right so then now we see say we uh T3 starts running makes a change uh T2 makes another change and then we crash so at this point here we know at least there's a transaction T3 and T2 there might be other transactions we don't know about uh but they they would be in in our at at the checkpoint
59:05 end all right so now we got to recover the database so we got to now populate after after we finish the the analysis in uh phase we would populate the ATT right and we have metadata to keep track of like for every transaction and here's the undue status uh we think they're going to after finishing the analysis phase we didn't we didn't see uh commit message before that so we know we need to undo them and the DPT is telling us that we know that there's uh Pages P1 P3 and P5 that have a bunch of changes that may not written at the disk we need to make sure that we go ahead and reapply those
59:38 changes so the first thing we got to do is say okay going back in now we when we do undo I'm sorry yeah we're doing undo here so we're going in reverse order so the first thing we need to reverse back is the change that T2 made right here so we go ahead and create a a CLR record for it right with information about what what change we're undoing and then we have now a pointer also that says what's the next up what's the next reversal we need to make for this transaction
1:00:09 here we need to make undo the next change for T3 right so that's the same thing again we just keep we're just undoing what what occurred in LSN 008 reverse that change and go ahead and apply it and then now at this point here for transaction three we know that there's no more entries uh for it because the the the next undo next is is basically null so we can go ahead and put the transaction end message here now in the original Aries protocol at this point here when you see the transaction end
1:00:36 under uh undo phase you you would actually do flush the flush the the the redhead log and flush all the dirty pages so this point here you would know that if I crash come back I at least this point in the log I I've PL all these changes most systems don't do that because again at this point you're not actually running real real qu real queries and transactions
1:00:56 so why make it your recovery slower if you again if you assume you're not going to immediately crash right right after you crash before then why why spend that extra effort but again the original protocol does this then I update my flush LSN to say here's the you know if I do that flush here's the the log entry for the the the latest record that I flushed
1:01:18 out but then now let's say there's a crash at this point so we crashed during recovery so now we need to recover from our recovery right and this is what Aries got correct and a bunch other systems did not so now when we come back it's basically the same thing before we'll do the analysis phase still the first Che you know the latest checkpoint is the one at the top here so the the at is basically going to be the same thing
1:01:44 right um actually you you could get rid of uh sorry T3 T3 is done you can get rid of that uh say this this is this is what you see in the in the ATT at at the checkpoint yeah give it a T3 give it a t T1 at this point you know they're gone right so now I need to make sure I undo this change at T2 because I didn't see the undo change didn't see transaction
1:02:07 for T2 so go ahead and just do the same thing I did before after reapplying all the changes and now I go ahead and create this the the uh the CLR for for this one here pointing up there that I didn't undo yet once that's done then I know that uh or when I'm following this back I know that I've already appli this change here right so now I go ahead and the transaction end and this point I'm done and now the days is in the exact same state it should have been at the moment of the crash with any any partial updates from aborted transactions are all reversed and you can then now you now see the outside world our transaction successfully or systems
1:02:42 successfully back back online and you can start accepting new changes yes I might be misinterpreting this but can you explain again the dirty page table is that the set of pages that were changed but not committed and then written out to this
1:02:57 so yeah so it's I I should make this more clear because I'm like I'm showing this in like like at the checkpoint end it'll contain all the the pages that were modified at the at the when the checkpoint started
1:03:12 right then after you complete your your analysis phase the dirty page table will contain here's all the pages that were modified uh and I don't know whether they ref flesh out the dis and then here's the oldest log entry that corresponds to the change that they made so there may be subsequent changes afterwards but I only care about the oldest one because that tells me how far back I have to jump in time and then in the redo phase you're going to redo all the changes yes so I'm not it's not clear on PowerPoint here like I'm showing this like is it after after the analysis after the redo right ignore that for now just saying again after the redo phase I would know here's the
1:03:46 here's the here's the pages that were modified that may not WR been written in disk and the you don't really need that to do the do the undo because the undo thing all you care about is here's the transaction in my actual transaction table that did not make any changes and I just use the undo next pointer to tell me how far back of the chain I have to go back to make sure I do undo all their
1:04:12 changes right the right so so when you start undoing all the changes you would you would say Okay since you're recording here's the undo next say you get to the CLR here this points to 004 which which is up there and that's not a CLR that's a regular update I'm not showing the previous LSN because it's PowerPoint but you now you have the previous LSM so now you can say okay I've UND undone this this is then going to tell me that when the next thing I see is going to be a real operation that I modified therefore I know there isn't anything else for me to
1:04:44 undo so therefore this transaction has has been successfully rolled rolled back I can go ahead and add the transaction end yeah so so I apologize the at and DPT aren't aren't clear here like what the actual like going back here I think the previous
1:05:00 slide right in this case here this would be after the after you done your your analysis you would you would drop out uh you would drop out T1 because you would see transaction end for that and then you would drop out T3 because you would see the the transaction end for that as well so the only thing in ET after doing redo should only be T2 and that's why after the crash here when you come back you say okay well the only thing I need to make sure I undo is is T2 because I know I've undone everything else because after the analysis pH I would see okay I saw a transaction n for T3 transaction n for T1 T2 is hanging out here let me go clean him up and you're
1:05:41 done okay so some some obvious questions Bing upon what we talked about before so if I crash during recovery during the analysis phase do I do anything extra special no I just come back and do recovery all over again right analysis phase didn't modify anything right maintain some internal data structures in memory that get blown blown the crash so I had to start all over again if I crash during redo do I do do I do anything special no right because in redo you're just blindly overwriting
1:06:11 whatever's on the pages anyway you're just making sure you're doing it in the right order so if I crash during redo I just restart redo all over again if I crash during undo then I can be a little bit clever and and
1:06:28 and um sorry if I crash Jo undo again it's the same thing where again I I just in the redo phase if some clrs made it out the right hand log got flushed out well they would get picked up in in the redo and that may that that means in my undo phase I may have to do less work because I may have got some things done the last time I went through an undone things so no matter how many times I I crash and restart over and over again as all as long as I'm making a little bit more progress the undo then eventually
1:06:58 I'll get through everything that I I reverse everything that need to be reversed and then there isn't going to be an undo phase because now my redo phase is contain all the transactions I need to undo so can I make redo faster well again if you assume you're not going to crash immediately right after you know crashing which is not always the case sometimes the the crashes are correlated especially in older Hardware then I don't have to do the the the IM Media fsync or the flushes as I'm going through and redoing things I'll just assume that they're going to flush out
1:07:30 eventually right to make the to make the undue phase actually even faster no system does this there is there is a paper though uh after I do my redo I know all the trans actions I need to reverse so in same way we did that mod log approach where where we had like in my SQL they had like here's all the changes I need to apply to a page so before you read it apply those changes first or the B Epsilon tree was doing the same thing I could keep track of every single page
1:07:57 here's all the changes I need to I need to undo before you can read this page now if any transaction comes along and says okay let me start reading this page they'll get blocked while you undo that change but now I don't have to wait for all my my page to get undone before I can start running new queries so you can sort of lazily incrementally reapply the changes to to avoid this
1:08:18 problem of course now if you rewrite your application to remove any long running transactions then then your undo phase finishes really fast and and the recovery time will will be much slower but that requires you to rewrite things and most people aren't going to do
1:08:35 that okay that's Aries that's recovery not too bad huh the devil's in the details making again when you go read the viral Aries paper they're talking about like dealing with like bad sector rights and things like that it's very very low level in a way I don't think you need to uh need to understand the the core concept of areas
1:08:58 um but like this is the basic protocol again that every system is going to use right buff buff is going to use steal no Force we can flush out dirty pages from from uncommitted transactions as long as the log records have been flushed out and we're not required to flush out all pages upon commit because again the log records will have enough information for us to be able to reapply those changes
1:09:20 we'll take fuzzy checkpoints that don't block transactions while while we're taking the checkpoint we just have to record here's all the pages that were dirty at the checkpoint started and here's all the transactions that were running when the checkpoint
1:09:29 started when we redo we run back to the earliest log record from a transaction that was active when our transaction when our checkpoint started and make sure we apply reapply all their changes simp optimizations keep track of like you know look since we know the the page LSN and the LSN of the log record that that we're trying to redo if we know that the page LSN is older sorry newer than our log record we know our changes already been applied because again there's no entry that says in the log says this page got flushed out it's the lsns are telling us are giving us information about in what order things got flushed
1:10:05 out undo transactions never have to commit but we have to apply clrs to undo them and then those are just regular log records like before so that if we crash come back the clrs we would know whether our CLR has been applied or
1:10:17 not and then the log sequence numbers again it just allows to provide the ordering of these things okay all right so at this point in the semester you can pat yourself in the back because now you know how to build a single node database system congrats right you can build a seq light you can build a postgress you can build my SQL whatever right you can build your own duck DB not saying it'd be good you could do it right uh that's fine right
1:10:45 so starting on Wednesday and again my P student W will will present that now just to make your life even harder and start with you know introduction how you actually be able to distribute a database system so all the same techniques and things that we talked about this entire semester these are still applicable in a Distributive system you always want to build a single node system first push that to its limits and then when you realize oh like this is not working we we got to scale out then you go to a distributed
1:11:12 architecture you understand you need to to walk before you can run so you need understand how so all the things things we we talked about before concurrency control all the recovery stuff we just talked about how to run you know queries and xute uh two queries and run transactions all that still applies in distributed system it's just way harder now because the node you might be talking to might be gone you know it was there 10 milliseconds ago and now it's missing what do you do or I got two transactions running on two different two different nodes they're on different sides of the world and they want to update the same record what do you
1:11:45 do very few people need distributed databases most databases or like for the 99% of people can run on a single box right and so the rare cases is that you you are a Google you are a Amazon you are a you know Microsoft then you bring a distribut base then you then you pay a lot of money for you guys to to make this distributed
1:12:05 okay I need something refreshing when I get fin manting a like SMI and won one court and my thoughts hip hop related write a rhyme in my pants intoxicated lyrics are quicker with a simple Mo licker since I'm a city slicker play Waves a picker RS I create rotate at a rate too quick duplicate feel a breeze at escate mik's at Fahrenheit when I hold it real tight then I'm in flight then we ignite what starts to boil I heat up the party for you let my girl R me my mic down with oil Rec still turn with third degree burn for one man I heat up your brain give it a suntan so just cool let the temperature rise to cool it off with s eyes
1:12:51 [Music]