About the talk
How to write a isolation/TAP test to validate crash recovery such that the server crashes after writing a commit record but before updating CLOG? How to test whether a terminate signal that arrives exactly after writing a PREPARE record is handled correctly?
It is possible to come up with many more scenarios like these where existing test frameworks such as regress/TAP/isolation fall short. A new proposal, fault injectors (https://www.postgresql.org/message-id/flat/CANXE4TdxdESX1jKw48xet-5GvBFVSq%3D4cgNeioTQff372KO45A%40mail.gmail.com), offers such fine grain control over the behavior of a running PostgreSQL process. Faults are points of interest along a code path. Actions such as ERROR/PANIC/sleep/etc. can be set for a fault using a SQL interface. When the fault is hit at run time, the process takes the associated action. Actions can be set or unset using SQL commands. The idea was discussed at 2019 unconference in Ottawa.
As part of the talk, the proposed fault injector framwork will be discussed in detail. We shall see several interesting scenarios that can be easily implemented as regress/isolation tests with fault injectors. We shall also consider one or two scenarios from audience and build a test case using faults during the talk.
I am a Greenplum developer. Since 2013, I have been looking into PostgreSQL source code, hacking it and learning a lot from it and the developer/user community. I look forward to continue this learning and rewarding experience for many more years to come.View the profile
So yeah, but you come to the dock, the last talked of this conference. Let's let's hanging for 40 more minutes hang in there. Okay. Yeah, I my name is Elsie. I work on greenplum. I am an engineer, I write code and tested out as well. So the last part is important and sometimes it is the other way around I ride the test first and then write the court to make the best fast. So this is a typical development philosophy followed at which is commonly known as the test driven development and why we are talking about this today is because port injection.
Facility is a Is an air ticket facilitates writing tests which are otherwise very complex and very hard to to run in a reliable way in a repeatable way. Let's see what you're going to talk about today through existing test Frameworks. That a typical postgres developer has to has Advanced Disposal and then we'll see some test scenarios which are otherwise Very difficult to, if not impossible to write using what we have already. And that's where the need for something like fault injection, and then we will go over the fault injection proposal, which is
already submitted on the Packers mailing list the patches out there and then we will write a couple of tests using default injection facility. Okay, so the first promote that we are going to briefly talk about is the regress framework. This. So if you go to the store supposed to look at HRC test, relays you'll see a bunch of tests and the lowest rates are SQL commands. And each SQL, command has some out each SPL filed, which is a set of SQL. Commands has an expected answer file as well to service writer. Typically
write this set of security, and run them against the system, was this instance and then generates an answer file manually inspected, if it is, okay? And then freeze it as be expected answer. And then what happens, if any changes that is made to the post? This could be the same set of sequel has run the results that are generated are compared against the Answer file. And if there is a problem then the test field if the results do not match, pretty simple and it is.
So for example, if you write your own maybe a planet, or a different table, so you can buy with your changes and write a test for that other command using English grammar. Very neat and fast. The next one is isolation framework useful. When you have when you want to test concurrent scenarios concurrency scenario a couple of sessions, one or more sessions running concurrently and interacting with each other to achieve some what specific interleaving of concurrent sessions. And we will see more about
this. So, each session is defined as a set of steps and step can have one or more SQL commands. And then finally, the best is nothing, but a permutation of the steps that are Define and each step has a label. So let's look at an example to get this to get a better idea. Yeah, it's not too small, right? You can read it hopefully. So yeah, what we're seeing is a very simple isolation spec. We call a test written in this isolation framework at the spec file and that spec
file. This particular spec file contains two sessions S1 and S2 an s-1 session comprises of an altar command and it's Stewart vs cilic. And you can see that session S one has two steps behind and the other one and commitment. These are the labels for those steps and Daddy and the last line has you can use those labels to the ocean which is what you see on the last night. So what this means is the test Runner would execute the the SQL statements exactly as identified. By the permutation
so that they would create two sessions against the instant that is under test. It will in one session which is that one, it was Ronaldo in the second session. It will run select two. And as you can see, alter acquired Texas exclusive lock on the table would have to wait until the altar finishes. And once the station, it's the second session which is collected will go ahead and get the results and this any questions so far. Is this clear enough Okay, I'm even for this test framework. We have an expected
answer. They'd sodium sulfite comprises of the results of the SQL commands that are executed by Parliament by a permutation and you can have as many permutations as possible for the given steps, and you can have as many as you like, in a Spec V. the third-best framework and this is the last one out of the existing test Frameworks is that it's it's called ice pack and I do not know what that stands for, but it is Apple based this framework and it is because it's all based, it's much more powerful. You can express a lot of things and is currently used to test
different scenarios which are not really SQL commands, but they are the functionality of the database. For example, initializing a standby, or initializing the database for that matter. Anybody be setting up streaming replication. Are you going to have some fancy things like killer back in with secret and do crash recovery and see if the things are saying after the crash recovery is complete. Thanks. This is a much more expressive test framework. and, I know,
it involves learning curve because you have to know for which I did not know. Okay, so yeah, we got through the brief overview of the existing test framework, this spring works. And now let's think about some scenarios So the first one is bring down a synchronous stand by while a master back and waits for, to listen to be flushed. Know what that means. Let's say you have a standby and master and the standby synchronous, so that application is synchronous and asynchronous
by definition. What that means is a comment that happens on Master. Waits the comet is not complete until the standby confirms that the comet Ellison. Ellison is lock sequence. Number meaning the wall right ahead. Log until the comet record has been flushed by The Stand by, it will keep on waiting until the standby confirms that it has flushed wall up to the, Tolleson. This is the definition of synchronous replication. What happens if the transaction has written a complete
record locally on master and it is waiting for the stand by to order confirmation that it has written the government record and doing this exact point. Standby goes down. All the replication bricks at no point, it should happen that the comet should be the transaction, should move on without the confirmation from standby because that application is configured to be synchronous, do we have a test for it? Or how do we write a test case, which is repeatable for
this scenario using the existing test Frameworks? Bet even the top framework which is much more expressive is not sufficient for writing such a scenario. Let's look at another scenario. Server crash after writing, call record, but before updating the commitment. So, this is a transaction, a whole bunch of things happen. That one of the most important things, this upcoming War record, is written to the disk and then the other most important thing, the second most important thing is
the, which is a separate login is updated with the status of this transaction that this transaction is not committed. Describes what happens if the server crashes, After the first, after the, current world record has been written. But before the dog has been updated with the transactions. So, Upon crash recovery, everything will be fine. It is really the case, but don't really think that we should have a test case to cover the scenario because I don't when you changes happen to the server code based,
we don't want to request in this particular case. So I can just see that he was also very difficult to implement using diaper regress or isolation framework. Because how do you let a back-end called after writing, to record and before updating their status and then crash the server. It's very hard to get this thing. Let's look at another one streaming replication behavior. When the replication connection, brakes intermittently. If the streaming application is set up because of some Network blip,
there's a problem and the connection brakes for a short duration and then it gets reestablished. I do understand by side. The standby keeps on reconnecting keeps on trying to connect if the connection breaks. So because of network and the standby makes another attempt to establish the connection and that succeed, in this event, we should not have any changes to the synchronous, to the application behavior. And this is a genuine bug which has already discussed on the meninges. Is that If the sign by is having
a lot of Replay light has been built on the stand by doing two things, right? One is it is continuously receiving the changes from the master receiving, the changes in terms of world, record walleye stream, and those changes are written to disc and the second process on The Stand By Me, continuously replaying the changes that have already been written to death. So that the stand by the sink with a master data directly. But it is a common scenario that the replay lags behind because under
master you have concurrent sessions writing to the right ahead. But understand why there is only one replay process. That is replaying the wall. So it's you must have seen in practice that standby falls behind and is very play is lagging is assigned by his liking and replication collection mix. And if you have synchronous replication set up, then there are some interesting changes that happened when the replication connection is re-established. I'm not such a bad witches. As I mentioned, it's already been discussed on the mailing list. We have a patch to fix it. But
the reason I'm mentioning this particular also, as you can see, very tricky to implement using any of the existing disclaimer that we already have. Yeah. Yes, that's exactly. True and fault injection is an extension, so you're absolutely correct. so, Yeah, a couple of more scenarios. Just throwing it out there that are too many things that are complex to in Plymouth using the existing case trimmer group, it is another example, where you have to have multiple transactions, a lining up to a tube update. The
performance optimization guarantee that this performance optimization is taking place in presence of concurrent transaction. So these things are very hard to implement using the existing. Just remember be and primarily because existing sql-based you need to rely on locks to be able to synchronize the two or more conference sessions. What's a Saskatchewan cash? Entry utilized are. There was a cash miss. This is another thing that you don't know. I mean, how do we test if the
organic that has his cash was being used and if you have some significant playing around with some scenarios pretty easily that are hard to get, So yeah, hopefully with this is I think we all are convinced that we need something more than what we already have. Right. And that is what injection is one option that we are trying to provide is Fault injection. El Porton, Dixon involves defining a fault point. I know fall point is nothing but a location in the source code that is labeled as for the test developer.
It's nothing but an interesting point in the source code and the idea is When does sort of blind in the circle is reached during execution. A certain action will be taken in place of just executing that line of cord. So that's what is basically the idea and this involves changing the source code. So, we have a flag, which is labeled as fault inject and and there's an extension that allows for SQL interface to enable pre-configured fault. In order to create biscuits using fault
injectors. You need two things, you need to modify the source code and Define default point, and you need to write a SQL statement that will enable the fall point and all of that. So, the enabling of the five-point can happen using any of the existing case trimmer Which is pgd class isolation or tap. internally, the way it is, implemented is view, shared memory Quizlet shared memory to Remember the false that are enable during testing. So when 423 during execution, if the fault is unable
to identify, whether the fault is unable, and the action should be taken, is to look at the schedule and see if the fault is enabled, if it is any better than the corresponding action. The action is also recorded in stored memory. And that action is taken. And we have a patch for this. So feel free to check it out. Let's take a look deeper into how it works. So what do you see on the screen? Is a definition of Heap, insert fault. He's been served as judge, the surname that I have chosen but it's a good idea to name it. According to the function that function
Northam name that indicates the purpose of the fault, in this particular case, and right at the beginning, we have added this fault, call Heath enzor, So this is how a typical fault definition looks like. I need besides fault name that are two parameters that we should talk about when is the database name which is empty. In this particular case and the other is the last parameter which is really getting listen name, which means, which is helpful to qualify the fault and we'll see in a moment. How that
works. And this is the SQL Extra-Terrestrial interface to inject the fall. So in this particular case, we are, as I mentioned, this is an extension. So first you need to do create extension fault injector and and then you need to invoke a function call inject fault. Okay, so In this case, the error argument indicates that the action taken after this point is it would be about the transaction with elog header. I'm the last argument, my table, that's the table name and that's where the default regarding can
be even more qualified, so a subsequent insert. So, once this select inject fall function is executed there after the back-end processes, that is in Heat insert, if it exists inside subsequently been under insert is happening on a table which is named as my table. Then the fault will be triggered, which means the other action will be taken, and the transaction will be aborted. This is how it works. Any questions or comments or something not clear? Okay. So fault actions that are a bunch of old actions that we can include
that. We have already seen. There is a skip action which means literally do nothing and it is useful to remember that a particular location in the cold or a particular function was called for example of a particular Branch was taken. So, this is useful for that car. If it is also useful to have some custom logic, when a particular fault is Then they suspend and resume reset. Reset means to remove default. So in the shared memory, once you inject reset, the memory will be cleared and the fault will
no longer have any effect. And status is how many times for disregard. Let's take an example of just one more example of a skip fault. In this case modified create check mind function in the check pointed and adverb at the beginning of the check minder function, to Pediatric brain function to give Define this Vault with his name that checkpoint. And it's a skip. If the fault was injected with the skip action then what would we do? The custom logic that we would perform
is to skip the checkpoint. And this is useful to test certain classically scenarios. If you want to test replay of a particular world record that I want. You don't want a pointer to interfere in between your test. So this is a way to let the checkpoint checkpoints. To bring back to hip insert fault, we will have a few more interfaces that we have is wait until triggered fault. So, this is also an interface provided by the extension, which means that the test will block here at the statement until the back and process Executives Heap
insert fought until somebody else's executive in third ball and that gets triggered exactly one time. So the last argument, one means it will wait until the Heap insert for disregard one time. And remember the action and its status. Everything is maintained in shared memory, whenever the farthest hit, all other back and know that the fault is it and that's why this works. So the other interfaces status that we saw, it gives you in table form editor, text indicating how much how many times of our disregard and what is
the latest complete or not? I'm the last one is reset, which is to remove default. And this is interesting. So in a Synchro if you have streaming replication setup, so this interface comes in handy where if you know the board number that the standbys listening on Ford number in whose name then you can use this interface to inject, default understand by this, this example is rather incorrect because he will never be on standby. You can have some different fall and injected on the stand by using the stand by. Stand by for the walk. Internally
happens is this function would establish a secure connection and understand why. So the standby shared memory will have the fault as enable flashlight. Yeah, it's a sport status is you have triggered which means it was reached during execution by some back in process. Right? And the action was the NBA action was taken. Indicted, but not triggered means the test has executed inject fault, but the fault has not been reached during execution by any back in process.
Completed means it was the fault of the back into some guy can process reach the fault and the action was taken. It was triggered and maximum number of times it was too good, so it will no longer be triggered, meaning, even if another back into the system reaches that point, no, action will be taken if we continue as if nothing was no fault of hers. Enable, Yeah. So you left. Let's write some tests. Let me do a time, check. How much time do we have? Okay, so this is going to be interesting. This is this the first test that we are going to write a speculative insert test case
and You've, you've seen this insert into on conflict, right? Have you used this before in certain conflict? It allows you to do some other action, if there is a country in violation during insert insulin involves, but those conflicts are detected at a particular point. And that point is when the insert has updated the Heap. Heap her table on an index. And that's when the conflicts are detected. Because of an currently running transaction. And there is a test for it already committed.
And that test is the you can check it out. And to be honest, my head hurts looking at the test because it's so complex that uses advisory logs and goes on and on and if you look at the discussion on the mailing list through several Generations, the first it seemed it was okay but after that it would fail intermittently for four times. So it's very hard to get such a interleaving of transactions. Reliably do you need something for the injectors? And let's see. Let's try to ride the sub. There are looking at the
problem. The problem is. We want to achieve and interleaving of concurrent transactions sets. That one transaction is exhibiting insert into on conflict, and it has completed inserting that into the Heap table, but it is to insert into to update the index. That's when we want to suspended the transaction. So we can use suspend fall here. Let's first. Look at the set up so it's just absurd. Test is the table name that you have chosen to Unique index on it, so that the
conflicts generated and and then we Define to 4001 is the suspend fault that we just talked about, which is the name of this part is insert in Dexter police, and this is the function, with exactly this name insert index people's, which is exhibited right after the fault. So, you right after the temple has been inserted into into the heat table and dysfunction update the index. So, we suspend the WindSeeker suspend fault, which is at the beginning of this function, that's
no action will be taken until the fault is resumed. I need inject a second fault, we need another fault, which is cheap, inserts speculative. This is a function and you can. So the department of this part is that the speculative action was one conflict action was taken. So this was indeed taken because it's no accident. No, no action will be taken when this fall, is it? But it only in the shed or later, when you check the status, we can get to mine. This call was hit and then the test can figure out if I should pass or fail. That's why we have
to force and the table and an index Define. So as you can see here, the syntax resembles, the isolation syntax that we saw. So this is an isolation test kit, And we have two steps Define here and both the steps as one absurd as to absurd. And so you have two sessions Define here. So these two sessions, this is not a complete Spec V. I'm just included. A stupid for better than what we have is two different sessions and both of them are doing insert into on
conflict, both of them are executing juice. The debt is another step, which is to inject a resume fault, which will be in isolation, remember that you can have multiple sessions. So in this particular aspect file, we have two sessions here and then the resume fault. Would I would like the suspended and update the index? So, this is not a complete spec, but you get the brief idea of what's happening, right? Let's see. Let's see the permutation. Silver. S1 absurd, and I'm 10%. And that's a new enhancement that we have added to isolation chamber.
Which allows the isolation framework to determine that this particular command is going to block. And the reason for blocking is something other than locks because today that isolation framework if a particular command in blocking and if it blocks without for some other reason than waiting on a lot, it will just fail a test. So we had to add, we have to teach isolation framework that there is another reason to 4128. so, S1 absurd. Remember what is absurd? It's just the insert on contract action. So this is going to wait because we have injected the suspend fault.
So it's one update is going to block because of the suspension by. Then we do, it's too absurd, which is again insert on conflict but from a different station. And as you can see, this station should succeed, because there is no reason, this will block because the previous transaction has not committed. Yet, inserting some conflicting temples at the transaction is still in progress. So, its effects are not seen by the S2 absurd step, so it's too absurd. Step is expected to complete without conflict and then we unblocked This one and then it detects the conflicts.
Thanks, and the last, the last, but one step Bridges S14 status. That allows us to validate that the speculative in the speculative update. The conflict resolution step was taken and results that were finally ended that finally ended into the table. So yeah, this is how simple it looks like and if you have time to take a look at the other tests that I mentioned here, which is currently on Master this one. It's very complex. So if this is speculative insert test kit. okay, we have one more test to write but less than 10 minutes, so let search through, unless you have any questions on this
particular test, Okay, the other tests streaming application test and it relates to the bug that we talked about earlier during streaming replication, if you have synchronous replication setup. Right. And there's a replay like buildup on the standby and what happens is the Comics, Keep blocking on Master until the replay is finished. And that's wrong because because then if you had synchronous comic set the remote right after disconnection, suddenly, the behavior is a synchronous apply, what ends up happening after
disconnection and re-establishing. The connection, it's on the master. Keep waiting until the entire replace finished. And that's, that's the book. So let's write a test case for this discussion already on this problem. So here's the setup. Jama. Masjid running on 543 234 Port. We are standby 21, 5433 and the synchronous replication set up between master and standby. And we are going to test for this is ridiculous. The simple sequel based isolation, interleaving of concurrent connections.
So what parts do we need for this first? Is we need to induce replay lag and for that, we need, we need a fault, which would suspend the standby, standby replay process. But that's not helpful for us because we are writing a test and through sequel interface. There is no way to change a guck on a standby. It's so let's use the remote fault interface to to induce this delay and then we need another fall to terminate the connection Terminator application connection.
And as I mentioned earlier, logic, understand why is that? As soon as the standby to Text data connection, is broken it right to reestablish the connection. So we don't have to do anything. After disconnecting the connection will be re-established by the standby. So, let's look at the test. This is again, a snippet from the desk is not full complete test. So, I hope this is not inject fault. We domain Loop and sleep. Sleep is the type of fault we're using here and the number 10 is
the seconds to sleep for. So what this means is, once this part is injected, the standby will start sleeping before it applies. A world record in its main Loop. So after injecting the fall, I am, by the way, the local host, and 5 4. 3 TVs to last arguments indicate that this is a remote for it and it will be injected on the standby. And then you have the interview. We have this insert command which will generate a wall log on the master replaying. This wall will cause the stand by to sleep for 10 seconds and then we kill the stand by
you through another fault again. We kill the weak in the wall standard process so that the replication connection is broken and this is the Fault again. So to wall, receiver Loop in the wall receiver, be injected fatal for fault which means that the one with a V and The standby postmaster will restart the walrus. Walrus Eagle process and replication connection will be re-established, and the last part is the validation step. so, the last insert statement, tries to insert into the, the table with the timestamp, which is now Rights of
the value that gets inserted in the stable is the time stamp at which the insert command was executed. And if the is ended up waiting until the replay finishes with the what was inserted into the table, we can find the replay process ended up waiting waiting for the reply but after this insert and compare the timestamp again for select now. Minus the what is recorded in the stable? If you do this, then the difference should not be greater than 10 seconds, right than that.
If it is greater than the test, would fail to defend, So yeah, that's how it looks like. It's a simple test that allows us to test the Fairly complex and manifest Urban that's it. So that's all I had. Please help me with the bags. Any questions? Yeah. Yes. So I'll order dear friend Michelle tequila has renewed it. Thank you very much. And He thinks it is too intrusive. Yeah, that's what it is. Stuck at this point. Because you just cuz I believe you order them
and Branch love. Yous mailing list of drugs is used submitted patches on the meaning list. That's fine. Yeah, Community company-sponsored. When they make patches available? If anybody grounds out in after, you will not get to the experimental package right now. They will need to look at the mail mailing list and download the attachment and then apply the patch. Flourishing communities to security patches for the committee members can decide rather than going running around
lots of opinions on this topic within the community. Yes, I totally agree. This is that. It regretted the process but this is how it is. Yes. Oh yeah, thank you for asking that question. This is already in Greenville and we use this all the time. And in fact, this last year at one nice thing I thought of why is there is a non-conference day during the Ten Conference, we ended up discussing for injection exactly this problem, but in a very informal setting and we showcased
try to Showcase what we have on the green side and there was significant interest in the community. what index day before we went and ordered the batch from greenplum to Poydras and submitted Drink, this has already been there greenplum 6 5. Yeah. Yeah. No, it's not. His fault injection is Streaming, everything is available in swimming. Replication be used for master and standby in replumb for for like Sims 4, 5 x, and for the fault condition has been there since beginning.
Buy this talk
Buy this video
Our other topics
With ConferenceCast.tv, you get access to our library of the world's best conference talks.