A Path to Event Sourcing With Amazon MSK - Level 300 (United States)

Hi, welcome to this talk on a been sourcing with Amazon, managed streaming for Apache. Kafka My name is James Espy. I'm a principal Solutions, architect with Iris by steer and Sydney are working at Financial Services team helping as dry as Banks and insurance companies, get the best out of the address iPhone. Given the finest background, I'd like to explore with you today. A sample banking application. That demonstrates how you might Implement eventual sitting on their diverse platform by building. On top of Amazon. Many streaming for bocce taco Are there any objectives for this

talk? The water water has been sourcing, and how Amazon many streaming for Apache Cafe can help. I'd like to encourage you to think differently about how you catch a store and price listed. I don't, I want to explore the design and build process behind an event sourcing demo built around msk. So what is events? A good place to start, is by comparing it to what it is. Not. In about a relational database. As you often say, patent will you persist in today's by updating them in place? Here we have an invoice record. Got three columns, and we go to status of you should start with

30 days. Later, we're on a job. That identify that invoice hasn't been paid yet. And we update the status in the enrollment at table. So I would you 10 days light on top of pies at invoice. And again, we do in the update that that column in the in the particular road to paid One potential issue with this protein bar, with lost some of the rich history of this entity outfit on the fact that this invoice is in the state of Virginia, for. Time, is useful information to me on my way to run some analytics down the track that identify as witch costume is having trouble paying on time.

Let's compare that with the approach taken by ventilation which is to store an ordered sequence of State changing events. So we start off with initiative and where we create our invoice with the same cones, we sold before, and from that issued event, we can maintain running points on Snapchat with summary of what the invoice currently looks like. 30 days later, we're on the same job that identify as that this invoice is currently of the Jude. We tried another Jew event that update the status to evict you and we outside at running summary on snapshot

with that information. Only 10 days later we try to pay the rent that, I update the status to paid and we update our running summary with the status of flight as well. I would be surprised. We've got a complete history of the scientist the other time. This allows me to run analytics on all the days that we have. And interesting. It also allows me to rebuild that summary snapshot by replaying the events in order to a different city costumes. I can maintain two different presentation, lettuce

or if one became corrupted I could rebuild it from a replay of all the events as an example. At this point it's worth pointing out that event sourcing. Usually comes as part of a package deal is usually a few of the patterns and designs that you'll say used in conjunction with the vents open. Play something to mind, driven design set. Reminder for design is a software design philosophy that focuses on the demand motive behind your application. What are the terminology that you will encounter in reading about event sourcing? And

looking at event sourcing Frameworks comes from the land of Des Moines River design. So if you having trouble understanding some of the concepts old times that you say in relation to against all saying, it might be worth having a radon to my dream design saved. Some of the concepts in that space shuttles. What? Second patent here is command query responsibility segregation. So this is a design patent that allows you to step, right? You'll read them right past the data. Event sourcing is a really neat way of helping, helping you achieve that. And I finally distributed

log is usually the way that the people will look to implement or stole the events that generated in the van. So she approached a bit lighter on the section will talk about what distributive logs are useful for event sourcing and in particular, Apache Kafka. So why is insulting Like most about event sourcing is its ability to help you to prove your data architecture. And Amazon we have a I need several Concepts or mechanism called one way and two 8 oz for helping with decision-making and you are they behind this is that we making a decision here that we can potentially

come back from in a reasonable price, might acquire down the track. No way possible. We would like to make it to Idaho decision that we can come back for him as a place where one might just do a decision, which may have, I am over here and I duck behind it. I would argue that event sourcing with its approach to maintaining a complete history of data and the ability to replay it helps you from having to make a one-way door decision about your data. Additionally, it's increasingly difficult to anticipate in a long time. Usage patterns at the design built on. If your application

here it is on inside a Rock Texas. Five years ago, I would have been hard to anticipate the rise of the need to use that they do for training machine learning models as an example. older eyes of your production-grade nosql database solutions that you might want to be taken advantage of it now, And on that point, you would watch it that Iraq 600 it to spoil new patents of consumption is Technologies of all. By the time every year Daddy was platform is that a new services? And every year we see the new services in the Titus bites in particular? Are you like to be able to leverage these as

they become available? As an example here we have a showing on catch Ramon events. I'm storing that in Amazon, many streaming for capca. I found that we had multiple consumers, they can be building out a few of those events in Aurora is a relational model redshift. Is it is it data warehousing model? Turn my TV, 4K value stole an elastic search for an unstructured text by search. And then as new Services comes out, one form Amazon. Neptune is it is a graph database that we was

released couple years ago. I can then add another consumer to that event stream, that can go back to the beginning of time and build out a a graph model of of my data. If I walk How come and saying you here in Amazon is data? And analytics talks is the the idea that you should capture old data. So then if they like to help and preserve tree prices at title at Iran, how do I get to Norwood? Purchase to capturing year old died, are a cooler. I think that event sourcing provides

easy to reply and reprocess over time. So just take the box on capturing your get some more thought to how you might capture that road that are in a way that made it easier to manage favorite song. And then another major benefit is event. Sourcing cqrs together, provide some really neat architecture benefits. The diagram that is representative of the time. Are they going to stay later on? So we have A set of containers that the back out rot path that are running an address for gate from those containers. We we process the commands to come in from

the clients. When I rock pop and we stole the event that I generated from those commands into Amazon streaming for Casco, we didn't have a set of containers that process those events and builds up a summary of you in. That was an elastic search. I don't have a second so I can finance that back at repoff, it rated that it for now. Let's take search database and presented back to the consumer. 3 Spartans, allow you to scale with elastic lien independently, the the rights read and processing of those events. And not just a

Skylander that so you can prioritize or provide quality service for particular pop. So if Gary saying that the rock pop, your application has a certain level of light. I see that isn't active, bye-bye Raiders. You can achieve that with this patent. Any phone that you can separate the development. If it's, he clean the brown technology snacks. So if you had a set of developers to exit 10 elasticsearch, they can build the, the review, from the last state statue that having to know about command processing and Amazon. Registry me to pick up

So why not event sourcing what what are some of the challenges here? So it's not a New Concept. Why isn't the default way to think about your data supposed to maturity? So some of the tools we talked about today, I know the way around ten years ago, they're only just heard of getting to the point where you're constantly feeling comfortable to use them at Skyline in production. There is I'll give her your lack of framework support. So there's no killer framework, watch bring it home tonight on the jvm platform. That is really taking

off then, and one hot summer months. And your privacy it was always considered more complex. There is this moment leaving Pops in a an event sourcing cigarettes approach to to data processing. I would argue that with infrastructure is covered in and the level of automation that you can chief on the iris platform that some of the complexity heater is going away. It's much easier for small team to manage more moving Parts on the end of his phone and the benefits you get from the dock section has become increasingly more important as as data becomes

more important. Are the time before we jump into the Gemini, let's have a chat about what Amazon, many streaming for Apache cafeterias. So Cassie Kafka is an open-source stream processing framework originally developed at LinkedIn and now donated to the Apache software Foundation. Paris concept behind Apache. Kafka is a distributed immutable and on a log. Produces add messages to topics, messages are guaranteed to be ordered into NYU Concord back in time and edit messages. You can't insert messages in into a certain

spot in the previous entries accounting at the pain to the end and then consumers can make their way through that ordered sequence of messages at a right of the renters. I can go back to the store and replace. I walk to the events and concerts that we talked about earlier. Every look at multiple contingency, we Brothers telegram, we saw earlier. All these consumers, can be a different logo sits, and where they're up to in processing a particular Topic in Apache. But running Apache capture is not the easiest thing in the world. Solo. Customers

have had challenges running large-scale, patchy cabinet, appointments, on nitrous setup is difficult, shaving. The availability that you like is difficult Skyline. All the notes in and out the ability to monitor. In the same way you can a relational database support saying around it that it was needed just to run the platform. Tell me so many streaming for Apache. Kafka is a fully managed and Holly viable Patrick Africa service. For my device is fully compatible with the Apache Kafka open source project

has multiple versions of patchy Cascade and Ron will provision the cost of for you. I hope you can figure the cost. And I hope you do my tenants on the costume. Halle Bell mobile. So it runs across multiple availability zones within the address within this region. That's how I secure. So there's multiple Weiser security for walking down your ear, Apache Kafka costume. 34 night without you, this is what it looks like. And tons of fuel. What's up, where to find networks on the left, we have a costume of a

PC. Where to find network where the producers consumers and management notes for the token to an Apache cap gun cost with Ron and then managed service for Apache Kafka actually runs and it's only managed a PC, run, a cost for zucchini bites for you in a highly valuable way. We were on the couch with broken legs in the hallway, by the way. And then we present that to the costume is a PC. So put-upon network is the main points that that can be cold. Universal remote. Last year we we announce open want to train with Prometheus. So this is a additional capability for taking advantage of Amazon msk,

to use Prometheus, to do, monitoring the metrics that come out of Apache Kafka. I noticed all the way. I'm going to miss Kacey's eats its food. It was a mention for this multiple versions to Darius Management console whenever you are. I can be used to spin up and managed to custom and immigration with the right of your services for monitoring and and processing. okay, so let's have a look at the Del Mar As I mentioned it. We're going to be having a look at a sample banking application, that allows you to create

accounts and deposit and withdraw money. It is on process behind during this Tamar & Vince, housing applications in general car. Looks like the following and is it a chance? I want to get you across first. So I can man's is the instructions that your gation issued to the event sourcing system. As you process come in and she turned them into events which are immutable. Facts of oil in a sequence sequence of State changes, And we have an aggregate which is built up to you of the RNC and then our projectional materialized view is the

view that we build from our event. Been so lost sequence of events, to present to Consumers, Downstream example, we're using elasticsearch is out reed store. So, we have a projection Processing framework that the radar events from after and installed it in elastic search or you can use, you know, any sort of Downstream presentation liar that you want. Now, the right here, we have the general General design approach C model year did mine as a series of commands events and I will get some views.

Do you need to make sure that the events that you are generating, why not to create a fully-formed Aggregates and you have to complete history. If you'll aggregate or entity, you need to decide which fuse, you want to use which you Technologies elasticsearch. In this case, And then you going to design the command handlers to think about the code that would generate the events that you need and stores them in Apache Kafka. And it is on the event process is on the other side of Apache cap gun, that will trait your projection. And then following it is on the

right side that would read the title from that projection will materialize be I just want to look into Deuce simple machines. So simple. Machines, is a leading Australian technology consultancy at the intersection of date. Iraq text? You later engineering and Enterprise Subway delivery company specializes in engineering real-time data. Driven Platforms in applications. And interest interesting Lee, I have two of those projects of a walkie-talkie cobra. Look at first one being simple sourcing, which is an abstraction built on

top of Apache Kafka having a look at that in the demo today. It helps with the command handling part of your right path and Annette simple Saga which we don't have any look at today that you can check out any weight on it. Helps you with a distributed transactions Vanessa goddamn relaxers. So you saw a vision of this dog in earlier, just to quickly recap, we've got a supper tray, brought pop that supported by contain his running. And I do a spa, get to the Rock Park takes command stores at in Apache. Copter

has series of events. We didn't have a projection service rate read from Patrick half stole that in Galactic search. And then our rates are hydrates, are in elasticsearch. Number 10 is a tighter. So the client After we break that down before the calling issues commands, the first of the containers contains a command handling code, which generates events in stores in Catskill. We didn't have an event handler switch, read events from the patchy Capcom and creates a projection in arms, elastic search. And then we have Alfonso containers, which rates and protection from

search and returns. It made sense that died or so to the casino, Talk to sometime, I would have liked to show you first is the application functionality? Set. This amplification is three main functions. You get before, you can grade accounts, you can deposit money and withdraw money from an account, we have to create account screen. I'm going to create a new account with another being balance $3,000. Jump back to a list of accounts. Should see how you can sign up the balance. 3000

credit previously. I cannot go into those accounts. I can deposit some money. And I can withdraw some money. So he must have all this at this point would understand how they would implement this with a relational database today is it's how this might look using events in the objects that we we saw earlier. So the next time I saw you is the application project itself. So it's a great taste Java project, it's a multimodal project. We have Teresa modules that

represent the tree container services, that drive at Rock section. So we've got a ride. I p. I e. R, r, e d, a p, a, i, and a projection service. An interesting, Leary has a city cat in front and CK part 1. So these these two things. So lost your debit to look at Sydney Summit on the Edwards. Cloud development kit, which is a means of creating your infrastructure from code for describing of Nations fax from code. And I talked about reading both a pot one, definition antonym for Destination and this is one way of addressing some of the

extra moving parts and complexity that comes with an event sourcing architectural. So you can order make and manage this on your own. We have a fall hear that, that talks to the pipeline, that it runs and bills my application. Soccer iPod wine. Definition hearing code that has full stage, a darker stage that goes and kills my containers publishes and vcr. And then I'm going to Stage that takes my infrastructure definition signs that a new confirmation stock, and then lock up the

cloud formation deployment stage. Have a quick look at what that looks like in any of his console when are the poor that that part 1 definition using City. Can you end up with a pot wind definition? He does. What we do in the description for the sole stage listening to my kids, I propose a trade for any changes and we'll rebuild when it's text. One bill is my containers what is under a c? R stands up the the infrastructure definition at the end of that all will have

Tupac Mason sucks at his confirmation stocks in my part 1 itself and then Missy cloudformation stock that the pot one feels and deploys which is the infrastructure one which has Martin titas Taco Casa in the elastic search Costco. So if we jump into the service is quickly you'll see I've got a Three pieces for get Services. Running representing the radiator. I write a prediction service and a task for each. I've got a cast Acosta that's been stood up and I've got and elasticsearch.

Cluster has been stood up as well so silly small. Same you can't you can manage a much larger and bigger number of moving pods. Okay, so let's jump back to the to the coke. So we have Read write and prediction service. So the right path is probably the most interesting. So this is where we are processing a command. Turning them into events. so, we have a Call see that. That keeps keeps. That all off. The first thing I want to point out is, you know, what sucks about my task Acosta, you then

have discolored here that uses. Yeah, us 80 West to go and retrieve the list of broken eggs from a calf cost of a RN which gets injected into the. Container is part of the cough medicine stock definition. So this is, how am I can't? I not discovered his my cock Acosta. Then we jump into a simple sourcing Library, can help. So the primary goal of the rated rot API Is to take commands persistent, turn them into a series of State changing events. End. What is the average time to do is remove what we called the undifferentiated. Heavy lifting. So serialization of your messages, the communications

comes with casca. And let you focus a hundred percent on your building just the command handling card. So we have a Mentalist app, closshey, that especially pulling a soul together. We have to realize is we have no more night at command and one which is the talk about the rate at Johns. So let's go, let's go. So, I can man and knows how to process three types of commands. As you mention is, the create account. There's the deposit and then we throw command thing way, each of these command handlers. Comes

with it, what is, what is the commander about the prices? But also it comes with what is the current snapshot aggregate view of the entity that we're going to affect. So this is useful for doing things like ever handling and thought I seen Lord take so I need someone is the withdrawal. So I can take it the current balance from the car and Aggregate and I can check to see if the amount that was trying to withdraw. He's going to push this account into a negative balance and fireworks to, this is how you do

validation and there are handling within a, an event sourcing eventually consistent approach. This isn't all the area where the temple do some laundry helps, you at the Elks. There's a mechanism for maintaining inside of your order. The current snapshot to use when your oil prices in Yoakum in. and if we have a look at what size, France flights are part of the design process. We talked about was modeling your commands and events so we have an account command and to be here, Three cops come and see

her account, command the deposit command and the troll and they're fairly simple. As you can see, create account has a balance deposit, Sikeston amounts and withdraw, it takes them out. And then and I may have the resulting events mortal. What is well with your family again? So we have an account created eventually gets generated from the credit account command which has the account name your balance and the time it was graded simile with depositing the throne we adding the time

that these events occurred Off the back of that. We've got out front pop covered. We didn't have a projection service that processes these events. and we have, Two projections that says to consume his head. All the events one builds and accounts on review in elasticsearch L on bills and account transaction view. If you have a quick look at my face, looks like so it has a button that she wear tights in the events. And then based on which type of event, it is, it will take the last week. Search cordingley,

sorry, for patrolling deposited and created. A cult of elasticsearch to talk like that index with in this case, if you count 9000 sequence, Okay, and then finally, Oswego doubt, elasticsearch index is updated. We've got a really simple radio POA. The radiator is for you have a We have a set of main points that I presented using the Java spot framework, which is a really simple, has to be dressed if you are and it goes off and Kohl's elasticsearch to retrieve the data. When they say Pyaar Colgate, Today, we have

all three services. For processing commands for the Right Path of Radiance projections on materialized view from elasticsearch and the building out materialize fuel projections in elasticsearch. And we have a automated infrastructure as code approach using cdk2 to make it easier to manage, always moving pots. So quickly recap, what we covered today, We had to look at the what and why of events. So seeing as a date or architecture patent and we've explored how Amazon, many

streaming for Apache, Kafka can help you Hopefully filling tires to think differently about how you capture a store and persist later in the future. every time you look at some of the design considerations behind building an event sourcing application, Thank you for joining this talk today. I hope you feel inspired to consider event sourcing for a future application bill. As we move towards a more real-time an event-driven approach to building software, I believed that events or same as y processing and storing it that will become increasingly relevant and well-supported. As mentioned in

the talk. If you'd like to look at the source code to the dental application, a link to the GitHub project will be shared in the follow-up material. Please enjoy the rest of your conference.