Events Add an event Speakers Talks Collections
 
Duration 41:44
16+
Play
Video

PGConf India 2020 - How to boost & scale Postgres - Denis M - GridGrain

Denis Mekhanikov
Software Engineer at Facebook
  • Video
  • Table of contents
  • Video
PGConf India, 2020
February 27, 2020, Bengaluru, India
PGConf India, 2020
Request Q&A
PGConf India, 2020
From the conference
PGConf India, 2020
Request Q&A
Video
PGConf India 2020 - How to boost & scale Postgres - Denis M - GridGrain
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
101
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About the talk

Businesses are accumulating data at enormous rates requiring huge amounts of storage. Due to this, many industry players tend to predict Postgres' extinction. Nevertheless, most businesses have Postgres deployments for decades and continue doing so regardless of the advancement in big data, mobile, IoT, and all other revolutions that contribute to data generation.

So, how does Postgres manage its way through all technological storms and revolutions, and rank among the top 5 databases? The answer is the evolution of technologies designed to support relational databases. Whether you want to scale to petabytes or tap into RAM, there is a solution for Postgres. Let's review these practical solutions, including built-in caching, sharding, and in-memory data grids like Apache Ignite.

About speaker

Denis Mekhanikov
Software Engineer at Facebook
Share

So my name is Dennis I work it's a great game systems as a client service lead. How many of you know about gridgain or Apaches night? All right. Nobody is perfect. So I didn't to make some kind of protection, what it is. Well, basically that's an in-memory data storage. So that's basically it says, with a transactional consistency and some additional functionality built on top of that storage. Everything is kept in memory and the Charlotte across multiple know. So you

can scale your data integration with the external data storage and support. The sequel will come to that. I'm not going to teach you a great game. See how you can boost your progress, installation without use of the game. And when we come to remember that Greece will see that the in more detail. So, This is going to be our agenda, somehow doesn't have its first, we'll see how you can benefit from cashion. What is the cash in the first place and how to make use of it to in your car into

it? If you have for installation, how to make it, work better, faster, internet decrease, your relationship your operations. If you want to store more data than one single note, so I can I compete. And finally, we will go to in-memory data grease and see what grade game is. How you can use it even though if you only use postgres, how you can make it like work faster together integrated with with great game. So, let's start. Happening. All right. Yeah, so what starts with catching techniques? So we'll see what the idea

is to preserve. Some hot subset of your data in more performance and usually more expensive for the data. For example, if you have for the storage, then it's you'll probably have four more expensive than smaller, set of memory, which is faster. And it's better to keep for the most required data in Bethlehem in the memory. And this is what to that idea of cash and is about, there is a table like a column. You can find that latency of operations, you can

see on the top row that time. Or for once, if you say, All navigation and go so you can see it. Like, what are the latency of different to access times? Two different date Story show on the right column, you can find the scale play Cincy. Like it wants to be a cyclist taken for one second. And you can see how it's related to all different kinds of data access is related to Once. If you cycle, is it good to memory and sell a memory Centric Solutions, you stay in nanoseconds latency. And this is where you want to aim at it. So

you're not going to take him to check on flight cash CPU. Memory. You can actually benefit from storing data. Is it in memory? For example, like if you have progesterone, if you get stir doing like a hundred nanoseconds, you can also make use of entertain which allows you to work with your persistent data. But with within the name is, I can't even see. It's like a great technology but I'm not sure that it's always performs to Perfect 2 with all the work clothes. For example, I'm not sure about the latency of the threads Random Access but you can actually shoot

for their this. So I can kind of relationship while working with me. But as you go further or if you move to to access, send the list of this kind of phone. Powers from minutes and this is huge and settled idea. As much data as possible answer actually need to use any external to. So to achieve that, it's always that. For example, if you do a lot of related Aquarius, like if you do over and over again, you can just like, Howard or remember what the result of all that Clary, not to going to go to progress it again to request something. But just remember that, if you request

this data, as a result of side of it is that if they to changes, then you need to involve the death cash, answer it. Sometimes it's hard to tell, which cash records, you need to drop a few change them certainly They to record. So you need to introduce some kind of expiry or something like that. So it's doable but it's kind of fun. Another scene is Glen kashin which is implemented inside to oppose grass. So if you ran some complicated query and other at different steps, all of that you are first, you build a clear plan.

Executed the part, where you build, a clear plan actually takes time. And so instead of doing that over and over again, for the same for, you can just memorize it and to preserve its in memory until you just over, it's done inside. Pull. Your sister is a, it's a configurable and achieved using I alone just configuration but that's it can be used in any database and the most interesting part is I would say it is the relation Cassian or like, they take cash in or

indication that are used to actually produce the result of your queries. When you execute a query that needs some data from this, then that database will go to this couple some information to memory and if you do something related to that to the same data sites again, then you don't actually need to go to the Disco because you are already have for all this information available in memory and Your database. There are two kinds of roshes for Progress memory and

notes of floated when when you finish using all the data pages. And another thing is always kind of doing the same theme, but it's on a different player. Application layer, nothing. But I was just a player, but on that OS, operating system itself. The page is that it's Reese from dick. So are you doing? You don't need to go to the disc again every time the tree. It's hard to find the perfect ratio between these two catches for example is buffer is not able to all the time and the size of it is defined by the amount of memory is not used by other

applications and running more applications to see how your performance of your database De Grace when you get less memory in the homepage cash. And yeah you can read out for about that and it's sad that you rarely get smashing provement. As you go Beyonce, 15% of your memory if you give it to share, its about 4 because the program utilizes Westland Westland will buffer, things that you can change here if you need to experiment and to see what works best for you. So you can eat Send text with it. Some time to try to find the perfect Erasure here

and has nothing to do with the latency the performance of your query. So that you're on on a single machine, you can configure perfectly your one machine, but as you go further, you will see you kind of machines to buy better. Equipment is, my ship is much more expensive than the other than I can just buy more machine. So the other two have for that the same performance, but they actually scale and they distribute the load across multiple notes. Let's see, what what kind of requirements for a Wii system? First of all, is a strong

consistency database, so we don't want to lose it. A list of this guarantees that include want to give out the asset guarantee. So otherwise, it will be a different kind of product. I need the ice guard. She's also, if you store data across the cluster, you need to be able to actually use it to benefit from a solo. Balancing is also required when you have more than one node in the cluster, then the chances are that more or this will happen because you

have more moving parts. And so she has some kind of pale over would be a little behind it. Let's see what there is to solution for. For a part of this requirement is just triplication between different instances of address. If you have what I call one installation of progress and to perform some operational, you can just set up the same instance and two can figure that application between them. So if you change dates, in one instance will be transferred to another sure. You can't afford a price on their own one part

and it will be transferred to another. So you don't scare that right operations. March by itself. Or its, you can use actually both copies and to hear the different configurations here, you can set up. For example, synchronous, replication. And in this case, You will need in order to finish decorations. You need to go to both instances and to change the date of there before you used to finish. So you can actually get the overhead for rice, but you get the failover because if one of their inferences fail than

you can use another one and your research operations can balanced across. Another downside, is that you don't increase your total capacity of your system. It's still stays the same as a single note of it because both instances will have the same day to Santa and you don't distribute, so your data cross them off for nose. But if you want to address that issue, I've been there is a solution in general. They look pretty similar products, or as it has data, which has been acquired by Microsoft. And

as far as I know there, there, there was suspicion of this morning from Microsoft about the presentation, fortunately, but I guess it was about that. Yeah. Hyperscale product in Asia. And so, that's so that's why my idea behind herself. As you have dates on those, those are just alike. And on top of it, you are set up an additional infrastructure like coordinators transaction managers load balancer. And in total, it gives you all the guarantees that we wanted to achieve. It gives you a seat load balance, see him at like 8 and see if,

if something fails, you have multiple instances of the same scene, so you, you are cheap for failover ye supports the Chardon. If you see that, Are you sure it's a limit of your a single note capacity, then you can just add more notes and give you your able to store more data in your system. Is that the song, the downside of it it's hard to to set up for a relief. You go to the creation of one poor girls XL. The first thing that are they writing the other

tutorial is that is going to take some time to configure. That what it means is that it's painful to actually spend the time until you taste, like them up steam to like there is a lot of consideration or quiet and not such a beauty in in memory technology. So sometimes you get license expires because so you need to like to not do something. Placements, you need to go to discompose dates of the dismissal in the memory. So if you'll send me to configure the date announced themselves properly to make them working up

to my waist is a good solution at, but it requires like a lot of time learning about that and said, maintenance is like a pretty tricky but it is good enough. If we have bones grass instance and taste, either comes feet enough data in it into self or a it says not performing tonight, which is usually a case when customers, and come to us. And they quite often say, well we have a relational database and it's not preferment enough help us somehow. So this is how we how we working in this, in this situation. This is what

game is the middle section is about a, what, what, what, what Apache tonight is there, you can see the bases later. It's a store. And on top of it, there is a bunch of apis available for your application. There is a key values of API sequel, set up to services, so, or perform computation. Everything is built on top of for them. Basic, where is a storage that you can configure external storage your main frame that you're using is in your system. Or it can be like a dupe for or can be nice nitrous systems because there is Portal Knights persistence,

so you can split configure Samsung that works with disc. It can be a relational database, it can be. So the blue squares is that what Greg, a proposal. So they built on top of a charlatan security and install an introduction. What Make use of it in your system and on top of it, you put cash in layer of them or which works purely in memory until Charlotte across multiple notes and you use it in your application. So you do kind of migration to use them instead of us and to the

night will go to be there when it's needed or like a makeup date. That's one of the ways, how you can do the migration. Another way is just enabling that the Persistence of the data into into answer. Kind of fun. Forget about your old mates storage. That's usually customers go the first way because so they are not comfortable enough when they see the new technology and they comes trust. All the data is going to be stored in that says single place. So usually start to visit like when they get old enough we was that they just tell my great effort into

to the Socure sure how the interactions happens. You have some betasept, it's a Sprint across. The middle section is, is the great game. And if something is missing in that, in memory storage, I can go to to the external rotation, make a read-through request, some keys, missing in the cash. And later, if you have to go Center than they were, just a usual way from from the, from the big stories that you can see, if you change something there, then we can go to the external storage and change it to in there as well as considerable. If this Behavior can be Tunes, you

can like batch Update Center This behavior is available. You can like not even enable Waze and just work her in memory of by loading, the data first and keeping it in memory. And also, there is support of obstacle is Nancy. 99 compatible. It doesn't have all the specifics. That progress have sold. A kind of migration will be required. Well, usually it's not a huge deal but somehow to, to make it to work perfectly with with. So it was about you tonight because it's a distributed data storage and everything that you keep on the single machine

are all the cars that you're on the single machine. Not always map perfectly on a distributor story but saw like some basic functionality available. Yeah. Well, what's important to know about? I distribute the storage is that you need to take care of a betta. Collocation, if you, for example, have four tables of cities and countries, it makes sense to keep all the cities from the same country on the, on the same note. So you need to make some how related data be stored on the same knows. Otherwise I say you will need to make her a lot of for cross notes or request to to, to to, to to, to

do with Samsung. Like a business that request. It's, it's true for for any distributed system XL. Also does some of this kind of for a vacation and yeah, it's pretty important to her to make use of its in Patrick night as well. This is how the process ends are there in the Preston of requested is implemented. First, the Nelda to did the request bills day execution plan, which will contain two parts, it's a map introduced. Like, my prayer requests are sent to the note. So that stores information, this is

executed their return to that reducer and so the final result is co-produced and to return to the user, That's always so kind of like my pretty special. If you don't consider that fancy Caucasian properly you will need to make a cross, note request on the map station so it will be a lot of communication between now. So it's important to make sure that all related data is stored together. Yeah, and it has no guarantees to say it has strong consistency. Yeah, if you're curious, if you can look up or how fast can meet you, a protocol has implemented. We have

it in and say, it works even with external storage. So you can have transaction fee for one of those pale and tecumsah. Write something to the storage. Yeah, you can roll back from section. So old the Grinch's. That's a normal of my assets. Database is also available in the Java API support for sequel is on his way. He's ever have it in better. It's not like, production-ready we are finishing it up, but Samantha mvcc in concurrence control that allows transactions for a sequel by the

month from the security, you need to keep it in mind. So you need to make some looking or just to use them for for for simple and if you have thrown database and it's then as well as coordinator of the transaction will first go to the external database ride information. There do that on there on the clock. And yeah if something goes wrong then the transaction will be rolled back from the external data storage but the idea is that you can get the situation. Either we can formation is committed to into ignite, but not

to external storage. That way or like, one note to come eat something, and Moscow oldest cases are, are supported them to that. Correctly handled. And now it's time for demo behind this demo will be like how to start working with by Suge Knight if you have her database in full dress. So I have a database downloaded from the internet. There is a sample sample database with the cities and countries. I'm fixing the Voltron, I mean everything, the whole transaction or not it does not guarantee that the

transaction will take me to tell open up some holes where you can see data can be there around 1 or between the second phase of the moon still, see some inconsistencies in the data. How do you, how does ignite Victor not true because entries that are committed to participating in the transaction of a locked by by the transaction and until you come into the transaction that very warm. So I won't be able to give it to get information from it. So I'm going, if the committee face hasn't completed yet, then

you won't be able to access the data because And tell Guerra, give me an impossible to get. If you read both, postgres and organized, then it's possible to get different results. So I can under condition under a race condition. Well yeah, usually it's the excess is done from one point but I can you if you access only ignite, then I say impossible to get to get to any inconsistencies. Because there is a single entry point is when you need to access both like fashion layer and the progressives itself, usually the

migration to do the system now like to the cash in layer in the first place. I'm going back to the grid. Multiple, is it multiple? No, the closest entrance is on the right. If so I can a single pole dress, you can have multiple instances but within a single transaction you can use only one that's like a limitation. Yeah. You can configure different Integrations with us but will be able to connect to know it's late, will be the connection will get down next to there by that coordinator.

The single will be open today. Alright, so we're running out of time, actually wants to go to the demo so that the date of sad that I found on the internet. That's is a cities and countries. It's actually a small and I'm going to show you how to boost your performance. I would like Ashley to show how it's so easy to start working with Ignite. So here are like three little bit, okay? Can you see what was written there? I can increase the font here. So yeah, the

table for example. Yeah, there is some information about cities. Are there named country District Center of population into your ID and four countries, where there is some information that we are going to who in turn Ivan's to start working with that word, that there's a two-week-old, what counsel I deployed on my laptop. It's actually available, but by the Linker Council good game, so I grabbed it the same thing, but I wasn't sure that I would be able to access Internet. So I just acquired the top walking it. There is

a possibility to install on your laptop or something. So there is no Imports on the database. You insert, you put some credentials to your database that you want to import. You just click noise. The scheme has that you want to import and say yeah there's some configuration. If you don't want to, if you just want to try out, ignite Sorento. Don't want to read about that. You can just keep it as it is, but there is a possibility to consider her about the performance of the

guarantees that you need. So we're going to have the cluster out here in the configuration stabbed and yeah, I'm going to change anything. You can go through the configurations that are important for from our progress and all I care to somehow but yeah, but it's not required for the first, right? You can just click share, saving. Download. And after that you'll be able to attend important that project Maven project will be generated until be able to open it in your idea into IDE and work with it. I did that before the presentation. I'm going to wait for a while,

their idea is doing the import. That's the thing that I just downloaded for, from what console, and expected it. The only thing I changed here is in resources in the secret properties. I changed this properties to make the connections are available for installation and she's driving us into their driver. Their story to read me. The tells you what, what, what to do. So you won't be lost while I'm doing that Direct Auto in your classes and configurations. So,

What is what's your rates through is like the XML configuration of Eden. I didn't change your single line. Secure. It's like eight configures that connectivity between. Now if you start more than one now and I can figure is that connection to Old Grist that lets you can specify in in the properties file. So let's start the few notes. I'm going to start to now it's just in my laptop right in from my ID and we can see that to serving those are running and now we can go to that

load package and start loading the data. There is some codes going there, a load cash on that. This is the date of saving the heat in the night so we can see. The older Imports is finished and now we can go to web town so we can see that they found my cluster. And so you can see some metrics for example, in the board here, you can see that you're I have for two nails and there are some cases configure to with the names. But here was so we're going to do is run some sequel Clarice.

I prepared some Chris to run on the cluster out here we are looking for for the most populated countries with the select. You come execute it to either in progress in the united world will be the same sodas. Are the results of the lads into actually to send to boss knows that requests are executed their ass and back to the reducer and my dad. That was all just aggravated. So, this, this is the result of this execution. If you can do the same from Council begin, come there. That's deployed into cloud. And you'll have

the same behavior, see that you can change something. This records, which is also available in both grass. You can find it here and for example, if you want to change something or something in that, I can do an update query and April it to syntax error here, that's awkward story. All right. Yeah, you can see that the that population changed and if you go to Supposed to be in changed in progress, all right, through our semantics is achieved. If you change town or something in an ignited, salsa changed in postgres, and you can also like deleting certain

you and Tracy, they will be a propagated to the purple dress. So yeah. Let's excellent basically. So thank you. I'm ready to answer my questions. Step. So if you're using a patch ignite VIII, eventually have to use the bills and I'm right because we will have to look at Ramapo State as well. And then put up a Checkmate, is where to buy a note to use a solution for example, cited and released into memory and just then we have the advantage and we will, well it in this case, you will still have the

eccentric solution as far as I know. You come to reconfigure, the bulgars to keep everything in memory of sound workout. You, you need to do to do songs like the initialization to boost her ever since from this girl, from stars or like somehow can figure the ratio between them this capacity and memory properly. it's like a it's doable Replacements and we well, we see that that there is no chance that you will go to to this cancer. Get the latest spy consist

is highly configurable. It's flexible and it's hard to find the proper a configuration that will make my car system would work out perfectly. So it was easier kicking my seat belt with my sequel with Oracle database, be stored in desk so can we can figure it out. So that only the tables with three smaller on which we were born. Plastic wading goes through night and day. For the other data those can use night and for the rest of the tables candidate, Calabrese directly route to go. See if there is a

possibility to change them ratio between the different base, original sin ignite. So for example you can locate a huge data region for some certain Cassius in ignite and two smaller data Region, 4 4 4. External memory says it's like you can configure how much of wish 70 Cucamonga. Well, in this case you can actually use postgres directly if you don't need to. 12 questions. Yeah, I think we can consider continuing

Cackle comments for the website

Buy this talk

Access to the talk “PGConf India 2020 - How to boost & scale Postgres - Denis M - GridGrain”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Standart

Get access to all videos “PGConf India, 2020”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “IT & Technology”?

You might be interested in videos from this event

November 9 - 17, 2020
Online
50
21
future of ux, behavioral science, design engineering, design systems, design thinking process, new product, partnership, product design, the global experience summit 2020, ux research

Similar talks

Nikhil Sontakke
Contributor at PostgreSQL and Postgres-XL Global Development Group
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Kaushik Iyer
Software Developer at Newfold Digital
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Abhijit Menon-Sen
PostgreSQL Developer at 2ndquadrant
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video
Access to the talk “PGConf India 2020 - How to boost & scale Postgres - Denis M - GridGrain”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
842 conferences
34129 speakers
12910 hours of content