Duration 28:22
16+
Play
Video

Future of Data Platforms: making use-case driven... By Ketan Umare, Staff Software Engineer, Lyft

Ketan Umare
ML and Data processing infra Lead at Lyft
  • Video
  • Table of contents
  • Video
Video
Future of Data Platforms: making use-case driven... By Ketan Umare, Staff Software Engineer, Lyft
Available
In cart
Free
Free
Free
Free
Free
Free
Add to favorites
19
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speaker

Ketan Umare
ML and Data processing infra Lead at Lyft

Ketan Umare is a senior staff software engineer at Lyft and founder of the Flyte project. Before Flyte he worked on ETA, routing and mapping infrastructure at Lyft. He is also the founder of Flink Kubernetes operator and contributor to Spark on K8s. Prior to Lyft he was a founding member of Oracle Baremetal Cloud and lead teams building Elastic Block Storage. Prior to that, he started and lead multiple teams in Maps and Transportation optimization infrastructure at Amazon. He received his Masters in Computer Science from Georgia Tech specializing in High-performance computing and his Bachelors in Engineering in Computer Science from VJTI Mumbai.

View the profile

About the talk

With the emergence of data as a primary business asset in the digital age, strong data platforms are critical to the success of companies both within and outside the traditional tech sector. Our existing data platform tooling, however, falls short of modern needs. We need tooling that can not only scale efficiently to the incredibly large amounts of data businesses depend on, but tooling that is user friendly and enables faster innovation. The production-grade container orchestration that Kubernetes provides has enabled Flyte (an open source data effort started at Lyft) to become the foundation needed to innovate in the data platform space, working towards a truly scalable, user-friendly, and efficient data platform out-of-the-box.  In this talk, Ketan (founder of Flyte) will discuss the future of data platforms, grounded in his experience building Flyte. The session will provide an overview of the primary use cases (e.g. ETL and ML), users (Data Scientists, ML/Backend Engineers, and Data Engineers), requirements (Scalability, Efficiency, etc), as well as industry trends seen in the space. Additionally, he will provide real-life examples of these needs and trends from Flyte’s usage at Lyft. By the end of this talk, you should feel well-versed in the critical

Share

Hello, welcome and good morning today today interesting topic for me. I'm going to prophesize about future of data patterns may be a few years down the line when I look back and I say what I think I need to go to the Future came from what I've experienced or the last few years at a company like Lyft and how how we were struggling to find the right balance between machine learning and data and the combination of both. I'm through these learning. So why do

you know some changes that need to happen and Carol have a joke, why is working closely with users using both our platforms? And hopefully I'll be able to do some of these problems and then I'm confused. All right, so I guess everybody's of Arrow must be the machine learning from our Carnival ship produced it up. 610 and Ella Street differentiation and in the differentiation is not only in our minds that it's also in organizational structures. Take an example that list diagram is the

graph of the three big. This is assuming section of part of the graph. I bought a rocket out of the boxes are represented or models or information between the sticks to see that extremely it's sometimes they're not even genetic just add cilantro. And this is a interesting cycle that happened. Where did I get spots on and on and on and on? Constantly crossing the boundary. We lose focus on what happened and how was it with you? Remind you is actually this is

more like the real world picture that happens by phone use that for sure, but they also pretty Smart Alarm facts, which may lead to more or are they made? I need to buy that argument then stick my neck Sgt. I want to say that email is actually the super set of data. And why do I say that turn side? I'm sorry if I've made it look smaller, but the focus was to show how it's evolving you yesterday and you dream about an ex and new store that in Italy accordion my house to another level

activation issue. Transformations are more specifically to to help you train that are models. So what are these transforms? I want to let me deal with this state. Are you did you did you sleep for you are in model in some cases? You may just use the model to in for information. So I'm Nebraska. Both of these result in more data get stored in the also want to monitor. This model tests are progressing and changing for drift if you depend on the date of the house for the groundhog day. Also record them on

tree. Where does light fit in in just a plug for flight flight actually is great for the dream. But you see that in the future on the right-hand side. It's great for doing it also comes with a feature store for for off light switch doesn't it comes with definitely works with model training a prostitute. most of them so let's talk about not a big thing to be ours as I was walking with the user's we got various set of requirements from them. What I've done is a federalist. The next number

one thing that we heard from the uterus that they want to manage machine user personas that used in animal parts. We always think a little bit of a scientist and then there are the other regular product ingredients because many times most of the products today have machine learning in them and they are very focused and they want some techniques to manage software and they won't be able to take those techniques to What do all of these uses one thing was, they did not want to stretch it back

if they would have it. They would never think about a machine that team building either the same model or do you want to work an isolation? Are you going to break up with you? Do you want to work in to be live collection and they want to be unaware of each other? They also an altar resources and scale resources on demand and and On Demand creating machines is actually simpler and cheaper rather than creating machines and keeping them around for a while and the usages you're not listening. eventually wants to move to

That you probably want to work on the Russians. Mmd1 access all of these resources to deal with any UIC airline are flags in the one last thing because machine learning and data processes tend to get very expensive very quickly. The other set of use cases run around, how do I use the truck? If you want to learn how are user typically comes to let me know where they're faced with a problem like that. Once they have that they want to quit this game it up either

syndrome. Once they have scared of they want to maybe create a pipeline. Once they have a pipeline they want to run the pipeline on demand, or maybe that's how it is. the torture of ukases was around how the experiment the experimentation is interesting because when you're running an experiment misses you more back flexibility later, if you want to try to figure out which one works better in blood work because even if you had a brain stimulation test showed that model A was better, but usually they

want to keep those two close together and you want to run the results of the you also want Something special in which some of the systems today in the world schedule yesterday explain why this happened? and as you scared, I mean Till yesterday was just was not available just or sometimes you will never use the data input. You may also like if you're doing maturation and mapping you may want to change how you actually cost structure of your observations extremely expensive Google. Send message using Maps affect the cost of that pie based

on. So you may want or like how I just received road traffic information. The photo of use cases that we expect from a system. It's a cycle dependent most attractive results. and then once you've got a happy that I'm happy to go back to the why they're doing this if they have some at home and said that they don't want to affect production and now you're right. It wouldn't do it. Like this is happening more and more. I see that they would be fine for some people. It's better to write

code in some languages so you don't want to use this application to go sandwiches. And they want to read it and if they're running something a titration. Alright another tape that very happy everything goes to production for the next set of used or requirements is essentially around what happens when they want to use want to get a history of qualification get notified if there's something failed based on which user trigger them. Alright the next set of the UK's is very modern. This happens once the system.

I have done all of this. I want to add more stuff to your system and one more stuff to the system. They wish to think that I want to just be quite a bit now and probably have a great month. End the last hour of use cases around extensibility inflexibility. This is it colder than the pattern have one of them is the teachers every day so that I can be like my uterus and to do that. Enter told you keep them from how can I make it happen? If anyone will impact any of music, I want this to be absolutely

constant light made it almost a piece of cake. He just went in and I do not even know about it best in class instead of relying on what just because So my prophecy is because understanding disqualified another to deny that reason is is the new API for extending don't you mean to do actually show it to the universe and everything? the number to is Every time there's a new use get you a contrasting Kostas, Nicole. I have to start up my dependents. Number 3 we always think of expensive

but I think it's not as expensive as losing valuable. This has happened to us, right and it's okay. It's not that expensive. Okay, so you followed this is better uniforms abstraction to actually usually in our terminology. I want to say that something that is a different way of doing it sick again, if you give a gift shady and it's it's not doing anything that has an awesome job. And it is designed from ground up to the article reputable unsecured. Implied what's the username is the insured map with the

remote and once it's running infection, they can retrieve all the results for all this article execution to like a GC pressure on Wednesday at 9 to 6 months or so on and then they can reproduce anything. That's a galaxy website that you can do. As long as you follow someone important. and this Christian Journey how do they treat this like has like any other language with a different book allows you to run everything we have to do so nothing really prefer everything rare lyrics

the packages and all of this is available in the specification. Any subsequent believe in you? I took her notebook Seattle. I forgot my pin from service. And then just a quick overview on the concept. So we have my tendency an organization to project and CR-V have and workflows tasks are the elect monad thing that executes and workflows are something that actually be together and both of them are beautiful. It also understand Amazon tracking using that and you like reading on something and heavy competition. You can just mark it up and

then he will not read. And then if you want to use flight, here's another reason why you should look up like you cuz it's really wrong impression that we have more than a million times and running every month in August and returning on put on more than 40 million containers on kubernetes. We probably by far one of the largest scale. And this is possible because we allow getting out to multiple different style design and sew. I was like, I was just one of them call this our day in the exploration of the use of your life

cycle has had recently on the right hand side. And then a lot of new features coming soon to stay too, but it's growing really rapidly to be quite a bit. Please join the love to help you with anything and I love her. I'm getting Maurice how to complain on the SEC channel, please. I appreciate it. Thank you. Any question.

Cackle comments for the website

Buy this talk

Access to the talk “Future of Data Platforms: making use-case driven... By Ketan Umare, Staff Software Engineer, Lyft”
Available
In cart
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “Global Artificial Intelligence Virtual Conference”
Available
In cart
Free
Free
Free
Free
Free
Free
Ticket

Buy this video

Video

Access to the talk “Future of Data Platforms: making use-case driven... By Ketan Umare, Staff Software Engineer, Lyft”
Available
In cart
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
561 conferences
22100 speakers
8257 hours of content