Geir Magnusson, Jr. joined fuboTV in early March 2018 as CTO. He was the Co-Founder of Sourcepoint Inc. and served as its Chief Technology Officer. He served as the Chief Technology Officer at AppNexus Inc. since April 7, 2014 and served as its Senior Vice President of Engineering. Mr. Magnusson served as the Chief Technology Officer at Viggle Inc. (formerly Function (X) Inc.) from May 2011 to May 14, 2013. Mr. Magnusson has a proven track record in building scalable secure platforms for entertainment, e-commerce and the financial markets. He served as Chief Architect and Vice President of Platform Engineering at Gilt Groupe. He Co-founded MongoDB, Inc., and served as its VP of Engineering. He is seen as both a Technical Executive and an international leader in open source software.View the profile
Sales Engineer with a demonstrated history of working in the high tech industry focusing on Media, Technology, and Entertainment solutions.View the profile
About the talk
Creating a high-reliability, low-latency optimal video-streaming experience for audiences is difficult. And creating that same experience for live sports is a monumental challenge. Learn how independent live TV streaming service fuboTV built and stabilized an OTT streaming platform in under 18 months, achieving +100% YoY subscriber growth as well as international expansion. fuboTV CTO, Geir Magnusson, and Google Cloud's Neeve Nikoo will share best practices/challenges on microservices, cloud media architectures, and performant cloud video systems.
Hello everyone, I'm Nico. And this is a gear we're here today to talk about fubotv and how they become a great streaming provider in under 24 months with Google Cloud. So just one more background. I'm a customer engineer with Google cloud in New York the meeting retainment division. So we're focused partnering with me entertainment clients. I come from doing this an on-premise world and I'm moving to Cloud. So kind of have a a good background in the space.
So advertising background in a video background if you remember a company called juice some number years ago. We were trying something similar. Just a quick agenda. Today. We're going to go a little bit into kind of video distribution different problems and considerations that we see in the market today and what kind of talk about Google Cloud offerings to that. There's a couple different solution Pathways that we can go and then we'll talk about kind of food boat and
their case study in kind of their approach to streaming and going to know how they were successful what their architecture is best practices and kind of talk about their timeline to current date and then some exciting stuff that they're going to be doing this year and moving forward so So why is live video so difficult how many people here have watched any type of video stream and it's been buffering and you waited on either a phone browser. Yeah, I would say everyone right
so like why is there actually and there's a lot of things to consider? So just real quick to kind of level set and kind of get a good picture of what's going on. So with live live video distribution, right? There's kind of these different components that you have. So first you have to actually require all your different signals right supposed to be a satellite camera fiber, right? There's many different ways to do. So then, you know, you're going to take that into an MPEG
transport stream and you're going to do a bunch of processing and coding transcoding right having your digital Rights Management logos that are needed right and kind. Get the video stream ready to actually deliver then now how do I actually deliver this globally, right? So I need to I need a partner with many CDN providers to then distribute to all the different types of device is write. This is a really really really really I'll just keep saying that high level of kind of the end-to-end video distribution workflow will go into more detail and Technical death, but it was kind of wanted to set
the stage, right? So what are all the different things that we have to consider? So this is been an industry that's been around for quite a number of years broadcast for this. This has been done via satellite where I was set fixed amount of bandwidth and it has been new protocols that have come out, you know, every 5-10 years or so if a sweet compress the data, right? So how much how much data can I fit with a with a fix bandwidth rate right now, there's many different devices so that brings a bunch of different problems and when I'm sending video over the internet I
have to account for all the different devices the different video formats that are needed based on the device that I have for the picture quality, right and in that workflow in order to do that. There's so many different partners that you're going to have to actually go through with so I can fool both case, right? Like they have to acquire all these different types of content from many different providers, right? So there is CBS. There's Fox there's all the different sports streams and that's just on the content part right not even including the you know, who they choose to partner with on
the video processing side to CDN side etcetera, right? So many different parts in many different companies that are in the entire workflow that can really, you know, make this a pretty tough problem to actually solve and then not to say how much bandwidth there is right. Now, there's different band with that each part in that work flow. And then at the end of ice right like I could have really poor internet because I'm a cheap guy right? I only have 20 megabits per second. How am I going to get high quality video? Somebody who's paying for the one gigabit line, right so many different
forms that we see and then the coverage area globally, right? So there's going to be people where there's Me. Maybe not infrastructure that is really best served in that area. So, how can I serve someone in you know the middle of nowhere verse a large Metropolitan City. These are all kind of different things that we have to consider when Distributing live video. So we're can Google help so we have the largest cloud Network. I had to show the beautiful scaled slide that you probably seen in 15 other different presentations, but in all seriousness, we have price of network
over a hundred points of presidents, and we really really think this can help with scaling large video, right? We have many different applications and I'm sure we all use and so we we really think we help with scale especially with live video distribution. So there's a couple different solution Pathways that you can go when trying to solve the problem of live video distribution. So kind of the spectrum is infrastructure-as-a-service. Can you see it on the left hand side all the way to kind of software-as-a-service. So we've seen multiple clients and partners of ours kind of do
both. It just kind of depends where you want to turn the different Sprite. So on the right hand side, we have a products recalled in the bottle, which is our video processing and kind of video processing sweet Cena NBC and CBS kind of use this before and then on the left receive different partners such a screwball who use our infrastructure-as-a-service solution set. So using different things like DK compute engine and another other gcp offerings that we've seen so just depends as a service gives you kind of
the most flexibility in your deployment, right and Can kind of really control different parts in the video distribution workflow or a software-as-a-service kind of gives you what you need, but you have a little bit less control on that. Right and we kind of see this with a bunch of cloud offerings today with serverless, right? And then also, you know compute engine self. There's just different methodology that you can do. So and I think that's kind of what makes Cloud a really unique and fun place to be we kind of give you the tool sets and you kind of can choose what makes
the most sense for you and your business in your application. So I kind of set the stage talking about you know, what the problem is different things that we have to consider throughout the video distribution workflow and kind of the different offerings. We think that gcp can help with now I'm going to hand it over to gear kind of talk about, you know, fubo their Journey on gcp in the reason that they chose a specific architecture in different components that they did. Thank you for the signs, and
I'm flexible. So how many of you are? Cord cutters that that get your TV consumption not from your bandwidth provider how many of you are fubo customers? All right, great. Great. Thank you. I want to So who bow is a OTT streaming service? We are sports focused. We call ourselves the world only Sports focused. It really comes to our origin as a provider that was delivering for soccer. And I think the the name came from a riff on on football division and VOD to you and we stay live television. These are the channels
that you would get from any other provider Fox Turner cetera sports stations, like like being Sports, you'll get NBCSN NBC stations. We have a locals. So just your local television Affiliates for four Fox NBC. We also have Regional Sports networks. So I don't know if nesting in Boston or SNY in New York for your for your local sports and it's available on on Anything that you would want to show it on we we are trying to make it really great in-home experience. So you can get it to your smart TV or
your your various TV connected devices like an Apple TV or Fire TV on but you can also get it on the web. You can also get it on mobile devices. I'm pretty much anywhere. You want to be as long as bandwidth? The Little Dipper backstory will get into some technical things in a moment original product Vision was to aggregate. For professional football clubs or this is there at soccer for anyone who's sorry? It's a foreign sport. That's really popular. What the clubs were doing is they had their on their website, they would have content specific to their team. So you could ever 10 and you can see,
you know game replays highlights interviews backstory kind of stuff. But you know, a lot of lot of soccer fans like multiple teams. So what beat what food I was trying to do was bring it all together one place where you can get this content for for all of soccer and it didn't quite work because at least in the market sense because Sports bands like live sports write anybody here is a sports fan. Do you tend to want to watch it live if you're willing to record it and like watch it next week live, right? So we will have to bring in live
streaming and while the OD is fairly straightforward because you can get the ass that ahead of time you can transfer coated for parrot, you know quality control make sure it's a good experience for the consumer and then put it out in your in your distribution system live is is kind of a real-time thing right is being broadcast when I say live I'll also use linear interchangeably for me live is actually True Lie when people are out on the field doing something or you know, like like an award show or something. Where is actually happening in real time linear just a general idea that
we have these dreams being broadcast at us and they need to be delivered as close to real time as possible. We cannot be in real time. There is latency physics and architecture by John. Adding is linear stream became a really big change for the business both technically and as well from the perspective of the business and I'll get a little bit of background on a business where we're cable company, right? We're no different in some ways. Then whoever your cable provider is we have to license the content from from the content owners box
Hallmark, whoever that that the cable companies do witches. We have to pay on a person scriber basis not with you'd but just if we have a quarter million subscribers, we will pay quarter 200000 times whatever the group per subscriber fee is just makes a pretty tough business because a lot of money goes into purchasing the content or licensing the content. I mean, in fact in our business, we don't quite even break even on subscribers fees subscriber fees plus a little bit more are giving all to the content owners
and our business is about making it up. advertising and other content up sounds on technology side. This is where we going to go into it in a moment adding linear. It was a really big difference right there is The plane is always in the air, right? You can never stop you can never pause it just has to work and we're competing with a system that you know has been over 70 years of development behind it and is fairly stable. Right? When you turn on your television on a on a serve a legacy
distribution system, it works and it works really well and for us to be successful, we have to deliver that kind of consumer experience without owning the infrastructure without owning the path the network path to the user and it's a bit of a trick. Apparently my slight change of self. Why is it hard? I think it's because we're all figuring it out. Right? Certainly. The content owners are partners are figuring it out. We are a different kind of place for content to be pushed. We have different requirements. We're in different places, right? We don't have satellite downlink
infrastructure. We don't have broadcast dinner's we are in the cloud signal acquisition partners are people who specialize in transport of getting it from the content partner to the cloud right? They are also figuring it out because Again, it's kind of a new thing to get into the cloud into in to integrate with these with these other parties are Cloud partners are figuring it out. Right? We do a kind of workload. That isn't very common in Cloud environments. Some people think little crazy to do in a cloud environment, but with the variability in multi-tenancy that you're going
to run into but we make it work CDN Partners same thing. We're certainly figuring it out and then consumers also are are part of the chain. They are effectively part of the distribution system in that there in home networks are part of the delivery path. They the different kinds of devices that they that they would need to to be able to have a good OTC experience is also something new to them. I will get the technical stuff in the moment. Our timeline is were founded in 2015 entirely soccer Focus we talked about how we went to VOD adding linear, but that really
wasn't a success in the market either because soccer any sport is sort of limited in terms of time and limited terms of interest rate. You have your season baseball season. I guess it seems to go on forever like like hockey but football seasons in the fall or early winter soccer season is limited. So what we need to do is to make it something that you would subscribe to an ENT fall year-round. He has had other content keeps changing. I don't know why but yet if I just keep jumping So Gallant Fox NBC read CVS, so now starting to
produce a fat content package. That's really for the whole family. We launched Advanced DVR, which is kind of a table Stakes if you're going to have a OTT service hit a hundred thousand subscribers 2017. Really focus now on on building a business out of it by adding server-side ad insertion which is sort of the main month cessation driver for the business. What we do is on every stream that's coming to every single user. They're all individuals streams and were able to insert advertising on a per-user person basis will talk about that later. Apparently.
It's a big event that I joined the place but I've been there for about a year. We were the first to do 4K streaming Ott. And we did it with the World Cup which was which was last year. That's a large event and played mostly by foreigners soccer. How are you going to soccer fans? Okay to put football and soccer we're going to Interchange that just we're not confusing anyone. So and then where we keep adding new content to the package, so How many people know how streaming really works it's it's not a continuous stream of bits being sent out you write. It is really a series of files
that are just being loaded over and and and bufferin then played at you. So I'm going to get us in the background talk to you about something called HTTP live streaming. It's a fairly simple protocol. Some would argue one. That should never have made it out of the The design room right? But the way it works is we have a source. Usually it's a pretty fat Source 30 megabits per second 50 megabits per second depending on the quality of it. It is encoded and what that means is we take that that sort of mezzanine
feed and we turned it into multiple bitrates. So there may be a 10-5 a 3-1 750k whatever you choose to do is sort of the variant ladder. It's called and the reason why you do that is it allows the player to based on the trouble is having loading content switch the different bitrates to keep continuity of the stream. You're going to lose quality of the bitrate goes down, but you at least get at least get the experience. Know what encoding is doing is creating little files that are called manifest. It's a text file that is a
list of the next series of video segments that are to be played by the player and then video segments. Are are Just Junk the video that are being produced and then dropped into storage place of these would be say S3 bucket somewhere or I guess I misspelled the next manifest take the Manifest and it starts loading the video segments when it has enough of them. It'll start playback within a keeps this process going while the videos playing asking for the next manifest every 4 seconds. It asked for a manifest and the benefits keep getting updated as new trunks are
being created off the encoder and Playback Eckert's right to this is fundamentally how Rachel s works and Ash is very similar to a different different stand. But it's basically give me a list of things to do start playing it keep fetching keep to give me the list and tell him I'm Apple thanks you for your exclamation. So I feel like I should be selling something here in the floor anyway. You can order the obligatory how this Maps into Google services slide ingestion is is on the networking infrastructure who has an excellent networking infrastructure. So
we bring a lot of streams right into us that way we do the the encoding of the transcoding using Google compute. We are dropping into multiple storages for redundancy multiple regions and and delivery Network such as you know, Google CDN is used to front speed the origins and devices pull into the way. We just described with your last lesson. Now it's what we do is a little more complicated than than the previous slide. There's a couple of pieces of this we've got sort of third-party inputs. That
would be content is the biggest one right there. Is there some content providers up there we get metadata because it's part of any television you want to have a guide like electronic program guide. So we need to get that information and that comes from a company called Gracenote Tribune Media Services is the subsidiary other kinds of data sources that can come in and then there is third-party services that we use for Bailey's are non-core things such as subscription management and payments. More than that in a sec. Then inside this speaker box is really what we
do. We have do you think of the platform divides into couple of pieces? There's video management delivery there is sort of program data and Discovery and then program, data and Discovery and then user and subscription management are the three big chunks and then there's finally the client application that we also build so we build apps for your mobile. We from her web obviously and then we're building smart TV app Samsung is about to be released in data next couple of weeks. And then Apple TD Tupac's cuz I called them Apple TV Roku
sticks Fire TV, whatever. What we're doing is actually a lot more complicated than the original hls slide for for bunch of reasons. First of all, we have permission and Licensing to worry about and not only is it are you allowed to access content? At all but is very specifically what content and a lot of this is based where you are the two things that fold into this or that the primary thing that folds into it is forthright management sports are interesting in that sometimes depending on the
sport. You're allowed to see your home team when you're away, right? So if you're a New Yorker you're here in San Francisco. I think with baseball it's called couch rights meaning where your couch is. So if your couches in New York, you can see anything your couch can see no matter where you are. I think NFL has something called the hotel rooms. Where is wherever you physically are so I couldn't I was at I couldn't watch the New York Jets, but Like why would I?
Just like the mattress has a big sign in New York. It was, you know, Welcome to New York home with nine professional sports teams and also the Mets. What are things we have to do is where you are create streams that that's witch based on his Rights Management stuff. Right? So we're going to go into a little bit of death when we're creating a video stream. It's actually a personal ice cream for you. Every time the player is asking for that manifest. We're not just serving of something out of a out of a CDN. What we're doing is a rights check, you know who you are and then we based
on Switching information that we have on the incoming streams synthesize a channel for you based on whatever your rights are. Whatever you're supposed to see to every 4 seconds. We do a computation for you. We get what the Manifest are for whatever streams could be contributing to do what your channel is. And then we write out of manifest specifically for you in return that Tell us what we talked about me say the real-time video stream construction and it will go to that in some def. We have to manage DVRs. That's a talk
unto itself. The rights metadata is is what I was referring to is how we construct these dreams encoding storage. We descend delivery optimization. What we can do is because we're also as we're writing is Manifest you have the option of choosing where you get the segment from right? So if I if Google CDN is the most responsive where you are at that moment we might write the URL to pull the segments out of Google CDN, but if you were maybe fastly was better for you weed then use fastly, right? So this gives us the ability to optimize the
consumer experience to go for the lowest latency possible. It also gives us the ability to easily steal around a CD on when is a failure and you know, like everything else they fail Progundating discount program, data and Discovery, we do a lat insertion personalization is something that we've been really focused on and it's really there the area of main focus for us in the consumer experience going forward we get this metadata schedule metadata, and we have to process it normalize it and then make it available to the front ends as well as internal systems for
channel construction and then subscriber data and in user management wear subscription business direct-to-consumer, we have to take credit cards we have to do something, you know billing music early for that but that that section and the only thing I'll add is all the different partners and components like if you look at the slide and just like the number of people the number of companies that are needed a discount. It just puts it in perspective like on that slide that I showed how much actually goes into this and we're going to go to technical that song.
I'll talk this isn't how many moving Parts there are advertising producing videostream personalized. So we usually see pissed off about a Google conference. We are primarily Google cloud and always have been. It's the wizard is the magical wizard. I thought it was I thought it was the I think I just trying to move us along here to use personally to sort of the more advanced proprietary offerings that you find in a cloud provider. Honestly, so fundamentally were there for the computer networking
storage with refined is excellent. It's really good stuff. We do use kubernetes. We do you speak. That is one of those proprietary technologies that we made the decision to go with his excellent. It is sort of the core engine behind everything we do is search personalization metrics analytics Cloud store for The video segments we have a lot of stories about that because it is a general-purpose storage. And as you can imagine we want to be able to get the video segments as you receive them from the encoder to discus fast as possible. So we
don't lose them. But at the same time you have to be available for reading immediately. Right because it's a live stream. So very sensitive to the latency there Tuesday to store as really for manifest in historical manifest storage. So I can talk about this again. I just want to do as a reminder for this works. What we've done is be taken at previous diagram and made a little more complicated because of this idea that we have to be synthesized in channels based on a bunch of factors one factor. I didn't mention is especially if it's important to the concept of
overtime. Right? Nobody can predict how long the game is really going to go in any sport that allows overtime so very often what we have to do and most people are watching it with except two things like be and where you're getting Sports specific channels on your local affiliate write your local NBC station here in San Francisco and they have a schedule that comprises local local programming which might be like Source number one and then there's the National Theater to be sourced number to and and what we have to do is we need to we have what's called study signaling information to
tell us when to switch between stream, so You may have local programming up to the start of a baseball game. And then you switch to the National feed to get the baseball game and then you might have to switch back. Once the game is theoretically over cuz then is Judge Judy or whatever when you have overtime or sending play like an extra innings. This becomes a problem because you do not want to switch back to Judge Judy when the game is going into overtime cuz that's usually the most exciting part of the game and it produces a tremendous amount of iron from your customers like true true
hate. Right nothing is worse than they dropping off like the third overtime in a hockey game and switching back to the original programming that they will come after you storing. And what's happening is this playlist decision is using some of the metadata to decide which source to give you when you ask for a channel. So in terms of a flow the player thing to an authorization service, hey, can I watch this and there's my credentials and and The off service gives it a token back. So sure here give this to the playlist service the player with the playlist
positioning system. Give me give me a manifest for the channel on the player decisioning system will will based on where you are in time and what's supposed to be happening choose the store stream and provide a manifest to you that tells you what the next I can also rewrite the URLs to get them so we can choose among the set of see the end that we work with which city in Canada liver to you. And then the player says to the right CDN CDN, then gets it from origin is not that interesting that part and then you ask for your next manifest.
The players list of security system can choose a different stream based on whatever is really supposed to be supposed to see and then you may choose even a different CDN because something happened with the first CD and that you're using this is really how we produce dreams and if you think about it, it means that every single stream that we produce is unique. Right. I mean they're going to look the same but they are in theory unique because your Computing them specifically for a given a given player. This is a business where we take all the money. That's the
consumers give us and then we take some other money that we get from their advertising rvc's and we give it all to the content owners write. This is not in itself a sustainable business model. It's it's a very difficult business to where the financials come in is really advertising is the core monetization for the company and its two is an interesting technical problem. I used to be to see if you owe back taxes when I left off Nexus. I thought I could have done my time and advertising. I will never go back to this and I find now that really most of my attention is being spent on
advertising and Google knows nothing about advertising so we don't know anything about Google. As he said it's every video stream is is unique an independent and that gives us the ability to produce advertising that is unique an independent three Chooser, right? This is kind of the though the holy grail for advertisers and to be able to Target you specifically and it's going to be watch a web web display where most of it is below the fold anyway, This is something your TV is up there. Right?
And this is not a distracting thing in the corner. I mean, I'm not going to apologize it for television works. But at some point you go to commercial break in the commercials are coming to these are ads that are showing right there is human that watch them. Do we can determine real time? Not A New Concept for anybody that has been doing advertising last 10 years before television. It is kind of a new idea and honestly the television buyers the people were buying advertising for television. Don't get it yet, but they don't really understand that we can
Target to the individual user. So how does it work technically? So if you remember what we talked about before the player talks to the playlist service with gets manifests and the cycle continues what we do is we put an ad Stitcher. In front and what a nice picture looks like it's at the proxy in some ways and when the player asked for manifest the last picture goes to the playlist Services manifest is produced and its return is coming back. What to get an advertisement there are two points there might be a little line in the Manifest that says she was out here and you can do it for 60
seconds. That's a 60-second add break. Now, they're still was going to be video segments listed. Those are the video segment on the original straight. Right? So that would actually be the advertisement that the content owners say CNN has on their Channel. But as part of the agreement, we have the content owners. They give us some amount of time for our say one to two minutes to 3 minutes. So when we get our ad break what'll happen, then is me to tell her we'll look at the Manifest manifest sees that an ad break is coming and then it makes an ad call. Are the calls we have a
proxy because we didn't want to connect we wanted to be able to. We have some control over in real time. We were calling which ad server do want to work with and logging and all sorts of stuff. It's obviously a good idea to do and what we do is we choose among the set of servers are at Partners three-wheel. You probably heard of it. It's owned by Comcast. It's a it's a very popular very big video ad server spring serve is a lesser-known an excellent ad server IPs is is a Amazon publishing system date. They do buying
talk to CBS ESPYs other demand Partners who figure out who has an ad and wants that specific person or that specific device or whatever. Then the app slot is filled up with ads. Basically links to add and I hope I'll has been sent back up to Media Taylor to the app Stitcher. These are MP4 is right. They can't actually be inserted in hls stream because hls has multiple variants at different rates with his dad sister has to do is take the advertisements that came back from the from the a decision inside of things and have to transcode
Burien traits that we have the mainstream's on which means that for the first time it ever sees an advertisement. It can't show it right because it hasn't transcoding you Contra transcoding that fast so transcode them put it into a CDN and remembers that is transferred to that ad. So the next time that ad comes through which it will the ad is ready to play and it rewrites the Manifest so that instead of getting the video chunks that are from the underlying stream to CNN. It actually didn't just gives video chunks that are advertising trunks, right? So it's a seamless process
the player generally doesn't know what's going on. Is it there either some berries to that you can do sometimes the player can do beaconing as well. But fundamentally from a playback respective. The player has no idea. It's just getting video segment video so I can video segments II happen to switch from original content video segments to advertising segments. And then when there's a 2in basically the it's where the the playlist service was saying. Alright add break is over. You got to go back to then me detail shops doing anything and just just keeps blowing through.
But lot of moving Parts a lot of lot of things that can go wrong if any of you have have work with advertising Technologies. I need our So I think that was it in terms of the overall Tech part of things that he just explained in between when you literally the game is on and then goes to half-time like that that like little blip all of that literally happens in the back end just to serve you the correct. So like it's a very very quick process that needs to happen in order to stitch all of that together to serve you advertise and
I hope I don't know if it's just me. It's a really really hard problem that happened super super fast like explain that over 5-10 minutes and that happens in life like that. So it's just the point in perspective kind of like all the moving pieces to do that. Sorry. So they're the best practices, you know, we've been on this journey for a couple of years we did with people. We didn't do a people normally do witches do on friends. They can control everything. You can imagine. We're really sensitive to network. We're really sensitive to compute. We're really sensitive to storage
latency on the cloud and I guess we have some best practices that we've learned and I'll stay together because honestly, Google when you working with a cloud provider like this, they are your operations team or part of your operations team whether you realize it or not and Our relationship is to the point where it really is a partnership. That's really about the last Point. Here's is They're not just a vendor right as you shouldn't treat them like a vendor if it's something core to your business and they shouldn't treat you like a customer or just a
customer right? You need to sort of being part of the same team. We have weekly calls that they participate with us and with other third parties that we work with because we're so sensitive to things like networks are networking team is on the call. It is not just the usual front facing customer Engineers we can get deep into their teams because they have a vested interest in US succeeding. other ideas is that what this is cloud systems are built for general-purpose use they're not built for us right there built for Thousands of customers multi-tenancy wide
range of work clothes. So, you know tester use case right? It's kind of obvious. We're very sensitive to latency and we have to design around some of the uncertainty relation see that we see in these cloud-based systems Cloud systems Behavior are subject to change. Sometimes just did a failure, but that's not that interesting cuz everything fails sometimes it feels like the system's actually change their there. Behavioral properties based on on I guess load like we've seen normally have no problem with hotkeys in a datastore and all of a sudden he's become
hotkeys latency increases or fail write. These are things you just have to be aware of will happen. And it's do do other activities within the cloud because these are shared systems and then this idea for proprietary technology carefully, right? Because what is a quick win today? Like I'll let me just use that because it's quick and I can get the system up is you know next to your text that and it sometimes awfully hard to back out of these things. We made a decision to go with a query very happy with it. It is a proprietary lock-in kind of thing. We are
kind of stuck. We're not unhappy we don't want to go but if you think about what it would take to move all of this data out of out of Google into somewhere else. It's not a problem if you want to think about I think that do anything to add on the last thing I think like what's was just kind of bring the point home was we when we get on like we we have these weekly calls, right? Like what is that Weekly call there's going to be like many different people that are always involves. Right? Like all those partners that you saw will have multiple calls with different partners and it's just
like really being aware of like the different services that are like segmented and who was all involved in where and I think she's being like an omelette and honest open transparent kind of, you know, not to be cliche but devops, you know, kind of culture and environment really helped I think like make us successful together because we understand that there's all these moving pieces and all these, you know, different parts and partners that are involved and it's just about like kind of like together like solving this problem and understanding where each each piece fits and I think like a
really really helped us like really both be successful and move really fast. So I do It's true. And it's I think it's great because they are world-class experts in the things they're working on and they can really get to the root of a problem when when there is one and there are problems. I miss our future we're trying to bring key core services in house will continue to build more new things on on the Google platform. Mostly cloudy. It's amazing that their marketing people at this through but it's one of
the things we're doing mainly driven by external forces that there are there are content streams that are coming into other Cloud environments not to Google and we have to go over there to be able to receive them and told them and then then we can even bring things over in personalization more markets not to Spain and then apparently there's something I know that he doesn't mean that I'm not going to tell all right. That's it. That's it. We left a little less than 10 minutes 8 minutes and says for any questions that you might have if you could do the feedback, we
really appreciate it. I think it's really important so we can kind of make sure if there's any anything to fix change wheel of Love honest feedback. If you have any questions the mics for their lineup will be here about 7-8 minutes left. So we appreciate your time. I got one question on DVR. How are you dealing with massive parallel DVR recording? lots of storage It's it's a it's a big expense. But but we have the meeting thing about what a DVR recording is in the cloud. Right? We we have all the
segments they'll and right because we're taking off the encoders and putting them in the DVR process happens in the background and makes copies right? It's it's hotworx and it's a large storage cost for its we use different tiers of storage for this purpose. Right? Because unlike the the front the first line of storage for the CD in Origins which have to be fast, but wish it was faster DVR doesn't have to be as fast. You can accept some late and see when somebody hits play on a DVR.
It can be pulled out of the sort of the nearline storage and start playing. Google has advanced auto service that checks a lot of the same boxes. Why not use them instead? They have a suite of services. They have an encoding service, right? They have a cabin at Stitcher. They don't do what we do. They don't have the Rights Management. They don't handle the things that we needed to do to be able to license this content and distributed right? Like they were there pointed out on the original Slide by content owners, right if I'm in DC what I'm going to do
and don't have to worry about Licensing in the way in the same way we do because they're the ones that are putting licensing constraints on us. So I think it's a good set of building blocks. We had to do something different fundamentally with the playlist generation and then honestly given how sensitive we are to a good video experience. We are trying to do more and more things like encoding in house. So if Lisa something goes wrong, it's our fault and it's something that we can fix if it's a third-party provider is not a lot you can do sometimes Declare just
to clarify a couple icons on the slide. I'm guessing that you use GCS for a video segment data storage, but I was curious what you use for the video stream meditative storage if that's GCSE or if it's faster or I mean like the Manifest. Things that go in today to store right now. Because we need them time ordered right? We are switching to a different system which again preserves at time order. We think it's going to be a tough guy is a Time Warner list of stuff
that is permanent until you roll it off the back if that solves that problem is well testing a solution that is based on a large Kafka cluster. Is a flat load input right? Because you got 800 channels are a thousand channels. They have a completely predictable and stable load. So that's actually easy to ingestion store. And the variable load is on the on the the the output. So how many people concurrent streams are coming in and it's an easy problem for something like Costco
cuz you confronted with a cash in front that keeps a the last 15 minutes of manifested memory, and if everybody's watching the exact same program, not a problem you just pulling another man. Thanks. How did you get a server-side connection to dfp to do your ad stitching on the for your proxy? What did you have to do to get a server side connection to dfp to include the RADS in your stitching on your server side. We just so what happened to the ad server?
What happened to his dad Stitcher makes the add call to us it goes to an ad server just a conventional add video ad server and that makes a call to something like the FP. So that might be somebody is buying Roku. Like they really want to buy Roku Impressions. They give us tags that come from DFE that server. We are Star ad server will call that just to see how do you want this and it'll return either that or not. I'll get package together and return so it's not like there's a connection between the add Stitcher and the demand Partners directly the ad servers making the calls to the man
partners and going back. We don't use the Azure ad server. Hello, are you currently using or have plans to use any machine learning services for Content delivery or a delivery? We're experimenting on on both of those. It's primarily in the area of a sort of recommendation and similar but there are some interesting uses for detection of advertisement detection of sentiment to understand what's happening in the game. If we could automatically know a game is going
into overtime by understanding what the game action is or the clock or the scoreboard it would make her life a lot simpler. Although it's a fairly sophisticated way to do it versus just having somebody watching hit a button. Hey, if you have a critical failure in the odd pipeline, what does the experience look like for the customer? Do you try and fail in favor of them all know this this whole lot pipeline are the thing that delivers the the adverse to the coastal range from heartbreak if they saw some kind of
what does the experience look like for the food at the best failure mode we have because if you remember where to take your watching CNN right CNN goes into an ad break their seeing it as their CNN is putting advertising the same stream on TV. So if there's a failure to deliver to get an advertisement or to transcode the advertisement then just the underlying stream just plays in the user has no idea something happens. That's assuming the picture doesn't do something really stupid, but they generally doesn't
It's your add Stitcher having to decode and re-encode so that it can. Yeah, yeah, so when an advertisement is first seen so suppose. It's a 30-second ad for Ford something. I don't know before and Taylor keeps a list of all the advertisements that it knows about and has encoded if it's not on that list. It will run a transcode process in the same way. We transcode the the incoming stream turn it into the bitrate variants that were using so that it can be slotted right
into whatever bitrate stream the player is playing right that means of course that the first time and is seen by by the app Stitcher it cannot be played to the user. That's the case where there is that failure mode. We we just don't show an ad in the underlying stream is playing but I thought I should have drawn and I'm sorry. This is late. Show me a little Loop here saying this is where the encoding happens. But then once it has in its cashed, it can be considered that next time decision systems that show that
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.