Brennan works on TensorFlow and Cloud TPUs.Nikhil ThoratSoftware EngineerView the profile
About the talk
Swift for TensorFlow is a next generation machine learning platform that leverages innovations like first-class differentiable programming to seamlessly integrate deep neural networks with traditional AI algorithms and general purpose software development. In this session, learn how Swift for TensorFlow can make advanced machine learning research easier and faster.
Presented by: Paige Bailey, Brennan Saeta
It's a pleasure and an honor to welcome. Our next speaker is Paige Bailey and Brennan Syeda who are going to talk to us about Swift for tensorflow a next-generation machine learning platform that leverages Innovation. So, please help me in welcoming page and Brennan. Good morning, everyone. So thank you for coming to join us during lunch time. We we very much appreciate you taking the time to come to tensorflow world and also to learn a little bit more about on page and the product manager for Swiffer TF and manager for Super Bowl
for the room. How many of you have used before? Excellent. And how many have you attended Paris and Tim and other session their workshop on on earlier this week few hands cool. So this is a great place to get started if you've never used Swift before and it's also a wonderful place to learn how swift is getting more and more functionality at is specifically for machine learning in for an exciting time. Do you want to get started running? So today we want to bring you a little bit different kind of a presentation since you actually know Swift at least to some degree.
We wanted to really dig one level deeper and and really explore Swift for tensorflow from its foundations. So before we get into the me to talk, when is it take a moment to acknowledge a few very important folks. So number one is this work would not be possible without the rest of the Swiffer tensorflow teamfit really was a huge amount of work to make this this platform. Awesome, but also not just the core team itself for their number of other teams that we collaborate with a Google Jack's team to the rest of tensorflow & Beyond
and so we really are are collaborating together and building on each other's work to make something awesome. So thank you to everyone perfect Google contributors Beyond. We have a lot of computers that are really making a huge difference in the project. And finally those of you who are familiar familiar. So Jeremy Howard and Chris lattner co-presented release year. So to begin with I like to start with a question. Why Swift for tensorflow my answer but
still on for a long time. I really love what kind of the the flexibility and often the user experience that you got using the language. It feels it feels very understandable to read the code on the page and it reads almost like a plane sentence and it also has you know, a vibrant ecosystem of data science tooling but something that I was always really frustrated about whenever I was trying to deploy python models into production is that often you have to do a lot of reading on that wave part of the part of you know, python
benefits don't necessarily translate so well to deployments in areas. I'm so Swift is to me at least kind of a happy happy way to get a lot of the great benefits of python, but also the benefits of a typed language That's right. And so what we are going for with Surfer tensorflow is Nibley hackable platform that gets you all the way from Research into production and most importantly that everything is customizable because although there's a lot of flexibility that you need to do research. You need just the same amount of flexibility and capability
often sometimes slightly different really make high-quality production application. That's what we're really excited about with server tensorflow, especially with us with platform how we can do interesting things to optimize for productivity. So let's take a moment to ask the question. What is swept and swept is a programming language that was created by Chris lattner pretty recently when he was spending some time over at Apple since then. Chris has come to join us then worked at Google you'll be hearing a lot about one of Chris's other projects later this week
Emily R. I believe it's in the keynote tomorrow, but the width is a programming language that spoke is done that extensibility component that Brendan just mentioned as well as readability usability and kind of production Readiness. That's right. So this is Swift for tensorflow from foundation. So let's let's really dig in and let's let's really go from the foundation's so I think it's an important question to understand. What is Swift at a deeper level is understand. What is a compiler? And so as used in a program computers, right? They're sort of two fundamental
things that are sort of immovable and that like to think about the world in their own particular ways that are sort of sometimes incompatible. You got 2 minutes on one hand. I want to think about the world in a particular way and you got Hardware which also thinks about the world and a programming language is basically an intermediate point in between these two immovable fundamental truths of Technology women's to express ideas for their also designed to be executed on hardware and their different points in that
space that you can pick just one is touch points, but we think it's really interesting one and we'll talk about why in a couple minutes. The one of the benefits of having a typed language is that a lot of the responsibility that might be pushed on you? If you're using a language like python is suddenly abstract it away. So I don't have to care so much about write a baking performance optimizations into my coat immediately. The compiler can just do it for me. Another great thing is that if you look at the screen right now, you can see an example of a few commands and split it looks
very very similar to python write like there's some let's sprinkled in there is some a few indications of tight. But other than that pretty straightforward to understand if your if your only experience using typed languages is something like cplusplus. This probably looks a great deal friendlier. That's right. And Swift is a very modern and Powerful programming language. So even though so if they are statically typed you don't actually see very many types in. Advanced Swift code right here waiting to finding a dictionary numbers from strings to list of integers on the
compiler can just figure this out for you. One of the principles of Swift is it has a very helpful compiler and you can Define higher-order aberrations, you know reduces, you know of a common pattern from functional programming across a sequence of elements and you can call this Dino X. Some of the guys in the dictionary and voila, you know, the number of high order functions together without having to write types in a very seamless in this manner. So what we're trying to get across if it's Swift is almost as high-level is python. So this is actually a slide that I'm borrowing
from Chris and Jeremy pesticide lectures and at the top you have a Swiffer tensorflow model and at the bottom you actually have a python model and if you look at the code, You squint a little bit there almost exactly the same thing, right? You got a few more stealth dots on the bottom and got some more curly braces on top, but they're basically the same thing so Swift is just as high-level almost as python. We also have here are lines of code in that example is that alright Swift
is that it's designed to be fast, right? It's practically in the name. So who here has actually looked at assembly in the last 6 months? If you aren't hands this time well be looking at assembly today. So if you don't know the godbolt compiler Explorer is really neat. You can plug in some Source Kona left hand side and it will give you the assembly on the output on right hand side. And here we have Swift. We're calling a higher order function reduce on a generic type. So it's actually an array of integers passing in the + function reducing over everything
and if you look at the Loop that is going on here. So you got your function Prelude you do a couple checks to make sure everything is reasonable looks just have these five instructions that form the main body of your you do the ad you check to make sure that you don't have any over clothes you then increment 1 to get to the next value in the you compared to make sure you're not reached the end of the array and if you haven't then you do look again. This is like the most efficient set of instructions you can possibly do to Come across in a ray. And so even though you have all these high
order things, you're not even writing types. The Swift compiler can help optimize it all the way down for you. No, actually it wasn't me who really know about performance and assembly a really helpful compiler will actually vectorize this operation for you and So Sick by default is safe when you use plus you're getting safe and in your ad that checks for overflows, but if you do if you Ampersand plus the empress and the second pilot will help him vectorize your operation for you to make it run extremely quickly. So
something's designed to be safe to be high-level. But of course also fast. The programming languages are all about atoms and composition. And so the atoms in Python are Hico that implements python objects in a composition is an interpreter that let you combine these he calls enduring sting by Great Lakes python array python dictionary. They're very very carefully and playing in Python to be as quickly as quick as possible and see when you get to build build your whole universe on top of it. People Plus hours makes a different tack. There is no see underneath cplusplus, right?
It's just deepest parts of the atoms in C. Plus plus are in float Series in planters Adams as well and you can develop your own structures in C. Plus plus likes classes after the world. What is you think about C plus plus? What's the real difference between string in Array they both are sequences of things, right? And the programming language is actually evolved since when C plus plus was initially created and we can do more interesting things today. And so Swift built on top of a lot of this programming language evolution in design. So Swift is Lily syntactic sugar
for all of them remember getting back to that picture. You got humans. You've got Hardware programming languages are in the middle. You got text editors to constitute a human's interface with a permanent language compilers to interact with the hardware. You can just get rid of a lot of extra intermediate things and by doing so you can make a programming language is more powerful. And so I The Primitives in Swift are LVN instructions in a composition are stretching classes. Just like C plus plus button Swift instead of float in instant being
defined by the language to find the standard library on your own in in float. Now this may seem like an esoteric question that that isn't terribly important, but will as it turns out the definition of float is something that's in question these days for those of you who are not familiar with TPU but cheap used to find our own version of floating Point format to find Bigfoot 16 too short for the brain floating Point format that was developed as part of the Google brain research efforts. We found at the FIFA 16 format is much better for both
efficiency in the hardware and for training neural networks. And so sweet to the language is especially interesting for us because it lets us Define a B-flat 16 type that works exactly like in spin float in Russian language. So what does that actually looks like? It can't be super weird with his black magic will again this is Swiss from the foundation. So so let's understand. What is the implementation of you can totally hear sort of what it looks like, right? You've got struck slow cuz your composition is strikes and you to find the value type which is
defined in terms of the LVN Primitives with built-in 832 Particles in just a normal function is defined in Swift and it's implemented using the album primitive. MP + 16 is Brendan mentioned is incredibly important to us specifically for TP use on but we're seeing even more so kind of exotic data data formats and started requirements for floats for other specialized Hardware. So we anticipate that this will become even more important in the future and it's a place that we're really excited to explore more with
Swift. Even though in and Float are elements in the standard Library. They're just as fast right as we looked at the assembly instructions before but it also means that Swift is language is super flexible, right? You can Define extensions on instant to find me is odd property of in jupyter Notebook. You can define an extension on Boolean symbol which returns a Unicode character string thumbs-up or thumbs-down. You can compose all of these exactly as you'd expect to print out, you know, these these silly true-false isn't an emoji.
Let's Twist Again, even though using all the time of operations is just as fast as he and so these notebooks cuz you check out from Jeremy Howard and Chris tell you how you can build a map insert that is just as fast as he perfect performance parody. Russell about Swift let's talk about deep learning to get there. How to understand a little bit about linear algebra and a little bit about math in order to effectively architecture experiments. I'm so afraid this
next slide. We have a quiz and I promised it isn't too it isn't too scary. But it's somebody in the room just shout out what you think be will print based on this play. I think I heard at 3. So that is very good. This is value semantics. This is it. This is how integers working python it is exactly what you expect. If a is 3 G is equal to a that means bs3 matter what you do with a real steel 3v is still three. We're all good. I'm we have a python lift and this crazily enough whenever you add for wood prints 3 and 4 for the values of
of be so this is not value semantics in this offers are a great many frustrations for developers. Is there attempting to architecture experiments designs being very very SAS programming language from the same time really push forward value semantics and sewing Swift arrays actually do have value semantics and behave exactly as you'd expect. So here's a copy paste to show you that I don't have nothing up. My sleeve is actually three and four, but if not just a raise but
dictionaries and so do all other sort of high-level types many high-level types that you build can have values minutes as well as important for how to write neural networks and do machine learning and differentiation. Let's leave this for an ectopic automatic differentiation is one of the sort of products defining features of Swift. I mean, it's one of the things that were most excited about extra. Programming language. So we showed in one of our slides previously. I believe a function that was prefaced with that differentiable. And this is sort of functionality that allows you to
differentiate not just sort of aspects of machine learning models, but literally any function and switch you can take the gradient of so do you want to show an example friend better to show not tell? Let's see here. We are in a Google colab versus a host in jupyter Notebook. And here we Define a function that just takes two doubles math operation and Returns on a double and soda make this function differentiable. You just annotated at differentiable. And what you've done that and run the cell the compiler has now
constructed the the Ford past but also the backwards pass for this function automatically. And so to then figure out what the difference relive that function is. I just called the gradient function. So you do the gradient at 0.5 of my function Miss get you the partial derivative with respect to a Of course, you may want to take a partial derivative with respect to both A and B. And so that is exactly as you'd expect so you have now both points. When your training neural networks, you sometimes also want like the value of the function in addition to the partial derivatives at a particular
point. And so they do that instead of calling gradient. You just called value with gradient and this will return a result that has a tubal to insist. You got named named tuples absolutely the value in the gradients and and you can manipulate them exactly as you'd expect. And if you're coming from python lands or you you stuck around for the presentation a little bit earlier on this would be sort of taking the equipment place of gradient tape in tensorflow. I'm so instead of having to create a gradient tape and collecting your variables on that. I'm you can just go ahead and do the the
annotations for your functions and the thing that's really so these aren't just on tensor of double that are you know, Wendy Wonder Years do tensors. These are just regular built in InTune floating double in and Beyond. That's how you write a function a custom function and take the derivative of it. But you know, we really want to build this next-generation platform that allows for maximum flexibility own types that can also be a differentiable fletc how to do that. So here to find a point in 2D Space X and Y
and has Market is differentiable. We can then use we Define some properties on it right to the function. And hopefully he'll pore function and you can then compute the gradients of you know, this. Products this point. It was itself and so there you go. You didn't get the tangent of this point about space. It's a good idea to Mark your functions as a differentiable because it helps the compiler catch errors for you, and it's also great from the documentation. And here we sit example of the classical guitar for us. So page. This says error can only differentiate functions
results to conform to differentiable but it does not confirm differentiable. What does that mean? But you can't add a little incremental step change. That's exactly right. So integers right to derivatives are about infinitely small steps and integers don't lie to take infant ugly. There's nothing in between into this function is not differentiable in the compilers very helpful and telling you how you can't do this. And this is why instead of silently giving you a wrong answer. So if we were moving a compiler Conair 4S great we can keep going on but let's say we want
to take things to the next level. What if we want to Define an additional properties, they liked the magnitude property Quizlet defined by a vector from the origin to technical point. So this is a great excuse for us to show the Swift C interop sew-in Swift you can just import any arbitrary C header and uses symbols to find in it. No boilerplate no rappers know nothing. Go straight for it. So here we can import GMC and this is the square root function defined by the standard length
dinner Library so we can use that to the fine point the magnitude of tension on point, which is the square root of x squared plus y squared and hang it. What's the saying differentiate functions that have not been marked differentiable in letter to find another file. So it looks like you have to go and change this in another file. That's right. So square root is defined by the Tyler C compiler hasn't been taught how to take derivative. So this is a great excuse.
So I have to work all that they driven the square root of x is 1 / 2 * sqrt of X all we can to find my square root which underneath the hood just coffee and we can find the derivative right here with this function closure in included here. So we run that not compiled successfully or change square root to my square root that runs and flower off the races. We've not to find a differentiable magnitude function on point. We can then use this function inside other differentiable functions exactly as you'd expect everything fully composes. So here to find this silly
function taking points and doubles right you can mix arbitrary different types of point take the gradient of it. We can get our tangent Vector which is has a partial derivative with respect to both X and Y at that particular point and I love that you're able to sort of used all of the greats code that's been written to support mathematics. But I also really love that would Swift you get C & C plus plus in her up. Where is Brendan mentioned? You can import any header use it as is or extend it to meet your yusuke. So if you're working and Enterprise than
you have a massive existing people's Plus Code be like, you know Google and then it is really helpful to be able to not have to rewrite the entire wheel, right? Like you can just import things that people have created over time. Use it and then everything as part of us which program would compile down to it. I still file so you can deploy it on any platform Windows Linux Mac iOS Android devices even really is sort of the most multi-platform approach that it said I've seen for a mission
for tensorflow automatic differentiation system and encourage you to check out more details later, but let's move on to actually talk to you about neural networks. And here we go. So tell us about right and wrong that works in Swift swift. We just saw an example of it and it's pretty straightforward. You only have about five lines of code. It looks very similar to take care. And if you open up a collab notebook, you can see an example of a beer bottle by adding some layers here. We see a convolutional layer comfy
pulling layer, we flatten it and then we call we call a function with active or are we prep the class function without the French people in naked Sister Christian? So now it's actually use this as we would text Lee train this model on some data so here to find some random training data just because you know for convenience purposes, we instantiate the model and an Optimizer for it and then we're going to run this for ten Penny steps. So we do this in just a
few lines of code directly hand writing out the training Loop up here. The value of green peppers are lost and are gradient and then print out the loss and use the optimizer to update her model along the gradient. And so just like that we're training a model and you can see the last values decreasing as we have expressed here. Jeremy Howard did a great collaboration with Chris and with a swift for tensorflow team where he re-implemented fast AI on top of Swift. So if you're looking for higher-level constructs, you just want things that are kind
of out of the box. I highly recommend checking out those notebooks as well. That's right. But one of the common workflows fit anyone who's doing machine learning is they have the first person the model they train it and they figure out how to make it better, right? That's when the Great promises in machine learning as you can keep in coming in and optimizing so, let's see how we can optimize this model to make it a little bit better. So you may be following a lot of research and you may hear that skip connections residual networks are really good idea. And so in the next
30 seconds, we're going to convert this network to be a residual network with his connection connection. So we need then floats and I can't type and can you can see the autocomplete right with your productivity autocomplete that's helping to fill in the initializer parameters for us fix the shapes right now. We're all good. Do we now added the extra layer now, we need to add an escape connection. So let temp is equal to input. We need to capture the value after the flattened step. Let temptu is equal to dense of temp
service is going to be the first part and now we do our skip connection thence of 10 + temptu. Wawa we have now added an additional layer and added to skip connection to our model right here in an Uber notebook. So we know recompile that to make sure that it's working. Well looks like it is and so we're going to be in tension or model cuz it's not a different bottle and steps and looks like we actually got slightly lower her similar life values. This is good, but we better for this is sort of example workflow of house easy. It is to customize
models using server tensorflow. But for those of you for you know, just applying models to standard data sets the bread-and-butter workflow, but we're finding more and more that both research and production. You sometimes need to be a bit more advanced. And so Surfer tensorflow ability if it customizability what's actually see how you create your own custom layer also in a jupyter notebook. All right. So let's say if we're a researcher and we wanted to define a custom layer that we had to buy a sturms to ardenne players that have just one well in about 10 seconds. We're
going to we're going to write this. All right ready. Here we go. All right, great. We're just I'm alright. So here we Define our weights are weight tensor in our first bias turn and our second bias turn. We have our instructor and then finally the actual Ford puncture or college function retake the matrix multiplication of the important ways. Can you add in the to buy his term? So there we go. This should compile. And once it does that we got we can then use it in our new model. So instead of a double by since there
and books that doesn't quite work again till you see the Swiss compiler being extremely helpful. You have a quick typo here instead of label. It should be labels and it just tells you this is what you should be doing and you're Off to the Races. So we fix that up and we can run the training steps and we have a little bit of information and so exercises for the for the audience to figure out where that bug was, but will let you think about that. Anyway, that's a brief tour about how to use
Swiffer tensorflow as you know, ml practitioner applying and fitting against data sets and models. Excellent will head back to the slide. Actually, I forgot the one other thing right? So we showed how to write a custom model. We should how to write a custom layer. But if you're doing research into different optimizers, you also want to be able to customize that until optimizers in Swift are just pure Swift Code. And so here we have the Salient bits of the socratic gradient descent Optimizer that we were using just now so it's just so you know,
10 lines of code to implements momentum nesterov and and other Advanced features for a stochastic gradient descent Optimizer. So we've been spending a lot of time thinking about what are the right apis and one of the right ways to factor machine learning to make it easy to use powerful and flexible and we're pretty excited by where we are today. We're always looking of course to improve so if you're interested in pushing, the limits of the city are there should a bunch of other things that we'd love to have you try and give us feedback and it really did help us improve
our so we mentioned before that. One of the biggest sword wins for python is that an entire ecosystem of data science products have been built around it. The good news is with which you don't have to give any of that up. So I think Brennan has a demo coming out showing that you can use tooling like matplotlib or numpy or sci-fi or any of your favorite python packages directly from Swift almost identical to the same way that you would run it in a python jupyter notebook. Is that correct?
My secret notes. All right. There we go. So here is a jupyter. Notebook are actually Google colab where we import tensorflow and we also import python. I want to him for the python we can use this python object to import arbitrary python libraries with no rappers at all. So here we can import matplotlib the pipe lot actually type from there and numpy in Riverside needs to plot in in Pete. And after that you can write Swift Code and Swift Code that looks almost exactly like python.
So here it is empty. Living Spaces to plot X and sine of x + x + cosine of x and you know, just to show you this really does work and I'll run it and Wallah you're using pipe lot from that pot lid and numpy right from Swift. In short you don't have to lose all the great python ecosystem while taking advantage of all the capabilities that Swift offers. Not sure. What about what about a more whack your library I mean either kind of wacky, but you know, one of the really
one of the important parts of the ecosystem are all the machine learning libraries and in one of my favorites is actually open a gym. So this is a used to train and evaluate neural networks. Are you doing research and reinforcement learning? And so you're not the python library, but as it turns out with her for tensorflow you can just use it. So here we can use Python to import Jim to find some hyperparameters. Define are neural network using Swiffer tensorflow. So just like you saw before we got too dense
layers. We have some code to interact with the python environment and including time in the next batch. But Siri use that Jim library to make the carpool environment. I was intentional neural network and are Optimizer for it and they're going to train it for many steps as we need again. We see that the Swift compiler is actually really helpful. It's telling us an online 24 instead of one hot label do probably mean probabilities. If I take that right we should now see this training. So here were you using a neural network to find answer
for tensorflow training in the python openai gym Library totally seamlessly right in jupyter Notebook. Text Dalton, of course, you can take the results that you just you know, the intermediate value theorem computer and plot of The Notebook only works all back and forth. Does that your workflow doesn't change right? Like the only the only sort of incremental steps that you need to add is importing Python and assigning excellent So that's number of demos and and notebooks so you can see sort of the state of the world today.
But let's talk a little bit more about what's been going on in the future directions that were heading. So we have seen trop and we've been working on C plus plus in traffic as well. So we're sort of an example of what we have. So in the left-hand electric column you to find a C plus plus pipe header file excetera in on the right hand side. If you do find this in example. Age, you can import it in sweaters import example, and you can call these messages directly on it with a number of features already supported including instantiated templates, because it turns out C plus plus is
not quite the simplest language. But we're seeing that it's it's available to be useful for a phone number of things already today. One of the mantras is flexibility and mobility. And so we've talked about reading Casting Networks customer optimizers. What about custom kernels as well as it turns out we've been working on that has some of the work that is preliminary work on the team, but here you can Define 1D average pooling in pure Swift And it's actually runs on both CPUs and gpus in the Prototype that we put together. They were still interested in getting the
fundamentals of design. Right but this is sort of an example or the direction that we're trying to go with Surfer tensorflow with infant hackability so that you're not limited to what's in the box for you the whole world's our oyster you get to program at any level 2 stock doing whatever you need to do your never limited and it when you're building on the surface answer for popcorn. Always had a lot of amazing things happen. If part of the community that's been going around Stafford tensorflow. So we have in development model garden with a variety of supervised learning models and some
unsupervised learning models but one of their collaboration but I've been super excited about has been with the dietlein team of as part of alphabet. They recently released open steel which is a collection of reinforcement environments and I'll go there a subset of those have been ported to Swift on so that's a great place. If you want to check it out if you're really interested in reinforcement learning So if you're interested, how do you get start in page blank and get started? ORD / Swift
and I also encourage you to look at Jimmy Howard's Swift AI implementation of fast AI That's available on GitHub. They're going to take a moment to really highlight some the community projects have been going on. So we talked about open Spiel, but Anthony put honey OS is done amazing work with swift RL another reinforcement learning library and swiss Atari learning environment to make a really great. There's food plots with an L that have happened as a result of the Google summer of code project those projects that have happened this Summer that
page is organized that I think are really fantastic. So say I came out of the past the expiration we have tents are sitting perfectly Posh go to join the team this past summer and it's a really great research. Everything is open-source. We'd love to have you join our community. We have open design meetings every Friday at 9 a.m. Pacific or 1600 UTC and we'd love to have you all doing it and really helped shape the spot form together with us. Recently released 0.5 with a number of improvements and stay tuned for our next release is coming up and with that. I'd like
to thank you all for listening to us. We will be around here and in the hallways the rest of the day. We have a load of those of you here in person to conference. We will be hanging out at the active tensorflow or Cafe in the expo hall. Come to your questions feedback. We love to hear what you're thinking and and see you at openside meetings. Thanks.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.