Laurence is a developer advocate at Google working on machine learning and artificial intelligence. He's the author of dozens of programming books, and hundreds of articles. When not Googling, he's author of a best-selling Science Fiction book series, and a produced screenwriter.View the profile
Kaz Sato is staff developer advocate at Google Cloud. For machine learning (ML) and data analytics products, such as TensorFlow, Cloud ML, and BigQuery, Kaz has been invited as a speaker at major events including Google Cloud Next, Google I/O, Strata NYC, NVIDIA GTC and etc. Also, authoring many Google Cloud Platform (GCP) blog posts, supporting developer communities for Google Cloud for over 8 years. He's also interested in hardwares and IoT, and has been hosting FPGA meetups since 2013.View the profile
About the talk
With the IoT market set to triple in size by 2020, and massive increases in computing power on small devices, the intersection of IoT and machine learning is a trend that all developers should pay attention to. This talk will cover three core use cases, including: how to manage sourcing data from IoT devices to drive machine-learned models; how to deploy and use trained models on mobile devices; and how to do on-device training with a Raspberry Pi computer.
02:42 Simple sensors that give smart solutions
06:52 Environmental monitoring
09:08 Data security
12:44 A shopping cart with a smart camera
17:55 Smart shopping cart
19:15 Smart shopping navigator
22:20 TensorFlow Lite design
Hi everybody and welcome to this session the last one of the day everybody having a good iOS so far. Nice excited for the concert tonight. So thanks for spending some time with us beforehand. I'm Laurence Maroney. I'm past here today to talk about machine learning plus i o t equals a smarter world famous computer science book. Anybody know what that is algorithm suppose data structures equals programs by Nicklaus worth. It's one of the Seminole War and Tom's died. So we said hey we want to do with a little bit of a red van that and it's kind of it's also really
about what's going on in the world today that the two major Trends going on the but one is the rapid rise of iot anniversary was the rapid rise of AI in machine learning and the many ways these are kind of you know that they're intersecting with each other to create some really cool new scenarios. So we wanted to talk through a few of these scenarios and give some demonstrations. The first one will give is like, let's take a look at like really cheap iot sensors that like will do something like that in this case. We're going to take a look at those that measure air quality. How they can push
data to the cloud on how we can use that for machine learning and then we also look like how the cloud can be empowering to for collecting and I think your mother what can we do about on device inference with them as well. So we'll take a look at some of the examples of that and some pretty cool stuff that's happening there 31 billion connected iot devices by the end of this year. That's a lot of devices and those devices combined we found the same research to showing at 27.6%
compound annual growth rate of the data that they are pushing to the cloud. So 31 billion by the end of this year and every year the amount of data that they're growing is I think it's growing by 27.6% this the usual things real things filtering sorting carrying all that kind of stuff, but we can also use that to start training some models of those models machine learning models can then be used to generate intelligence and to help us to start making Intelligent Decisions based on the data that's being generated by our devices to the
first scenario. I want you to look at was thinking about simple sensors that give us smart Solutions. I just wanted something before Isis on it. You have to remember that way and there's like a device. The one on the left ear is called in esp8266 and this little chap actually has gpio pins on it says input output pins. I know what that's a Wi-Fi circuit on it. And you know, when you combine that with something like the one on the right, you know, that's an air quality sensor on a gas sensor. Now, you can start putting together
something very quickly and very cheaply that will push data to the cloud and I'm going to go off script for a second and an interesting story and I had prepared this talk and we were rehearsing it last week with like the VP of our division. I'm one of the feedback he gave us was like, hey, you don't have any small simple sensors on their you're doing off the cloud stuff and even left a mobile stuff. We don't have any iot sensors. So you should really build one and put it in there. And this was like last Wednesday and the environment that we live in now, it's so cool that we were able
to thank to Arduino and thanks for the maker community. Be able to build a demo. I'm going to show up shortly, you know in just a couple of days and the longest part of that. Can anybody guess what the longest time in building the demo was? I know it was actually testing it was waiting for the parts to be shipped from Amazon. I'm an Amazon Prime subscriber and it took two days. If it only took three days to build the whole thing. So, you know, it's an amazing time to be alive. I think you know, they're given amount of things that you can do the amount of creativity and the kind of
things that you can build. It's just I find it so cool, and I'm really happy to be alive right now. So let's take a look at it like an example of how you choose one of these I consider a building like this one and saying it building like this we can put sensors like this for air quality around the building and went around the building. Then we can start tracking the air quality in that building so we can tell the stuff on the south side of the building the air quality is in grey, you know, and then on the north side of the building the air quality is very nice, but would usually happen if
you work in an office like this people on the southside of going to kill him to just the thermometer adjust the thermostat the fans will kick in and then people on the north side of got to be too cold, you know, it's great if we could intelligently adapt the building or Monoxide leak and with our senses are going to show that carbon monoxide leak and Identifix how well you can see it here, but the leak is primarily centered around the exit and if people have been generated being driven towards the exit they might be driven towards danger, but when we got Chiefs answers
like this, we could say hey look instantly we can have an alternative exit. Sorry. Further than think about scenarios where machine learning comes involved and we can start predicting things. Like maybe we can predict the best times of the day to keep the fans on to maximize air quality and reduce our energy by Prince good for the environment. We could if there's a gas leak or something like that we could predict the path of the gas leak best on type based on testing and based on machine learning models. So emergency responders would know where it's safe to go and where it's dangerous
to go and you know, we all have our emergency responders and we want to make sure they're safe or maybe we could look at what the impact by tradition by modeling. What would be like rearranging our office? What if we have more people sitting in the north and lessen the sound so what happens if we host a big events and you know, this building is actually that one of the offices in Google Seattle and open that top right top left-hand side there that Rim call center of the universe is actually where we host a lot of events and it'd be good for us to understand the air quality when we do so So
if you come a lot of devices like this was a product called Cloud iot Core and the idea behind Cloud iot cars that's fully managed and allows you to easily secureconnect manage and ingest data from all your devices. And then once you have that, then you can start doing interesting things like pulling the data out using jQuery using Cloud. Ml using Cloud datalab using Cloud Studio to be able to that build models on and run inference on these models and have analytics behind that environmental monitoring and what it would look like if we can switch to the wall fishing.
So this is a device I was talking about so this is a simple Arduino device. I'm not using the Wi-Fi in this case. I'm just using a wired one. So I have a shield on top of the Arduino with a with a wired internet just so we can make sure that the demo would work here and then this little device is the air quality sensor on so it's like it's right now it's measuring the air quality in the room on so that's measuring for particulates measurements of gases carbon dioxide carbon monoxide natural gas all that kind of thing. So I kind of feel it in the air that have any ideas how we could do that.
I'll just breathe on it. Cuz we all breathe out carbon dioxide, right? So I'm going to breathe on it. But we wanted to do that in a session and if we switch to my laptop. Sell my laptop we can now see I have a Firebase running here and I've using Firebase cloud firestore cloud firestore. I'm actually detecting and I'm getting the readings from this little device so we can see the readings up here. Once they come up look here at the reading I-57. That was a moment ago. When I breathe on it 54, I've actually gotten it up to 70. I must have eaten something bad about 42. So what's
going on here? How is this being done? Well with Arduino, it's super simple to write code and see can we switch back to the slides will be a little easier to see the code. So if I go back to the slides when you see this is the kind of cold. Do you have written in C on an Arduino? And all I'm doing is I'm reading Penn zero that's this is plugged into Data Penn zero on there. I'm reading. And then every two seconds I'm reading it again. I'm van what I'm doing with that is I'm posting that to a URL and then. You were right as writing. It's a Firebase for me and that's all you have to do. It's
really that simple to build a sensor to start monitoring air quality very very cheap very very simple. And then when you activate the data from all of these sensors together, if you got five thousands of them in the cloud, then you can start building some really intelligent solutions, but there's something to take into account when you doing iot enough to think in terms of security. So data security is very very important. Obviously, these things are just generating data and pushing it up to my database. What a smart Factor might be able to figure out how I'm doing that and then get
access to my database. You got to think in terms of data security on a pattern that I usually like to follow for this Unforgiven reason to is instead of Missing from the devices directly up to your databases have something within your firewall that they are pushing food and let that act as a proxy and then that prophecy will prophesy out to your databases for you. So then you know, you have a secure connection between that proxy and the databases and then hopefully that secure connection will be enough. And the reason for that is also to do it as an internal thing. Was that the power
limitations on these things mean that a lot of these will only support HTTP. They won't support hdps. That's why my Kodak moments ago. If you looked closely you would have seen that I was just calling and I was just making a client. Connect server, 80, you know in that case cuz it's just that open HTTP connections that was to my internal proxy and then the internal proxy was sending it out to the cloud so things to consider, you know, very cheap very easy for you to build systems like this one coding with something about the Arduino and see is super simple tons of apis in front of examples.
Adopt and that's what I did. In this case. I've never touched an Arduino before last Friday and it took a couple and I'm not particularly smart and it took it it just a couple of hours to figure out how to hack some of the code that they had provided to be able to build this and write my data up to Firebase take a look at it. Not good to have our devices such as if you take that picture with the camera. You may want to get some object what kind of intensive law provides an API to do that that is object detection API to buy
labels of it or you could have the boxes and you can have your squirrels and it's really simple to use the API example, you can just download it and you can have your dictionary. Cancel Smule T-Mobile buy more than you can pass your images to the dictionary and that it so you would get the output dictionary that would have the detection with such as the number of the app what class is object angry boxes. I'll just for us it's so simple so you can help you out with the
camera at the shopping cart do that. You can take a picture of the east side of the shopping cat and applied object detection API to what kind of food what kind of the items to your hiding or how many of them you have in the car and we can use the Cloud iot Core a combined Vizio flat in machine learning energy to build a system that is that provide for protection. Example if you want to build a smart camera system for the shopping cart, you can have a broccoli pie and a camera attached to the shopping cat that used to call the I-84 to
collect all your Data Squad Gator of storing data on the Frog Hops and which could be here packing for your server and we could use the Google kubernetes engine UGK. He has an orchestrator for orchestrating everything happening at the clouds syde, you could be sending the data to the MLG. That's where we are. We are running the protection with the Albany protection API. Let's take a look at it. So it's right to use to your auntie webcam to show how a laptop. So this is how the traffic is very
small. Meet your car and we have a Raspberry Pi and display and the camera. So if you put something by diffuser fake eggplant I bought it in a Japanese store also known as an aubergine for us Europeans or tomato. tomato Should I bring my working maybe the network issue? Thank you. Play Store easy to detect what kind of the object and how many of them you have in your shopping have so please go back to your flight. So that work. So that was a very simple Mario where we just counted the items and Cross by them, but you may think
that's what's the point of using Cloud that is true and ear protection API just inside sensor flood light bulbs and you don't have to use the crowd outside recalls, then you can extract certain time because you can't just hoping I can see your identity card hasn't something that can predict what will be the next to items. You would be putting into the shopping app. That would be a different thing the location proximity starting from the vegetables and fruit and meat like a chicken and spices and faster.
And heating also have a small display with it showing diversity recommendation with your next to item to tasks. That would be working as a navigator for the Shoppers as a smart shopping in Navigator. And what kind of machine learning model we should design for implementing the dimensions of the Navigator at first we have to represent what kind of items you have in your car. So if you have any tomato in the car that we may we may have it back to her that we couldn't say tomato in your car. In this case. You would have a 1. Fever as a value investor. And if you put eggplant in
the past you would have another one of the about you in the vector. Be happy that you have another one to do before chicken. So this multiple of 3 and we like this for you to have a dictionary for shopping items and you can call you what the function to in call the shopping items as a vector and then you can have them out. You better not like in history we have refused to me and how do you detect your changes in the history? In this case? You can use to your conversion
and competition in machine learning is usually used to detect a certain patterns in a lot of group of the dictator. If you have a 2d images you can apply to the population to find us some shapes of hottest in the images. That is how she works. For example, you can apply the body composition on the card. I can pick 3 pick a detective that happened inside the car and then you can park in the output to get the result of extensive rock. You can lie to call like this. You can call the 101 the function to apply the single dimension of completion and
then for the flattened to get a result. So now we have the card I can change history and then we may also want to taking other factors such as his narrative website for winter or summer what time of day because I want you to be fine food items based on this is not get birthday to Summer hot day or where they talk all day. So we put everything into a single MLP modulator project which is a classic or newer rock with you predict the next to add players of the MLP and that pay
so He put the eggplant. Damn, it should detect the egg pan. I See Your Shadow So did you are watching do you say nice to meet you? I'm watching here. So at the right time you see the what kind of recipes and I can study comment that so it looks like the assistance you, Auntie pasta eggplant baked and I want to make a pasta so I would put the tomato. Navigate to the show the other things you have to ask would-be. I already have it, but Advanced tomato, and I want to make this year.
I'd like to make a pastor. The mouse is also working. I'm not smoking. But somehow it doesn't show but Pastor has chicken so so that you can get you can find the all the items to make the eggplant tomato pasta that how it works. Thank you. So, please go back to the screen. Just right. So after you have seen on the demonstration by showing the recipe ideas and next item to add it works just like a car navigator for the Shoppers so they can just follow the next to
Field Apartments. The all the items required to cook is gas at in the recipe for the iot device is not only but also you can analyze it and a Google play the part that so-called glued nails and they have office of everything Sonic it half. So if you're interested, please go to the gate hand and surgery smart shopping navigator to find out what kind of car do you drive? And this is also they also provide your product information for the Magellan block where you can have the user interface to build your whole data pipeline for quitting the iot data and analyzing it and learning. I need them all y'all
so if that I directed at the station to Lourdes. The third scenario that we wanted to look at was like on device in France and training. I like this a few different types of devices you can do in France and training on. So for example, you can do it on a mobile phone or tablet. You can do it on a Raspberry Pi or you can do it on an Android things device and there's a really cool video of doing it on a mobile phone or tablet that I put it this QR code. So take a look at this carol code
watch this video. I'm not going to show it here. But the idea behind this video is that if you went to the tenths place, you would have seen it but it's farmers in Tanzania in Africa who don't have Cloud connectivity and who rely on a plant called catch Salva that if the cassava plant gets disease it can kill an impact in entire crop and it's hard for humans to eyeball it and see it but they built a machine learn system that only they would stay away that phone over the leaves under can diagnose diseases early. It's really really cool. So that kind of on device and friends without
any Cloud connectivity is possible through something called tensorflow Lite with tensorflow. You can also do in France and training on a Raspberry Pi and I have a car here that I'm going to show in a moment that's driven off of a Raspberry Pi and if you were in the sandbox area would have seen cars like this one's self driving around but basically there's a little camera at the frontier. And what I would do is I would manually drive the car record what I'm doing and then based on the Telemetry that I'm sending to the car to drive it around on the corresponding video that's giving us the
features and labels that we want that the car will then remember and then use that to self-drive we switch back to slides for a moment. I'm in finally does the third one which was showing him on with his son and rice things device. All this is made possible because attentive low-light and I strongly recommend check out tensorflow light and check out the talk sometimes but the idea behind sensor flood light is that if you want to train a machine learning model, we have a bunch of converters that will then flatten that model and shrink it to make it more mobile-friendly does an interpreter car
than which is used to execute that model so you can do inference on the model. So for example of things like if you want to do an image classification, it'll do that locally on the device without round trip into the cloud and then it's all the stuff is needed like operation kernels and Hardware acceleration to take advantage of the mobile Hardware that you have. It runs on Raspberry Pi AI runs on iOS and of course that runs on Android and it works really great on a pixel. I have a pixel 2 up here and if you want to do it for the demo of it come see me off to his and I can
show like real time in France on the pixel doesn't really work on the stage that well. Here's a video that I shot of me doing real-time inference and you can feel like it's got all the coffee mug correctly and here, you know what this coffee mug kind of looks a little bit like a mixing bowl. So it got like at some point it felt like a mixing gold and I know that's a horrible product placement for my couch show coffee with a go there but I have to do it and it was able to recognize a mouse and stuff like that. So this is what it would look like running on Android. And so for the self-driving
car that I've been talkin about so you can see on the right is the Raspberry Pi and on the left is coming call the pulse wave modulation controller and that's what actually drives the car itself. So I have pet supplies light running on the Raspberry Pi and that's what we'll actually drive the car. And this is built on a project called donkey car. And if you want to go and take a look at donkey car and details about the donkey car there at this URL. So it should we take a look at what it would look like a choo train the car. So if we can switch to my laptop, please.
And so I've actually started up the Raspberry Pi on the car under the little web server running on that. So let me go and see if I can access it. Hopefully it'll be oh now it's refusing to connect. I know. Oh, sorry. I'm so just running the python code. Sorry cast to make it hold it so long. So this is actually booting up the web server that's on the car that I would then use the train and drive it. So we'll take a moment just to visit that up. What would happen if like so if I want to control the car when I'm training it? Obviously it's hard for me
to do it on it. So it has that little web server the web server is connected to a hotspot on my pixel on my laptop connected to the same one. Starting up that server. It looks like a service done yet. And here it is. So is Kaz moves it around you'll see that when and if I went to like try and drive the car, I made a bit of a mistake. If you've been over to The Sandbar she would have seen the cars that using a really small ones. I didn't read the specs properly and I bought this one and it's an absolute Beast. It goes really really fast. I don't dare put it on the floor and start driving cuz it
will take off and it'll and about three rows back. So watch out cause any see if I just want it looks like if I'm driving and training it, you know, I can steer and I can control it like that. So it's you know what it's doing right now. It's recording what it would see on the webcam from the car actually moving and storing all that stuff that would then use to train a tense of flow model for that specific area. So when you see the cars over in the sandbox there on that figure of an 8-track, so we draw. Call them around that figure-of-eight. We trained them in that way and then instead of
launching it in training mode as I have it now, you just launched and drive mode and they going they drive themselves. So you can build one of these as a donkey carrots. Okay, we can put it then we can build one of these using the donkey car project. It cost. This one is a little bit more expensive, but they generally cost may be about $200 total for everything and you can build your own little self driving car and it's all open source. It's really really cool. So, thank you. Can we go back to the slides, please? The motor on the car is really loud. So I'm going to turn it off if we
hear it was he? Okay, so that's what it's like to train a donkey, and that's the kind of with on device and friends. You can men have a believe it or not a model self-driving car. I think that's really cool. Then. The next thing was Android things that I mention xan's did everybody get one of these. Right. So these are just so cool. So this is like just a little it's this one isn't actually a Raspberry Pi it's a different developer for but a similar concept I'm with Android things being able to execute stuff. I'm with tensorflow Lite running on Android things. You can start doing inference
and it has anybody done the code lab. Have you tried it out to do in friends on these at try it out and build it and you'll be able to do things like this. So this afternoon back in my hotel, you know on this device. I kind of pointed it at a water bottle and I thought it was interesting that it gives you three things that he thinks it is a thought it was a water bottle of pop bottle or why does this year? I don't know if it's if it's because of the circular parts of it or anything like that for a measuring cup handle is
placed. I thought it would be really interesting to try and fill it with the handles Wades Place and I told him I think it was a picture cuz sometimes when I do mugs like that is it kind of classifies as a picture and then of course this one I tried on my Android to my little French Android and I thought he was a piggy bank a teapot. I guess he's a little teapot the way he spending or maybe even a spot lights and maybe it's the shape of it or something like that. But that's the stuff that's built in and like when you when you assemble this device that you've got that app is actually already
running on here. And that app is open source. I'm one of the really cool things you can do is that the image model that. Stop This Train for you can actually retrain that for your own images. I've been working on a version of it that I'm going to add to the car and maybe we'll be able to talk about it at a future IL where I'm retraining the car to recognize things like traffic lights. So if it's a red light, I will stop if it sees a green light it will go on all the code for you to do that is available with this kid when you get it. So give it a try. And for those of you watching online. Just
check out Android things you can you can buy these kids the relatively cheap and it's so much fun to play with. So I know the code lab that I mentioned if you're here and I owe you can go over and try it out. If you want to try it on your machines at home to you or else word is that this killer El codo, give it a in a have a play with it's it's it's it's a lot of fun. So in recap, you know, we spoke a little bit about internet of things and they are in about the trends that are happening that the explosive growth that is going on the growth of actual devices and the amount of
data that I producing and then the things that you can do with that we looked at like with a sensor data on an Arduino. We looked at Clouds Kayla and wheel update on device inference. None of the whole bunch of things that you can do to go and take a look at okay. I'm on slide animate that we go so things that you can try out number one writing data from your Internet of Things device to cloud storage this break details on it at this URL. If you want to explore the tensorflow object detection API, which is what cause was using here when it was detecting tomatoes and eggplant or aubergine jeans
and that kind of stuff. You can find details on that at this link iot in the cloud. Ml engine details for that are at this link. I'm this one tensorflow and tensorflow lights and all the mobile scenarios that I was showing here with Basin tensorflow Lite. You can learn about that at this link or offended talks if you're here. I didn't find any exploring Android things at South I find it's super cool, and it's really nice because if you're already an Android developer, you can start building stuff using your existing skill-sets when I was doing the thing for the Arduino here. I
have to get myself a crash course in see, you know, but if you would like I said, it's already an Android developer used to using Java or Courtland to build your Android apps that you can start building things apps without food. We just want to say thank you very much feedback you were I was always welcome would love to hear and set this URL. Thank you.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.