Events Add an event Speakers Talks Collections
 
2018 Google I/O
May 10, 2018, Mountain View, USA
2018 Google I/O
Request Q&A
2018 Google I/O
From the conference
2018 Google I/O
Request Q&A
Video
Integrate your smart home device with the Google Assistant
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
9.98 K
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About the talk

Come and learn more about all the new features of the smart home APIs that allow developers to make their devices work with the Google Assistant. This discussion will include an introduction to the APIs, as well as the new device types and device traits that are available.

About speakers

David Schairer
Tech Lead at Google
Mark Spates
Product Lead at Google
Michele Turner
Sr. Director at Google

David leads engineering for SmartHome/IoT in the Google Assistant. Before joining Google, David founded or advised multiple startups in the AI, NLU, and productivity spaces, including Trapit and Magneto. He began his career in the ISP and telco business, building Internet infrastructure at Concentric and XO.

View the profile

Mark is the product lead for Smart Home and IoT capabilities across Google Assistant and Google Home. Mark is also the founder iotlist.co, a platform for discovering IoT products. Prior to joining Google, Mark was Head of Connected Home Platform at Logitech.

View the profile

Michele has over 20 years of experience in high-tech product development. She currently leads Smart Home Ecosystem Product Management for Google Assistant and Home. Prior to this, she was the General Manager of the Nest Home Security Product line, and also ran Software Product Management. Before going to Nest, Michele was CEO of Dictionary.com, a top 100 web property that serves over 70 million unique users monthly, where she grew users and revenue 20% in less than 2 years.

View the profile
Share

Hello and good morning. Thank you for joining our session to talk about integrating your smart home devices with the Google Assistant unlock space and I bought a few of my colleagues with me and Michelle Turner. I'm the director of the smart home ecosystem a couple years ago. We had a vision in our vision was to have the Google Assistant be the center of intelligent interactions with iot devices. We selected the word intelligent for a very specific reason. We believe that users would demand that their assistant in their home understand

environmental context as well as the device capabilities when they start to control these devices to achieve this goal last year. We launched the ability for developers like yourself to integrate smart home devices using the actions on Google platform. Just a quick reminder actions on Google is our platform for developers to create experiences around education games and many others through the Google Assistant. When it came to smart-home one thing that we focused on was having a seamless easy experience for developers to connect these devices to

this distant. We also understood that we had to allow this connection understanding that we were gaining from the home and devices also to be accessible to our developers with that week also created the home graph. This is a very simple visual representation of the home graph. The home graph is the key to these intelligent interactions that we have with users. I'll just walk through it quickly. If you look at it, it's broken down into three main parts. The first part is the structure understanding the structure and the name of the structure the address of the structure

who's in this structure. What rooms are in this structure? What devices are in this structure? Is there anyone home? What's the presence of this structure? Then the next thing that we look at our rooms rooms are extremely important signals when you think about interacting with an assistant when it comes to contextual relevance command in the living room versus the kitchen kidney very different things. So when you look at the room level we're saying what is the name of this room? And what devices are in this room? The third and probably one of the most important components are devices when

we look at the metadata that we want to get from devices. It's things like names tight trait attributes and state. If you combine these three element you start to see how we're building our understanding our contextual relevance to help users get to the exact control that they want. So we wants to slash your how we don't how how we're doing that. We have over 400 Partners who have used a smart home API on taxes on Google. We hope that never goes up and let some of you guys leave and start to create some of these Integrations for us. The other cool part is we control over

5000 Smart Home Products on the market today. So if you walk into any of your favorite electronic stores or any of your favorite sites and look at these smart home devices. There's a good chance that the Google Assistant actually can control it. What we're seeing the evolution when we first started this I remember talking to David and we are all excited that we can say turn the light on turn the light off in the world like it's our Edison moment, but we realized very quickly that on and off wasn't enough. We we wanted to be able to have the user speak to the assistant

in the way that they speak to their partner their spouse owe their children or their family and its evolution is a very simple thing some from turn the temperature to 72 degrees vs. Make it cooler make it warmer and how do we achieve that? The first step that we took was we decided to make sure that every single device. We actually support it directly so that we had a deep understanding of how that the device works and what are the actual traits that we need to create so we can match those to the grammars that they use her what actually want. So 2016. We had a very small start we

have four devices. You can see in 2017. We brought so many devices to the market that we support directly and we understand with deep Knowledge from camera doorbells washers and dryers sensors air purifiers refrigerators this tons, but we're not done. There's a bunch more in 2018 that were bringing all the way down to tubs and toilets right? There will be a day where you talk to your tub believe it or not, and there's a treat for that and David to tell you about it. Let me just walk you through really quickly. How are we using all this data? How we using a are we using machine learning to

actually create a better user experience, but this is a very simple graph. But if you think about this light to the user to a five-year-old they pointed in a safe. Mommy daddy, that's a life what we want to understand about their light is a little bit different. We want to know. What's the name of that? Like what traits does it have? This is a port on all this is a port color brightness States and where is its location? Because that allows us to take the right action when a user says make the lights in the living room. Dim the lights in the living room a little bit or make the lights in the

living room brighter because what we're doing is we're going back to that home graph ever saying what's the name of that life? Where is it located? What's his current state? What's his current brightness of screen? Brightness is X will raise it by why this is how we are starting to really take this data that were having about the home and devices and its contextual awareness and make it a better user experience, but it just doesn't stop with lights life is easy. Think about all the other devices you're home. So you'll be able to say Hey Google raise the temperature little bit. Hey, Google

is a front door lock. Hey Google do the laundry dry my clothes or hey Google start back to me to remind me a lot of like me being my mom's assistant, right? It's like it clean the house, but these are the type of things that we believe you will start to see from the Google Assistant because we built this really cool back in with contextual understanding about the location and environment and deep understanding devices to give you a little bit more about how is a developer you can take advantage of this David a Get directions on Google. Thanks Mark as much as noted we spent the last year

for The Expendables platform. We launched it a year ago at Ajo. We had at the time maybe a dozen plenti partners. And now he has Mark said, you know 400 Partners 5000 partner 5000 devices a little later. I'm going to talk about how we actually do that. But first I was going to walk through a little bit about how this works we do. So for most of our partners in the iot space control, they've got devices that are wired to their cloud and so it's very very easy to take that cloud integration and plug in the door light

with a p I am most of our partners here. We say in a rather than six weeks that's end-to-end from start of projects to launch in the partners includes testing and evaluation includes whatever we need to do to make sure every We have partners that actually do the work and under a day because in many cases it's just that easy. And a smart set of a little more detail, you know, we use these concept called types and traits the reason we do this we don't say this is the platonic washing machine all washing machines aspire to this ideal device instead. We say there's a type of

machine type of device called washing machine that it's a washing machine type tells us a little bit about it and call it a washing machine. We can make it look like a washing machine, but that doesn't tell us what that washing machine does safe and sound settings has a you know, I reuse traits as the building blocks of the smart home and this lets us. Let's Partners innovate in their own way in New in new ways, which is what we did a tour. Sandbox mailbox over their mailbox here on stage. I want to show you how you build this later. But that means is mailbox in the sandbox of our partners in

building smart mailbox before we did this for the demo because all the paid for their camera light off for the light than toggle for the flag of object detection for male version of a new device based on the existing functionality we have and because we all put this in the Home Run for the hungry understands contacts understand State understand change and as you can well imagine a lot of the Google Secret Sauce in machine learning and intelligence starts to operate on the things we learn from the home raft restart to be able to take all these devices

can actually truly make the Smart Home Smart. All of this today you in a bit with one API very simple short you later with a cloud devices Metallica devices are there you can call us back at any time to do notifications the push of State changes the change the devices. So that's all you do a little rest restful API. We have an invitation to show you and a lot of documentation a full team of developer relations. And you don't need to do grammar. You don't need to do internationalisation. You don't need to do you lie everything out like that is provided for you and if you can imagine

with all of the other devices were going to be rolling out that you do the same. We do it all for you happens if Markus days until 4th of snow, this is one of the status quo going today. I'm going to turn it back to Michelle to talk about the fun stuff coming next. Great Thanksgiving. So as Mark mansion and David mentioned, we've spent the last couple years building the spot for about an hour an estate. Where were able to add a lot of new features to really build on the intelligence and help you guys create better and more integrated and interactive experiences at the end of the day we

all want to do is create solutions to problems that people have in their homes thing is that they're going to use every day and so the fundamental features that were adding over the course of this year are going to help you do that. There are four main features are adding for the rest of this year on something called requesting report State new object detection trait that Dave is going to show you and some new things for entertainment devices. So let's start with request think request think is really very simple. And right now if I go to the store

and I go buy another half-dozen a smart lights to go put my home I have to add those devices and they don't necessarily the fact that I've added this device is so not till I get synced up onto the partner out very quickly as to what we're doing with request is to let us know that these new devices about bored so I can buy sis. I screw the man we bring them. I pair them up get them going and they automatically just work today. If I did that I would have to unlink and re-link and that's a real hassle for most users and we've taken that uncomfortable step out for users made it easier

for you and made it a lot easier for your customers think one of the most painful points of smart home for a lot of new users. It's just Things started even with things like lights so making that onboarding process simpler and easier is really important and that's part of what request things does. We've also added a new API for report state report state is really important and it's I don't see why it's even more important than ever. I have spent a lot of time talking or number of you in the sandbox the last couple days about why we need for Port

state. So imagine a situation where you go up if you change your thermostat on the wall go up I shifted from 65 to 72, but we normally do on the back of this we pull you to figure out when that device has been updated but you guys all saw the new displays in the new user interfaces for the phone in the Keynote. Well now we have new user interfaces on the smart displays and iPhones that will show the devices and with report State when you push report state to us and into the home graph we can get that stayed information instantly. So there's no cognitive

dissonance between I just went over and changed my thermostat and my smart display is still holding an Old State of temperature information to report state is a very important one two, things are starting to require our developers to give us so that we make sure that that user experience for our consumers is spot-on the other reason you want to use the report state is so that we're not pulling you all the time which can get annoying and puts a lot of load on your server, especially if you have very active devices In the future report state is going to allow us to unlock a lot of

intelligent the state information that we can get we can work with you together to start creating more intelligent experiences, like understanding that in the kids room. The light is turned out every night at 7:30. So maybe we should just do that for you. So report States really important now to make sure we have great experiences for our consumers and it's going to be really important in the future as we start building more and more intelligent. The next speech I want to talk to you about is object detection. And this is one of my favorites. This is actually a trait.

We originally built this trade a few months ago. If you're familiar with the nest hello video doorbell at it will actually let you know someone's at the door. We did that integration using the new object detection trait. Let's take a look. if I can Rio Someone's at the front door before I push the button and it notifies out on the on the Google home mini through the assistant. What's cool about this trait is it actually changes the model and this is one of the things I really want to stress with you we're trying

to do with Google is giving you guys the ability to create more interesting integration. So with the object detection trade normally the assistant controls the device to OK Google turn on the lights right with this you take an action on a device and its assistant to give a notification. Oops, I just fired the sister. Sorry. So Dave is going to come out right now and show you a little bit more of what you can do. Cool. Thanks. So I want to go demo here at home notifications work object detection is actually a specific case of a

general facility were adding to the to the smart home. Their Partners can send notifications on object and notifications become another aspect of objects in devices that you push into the home crafts just like traits and attributes and stay in command until we're there a speck of aspect of devices in this case notifications attach two traits. So just like in you know in Java where you have an object made me see messages 1990 clear the error that throws when you declare a trait you can also see you also are saying hey this example of this washing machine

support the one cycle therefore it cannot send or receive depending on your perspective notifications of cycle are still washing machine has failed during the rinse cycle. Case of this and like like the video showed he built it first use cases doorbells, but we build these things is General cases the example in the lab in the sandbox is for male detection and here we're going to build something a little more interesting thing works show how you feel about

your cloud service here or elsewhere how the notifications are sent through do in this case nodejs in Firebase from the device itself and finally show you some code play Sunday by controls. This is basically the flow of this demo. So this is a smart mailbox really a pretty dumb mailbox, but it has a very smart Raspberry Pi down there that has some sensors wired in the box and again, not a shipping product demo. That isn't make sure you can build something like this. The mailbox has a cloud service. It send a notification

to a cloud service center part of our API. It called the assistant with one platform API call sends notification notification, then goes into the home and goes directly to the Google home. I have up here. I will see that demo. And of course all of this is driven by the fact that initially and I'll show you the code for this initially the mailbox know what devices are synced from the partner service to the assistant for my demo. And therefore in the hungry if we know that this device supports technically, it supports everything that the one in the

sandbox does but the flag the lights the camera so far no camera in here unfortunately, but all the traits are there. So I'm going to hop on I'm going to hop on over to the to the laptop screen here which of course locked on me and so some of the last year we have rebuilt this quite a bit last year. We were doing a demo that was a large that the command line. You know what you need to know from an online functionality for this. Now those of us like me that the command line or a

little sad, but for the vast majority of people we are now entirely isn't reactions on Google console for writing action packages. This is much more native for us. So I'm just going to walk through a very simple example of how you build a project today. Recommendation Nail Pro tip from what we've done with Partners II and it makes sense or no longer Auto picking names for you. So you don't get something like purple chaos giraffes at the playlist Productions. So that said that's called this. Read

that creates spin spin spin very slow cuz I'm on stage and it knows on stage with going to slow down. This is a project in and stuff. And now I have all these things so I can build you know that for the day after using conversational action, but you're really recommend even very strongly recommending notifications in for understanding and so today the setup is in fact, very quick. something random And I will then. And forgive me. My network is especially slow today. And if you have done this already off is the mechanism for account linking to existing service clouds.

This is pretty standard of all or nothing nothing out of the ordinary here. I'm going to make some keyboard mashed his keys at home. and of course, I don't that is protected by RFC that you know, there's an actual RFC protecting example.com in perpetuity No One Ever Knows that but it is real and testing is awesome. And at this point we are pretty much set up for going to go ahead and create an action. I am going to give it I need exactly one thing. I need the endpoint for the action. And again, this is a demo. I'm not going to sit this one up. I'm going to speak to you when I have set

up and at this point I have now deployed this and this will show up on my phone. Actually. I'm not it I have now deployed this and it will show up on my phone and it will complain cuz I don't know that you set up but this is my phone and will be available in a second to show that but instead I'm going to switch back over here to my stage demo project. This is the one who actually using and see I've got the action set up its pointed at a Firebase app. And the Firebase app here is a

show you the Firebase app. You can deploy. Let me get this a special tool. Do I need to use do I need to use Android settings? Do I need to use we do I need to use the auto thread on Google has asthma but all these does a number of technologies that you can use to be a completely full stack developer and everything you do and if you want to do that, that is awesome. But it is a completely and totally not required. If you have an existing Cloud, you can keep it this with any infrastructure that you need internet and

a Json API using Firebase if it's using the Firebase database to actually do all the triggering you can see are trigger types are traits here right now for object detection. And so on and so forth that is that is how we Implement in Firebase database the actual representation of the commands that meat that we won this too and so I'm not going to show you when I'm going to show you a little code we have this is implemented as a no Jazz application. This application is a slight variant of the one that's in the coat lab. So if you go over to the cloud Labs by the front of my o there is a Smart Home

Depot lab is building a washing machine. It is very much of this code. There is this is all so I'm just so you can that you can play with it yourself and use this if you want as a reference in imitation, this is not a full you would not want to use this as it is for your iot product in Market cuz it doesn't have a persistent database. It doesn't have a lot of the user handling that you'll want for for you're obviously one product imitation including a lot of these traits here. Answers to show you a little bit but we're going to demo here is primarily object detection.

Like like Michelle said but instead of doing mail because he seems our extensible. I'm going to show you how you might build a product that detects like in my in my in my neighborhood. We have some other needs for this sort of thing and I'll show you how that works. But first, let me show you how you declare this. So we have a sink Handler the same Handler is basically just as on the OJs. It's they building the payload. That's actually my completely uninteresting the device list here in the references building devices. So this is a fixed list you run this and because of the database

that defines a number of objects for you, you can use Protest. The Hero in the washing machine is I was saying this washing machine implements often start-stop and modes and taco. So it's going to have modes for permanent. Press hot cold war. Quiet mode or bleach mode or whatever and it has a sensor built-in and then rebuilt down here somewhere a mailbox from say these are all the traits that we have used. Our mail Taco here is is a little things all the things that the light in the box and the sensor and the flag and injection in

the cameras theme support of the sea all of these in action go to the rest of that. Let me show you the notification notification in this case again. No JS, very simple. It's taking a pass through from the payload from a HTTP endpoint is a box calls our invitation in Firebase and it's simply that has ignore the fact line. It's simply sends this notification back through. Freeport state report stated notifications use the same call. This uses a Java web token to call us at anytime now you just think that's how you do stay before adding and it's how you do notifications. And

this is running. It's running in Firebase. I am going to show you if you can switch to my phone for just a second. I'm going to show you I'm already set this up. So are you connected to this agent you can see about all the virtual devices this when advertisers which of the devices in the sandbox. The mailbox here at the resort accident commands on the mailbox right now, but I want to show you object detection detection. In this case. Like I said, we implemented instead of doing mail. We hunted animal detection because I

actually have two anyting doorbell face recognition, whatever this was supposed to be originally a live demo demo said no that was a liability and I was being grossly irresponsible. Thanks. There's a raccoon in the mailbox kind of a lame. Kind of a lame demo. We are reusing the the detection Logic for mail and it's a little harder on fuzzy animals cuz it's not fuzzy logic. So it's a little tricky there, but you can see just like with the nest facial recognition which to be fair is many many many more thousands of engineering hours

of time. Then that raccoon demo partner can do what you do best. You build a cool device and will handle all the grammar the understanding the notifications. It's go to go to now you have access to the ecosystem in the house. You don't just send message to the phone. You can do notifications around the house of things and we'll see over the over the coming coming weeks and months will the adhan notifications of a lot of different types to existing traits for security events safety events other notifications, you can imagine, you know, Lindemann the oven timer is going off in the other

room. You want to know the roast is done that was such a notification that are available and you can now be too late. Ecosystem and these are things that we were significantly over the coming months and do this with all of our ecosystems. You can run it on its ass on Fire based on your own cloud on your existing infrastructure. You can run anything you want on your client. We do have Android things you should absolutely in this space is go and see the Android things sandbox to some very

cool stuff. We have some cool projects where you can easily integrate Android things with this morning ecosystem will be doing a lot more of that. Someone just showed me this walk through the election last year, but this walkthrough, you can find my searching for home automation and Android things somebody did a very easy walk through demo of Android things running directly with the API and you can do that walkthrough. We also have we also have several open source projects natively integrated today. So over here we have

home assistant somewhere out there in the middle and we have openhab which are both right there in the middle which are both open-source platforms that are already integrated. So if you want to play with smart home devices and don't want this on awesome API, you can use those for your projects as well not even asked for extensively in the last few years or last few months 6 months is we have done a lot more work to integrate developer tools into the Google

ecosystem for smart. So this here is the stacker. I replaced it back to the laptop. so this year it's hard to read I apologize is the is the staff driver for this account so he and all of the errors that we see on our side and I had to generate some yesterday to get this but last week of these errors are things that you would not see because they're outside of your system recall you be coming to call the mailbox and how they are serving mailbox failed

you know but if I open Office failing if sync data is wrong if the notification that you sent me to deliver it after you've given all those things are errors that we don't have real-time reporting so they're not going into stackdriver you can integrate this with your service you can use this to be bugging you can Pub sub from stackdriver I just a clever alerts and all of that is is not available and highly recommended mix and developing process much easier that's really end-to-end on the demo I just said please visit our visit the

Androids are the least we have we have we have our development teams ready to support developers. We have folks as a person's online and have a dedicated team of folks. Just building new traits build a new device support every day. You have a lot of stuff that is available, but not yet. I'm a public website. So if you see something that you want, it's not there talk to us. It's probably tell me what he built. And if not, we'll help me build it for you. So talk to us and that I will turn it back to Michelle for one more thing. One more thing. Thanks, dude. That was

awesome. Direct me to have a name and have a name. So we do have one more thing. So we are announcing here that we are also opening up new apis for control of entertainment devices. And if you been to the sandbox, you probably seen the demo that we have in the booth of the dish. Hopper box. That's it currently working in there. That's our first of the nation will be launching in a few weeks. Take a quick look at it for those of you who haven't been there yet switch to HGTV on

the hopper. So there we go. It's switched a sticker to me. Switch to ESPN on the hopper. Yeah, by the way, that's marks in my cool little set up in our office at the mini smart home. So these New Media entertainment for devices that aren't Android TV and are not cast enabled and there's millions of them out there like the one in my living room. That's a smart TV that has no voice interface. There's no mic. There is no speaker. What we want to be able to do is light up the assistant to make

home control is here for those types of devices have specific to entertainment devices. So we have transport control music ability to move the media rewind fast-forward Etc. I change the volume up and down at the state of the media do recording understand what channel you're on used to switching to the channels. We played a lot of ESPN yesterday and in some cases. We are also doing media initiation and this is by Early Access partner only. So if you happen to be

at a company that's doing entertainment devices. Please come talk to us will be in the sandbox and happy to talk to you more about this. one of the thing that I wanted to mention in addition to our apis into touch on lightly toast lightly covered in the keynote, but we now have a number of surfaces in which your smart devices will be shown so not only do voice control on a variety of speakers, but now on the phone you'll have the new assistant app that will show the status of your smart device on the phone and also on all of the new intelligent displays so

multiple services for your smart device to be displayed on and interacted with Tamara mentioned this earlier that we are up to 400 Partners in over 5,000 devices. This is just some of the partners. We've launched Cincy ass, which was in January very fast with new partners like LG Hisense Vivint Panasonic would really interesting integration. So if you're thinking about getting started with a smart home, please come talk to us as David mentioned. We are in the sandbox where the Google Assistant sandbox with his tent a Wheels have a code lab going where you

can go try this out for yourself do some of the David just did get your basic project up and going if you are already a developer with us smart home, and I want to understand more about report States. We got a collab on that as well, and we will be hanging out in the sandbox after this. So please come and talk to us. Thanks very much.

Cackle comments for the website

Buy this talk

Access to the talk “Integrate your smart home device with the Google Assistant”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Access to all the recordings of the event

Get access to all videos “2018 Google I/O”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “IT & Technology”?

You might be interested in videos from this event

November 9 - 17, 2020
Online
50
21
future of ux, behavioral science, design engineering, design systems, design thinking process, new product, partnership, product design, the global experience summit 2020, ux research

Similar talks

Hiranya Jayathilaka
Software Engineer at Google
+ 1 speaker
Jen Person
Developer Advocate at Google
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Jonathan Gerrish
Software Engineer at Google
+ 1 speaker
Stefan Ramsauer
Software Engineer at Google
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Aylin Altiok
Product Manager at Google
+ 1 speaker
Nick Felker
Developer Programs Engineer at Google
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video
Access to the talk “Integrate your smart home device with the Google Assistant”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
843 conferences
34172 speakers
12918 hours of content
David Schairer
Mark Spates
Michele Turner