Robert is an engineering lead on Android Things. His team is responsible for Smart Display products, from prototype to production.View the profile
Shikha Kapoor is a product manager currently leading the Android Things roadmap for Smart Displays. She has an MBA from University of Chicago Booth School of Business, and a Bachelors and Masters in Computer Science from IIT Delhi. Prior to Google, she has worked at multiple startups, and also co founded a hyper local services startup based in India.View the profile
About the talk
This talk will walk through the journey of building consumer devices like Smart Displays using Android Things, how platform features were ramped up to support product needs, and the vision to help OEM partners reduce time to market by leveraging Android Things.
Thank you for joining us here. And I owe today. I load everyone on the livestream as well. My name is Rob. I'm an engineering lead on the Android things team. And I'm joined today by chica. Who's the product manager on our team? This is a Lenovo smart display built using Android thinks you can think of it as a visual Google assistant or like a smart speaker with a display. You probably heard in the keynote already or maybe in Vince's talk earlier today at these devices and others from JBL and LG will be available on store shelves starting this July just a couple of months from now.
Today we're going to tell you the story of how we built these devices from for Prototype up to the production devices that will be available in a few months. Before joining the Android things team. I spent most of my career writing software that ran on other people's devices probably like many of you but building a new piece of Hardware. It's scale is different from the kind of work I had done before and it came with a lot of new challenges. And today we're going to tell you about some of them our hope is that you'll leave this talk with a better idea of what it's like to build consumer
Hardware at scale and how Android things makes it easier than it's ever been. Not that scary went, you know what to expect. So I hope that our story of prototype to production and help you build a device of your own. If you are a software developer thinking about billing consumer devices had a few things you probably care about you want to build your product features without having to worry about dandelion platform. You also want to be able to swap out and the line Hardware seamlessly without affecting your application layer has the technology electronic needs evolved over time. You
obviously want your device has to be to take your I think it's obvious why security is increasingly and essential requirement for smart devices. They hold the keys to our homes and our personal information what's wrong with the less-obvious is why it's hard to provide this kind of security. It's incredibly expensive to harden products out of the gate and to keep them updated overtime. Meanwhile, there's a man's cause pressure from an customers and it's hard to immortalize cause without large-volume Vino the side Google since we've been doing this for many products. Coming back to you.
Finally if you're building a consumer device. You'd want to manage release and update channels over time. I know things was designed with all of these needs in mind. Be provided turn key hardware in the form of pre-certified system on models or what we call a song the song includes the CPU Wi-Fi bluetooth memory and other critical Hardware, think of this as a small Computing platform that you can use to build a variety of iot devices. We bring up the operating system on new song tested and certified suitability for production devices so that you don't have
to do any of this later on in the dog will give you an inside look into some of this word that Google handles for you you provide application coat and white of the peripherals and Google takes care of the rest. What do you see here are smart displays from Lenovo LG and JBL as Rob mentioned. They are built using Android things. In fact, if you look under the hood all the products have the same song which is pretty cool. These devices have far-field microphones that can hear you from the other side of the room and built-in speakers that can answer back across the room just
like a smart speaker. But now we also have a touchscreen which makes it possible to see visual results. This is why cool of the result happens to be a list of restaurant nearby automatic showing you your commute conditions be announced this earlier this year at CES and we're very proud of the reception be deceived. Let's dive a little deeper and see how we built these. About a year back. We are Google were wondering if we could build a visual Google Assistant. This would enable deep integration with other Google services such as YouTube photos math answers.
Building a consumer device though is typically a long process, but for the fast-evolving space like smart home devices time to Market is extremely important if you want a quick turnaround time, and this is exactly where Android things shine. Our goal is to make it possible for software developers like you to build consumer devices at skin, but that's a hard thing to get dried in a vacuum. It takes Real World experience and billing and shipping a lot of devices to know that you've really done it. This project would give us the perfect opportunity to do that in this case. Our
customers were software developers from other internal teams at Google such as the Google Assistant smart displays while they took the lead on welding application software. We hydrated on the platform as a result. We were able to get these from prototype to production super fast, and we now have a stronger platform for you to build on. We started by building some proof-of-concept hardware for smart displays are going west to Lonesome that and to use the device to take off daughter discussions across the company. So we took a few weeks and airpods Adventist. This is our first proof-of-concept
Hardware. This is running on some Hardware that we had use previously to deliver an Android things powered smart speaker. I built this case with my son cost me about two Pokemon cards. He still wonders when he's going to get his brake back brakes back though. This device doesn't have all of the hardware that ships on the final devices. For example, it's missing a camera. It doesn't have the volume up down buttons or mute of any sort out, but that wasn't the point of this device for this one. When we were trying to do is show just the basics of what we thought we could achieve on the with
smart display product to get some conversation started are the mic on this one in particular was kind of tricky to use. It wasn't meant for this kind of abuse case. You needed a quiet room and you had to stand in just the right sweet spot, but with a little bit of practice it was possible to pull up a pretty convincing demo with it on top of the Android things platform and the smart speaker code that we already had we built a small Android application to drive the display and show off some of the key use case. We thought we're going to be important for this kind of device. I want of the
first things we did with the displays to transcribe your query as you spoke. It was a basic proof that the whole speech recognition pipeline was up and running properly and it looked pretty cool in a demo next up. We added some support for rudimentary what we call a knowledge card that'll show a brief snippet of text and a photo along with answers to certain questions. This device couldn't play video yet because it wasn't it didn't have the hardware required for that. But you could stream music already just like a smart speaker. So we had at title track display and immediate transport
controls like pause and Skip and then finally we took a rough cut at the ambient screen, which is what the device does when it's not doing anything else. It showed a photo slideshow and weather forecast. So like I said just kind of a basic set of functionality that we could use to show people what we thought we could build out of this it only took about two weeks a small team of people to get this working but the results was actually surprisingly close to the architecture of the final products that are shipping in a few months the changes between what we've built here. And what
we're shipping are basically incremental changes fundamentally what we built in what we're shipping involves the smart display application at the top of the stack running on top of Android things on top of a certified what we call of some Android things brings the hardware supports the security the application framework and the update mechanism all the smart display application brings all over the Google Assistant integration and that the visuals for the device don't get me wrong. There was definitely a lot of work to do between here and what we're shipping this summer, but as I said, the
fundamental architecture is the same and it was a matter of incrementally building out the feature-set from here. Are we practice the demo routine with this device and set up meetings across Google over the next few weeks people got kicked out of the brick case that we built and we had a lot of great discussions and some debate about product features. We should support about Android things in about what we should do next. Overall the demos went really well and we got to buy in that we needed from Key stakeholders across Google to move on to the next step with this project. now
that the Was going to be real we can go for a few steps in parallel to scale up. Our efforts are business development team started discussions with potential OEM Partners In this case and we wouldn't go to channel equipment manufacturer reference to a company that will build and sell the device. They come up with the industrial design and the physical look and feel of the product. They also pick Parts like the display camera microphone and other audio components. If you were to build a device on Android things, you will be the one responsible for kids design and components election in
place of the step. You might find an industrial design work from home to work with by the factory and olfactory to build your product. A lot of details have been handled by our OEM partners and you're not going to cover them today. Intitle, we started finalizing the hardware requirements for the product the proof-of-concept device use the song that we had used to balance my speakers, but you can build anything else from it that has similar requirements as well for smart displays, though. We needed something more powerful to support use cases such as video playback. Fortunately. We had
some some options in a road map that we started evaluating. Finally in Filer to the Williams collection and sound selection. We got started on the application software Google Assistant back in and client software for smart speakers, but darling a smart speaker into a smart display isn't just a matter of plugging in a touch panel. There's a lot of new work around the home screen the video playback experience the maps and photos integration and essentially all the individual features that we were trying to Velvet does turning the application software from a humble proof of
concept device and reproduction device was going to take a lot of time. Most of the software work was responsibility of developers outside our team and they needed to get started right away these Engineers what software developers like many of you don't want to have to think about the underlined platform or the hardware. But because we are building this on Android things they don't need to like we mentioned earlier we can pick up a little. Plication Engineers work on top of the Android framework just like any Android developer would their application can run on any hardware with
Android things support. So at this point in the project, it's July of last year. We're about a month-and-a-half into the work. We got our proof of concept Hardware. We've got some business development work underway or got the hardware selection underway, and we're nearly ready to get started on the application software development in Earnest. But in order to really kick-start development, we need to get some actual Hardware into the hands of Engineers a product managers and other folks at Google who get them actually build this out. But as we said before the hardware that we used in that
proof-of-concept device was just a little too slow to use for more than just a demo. So we look through the songs that we had in our Pipeline and found one that met the requirements for this category of devices and it happened also be very easy to buy in quantity over the internet and we had Android things already kind of running pretty well on it. So we decided to use this as the basis for some provisional Hardware that we would be able to unblock the application engineer so that they could get started on that on their job while we finished out the hardware bring up. We knew that we
would Swap this Hardware out later when I got closer to production, but that would that didn't worry us because as we've said we're building on Android things and that gives you the isolation from the hardware that you need to be able to do that. The second problem that we had that with the provisional Hardware was that it was a development board kit similar to the imx7 kids. They were giving out here at i o which means that it had a display in a camera connected to the baseboard through flat Flex ribbon cables. You can obviously work like this but it's a little difficult to manage
if you have more than one on your desk for example, and we wanted to give these out to a lot of googlers after the first few weeks. We literally held the thing together that are at our desks with cardboard and rubber bands, but that wasn't going to scale to the quantity that we wanted to deliver for that. We needed a more solid and closer to keep everything in one spot. And by the way, if you do complete the scavenger hunt here in Ohio and you get an IMX 7 Kit, make sure to keep the packaging it turns into a handy stand so you don't have to build one yourself. Also, this is the enclosure we came
up with for our first provisional Hardware. Honestly, Android things didn't really help or hurt either way in the building of this case, but I had a lot of fun making it and I learned a lot in the progress in the process. So I hope you don't mind if I could go out for a few minutes and tell you a little bit more about this design and how we built it. It looks pretty playing from the front. The point is that it's just enough to keep the hardware together and give you a rough sense of the form factor that we had in mind. There is a camera module at the top some microphones that we had
manufactured locally here in Mountain View in the middle of the top. And of course the big touch screen in the front. Here's a quick view of the back the song and the carrier board are mounted in the back of this big piece of wood that I called the bezel here. Just like most of the rest of the hardware you can get to most of the ports easily from the top and there's a little access for the reset button on the side, which is very important when prototyping. I want to point out here though that pretty much none of these components match the final Hardware obviously said the song is
different. So is the cameras are the mics so is the display but again not an issue because we're building Android apps on this and they don't really care so much about this Pacific Hardware underneath Building this prototype was my favorite part of the project because at the time that deadlines were far enough away so that they don't make that crushing sound they do when they get a little closer and we were still a small team that needed to be Scrappy and pretty resourceful. So I literally built this case in my garage over a few weeks and the best part of that is that I got to buy a new
tool not just any tool but a 60 watt laser cutter with your course. I got off the internet. I literally learn how to use a laser cutter for this project is overrated on the design in my garage over the course of a few weekends is my first time using any kind of 3D modeling package to do this kind of design my first time ever using a laser cutter and I basically learned by watching videos online and the point here is that if this is the kind of thing you like. Yes, you can do it too and you'll probably have a lot of fun doing it. The material that I tried first was this birch plywood.
It's about 3 mm thick are you can get it in quantity from the net very easily 2-day shipping. I Instax it 25 or 60 you go for the bigger. If you go for the back to 60, it's maybe a dollar and a half a sheet or less. That's pretty cost-effective. When I first tried building this out. I wasn't very efficient with my material used to would take maybe two or three sheets of plywood to build enough parts for case, but eventually you want to get a little more optimized waste a little less material. And as you see here, I managed to fit the whole case into a single 12in Square sheet of plywood. I
did have to fit one of the sides inside of the bezel to make that work. It was a lot of fun to kind of it was a big challenge to figure out how to get this all into one sheet of plywood lot of fun to overcome that but laser cutters only cutting 2D which meant it's a little hard to work a faster like a screw into the into the design. So this one is all friction fit. I came up with the tab and Slot design where you could put it together by hand and they were Stay Together pretty securely in the best part of that was that it was actually really easy to service in the field if a connector comes
loose, so you need to change a jumper you can snap the thing apart with your hands. You don't need to find a screwdriver you fix it. You put it back together and you're good to go. We tried a few variations in other materials such as the acrylic you see here. I think it looks much better than it would obviously this semi-transparent. Red is really cool. But what we found is that an acrylic the thickness for us at least the material we had the thickness very too much across color and even with in color so that when we put some of the cases together, they just fall apart on a desk or they fit
together. So tightly that the acrylic would snap at the weak points, which turned out to be the 90 degree Cuts in the material. So we decided to go with the wood after all because it was going to be more robust for us. I cut about a dozen enough parts to make about a dozen of these cases. We brought him into the office had some people assemble them so that we could try out the design. Generator down a bit eventually settled on a final design and sent it off to an internal shop at Google with a laser cutter that can cut these in bulk. If you happen to work for a company that doesn't
have a laser cutter shop not a big deal. You can find plenty of places on the internet where you can upload your cat files, they'll cut them and send them back to you. If you've never worked with a laser cutter before though I recommend you try one out first before you send it off and commit to a football quarter get a feel for what works and doesn't work in this kind of design. I'm going to get in the more for this part of the story is that you can build an enclosure to and laser cutters are cool. So you should try one out. I one more thing before we move on though. I mentioned we wanted
to build a lot of these are we built 75 in the first batch to spread around Google and we thought that was going to be more than enough, but they were so popular that we built another 75 and even that frankly we could have used a bit more. So once you have this that this many devices and honestly once you have more than say one or two, you want to wait at disambiguate them to tell which ones which because it's 7 Point someone will have a problem. They'll say hey, man, can you look at this device and you'll say sure I'm busy right now drop it off at my desk and maybe they'll put it next
to the one that looks just like it while you're away and then you end up looking at The Scorch marks on the case to figure out which is which so I think ahead give each device has serial number keep track of it in the spreadsheet somewhere. So, you know how much inventory you have left to give out or Who's got what or maybe one history of individual devices are so we now have the hardwood for the type to hold over an application Developers for a short-term imperative. We were working on finalizing the song for smart displays the evaluate starting options in Android maps and
kick-started another series of discussions only this time it was with the soc vendors. Eventually selected a Qualcomm song called SST a 624 for the Lenovo LG and JBL devices that are launching this year. This is an eighth code 853 with an adreno 506 GPO that supposed to stay on camera. It's plenty capable of driving the smart display and of course a bunch of other potential devices that you may want to build a snowman on the SDS. 624 Google will offer free stability, Texas and security patches for 2 years without Sims 4 extended support.
To give you a more complete idea of the hardware. We are talking about Taylor has a photo of the top of the song you can see that I so see the Wi-Fi Bluetooth module is a memory wall mounted on a discrete board. Is the underside of the song the main thing going on here are those three high-density connectors with 100 pins each these bands make up the songs interface to the outside world. Has a little deeper dive into this home spin out interface the fence Gary Powers Epi lines and wide variety of peripheral buses in and out of the song. Wow, looks a little over
doesn't it? However, you wouldn't need to worry about designing this if you're able if you were using Android things to build your device has been assignment has been sanitized along with the rest of the song so that the underlined platform know what's expectations are in some cases. You'll also have the option of laying out the song says get flat onto your boat exactly as specified according to this interface. However, it's much faster and easier to just been the carrier board that the song model can plug into So if you get one of our developing Psalms, it'll come with that carrier board
that has some common eye opens broken out on do standard size header so that you don't need to deal with any of these high-density connectors, but we need to build build the device for real for mass production. You'll likely want to spin your own carrier board that connects to whatever Hardware you need for your device. For smart displays, we came up with a reference carrier board that we asked our OEM to follow you can see in the slide hear that in the middle of we have the song which again is a CPU the Wi-Fi and other important parts and the peripherals are connected around the outside
but we standardize on button and sensor Connections in this reference design by specifying which gpio as they should Connect into for example to simplify our work on the software side because we're working with 3 OEM simultaneously we wanted to avoid as much as we could unnecessary customizations across oem's. So everybody for example plug their volume up into the same. Gpio on the some one other thing. We specified in the reference design here is that the microphone mute circuit should be external to the Somme and implemented in Hardware. I so that when you mute these
devices the mics are electrically physically disconnected from the Somme so that even in the face of some sort of successful software exploit in the future. There is no way. Dot to unmute the device and software similarly. We specified as part of the reference design that the camera the camera mute should physically cover the lens of the camera so that there's no way to unmute the camera. So to speak in software. We did leave some room for OEM customization. For example, they could select their own microphones. They could build their own audio output stages and they had some
leeway on a display Choice as well. This is the next revision of our development kit. This is what came after the wooden kits that I showed earlier. This one is built according to the reference design and it's using a lot of the production components that we have in real devices this time around the entire front assembly was put together by an outside firm and we didn't have to do any of that this case happen to be an acrylic as you can see. It used to friction fit romick grommets to connect a standoff to The Back Abbey sometimes came loose if you over tighten them, but you could fix it
with a little bit of super glue. If you had two from the front I what we have here is the exact same touch panel camera and microphones used in some of our production devices. We added this acrylic back at Google which held the speaker in place and really just gave the front something to lean up against so that it was at a similar angle to what we were expecting for the production devices. Inside these are essentially the Qualcomm SDA 624 some plugged into a carrier board that matches the reference Design This is actually one of the earlier versions which we called the fep one man
in these versions the amplifier and the display interface. We're on daughter cards that plugged into the carrier board, but on later revisions, we had it all flattened down onto a single carrier board at we completed these kids in the beginning of December of last year. If you're not aware at CES is July 3rd, June January just before the Big Show where we going to show off these devices at CVS. So it was a little nerve-wracking especially in those weeks leading up to the delivery of
these devices where the Cs was getting closer and closer. But in fact the application Engineers had gotten off to a great start with the wooden devices and we sorted out the platform bring up on the Android things side relatively quickly. So it's really kind of amazing that one month before this huge demo at CES. We were able to completely swap out the hardware. We move the entire development team platform and application Engineers from those wooden prototypes over to devices running on a completely different SOC without a hitch and this is because thanks to Android things the smart
display application is totally independent of the platform. You can see how this was useful to us because we had this kind of provisional situation where we needed to bring up temporary Hardware before the real stuff was ready, but this can be really useful in production devices as well. You may ship this year with a certain profile a certain piece of Hardware that you need and then next year. Maybe you want to cost it down or you may want Add new features and switch to a more powerful piece of hardware and because of this you can easily reuse the application software across a wide variety
of Hardware platforms. surrounded as a reference Hardware was being completed p.m. Partners Lenovo, JBL and Elsa. You were busy building out their own devices based on the reference design that you showed before we'll even closer to see. Yes. They got them to just about 2 to 3 weeks to spend but again, the transition was relatively smooth. We hit a fuel pump. For example for devices the camera look green between eventually traced to a missing capacitor, but essentially nothing that you couldn't sort out on time. CS was a lot of work. I remember her
saying whoever it is who should use it. So close to Holiday is this is a picture of our wardroom that we set up a few weeks before CES. We spend a lot of time planning the demos adding features and fixing last-minute bugs on the device and on the backend servers that make it work on the demos. We plan to show Rich integration with other Google services such as YouTube photos and Maps. This is a picture of a little kids room that is set up in Mountain View vet sadness cameras pointing at it and we could check in on a little Android baby all the way from Vegas. I had work and careful
planning paid off and the demo is pretty much work. Like they were supposed to we had some issues setting up the venue and a few Hardware glitches, but he brought backups expecting that to happen. The reception the gods from the press and users at CES was very encouraging it work on something in secret. It's hard to gauge how the outside world is going to reactivate getting this kind of positive feedback was a great shot of motivation for the team and helps for the galvanized effort at the CES. It was time to get out of the devil mindset and back to the work of shipping.
At this point we were flush with Hardware from Orion partners and the program was ramping up even handing out the single software Bell to all our developers, but now we needed to expand our fortunately Android things makes the super easy using what we call as channels are Developer console allows you to manage release and update channels over time. You can get a lot more details on how this works in Adam and Eve east of tomorrow. So what's my displays we established multiple channels for an internal user segments depending on their needs for example, or developers have medium to
high risk tolerance and can replace components as necessary. We have a dead channel for them. We also have a cannery for hydrastar indemnifies very rollout daily dose. Of course, we have a different channel for turkey. Well testing teams so that they can give them early access to bills that we want tested. At this point we also have hundreds of googlers why testing out smart displays and giving us feedback on teachers these people need more stable channels and have no risk tolerance. So we roll out table weekly bills for them with known good functionality and we skip building K psoriasis.
We are getting close to the end of the dog, but I want to cover one more thing before wrapping up in this day and age if you are shipping software, you never really need to be completely done with it. If you're building a website on a mobile application. You can roll out back taxes whenever you will but when it comes to Hardware, you only get one shot at it. Once it's in the hands of the user that there are some things that even software cannot fix that. Point. So you better get it right. We also just cannot keep the factory running continuously dough making small tweaks here. And then
you need to purchase your building materials in bulk. And since it's still early in the process. You're not going to be ready to buy in mass production quantities also the tooling and testing take some time to set up and change. So this process usually happens across a few bills that are standard in the hardware development life cycle. You can read up more about those on internet, but I'll cover it and now we play today. So we already talked about the Prototype is the next phase is the EBT or the engineering validation test for this phase. You're trying to validate that you got your
schematics, right? You don't have any layout issues and your compliments work with each other the way they are supposed to if you're still considering different design options you can I play it on them. This is a good this is a good time to do. So, this is essentially represents the first attempt at building your bored the way they'll appear inside. Your final device closure is typically still not final at the stage. So you may want to keep the boards for glutes through just to keep them from flipping around. The next stage of the DVT or the design validation test. The purpose here
is to verify your production processes and equipment to ensure that their meeting your final specifications the devices regard for CES. What are what we got called l'esprit DVD Davis off tool which means that the most used to cast the plastic sweat made of a soft material like Silicon again until you get to a point where you know that there are no fixes that would need a Hardware Easton you. The quantities that you get more confidence and eventually want to get to a run with Hart tool Plastics. The btter the production validation test is typically the last belt that
transitions into mass production. Hopefully by now, you've logged in the final software that you will be flashing in on these devices at this point. You're also hoping that your factory is pretty much running on autopilot without any manual intervention. Finally our product is ready to go to mass production which means it's ready to be on shelf at this stage. You don't think of production and focusing on making process improvements to improve the hint obviously want to continue testing it for the next few months though as well as established some sort of reliability testing. So damn, that's
how people smart displays from prototype to production to dehydrate having Android things as on the line platform help us and every step of the way first time use Android framework or software developers could focus on building application features without having to worry about the underlined platform. I system-on-module or Sound Solutions allow them to swap out on the line Hardware seamlessly as we moved from one prototype get to the other without affecting a player and yes while we needed it to do a lot of work to bring up the songs for smart displays. You don't need to do any of this.
That's my display song. They'll be ready for you soon. I update and device management features allowed for fast titrations and easy maintenance of multiple devices on the field and finally. Are security features mean that our OEM partners and users don't have to worry about the security of these devices. We have a lot more to show you what I owe or come and see us at the Android things sandbox. Don't what you can see on the left side of the map here entire. Dome is filled with Android things demos and it's open after hours as well. You can see how we leverage machine learning
Firebase and the cloud platform and you can also have an up-close look at the prototypes and the devices that we talked about here today. By the way, you'll notice Android things powered devices outside of the sandbox known as well. There's survey boxes all around I owe that are built on top of Android things and a large human-sized robot roaming around ready to take a selfie with you so you can come to our code Labs as well Chao serve in the middle area of this map are we have codelab showing how to work with tensorflow how to work with cloud iot and how we made the survey boxes that are
around Iowa and even more there is seven more sessions on Android things coming up today and tomorrow may go even deeper into some of these topics. I'll finally have some fun and I hope we have I scavenger hunt going on showing at the link here G. IO search I won't give away too much, but you'll run into some machine learning and has flowers, which is pretty cool. And if you solve all the puzzles and make it to the end then you'll get a piece of developer Hardware to take home for yourself with that. I'd like to invite you to go build some hardware, and I hope they were leaving you with
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.