Helping growing teams build cool stuff and get it to the right people more effectively.View the profile
I'm an applied scientist with expertise in natural language processing, machine learning, and data science. My interests lie in finding simple, pragmatic solutions to real-world problems dealing with unstructured text. I'm interested in working on problems that involve balancing accuracy, engineering complexity, and understandability.View the profile
About the talk
While most developer resources are focused on a limited set of projects, there is a large set of mission critical processes that still need attention. Empowering business users to create the apps and workflows to solve for those use cases is the next step in your digital transformation efforts.
Learn how a securely governed, no-code approach with AppSheet can expand your team’s capabilities and boost productivity. See how applications with dynamic interfaces, automated workflows, and machine learning models can easily be created using data in G Suite, Cloud SQL, and other enterprise data sources.
Speakers: Peter Dykstra, Tony Fader
Google Cloud Next ’20: OnAir → https://goo.gle/next2020
Subscribe to the GCP Channel → https://goo.gle/GCP
product: AppSheet; fullname: Peter Dykstra, Tony Fader;
event: Google Cloud Next 2020; re_ty: Publish;
Hey everyone. My name is Peter and I'm part of the app. She product team. Hear a Google Cloud. Thanks for joining the session about how to build a high-powered. No code apps without sheet. I personally work on making app sheet as accessible as possible for business users that don't have coding experience to build apps without she. And I'm joined by my teammate, Tony, who's a senior engineer focus on building a, i into the platform to make it easier to use as well as into the end applications that people can create to make those apps themselves. Really powerful. Thanks for joining Tony.
Thanks. So today we're going to walk through the basics of how to build an application without writing code in order to digitize a common business process actually do that. We're going to look at a few ways to make the app intelligent and even more efficient by adding machine learning to it grateful for a little background, a real serious. John's that many businesses face is this growing demand for improving their internal technology, but they frequently lack the skills and resources to update and build those Solutions internally. Software that comes out of the box, rarely meets all the
unique requirements of that. A business has is a developing custom Solutions becomes necessary real quickly, but it's expensive. It's time-consuming and frequently just doesn't get done there. Just aren't enough developers to help tackle, all the opportunities for updating and modernizing process. Yeah, and it's even harder to use AI in machine learning to make your business workflows smarter. The skill sets for these are rare. It involves unique infrastructure for building your applications, and considering these things just takes time to time it. So this is just over head that most
organizations don't have Intelligent no code and platforms. Help solve this by Annie, Lee anyone to be that app. Developer AI expert, no matter what their background is baked into the Noco platform helps in three ways. First creation easier for the user, by guiding them along the common pathway. I say automatically doing data entry. For you answer, that makes governance even easier and more reliable in the long term. No code apps, including all sorts of powerful functionality including collecting or managing data. More efficiently automating, business workflows and sharing
insights with teammates thinking of ending things like business intelligence with dashboards charts and reports. Yeah. There's there's opportunity for creating apps that use features like those really all throughout an organization could be on the field with customers. It could be managing projects you could be in manufacturing or distribution facilities or in the office and no more corporate setting but it's the most valuable opportunities are with deskless workers. So these teammates don't necessarily sit behind the computer during the day are more
likely to experience, friction trying to collaborate with other teammates or access important information just in order to do their job effectively. So let's look at this example, how to get started building a no code application, that helps a manufacturing team performed quality control, inspections, more efficiently from their mobile devices. Specifically, let's digitize their inspections, taking it from a pen and paper process to a mobile app and then Tony can help take that app and make it even more efficient by building a machine, learning model into it, with
the, with just a few steps. So, how do you get started? The, to start the process of connecting a data source to app sheet to serve as the back. End of that application? That will define an interface in behavior of that app in app sheet before, we share it with the team. Once it's in the hands of end users, those updates that they're making in the application gets sent right back to that original data source. So let's open up a cheat and get started building this particular app. In order to get this app started, we want to make sure that we have the data source. Ready to go. In this case,
we're going to use a Google sheet here. And this Google, she has three tables. We have some recent inspection data. We have the parts that are being inspected, and we also have the employees that are performing inspections. So let's connect this to app sheet to get this app started. Going to connect to Google. We're going to find this particular Google sheet. An apps, you can connect to that initial inspections table. Then we'll want to connect to the other two as well. While it's connecting to those tables. What's happening behind the
scenes? Is that it's looking at the content of that data. And then building a relevant prototype automatically based on what I can find it. So you can see, we already have a an application here that allows for creating new inspections for viewing. Those existing ones. And then also viewing the inspections on a calendar based on the dates in that data. Let's connect the other two tables. And then, let's customize this interface. With these three tables connected. Now, one thing that she was doing also behind the scenes, was intelligently building relationships between the
tables based on what it could find based on the fields that it's looking at in these tables. And so, right because we have a relationship, there are there any employees, performing inspections and inspections are being performed on different parts. This going to be relationships between this data and that's already represented in this prototype of the application. If we look at one of these inspections you can see that the part itself has been connected and I can navigate to the part details. When I'm looking at this part, it has related inspections that have been in performed on it.
I can also see the employee who performed this particular inspection. I can go to that employees details. So with these relationships established that just gives me a huge head. Start on what type of functionality. I can expose in the application and it makes, it makes it a lot easier. Then to now, customize the app to be exactly what I want. For my end up users. I'm moving to the US section. You can see that there were a few different views created, but I want this application be fairly I want this application. Be fairly straightforward
and simple, so I'm going to go ahead and remove a couple of these views and I'm going to customize The inspections View. And make sure that there is a dedicated form View. To create new inspection entries. You can see throughout the editor, there going to be suggestions for what features and what elements might be helpful in your application. In this case, when I'm looking at the view types, I have an automatic you type that has been selected. I'd like to take advantage of these images that
are being collected from these inspection cards. Use a card view. Actually to better highlight those images. But the advantageous about this card view, is that it provides a really easy way of defining, which Fields get exposed in the app interface. And so here, I'm just going to go through some of these different fields. Maybe I want to show the employee and the date of their inspection up top. And then the main image is already selected and I really want to highlight whether or not this inspection was accepted or rejected.
Until I make that the primary header. and then I want to make sure that if it is projected that the description of that inspection gets highlighted, I can I can include actions in this view to help navigate or initiate workflows. I'm going to remove those. It's nice and clean it. So now I have of you here on the right showing those existing inspections that have been performed that I'm starting this application with. Last up here is because these are inspection cards. I'll put a relevant
view icon of a card and I'd like alongside this a quick way for my application users to perform do inspections. When creating this view I'm going to make sure that it is adding new inspections to the inspection stable. I'm going to make this a form. And then I'll make this form icon something relevant. Like a magnifying glass and a plus sign. Great snow. This application consists of two primary views, a way of adding new inspections, and a way of viewing, existing inspections. I'm a go ahead and save this app. I
want to make sure the app logo is very relevant to my abusers. Recognize the application. And then because an important part of this application is really quickly, identifying inspections that have been rejected when I add a format rule, it just calls attention to rejected inspection. then this case I'm going to say forever in Spec shyn, where the condition Equals rejected. I'm going to apply a format to that particular condition field. And in this case, I want a Let's see, a little alert and I want the text to be
read. You can already see her on the right. They were previewing that affect. So I go ahead and save this great. So now I'll open this application up and preview it in tablet form and you can get a better sense for how those inspections that have been logged by my employees. Are visible and readily accessible. And I can also very quickly get to the details of the different employees and see inspections that are that they have performed or for example, look at different parts that are being manufactured and see related inspections.
So one reason why this is valuable to employees who said it's digitizing the process and it's making this information more. Reliably accessible long-term, another really important part of this application is getting the information in the into the hands of the right, people more quickly than than it previously did. So, one way that we can support that is by building a workflow In order to notify the right person of the right team as soon as an inspection has been logged as rejected. Let's build a workflow that sends a notification email to the right contact. Whenever a new
inspection has been logged that meets this condition And this case we're going to continue. Looking at that inspection stable. And we're going to say that every time a new row has been added. And the condition equals rejected. They're going to send an email. I'll send this email to the operations team. And a lot of relevant email subject. And then we can actually use a Google doc as the template for this email. So on the side here, I created a Google doc, that is just a simple layout. That
includes some of the fields from my updated. I can use this as a template for this particular email. as if I include a link to that template every time this work flow will run, it will draw from that template and send an email formatted based on that. Google doc layout inspection to this application. We can use a photo of a recently, inspected tags. Come off, one of the parts. This part has been marked as rejected and I'll fill in some of the details here.
Submitting a form in a nap. She dap passes that data right through to the connect to data source when this case is going to add a row to the Google sheet. And that added row triggers the workflow that will send the email notification to the team. So this point, the app is ready to start being used and I can get it to the team. By sending individual email invites, I can invite just a group from my organization or I can make it accessible to the entire domain. And then I can share a link that is accessible by any user on their mobile device or
on their desktop and it's already ready for the team to be using, so they can capture their inspections digitally instead of just with pen and paper. So they're already using it. After already collecting these tickets in the application, the real gains from this particular app or going to come by incorporating this AI model specifically using some of that existing data to build an OCR model so Tony can you walk us through that stage right now? So, Peter just talked about how to build this cool quality control app that allows you to move from a
paper-based processes through digital process, where instead of employees going out, paper forms, are there using this mobile app to take pictures of these tickets and transcribe the information in them? So what I'm going to talk about next is how to take advantage of the data that is now being collected via this mobile app. You can think of every time an employer uses this app, they take a photo of one of these quality control tickets and an interesting information. So we're building up that they just said now we can use AI machine learning to take advantage of the date of that we've
collected and further enrich the app by making it more efficient and easier-to-use don't. Let me give you a quick tour of some of the AIC teachers that are available in a cheat. So I'm going to navigate to the intelligent tab in the actually better than the first feature. I want to show you who's the smartest assistant teacher. So the smart assistant feature, enables a Google Assistant like experience inside of your mobile apps, but customized, do your business domain? So once you flip the switch in the editor, you get your own custom assistant that understand natural language. Give you a
couple examples. So the access the assistant, I go to this view that's available in the side, menu, in my app assistant. And here I can do things like this. Can you show me the inspection from last month? so distant takes my question like this and understands that in terms of the data and Concepts that are in my app, Can you show me? Just the rejected ones. you can see that it understood what I said and further filtered, the dated down to, just the rejected inspections, Can you show me more information about the rotary piston?
So this is just a couple examples of the types of interactions. You can have with the smart assistant teacher and actually I'd like to show you our or optical character recognition features. So let's go back to the inspection date of that. We've been collecting. You can see I've got a quality-control ticket image taken with a mobile device and then I've got the information that we transcribed down below. So in our current process, the employee needs to both take a photo and then manually transcribe. The information from the ticket. I'm going to show
you how you can use OCR to automate this process and go directly from a picture of one of these tickets. And I have a filled out for him based on the content from the image itself. So this OCR feature leverages Google Cloud Vision API business users have access to the years before, he'll let me walk you through the steps that you need to take in order to add this type of OCR capability to the app. The first thing I'm going to do is create a new o c r model.
So what else our model is is it's a way of looking at the examples of the data that we've already collected and coming up with a machine learning model that can extract from new images. But haven't been seen in the data set before. So the first thing I need to do is choose my table. In this case, going to choose the inspection stable, because that's the day that I wanted. Use this on to go from the picture of an inspection ticket. Do the extracted fields. Next, I need to specify the image, in this case, there's just a single column in my dataset called image. This is
going to be the inputs of the model, then I choose the output. These are the columns in the data set that I want to put it. In this case, it's automatically selected the other columns in the table like condition part. Number quantity, Zillow, So this is how I specify the model. And when I say what's going to happen, is if she's going to look at the data that's already in my table, these examples, you see over here on the right and it's going to use a machine learning model to figure out common patterns of where these values appear in the images. Cider press
save. Cool. Now that the pages be loaded the o c. R model has been trained. We can actually expect it to see what it learned from the dataset. I'm going to scroll down a little bit. You can see here below my mobile. I have the examples for my training set, if I click on them, You can see that it's found out where in the image, The Columns in my table are found. So, for example, that yellow piece right here at the bottom is identified as where the description is located in the image and
identified that the word rejected up here corresponds to the condition field in the same table. So it's gone through these examples that idiot regular you say. Yeah, I figured out how to generalize from it. And now we can use it to identify where these fields are automatically in new images to give you an example of a, to go back to the actor in America, look on the new inspection button at the bottom. You can see this is the form that the app creators have been using to fill out the app so far, but I'm going to upload a new image to The
Forum. This is an image that wasn't already in the table so that hasn't seen this before. So, I selected the image. And I can see that it's automatically filled out the rest of this form with values from the image. So for example, I got the date as July 26th 2020 from the upper right hand corner of the image. You can see, once you start using a digitizer paper-based processes, you start getting compounding returns. Once you collect the data set, it becomes more valuable because you can use AI, machine, learning algorithms on it. How are you don't have to be an
expert? Use them anymore, we provide no code interfaces to it, that allows business users to create their custom models for their particular business. Domain machine learning capabilities that traditionally only available to train software engineers Great. Thanks Tony. So in 15 minutes, we've digitized this inspection process with a custom app that uses ml models built-in, you know, an adult to existing business data and what we've shown can really be built by anyone on the team who sees an opportunity to innovate and who understands, you know how to
improve this particular process. So I just expand on a few topics that are related to what we just showed in the first is, curious know, based on this application in and where it is right now. Where do you think would be some natural next steps to take it to extend the functionality for this particular scenario. That's one thing I didn't talk about my demo was a cheese predictive models so pretty good malls. Allow you to automatically categorize records based on the data that you collected. So far don't forget ample in this inspection ticketing
scenario is that categorization to Route failures of rejections to the right team. I'm so this is another way that you can use. Cloud a high-powered machine learning without having to be a programmer. Great. Yeah, it's in the example that we just showed and then and then also the extension you just described. Is there a particular example of a customer that has built a similar solution and is using it with their team that you can describe a little bit? Yes, we've got
a customer from the construction industry who is starting to roll out an OCR power nap and see what they have is in a cheap power to where they have truck drivers driving, picking up goods and delivering them to work sites. And every time I do this, they have to use their mobile phone to take a picture of like a ticket or a receipt already using an app, she to have to do this and then a q, a person Downstream goes through those images that they were captured and manually transcribes, all the way to the date and the time,
because they're already have his workload built on and a half, they're able to just extend it further using the the platform's otrk capabilities. Are you a person doesn't have to go through and transcribe every ticket which would take a minute for every ticket and they have hundreds of these tickets. Now they can just actually have Go see our system automatically, fill out the forms and then I can just verify to make sure it's the correct. So this just can you just give us a quick synopsis
or description of what is around the corner for a i in the option platform. Yeah, so is a couple things I can highlight. So one is, the platform itself is becoming more and more intelligent giving greater suggestions as they build their app, making easier to go from an idea to a fully functioning app. A second way that it's a moving things in allowing you to build workflows based on about AI technology, for example, having a workflow where you upload a document automatically categorize routed to the right
person and classified trailer. And a third direction is a deeper integration with Google call Gene Waring offering to make it even easier for apps. Do you understand your unstructured data? That's great. Thanks for sharing Tony. It's really exciting to see. So how the platform they have some selves are continuing to get new AI features. So just to recap with Tony's described and and what we've shown here in in these couple demos is having a I built into the no code apps, as well as into the author environment. It really
just boils down to apps being easier to, to make and then being capable of, really powerful functionality. And importantly, all along the way because this is entirely in a no code environment. It's really easy to control with organizational level policies that are real simple to set up and they're easy to maintain long-term. So with that, thank you Tony for your work, making this functionality available and helping demonstrated today. And thank you, everyone who has join the session and are watching this video. I know I speak for the team,
when I say the word excited, to have new users joining, and creating really interesting and Powerful solutions for their own teams, there's a real strong Community about creators here at after she dropped Google. And then honestly all over the world that are ready to help share ideas and help you get started in the platform. So if you'd like to learn more and try it out, you can find apps using Google cloud or directly adoption.com. Thanks.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.