Events Add an event Speakers Talks Collections Sign out
 
Duration 18:52
16+
Play
Video

… or how I learned stop worrying and love the chatbot framework | Rasa Summit 2021

Heather Nolis
Software Engineer at T-Mobile
  • Video
  • Table of contents
  • Video
Rasa Summit 2021
February 12, 2021, Online, USA
Rasa Summit 2021
Request Q&A
Video
… or how I learned stop worrying and love the chatbot framework | Rasa Summit 2021
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
637
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speaker

Heather Nolis
Software Engineer at T-Mobile

Product development software engineer with a background of neuroscience research - with a penchant for absurdist machine learning projects and special needs cats. I'm a founding member of the AI @ T-Mobile team, which uses cutting-edge machine learning to revolutionize our approach to customer care. I have both a BS and a BA (in neuroscience and French respectively) from the Centenary College of Louisiana and am wrapping up by Masters of Computer Science degree from Seattle University by June 2019.

View the profile

About the talk

In just a year, the AI @ T-Mobile team has gone from creating models exclusively in keras or tensorflow with no supports to fully embracing Rasa - and not just for it's chatbot functionality. In this talk, I will go over how we've used Rasa to stand up a beefy customer service chatbot in a company that highly prioritizes human-to-human interactions - as well as the surprise lift our data science team has found from leveraging Rasa models outside of chatbot use cases.

Presented by T-Mobile Sr Machine Learning Engineer, Heather Nolis at the 2021 Rasa Summit (https://rasa.com/summit/).

#conversationalai #aichatbots #nlp

- Learn more about Rasa: [https://rasa.com​](https://www.youtube.com/redirect?even...​)

- Rasa documentation: [http://rasa.com/docs​](https://www.youtube.com/redirect?even...​)

- Join the Rasa Community: [https://forum.rasa.com​](https://www.youtube.com/redirect?even...​)

- Twitter: [https://twitter.com/Rasa_HQ​](https://www.youtube.com/redirect?even...​)

- Facebook: [https://www.facebook.com/RasaHQ​](https://www.youtube.com/redirect?even...​)

- Linkedin: [https://www.linkedin.com/company/rasa​](https://www.youtube.com/redirect?even...​)

Share

So I'm I am having all this, and I meant in your machine learning engineer at T-Mobile. And I am here to tell you about how I went from being somebody, does everything organically and rock Harris to actually loving shopbop. Bring work a little bit about me. That's necessary to understand how I got here is again. I'm sure I know I've been at T-Mobile since 2017 and I used to want to get my PhD in Neuroscience. I had a year-long project that was a study in wraps, all right, with having to measure and take your blood pressure and

collect all this data. And then it turns out that when I was done, they were like, Heather goodenow and Regina over to an analytics team and I was like, hey, I absolutely cannot let that happen. This is my data that I treated myself. So I'm qualified to do the analysts. See that you wanted me to hand it off because I have a control problem with computers and programs. By not really wanting to give up my data in the team that will work on it. We are called AI at T-Mobile. In our scope is, primarily customer care. So that means when somebody is calling T-Mobile

that they have an issue. If they are dealing with us on Twitter, if they are texting with us in the app, I don't go Liz to use AI to make that the most seamless transaction possible and whatever. And so, what this means is that we have a team that is fully staffed for real-time the ideas from the date of people who are doing the analytics to the products, people who are having ideas to the off, specialist in developers who are delivering it my team contains, every single thing that you need to take an idea to development appointment to support. And before I

can tell you about what I'm here to talk about today, which is pretty remarkable, you have to understand where T-Mobile has been. And so on in August 2018, I just want to set the landscape for what that looks like, so we can compare it to our current state. So in August 18th, 2018, T-Mobile these things called uncarrier, move where they hold industry initiatives that show, how we are different than our competitors and how it seems like, we are listening to our this commercial And it went on to say, I'm talking to robots is the worst. And

at T-Mobile never. We will never have a robot. Don't worry. Chime to T-Mobile. So you might say, why am I here talking to you? And if I've ever typed was that a company that just three years ago was probably Auntie robots and how could you have an AI team that is probably anti robot suit? Just have to contact Darren. We had a project called expert assessed and how it works is when somebody calls up T-Mobile, they say, I think I forgot to put my bill. Can I do that? Now, what we want to do is use machine learning to transcribe the audio into text classify that text. So we can

tell the topics that they're talking about and then go and get our T-Mobile expert, whatever information we can to help him solve the problem. So, if you see that there's like recommended responses that are listed over there, there is, it says, the topic is General payment up in the right-hand corner. There are account information Just pulled up and then marry Eden internal Wikipedia articles that can show the expert how to solve the problem. If they haven't done it, since I was. So this is the land that I was living in this building products like this and he were serving over two

million inside each word for experts. In the Carpenters are doing this. So we were building conversational neural networks that had some pictures of natural-language and some features of the actual out back and we were using these plans crafted neural networks to classify the topic of these conversations or to show their insights to people. And then I cheer today it's three years later and I am here talking to you about our chat, by almost the most unlikely person to be doing. So I did all of these hands by model work and I was at a company that

we don't like robot company from robots into having somebody be able to speak at a conference about robot project that we are. Very, very, very simply which we learned quickly that some customers prefer cell service. So the assumption that we made that people don't want to talk to robots actually fall for one-third of people whenever they have to call T-Mobile. If we would offer them a quick way to chat with an ivr system to solve their problem, 1/3 of people would prefer to do that. Then to wait 60 seconds to be connected to a light experts.

We also saw that in messaging so not calling but the people who want to contact us via Facebook or Twitter via are at the text messaging. The people who don't want to talk to him on the phone is increasing every year. So our volume is increasing through this channel that people realize I can wait, I can answer my T-Mobile problems on Facebook and then also people are getting more and more annoyed whenever they have to pull up the phone there, preferring to do so then it says, well we can only truly be listening to our customers that one-third that really doubt use a cell service

experience. If we are to do what we said, we wouldn't reuse go and build a chatbot. It's only had the idea, let's make a botched. And since I was already on this team that had this beautiful robust, internal topic model, we have a model that works. Let's just throw something on top of that and deliver some experience for customers that makes sense, right? Know, it doesn't make sense and I'll walk through the problems with that implementation and why we decided to abandon our love. For our homegrown models into

switch to the Raza framework how it works. For me, I had a team company and basically say, hey, I have 10 intense that I need. Can you make them quickly? They're they're, they're super specific and tensorflow model. They're cheaper than bottles and the architecture for artists East thousand, thousands of human crafted labels to create a new intent. So I was like, no, we don't have the data to create these new and since we started you say, okay, If I don't have enough data, we'll have time to create these new intends to leverage our existing topics. It's a

different kind of we ended up with 10 in that we were looking at. And I just wanted the math out the problems faced year. They've been in Houston for blue topic model. It has 88th and 10th is hierarchical. There's a Define taxonomy already exist. They are General topics like Network and make them went to run on a 10 message window to tell you really. But what has happened in the past 10 turns into a conversation about 2,000 utterances every single time you want to make a new a chance. But what I asked for was 10, Yuan Chen, some of them are overlapping overlapping with our old text

MonaVie and symptoms of relapsing with each other. They were highly specific and they're supposed to run on a single message and we had no date, no label data and no data labeling. So it's not very quickly if you try to treat our topics as intense we risk showing on since responses to customers. So one of our topics was General payment but the intent thought it was my bill. But there are so many things that could be classified as general payment that are definitely not pay. My bill. For instance, I won't pay my bill because I don't understand it. I'm

just checking to see if my payment has gone through and I want to change my payment method. So if we were to say when General payment is detected from the Malik to pay their bill and give them that body experience, we aren't selecting for the right people showing, if I want to change my payment method and we come back with great, you can pay your bill right here. It's a horrible horrible customer experience. And Then There came the Nuance between all the little things that the boche wanted to do in their experience. So specifically, I pulled out three of the intense that I was asked for

workshop for a device. I want to add a line and get a new device for that new line. And I went to Adeline and I'll bring my old phone and they wanted me to do this without your language. And so I pulled out some sample. Utterances that obviously are mine. So you can't see any Jester, a date of it, you can see very quickly have a natural language for all three of these situations with very similar. Buy a cool phone for my sister's lying versus by a coupon for my sister and Adeline for her to my account. I want to add my sister but she already has it. It was incredibly painful looking at this

and trying to think to myself how I would be able to do that in our giant topic model that already has 88 other intense. We had the idea of trying bras on and that was because I understand the machine learning up underneath as a machine-learning engineer is open source and I can check their coach brain works, though, it requires less than our current model and I was promised they would get to read all my refund favorite machine learning dislike me as well, and how do you can just use those? And then if we wanted to go down this path, we should probably be

messing about framework. I don't want to create one from the issue. Was that I don't get stakeholder by and spend time doing this to me and a data scientist on my team. Peter, we spent four hours Monday and just said, what if we spend instead of saying it'll take 4 months to get these? Tenants are in house and what if he stood for hours and plug it into Ross and see what? We come up with these are the actual results of that for our experiment. So many things like Send a message to see, there's not a ton of data in there, but but with only four hours of work, we were all

ready to show. Hey, there is incredible value in creating specific intent to the things that you want to show. And I don't think that standing up Raza would be double work. And so that was enough to sell them and they said, yes you're right. We should be using Raza. I had no idea that we could do something for our house, so incredibly long and actually got started was it was a stakeholders to try and get there by and based on the this experiment. Until then, there's the question. I've spent a lot of time on a beautiful handcrafted machine learning team

and now I spent some time on my ride, the team and what what's different when we decided to go. Again, just to remind you that our model would write on a window of messages. There's $2,000 ends is minimum for an intense and it has about 80% accuracy. And that was the world. We knew that Rotten Tomatoes are slightly different because they're faster and are out of the box after a few was slightly higher. But that's not the remarkable Parts. The pace is, what is absolutely astonishing with our tensorflow model that we had in Market. It was a market for 2

years and the team that supports it had been literally hundreds, if not thousands of production releases and only a handful of those included model update and in that amount of time, we were only able to add invalidate 2 + 10. I think you look at a bother and I'll you model and I spent 5 months in Market, we have had 43 Productions, releases, 19 of those have included the model and we have added since our initial launch 28 and 10 speed that I never thought was possible, especially creating a foreign language classifiers. And so, how are we able to

move that quickly? What about Raza does it for us? And so for me the number one thing, it's visibility will using rasa X disability into the actions that my model is taking comes out of the box. So as soon as I release a new model, I can be in there. Actively reviewing, every single prediction, is making it making sure that it's working in real time. And it's beautiful. Driving through the swamp logs and going into three buckets of having to use to pull out confidence, interval of stakeholders to do this very same thing. So my partners on the chair side of the house can open the same link of

meet and review conversation and then they can suggest fix this to me, without knowing how to use at all and it goes directly into my get repo so that a load being able to show stakeholders. Hey, this is, this is working its building to see if you can touch it and you can correct. It has In the most powerful thing for me because the hardest thing to build in machine learning is trust in your model. And the only way you can do that is by Shining Light on it, and I didn't have to do any extra work. So your expertise as our model, still doesn't have this type of interface. If I went to

pull these metrics, you can take me over a day to do the pie, start to get to get everything that I could see in Raza. Absolutely. And then, of course, the burden of this initial data says lessons. We don't have to have a date at Eurasian team to actually stand up intense. We can, we can create small engines on the Fly. Watch how they perform in production and had real data from our production log to create those intense. So my favorite here is, we have an experience where we are helping people pay a bill and when they when we offered them the link to pay their bills

sometimes system that we integrate with go down and there was a period of time where the payments leaks have you were sending was down. People were saying, hey this doesn't work for me and so I created in a tent that just broke and that let us know what our customers are telling us something is broken. And now we can use that incense and how often inspired to actually inform the team that creates the payments blink when they're thing is broken. Before they know because our customers will tell us something that we treated on the flight in route, the Xbox after reviewing conversations

with my products. I wanted to know. Do you like printed metrics for my in-house topic model? It takes is labeling to give me a validation dating site to find that the brother has some very quick cross-validation options. And that really helped us Target these incremental improvements, which I have never been able to actually be agile with my data science before and that's what this flight is kind of about where I finally see how to make Define machine learning agile by leveraging Raza

user experience tiger team that runs parallel crumbs to our software, to stop out there building beautiful action server integration, but we have a separate team that is our product owner, our conversation designer, the data scientist, in machine learning engineers and then bought two dinners to our Junior. People to research the intense eight Implement, our weekly upgrades to model and we have those people solely focused on a completely separate from that pushes these updates faster because traditional Can take an issue happening

in production immediately and we were able to eat fast and we also have a rotational software engineer, he comes and sits on that team. So yes you can understand if you're if you're a job developer and you're on a chatbot project and you never get to touch the bottle that probably doesn't feel good and said this allows for cross-training on those rocks of models. That way, all of our Engineers can understand the core components that they're working with him and it's no longer data scientist or a little island, all alone like you a lot of fun for

us will not respond to that requires a lot of API integration. And so by having his rotational Engineers, on our team, understanding what's coming in the pipeline and being able to focus on that were able to It's so. So, what's the impact? How does T-Mobile feel now? And I tried to think of ways to summarize it, and I think that there's there's one number that I really can't get over, which is customers mystery. We call the bike, Cassie Cassie on her. Own took three point four million dollars of care contacts. If you want to turn to lie and to me,

that's incredible. That's 3.4 billion dollars of Epic Air contacts that she's been able to handle all by herself. We have also sold a few more chatbot projects throughout our organization. Could we have some stuff coming up with markings and stuff with support because of an organization? T-Mobile is really bad. Now, you ain't these chatbots, we actually have multiple chat bot cheese now. So we had a team from legacies friends that came over and now they are working for the HR team. Building HR chatbots I'm so truly T-Mobile has gone from a company that said, absolutely zero. Lots of few years

ago with a few key projects with our company. That's just absolutely Embracing chop out the train to bring them in as much as possible in the next. My very favorite part, which is data scientists are beginning to leverage Raza models, to lessen, the burden of labeling. So a story here as we wanted to know the impact that iPhone launches were having her here conversations. And so a team came to me and he said, Heather, can we use your topic model to do this in my heart broke? Because you know, we talked about all three General they want something very specific people buying iPhones

right now and so instead of staying here so I can shoe horn you into the private file on here. So we can stuff sent data and figured you'd be able to say actually give me one hour of your time. I'm going to teach you how to build your own in Timpson rothlyt how to rip out the models and then I have some python multi-threading code that will allow you to classify your own stuff. So we can stop bothering you. So it's literally like that. I got to move from the Cheech and yet or give a man, a fish model of munching date of her people before to teach a man to fish model of teaching people how to

get smart about Incredibly accessible. That's the very shortest story that I could tell him how we went from incredibly antichat by, and you personally not wanting to ever work on a chat box fully behind this entire framework, because I'm only here, because of the great work. Like the 20 people I work with you. I just really want to acknowledge that.

Cackle comments for the website

Buy this talk

Access to the talk “… or how I learned stop worrying and love the chatbot framework | Rasa Summit 2021”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “Rasa Summit 2021”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “Artificial Intelligence and Machine Learning”?

You might be interested in videos from this event

October 7 - 20, 2020
Online, Mountain View
19
5.41 K
google, googledevs, it, machinelearning, mlsummit, network, platform, tensorflow, tfx

Similar talks

Bruna Nayara Moreira Lima
Chatbot Developer at PicPay
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Hector M. Fried
Co-Founder at inChat
+ 1 speaker
Rory Gianni
Technology Consultant & Co-founder at inChat
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video

Access to the talk “… or how I learned stop worrying and love the chatbot framework | Rasa Summit 2021”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
742 conferences
30258 speakers
11303 hours of content