Duration 23:56
16+
Play
Video

Use of AI to quantify the "safety of autonomous vehicles" By Edward Schwalb, MSC Software

Edward Schwalb
Technical Architect at MSC Software
  • Video
  • Table of contents
  • Video
Video
Use of AI to quantify the "safety of autonomous vehicles" By Edward Schwalb, MSC Software
Available
In cart
Free
Free
Free
Free
Free
Free
Add to favorites
12
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About speaker

Edward Schwalb
Technical Architect at MSC Software

Dr. Schwalb has received his Ph.D in Artificial Intelligence from University of California Irvine. He has more than 20 years of experience in implementing intelligent systems for a wide range of industries, including defense, consumer electronics, financial, and engineering. His data engineering experience includes building sizable financial data warehouses and automated load underwriting systems. He has authored a technical book, published in major journals, edited technical standards, and credited with more than a dozen patents. At MSC software, he was an architect of Apex, a product winning more than a dozen awards in 3 years. Currently he is charged with leading the MSC machine learning effort, including simulation tools for training and validating driving agents. His research focus is mathods to engineer inherently safe drivers, through quantification and validation of safety.

View the profile

About the talk


Share

And I join me here for the discussion on how we leverage AI for hazard analysis for autonomous vehicles. My name is Edward how many scientists at the company called MSC software? My background is mostly artificial intelligence automated reasoning and machine Learning Company is a simulation company. You can see here. We have had numerous products different types of simulations mechanical engineers and the vehicle Dynamics send lots of different types of simulations. The one that I'm talking about. He realized this product. Let me show you the video of what a product is.

I hope you guys get here. Yeah, yeah. Yeah. Yeah. Edward could you ask Siri? I hope you can see it. Okay. Okay, so then I guess I wasn't sure could you hear it though? It was the audio playing. Okay, that's fine. Okay, so I'm about to the presentation so d the company the company imaging software. I was acquired back of biologic Kompany Calexico on the end. So as part of that we have numerous other products of what I did and so here I'm going to focus specifically on

a hazard analysis for autonomous vehicles. And how do we address this problem? So let's start about talking about the scope of the challenge photos. Not in the industry. You mail me not know that humans. If you adopt the list of all the other people drive and divided by the number of fatalities on the road is about 90 million miles between Catholicism wheel drive 19 more time in the United States today, even though we have 30000 Fidelity's we do have more than 50 million miles traveled,

but if you do statistics on a vehicle by vehicle Basics because this is more important the tournaments driver needs to be as reliable as The number of Motor Vehicle drivers in fatal crashes not the driver and that would be more between three is 200 and 400 million miles. That's a big number and so we need to build AI systems that are not 99% accurate. Ninety-nine Point ninety-nine percent accurate, but 99.99999 many zeros accurate. And so the implications of that is that

autonomous vehicles would have to be driven billions of miles to demonstrate their promise promise rate of fatalities and injuries, and that existing switch for 2012 ZR2 baptismals. And so it means a physical driving alone cannot provide sufficient evidence for them showing at Animas Vehicle Safety. We know that yes, we do have people that are claiming to be on the road, but they have not demonstrated ordered a claim that they have demonstrated achieving the safety levels are Cubans. They do have a voluntary 60 statements,

but they would never argue that they are better than humans didn't have the data to support at least. I haven't seen it. But the option of this whole thing is you need to have your child tested because the only place you can run billions of miles is in Sydney Elation. And so what we do our products in our efforts are to make it practical possible and take the results of assimilation and make sure it carries to the real world. To give you more of a perspective on the industry. Please note

that if you do look at the list of all the crashes as reported by the national highway Transportation safety agency Administration. Less than 1% of the crashes are because of the vehicle failure. So there's a whole industry of vehicle safety and there's a lot of vehicle safety standards that have to do with making sure that the vehicle doesn't break. So how did control system works and how the steering and braking Works in one of those example is the rearview mirror that is very useful

for humans, but I don't think that's safe driving cars or autonomous drivers need rear view mirrors, but the bottom line is that industry as a whole is only addressing the failure portion Which is less than 1% of the problem and the reminder is basically behaviors multi-agent behavior of the different traffic participants. And so what it means is that when we increased automation, so if you familiar with SAU levels of automation Level 2 minutes. Amation for limited

automation tall but level 3 and above means that the vehicle starts to respond automatically steering automatically breaking automatically for example, advice frequent cruise control is equal 2.5. There's some left over to mention, but it's very limited. It's what that means because we are looking at multi-agent hazards that scope of residual risk decreases more than 140. And again, yes. I know it pelicula Master Cleanse never have autonomous vehicles. Be careful. What you sign in with the small print because

they relieve themselves of reliability of liability. And so he noted to make this thing work. Please realize that on the road and in simulation and is very difficult to control individual can see post birth control individual come mostly because the systems are like system listed for example of a system friendly not causing no determinism issues. But in order to achieve the results, we have to control the aggregate statistics to need this is an example where when we test something we cannot

guarantee outcomes for specifics and I would be enough to be able to guarantee the overall statistics. The question is what is that statistics? So I'm one of the first thing to remember is that statistic is complicated in the promise complicated. I'm going to show you three examples going to talk a little bit about some solutions and some experiments to be conducted as you can also contact to convince yourself that this is the case. This is just for informative. We're not selling anything to anybody just some interesting things. So one of the examples I usually give is in November 2018

NFL team has deployed in Las Vegas in a truck backing up and slowly but surely eventually Grace the shadow even getting closer and closer until it Grace meaning that the truck scratch the bumper in the front of the Shadow. So City officials said that the Shadows VW bus supposed to do in that sensors register the truck in the shuttle stop. It really is this the requirement really what we expect from a tournament Shadow by Passenger Zenith at the track is backing up into a

text with if you wait for it to get squished. What would happen if I would be if your feet trailer passenger said that we had about 20 feet of Industry behind us and most women drivers without for the car into reverse and use some of that space to get away from the least. You know, the horn in male prison is harder to miss now. Remember. There was no requirement for software Engineers to guess what they could ever do when a truck comes and crashes them and it's very difficult to come up with all possible

answers to the situation. But it is required. It seriously is required and according to need more than 3% of the accidents. How do you spell turn so it's not an edge case. It's a very common case. We talkin about. easily 1000 accident to your easily Here's another example. And so so I guess before I switch over to point ears. We have a solution in and you're encouraged to reach out to see how we actually automatically find a the right action until Chief accident avoidance.

And how do we train for it? I don't have time for this detail here. But here's another example, which I bring it to you because if if you are a US base person are you if not familiar with the driving laws in United Kingdom. I would like to know that the United Kingdom Death Row take a driving test requires that you identify hazards on the road against the video with a clicker. So the video is playing and you are supposed to click when you find out hazard. Is examples the training videos will give you his they would Mark those potential have the things that could go wrong

as yellow circles service Angela Bassett the pedestrians on the left side could jump into the back of the car from the sidewalk onto the street. There's a small is mental status can come on the side street over there. You can see the vehicle in front of us may stop and then there was parked cars on the rides are so that means pedestrians can emerge from between those cars to walk towards opening the doors and then you can have drivers are open the doors of the scars. The only something potentially potentially would happen if he needs to detect those and and then estimate how far they

are. So for example, when when you look at the vehicle in front of you that has dropped it in there like a few friends a few seconds later then. Vehicle in front is distance away from us we have to Go down to avoid crash. If you don't slow down, we will arrange that vehicle and so we will have to slow down and accelerating out. His marriage is complicated. I'm doing this to you. So you understand. There's an interesting requirement to not only detect those hazards but also handle gracefully that the

actions that mitigators action mitigators accident today after I want to talk to you about here is that we need to have those has a detector went for very very then give you a perspective. When you drive you rarely encounters on the field. You really encounter pedestrian coming between the motor in a park cars and in many cities in the United States, we don't even have product cars that you have to worry. You just go to parking structures in this many many other events like pedestrian jumping in front of the car. I got these are rare

events, but yet They need to play system is to be trained photos. But why is it challenging to rent for event? So let me give you an example of why if somebody tells you that I have a has it got to do with it, which is 99% accurate. Is it good for the answer is not really it's actually very bad because if if we have something that happens. 1000 or ten thousand a hundred thousand driving hours then if you just care if it doesn't happen, you will be 99.999% accurate to say something is rare and only half of 0% of the time guessing. It will never happen will give you 99.9%

accuracy. So this is a picture from something called smote s m o t e would use a synthetic. Naughty gossip sampling technique where in machine learning you would say. Well if my if my minority class is very small and rare, then I'm going to place more examples of the minority rights. What does red represent the minority class in the green. Represents the newly generated data that will represent those hazards that we would like the train. But that will not work. 4-H woodwork for verification and I'll explain in a minute but it does not work for

training because if I train for the wrong distribution the performance our observance raining and the model that I'm getting is only good for that is skewed distribution is not going to be good for the true distribution. And because we are talking about increasing the frequency not 10% but rather a hundred million x that makes a big difference this killing now makes the motor not as good. So that's part of a chance. Do you have to train for very rare events? Why did we doing an experiment with it? And it could do it yourself if you want. Imagine that you have

digits to detector supposed to Hazard on the road just to give you very very very simple. And so if you want to do a test, you can take the exit in this data set and then generate a data set in which the minority class. Let's say, we are classifying all these years you what are the zeros the minority class would be 0.5% of the majority class. For example, so 99.5% We have any other Vision zero and then only 0.5% Bigfoot 0dia. And so what we need to do is we we tested the mechanism where instead of screwing the distribution we are generating more digital

amplified the scenarios be like in PCR genetics one of the major. Breakfast is ability to replicate the DNA to have sufficient quantity that you can do experimentation. And here's the situation is very similar. We need to take those rare situations are events that we need to amplify Damon and generate more of them so that we can do so many for training and so here's what I'm showing you the results of what happens if you do it correctly or incorrectly. So the Blue Line represent the four

metric we represented here the metric of true positives that do negative or positive plus negative. It would accuracy is the sum of two positive and two negative. If you would have done any straining traditional way just like we had talked in the books and we get 99% accurate. And so what we can see is that Movie can see is that the the learning that we get with? The blue bar is very reasonable. We get pretty good two positive and two negative because we are giving it the entire data set fully balanced

as you know as the original. Training was designed to be but now we removed all those zeros, and we only expose the training algorithm to 0.5. A percent of a 0 digit. Everything else is not there right now. We asking it how well is it going to learn to the orange one represents? How will it learn so you can see that? It's still pretty well, it's 50% Actually you still get roughly 50% positive rate which is of course better than 0.5% still it's it's learning something but it's only 50% accuracy

report to positive way the truth the true negative it obviously is is very high because the minority classes less than half a percent. So obviously we're still keeping the in a by guessing most lead Hazard is not there. It would it would still show very high results but was very interesting as it gets very very large number of false negative at what is the first negative plus negative means something on the road that you didn't detect and you could crash for it. Is very interesting. And so if you do the application to get the green bar,

which represent? Okay. Now this is what happens if you use this PCR technique that basically says yeah I can I can I believe I N I remove my first negative dramatically so I can be there tomorrow 208 Darkness me some statistical approach. If you look at his truck from dazed and perspective the probability of a crash is the summation of the of a crash for a given scenario * the power of that scenario. And so if you would send this to using house as you can for the decomposers, you can see that there's a

probability of the cash for giving Hazard which is called controllability. How well is the Uruguay to my voice accident? So that control business documents avoidance component and then I have a listing for the exposure How likely is The Pedestrian to be on the freeway or How likely is the event of uber to happen to us? What we doing insulation? We can automatically generate distributions that will increase those probabilities of Hazard switch would learn what the emerging as it is and we generated so many different type of others that are natural just crashing. So it wasn't one of

the examples I want to show you a quick video. So what you see here This is Tesla. As you can imagine and what happened is the engaging at bicycle sport or delete vehicle is moving out of the way and we have a crush now mind you all the vehicles in the industry have this problem Rochester. We can't even do this not to mention autonomous. And so this is showing you this very simple songs. You have to find out exactly what combination of parameters make silly school over here. I don't have a picture

of Cheyenne because for whatever but here we show the exact definition of that scenario and I would like to tell if more than 1% of the crashes are represented by this example and hear a different reasons by vehicle might be stopping here waiting for pedestrians to Cross or maybe we have a bicycle trying to go over or maybe a porthole of 300 Road such. And so there is a desert. We need to pay the machine knocking on doors very best to to know how to do. So, how do we vote? We have to leverage distribution in one of the experiments that we did we said, okay, let's train audio

Network to detect the distance to the vehicle in front of us and see how it performs to be traded on sunny days images in Senoia, and then we tested it on gas and snow and then we found out that the network overestimates the distance that says it's higher than it really is when I taste it on distributions other than what you were straight on it and it ended cycling on sunny days and I just returned askance know and we can see the shift. This shift is again, I'm safe because we would Breakthrough out later than would. Otherwise what if we have another test so the point here

is that the accuracy by itself doesn't matter as much as you when I get to buy us here is a distribution matters more to instead of just showing a single number that says this is my accuracy or this is my fourth president. You need to have a distribution perspective because you want to know how have I farted why would be a 60 factors that need to add yogurt to make it work. So how would take we were multiple simulations in the cloud and what you see on the top is the exact the distribution of Hazzard before we run out something I'll go

with them and then in the bottom how we generate only Sand Island have the hazard showing so you can see the huge parking in likelihood of being of the scenario occurring on the on the Target app challenge Hazard you use case we need to fix your approach. Singapore photos in iOS to make it work and do that thing in the physical what would be that we would need to do machine learning with train. Our algorithms are very very vents because we're talking about

less than 0.1%. And I intend we need to do that over billions of my of the equivalent of but the picture developing and testing using those techniques needs to be a first-class if you just put a vehicle on the road and then you measure what happened. You have no chance that's just not enough physical combinations to cover all the multi-agency. So now you need and you need to work with AI that drains and validated some extremely versatile. So if it looks okay even 99% it's terrible. If it is

99.999999 percent, correct? That's good. So I guess the message would be that we need to focus on for the Mystic simulation of soon as we can. Just look at individual cases.

Cackle comments for the website

Buy this talk

Access to the talk “Use of AI to quantify the "safety of autonomous vehicles" By Edward Schwalb, MSC Software”
Available
In cart
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “Global Artificial Intelligence Virtual Conference”
Available
In cart
Free
Free
Free
Free
Free
Free
Ticket

Buy this video

Video

Access to the talk “Use of AI to quantify the "safety of autonomous vehicles" By Edward Schwalb, MSC Software”
Available
In cart
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
561 conferences
22100 speakers
8257 hours of content