Zulfiqar earned his Masters degree in Computer Science from University of Washington, with his research primarily focused on Deep Learning based solutions in the cyber security industry. He collaborated with Infoblox, an IT automation and security firm, during his graduate degree to work in the Explainable AI domain. His research project titled “Interpretation of Deep Learning based Domain Generation Algorithms classifiers” focused on visualizing the feature extraction process of trained DGA classifier models currently deployed by Infoblox to provide a detailed analysis and better interpretability of neural network models utilized as DGA classifiers. During his summer internship with REI Systems, he worked on a Computer Vision based solution to assess and evaluate the damage in areas affected by natural disasters from satellite imagery. Zulfiqar's research interests include XAI, Natural Language Processing, Recommendation Systems and Computer Vision.View the profile
Farshad is a machine learning engineer / data scientist at REI Systems, where he builds machine learning systems to solve business problems of REI Systems' clients. He also worked as a data scientist at Cisco. Farshad received his PhD degree in Materials Science and Engineering with concentration in Computational Sciences from North Carolina State University.View the profile
About the talk
The assessment of building damage after a natural disaster is an analytical bottleneck in the post-disaster workflow. Currently, the response and recovery efforts often require manual review of the aerial imagery of the areas impacted by the disaster. Such reviews might take days or weeks depending on the type of disaster and adversities caused by it. The goal of this project is to develop a solution to automate the analysis of aerial imagery. To reach our goal, we have built a pipeline for large volume data processing and applied a deep learning framework to analyze and classify the aerial images.
02:15 Problem statement
06:28 Semantic segmentation with U-Net
11:42 Instance segmentation
15:50 Input images
Thank you everyone for joining us today. And are you seeing my screen make sure everyone is see my screen. Cancel screen. Thank you everyone for joining us today for a presentation on natural disaster damage assessment on satellite imagery volume spacer. I'm a data scientist all your systems and provide solutions for federal government client. I received my PhD from North Carolina State University and I did computational science research at North Carolina State University. I invite my college to introduce himself. My name is Sophie Carter Hemet.
I'm currently an associate data scientist from REI systems. REI systems is a federal government contractor and we support Garmin missions to Salt different tasks ranging from Grant Management solution to their AI needs. But today we would like to discuss about the natural disaster issue the United States suffers from different kinds of disasters off the country. And on this graphic in the next flight or we can see the number of billion-dollar disasters that have happened in the US over the last 30 years. Of course, these costs are adjusted for inflation, but we can see that
the number of disasters are going. Drastically and whether we want to pin the cause of these disasters to climate change or not. We cannot deny the fact that the number of such events have increased drastically in The Last 5 Years. So the recovery efforts also grow with the scale of these events as well. And therefore we need a quick and efficient recovery process to help the government respond quickly to disasters and allocate help without any delay. But before we begin sending any help, we would have to evaluate the situation correctly and assess the damages. On the next line. We
seen at the current damage assessment effort involved manual review of aerial imagery of the affected areas. The images are taken by the airplanes as soon as the weather conditions alarm. And once the images are ready. They are manually and liseberg review. The reviewers have to manually scroll through those huge images zooming in to see the details of the building this process very labor-intensive because the images are very large and depending on the nature of the disaster and the times of adversity caused by it analysis efforts might take days or even weeks. Our goal of the
project is to develop a deep learning solution to automate this image analysis process to reduce the response time in the next light. We can see that this will help allocate recovery resources to the areas that needed the most our solution is designed to ingest for images and run them through a deep your own network model and detect damaged structures. The pipeline can be automated to process large areas in minutes as opposed to days or week gift for my new review in order to develop a solution first. We have to come up with their own custom data set which do not exist before when we began to
work on this problem a year ago in the current light. We can see that the custom data set that be prepared for a deep learning framework consisted of images released by The National Oceanic and Atmospheric Administration website. The knot website released woman has high resolution satellite imagery of the hurricane affected areas in the US during the last 16 years beginning with Hurricane Isabel. The custom data fit was prepared by utilizing the open source data available on the Noah bedside in the aftermath of hurricane Michael which in the year 2018 primarily struck the State of Florida
and Georgia and the us we caught these images to a size of 300 plus 300 pixels from the original imagery with scan a larger area. Remember cross 300 crop images typically contain one or two buildings at most while as the original imagery contain hundreds of buildings. In order to utilize the images for a deep learning Frameworks. We annotated the images with cocoa annotation as we can see in the next life. We utilize a tool to and get the images incurred on the roof of the buildings and label these annotations with the title roof damage or a current custom data set. We have
included only the roof damage moving forward. We would also include for the labels for other types of Damages such as tree damage Andrew damages. This is how we provide the data set for a deep neural network. And now for shadow cover the techniques that were utilized to solve the automation of analyzing aerial imagery. IKEA sofa car talk to you about our mother. I'd like to overview for most common conversation tasks on the top left. We have classification, which is the task of finding out whether an object of a particular category or class exist in an
image or not. Next we have object detection on the bottom was also there localized birthday drawing Bonnie boxes around the red boxes around the bones in the bottom left image or bounding boxes that localized them semantic segmentation keeps us more granular information by detecting other category for each pixel in the image segmentation divides pixel level classification. What's also classify each instance of a class separately multiple objects of the same class. As a single entity.
Houston segmentation treats them as distinct individual objects or instances as you can see on the top right-hand volume right images in the following slides good present to different models that will be at the Hartsfield for detecting damage roofs with cars from to semantic and instance segmentation past respectively. We have use unit architecture which you can see on the slide to build the semantic segmentation model. It's a fully convolutional neural network whose architecture is designed to work with you. It's
raining images and yield more precise segmentations than traditional convolutional neural networks. It's made of a contract to this Lake and it's expanding pass the writing the Contracting pass holder that captures the context in the Infantry unit and is made of a convolutional and Max full in layers and layers which results in using the number of feature marks while increasing their heart and bits. Find Allie Avant by Frank Ocean processes official maps to
generate a segmentation math and those categorizes each pixel of each picture of the girl. Also, skip connections between these two past versamatic features maps from the Contracting path of the network are copied using the expanding connections. Dennis is the unit for piriformis into this Thursday prevent Vanishing gradient issue second. They help avoid losing pattern information during obstacles. Play improve the performance of unit event feather be incorporated Resnick blocks
into the backbone of unit. As you can see in the picture business box itself is a convolutional neural network. It has a simple idea with the altitude of two successive Evolution deniers, as you can see on the picture on the right of the Knicks players using this different shortcut or the vanishing gradient issue learning capacity in general. Before building a model we modified original images by orale masks that be obtained from cook lima station to on top of them. We can
modify images to the mall. So we talked about our mother architecture in previous slides what you might ask, how do we know if a mother performs World on image segmentation to answer this question. We use diced similar efficient. It's a ratio of two areas where the numerator is two times the area of overlap between prediction and Francis regions and the denominator is the sum of the areas of those two regions a closer The Dice Corporation to Vaughn the most accurate is the predicted segmentation math definition.
Let us get the results. Retrain out somatic segmentation More Than This richness Tuesday blocks for the backbone of unit 450 stands for 50 hidden layers in the architecture. This model gave us a Dice Corporation of point 84in is not close to Von is related to the quality of a data set to some extent. When you see that these small dogs in the post their protective mask and also the original image, but there are missing in the ground to the unit are honestly like
this part of the roof as a part of the roof as a damaged area, which is a tree shadow on the roof as we can see in the original image. In the previous slide image showed some roof shingles are removed and some whose papers are exposed here. We observed a different kind of damage are some Framing and privates are exposed to the exposed. And unit cannot also mark them as damaged regions as we can see here. And also there is a little. Hear that exist in the original image doesn't Mark defries as you can see in the image as
damaged regions are all you have found Houston performance of unit for identifying damaged areas of building roofs aerial imagery to enhance the performance of our model. We need to expand options that to include different types of Damages is more precise and tissues at this point. I have to take you through the second mother we have filled for roof damage identification. The salty automation of analyzing aerial imagery in addition to semantic segmentation like
for hud-vash of disgust. We also implemented the concept of instance segmentation Retreat the difference between the two concepts semantic segmentation treat multiple objects of the same class as a single entity in the illustrative schematic on the slide on a plank semantic segmentation to the original image, the resulting image indicates that hate these are the balloon pixel on the other hand instance segmentation treat multiple objects of the same class as distinct individual objects are instances in The Illustrated schematic on the on the screen on a flying instance
segmentation to the original image. The resulting image indicates that there are seven balloons at these locations and these are exactly the pixel that belong to each one of them. Instance segmentation is typically a heart problem to solve Dan semantic segmentation, since they treat multiple objects of the same class as distinct individual objects and instance segmentation works better with data sets where the object in the data set have quantifiable speeches at that distinguish it from other instances of his own class in article used with the data set where the images of a revolving
around roof damage is there is no set of quantifiable features that distinguish different types of roof damages. We were aware of this limitation when we started the project. However, we still wanted to experiment with the concept of instance segmentation to compare the results with semantic segmentation. On to the next light we utilize The Mask rcnn algorithm to solve the instance segmentation problem in our project. There are two stages of mask rcnn first, it generates proposals about the region where there might be an object based on the input image II it predicts the class of the
object you find the bounding box and generate a mask in pixel level of the object based on the bus teach proposals are connected to a backbone structure the backbone structure that be utilized is a peach a pyramid style deep neural network. It consists of a bottom apart with a top bottom battery and lots of connections the feature phenibut Network that be utilized for the mask rcnn is a residual Network, which is the saying that they use for the unit model. On the next light we can see the architecture of the Mask rcnn model at the first stage of the Mask rcnn a
lightweight neural network called region proposal Network as can be seen in the architecture diagram scams the entire feature pyramid Network top bottom part three, which is essentially a feature map which may contain an object while scanning the feature map is an efficient way to detect features. We need a method to bind the features to withdraw image location. This is where I could come and play and kisara set of boxes which with pre-defined location skills related to the images ground truth classes and bounding boxes are assigned to individual anchors according to an
intersection over Union value. ivanka's with different skills binds to different levels of the feature map the region proposal Network user these anchors to figure out where in the feature map that would be an object and what the size of its bounding box would be At the second stage another neural network takes the proposed regions from the first age and assigns them to several specific areas of the feature map level. It scans these areas and generates object classes bounding boxes and math. The procedure looks quite similar to the region proposal network with the only difference that
in the second stage without the help of anchors if utilizes and how do I align operation which is similar to be alright pool operation to locate the relevant areas of a feature map where there's a branch generating mask for each of the object in PIX11. On the next on the next light we can see the result of Tomas Garcia in a garden. These were the input images containing buildings with roof damage that be provided to the model. On the next flight we can see the production masks applied on the images. We can see that the number next to the label indicates the confidence level of the
monsters in the morning in the current images. We see that the model predicts these masks with 100% confidence. However, we can see that the pictures captured in the production must also include undamaged areas and shadows as well. This is the key difference between semantic segmentation and instance segmentation, which would be covered in the next few slight. On the next flight we have the evaluation metrics used for the mask rcnn to evaluate the results of the master scene model. We utilized the intersection over Union parameter also known as a jaccard index the accuracy metric
that be utilized was that the intersection over Union with a 50% overlap threshold would count towards Texas food production the results we receive from the bounding box accuracy was 56.6% And the accuracy for the segmentation was 55.2%. Let's look at the unit predictions for the same images. On the current slide we can see that the original image consists of one building with several versions of the roof damage. We can see the democracy and then I'll put a prediction mask, which is larger than the ground Fruit Market Niche.
We observed The Mask rcnn include undamaged portion of the image and also includes shambles of the object presented the image much more concise and precise diction mosque in comparison to the ground truth the fundamental difference between semantic segmentation and incense Sigma notation in our youth kids wearing our data set revolving around roof damage has there is no set of quantifiable features that distinguish different types of roof damages. An instance segmentation model would not be able to distinguish between
several instances of the same roof damage class and therefore group sit together in a single mask. On the next flight in order to improve and enhance tea in a result of the Mask rcnn model. We need to perform the following steps. We need to further tunde hyperparameters to deliver more precise and concise musk. We need to enhance the image pre-processing to remove noise in the images that your shadows in conclusion. The mask rcnn is not the ideal deep neural network for identifying roof damage has a mask rcnn will prove to be a useful algorithm for us to detect other damages
damages Android damages where the features that all the object are match with a distinguishable balgruuf damages or the instances with lip gloss and it can be used in addition to the unit model to identify all the types of Damages except roof damage. Richard will not discuss about the xu2 competition that we took Parton. Now I shipped a year and talk about ex Mewtwo competition in which we participated shortly after we finish our adventure this semantics and instance segmentation. This competition was announced by the defense Innovation unit, which is a TOD
organization. The competition was aiming to solve a similar problem that you were working on. What was two main differences difference is that the data set which was consisted of pairs of images for before and after even as you can see on this slide Another major difference was that the post disaster images relay valve is a survey to degree from 0 to 3 0 corresponds to no damage and swimming destroyed building. Based on the requirements of the competition we modify our previous Solution by breaking it down into two parts
semantic segmentation model unit. As you can see here and classification model using Resnick convolutional neural network or damaged classification email out of which we extracted building polygons, then be over late the polygons on the post disaster image next because out building locations around them through the classification model to identify the severity of the possible damage to them. either you can see the results of our mother on a San Antonio para images from pre and post disaster images there Artemis image shows the mass obtained from the
segmentation model which highlights the Dominus buildings in a problem. In this case. There was a fire which worked out some parts of the neighborhood you can see in the middle image that they building on top of a portion of the image are missing that it is compared is the pre-disaster image. To evaluate a model be used as fun as score as required by the competition and I'm 68% bonus code is the weighted average of Dimensions course and amounts to 74%.
Laura van store for damaged classification to misalignment of buildings in Prien potion disaster images. Perfectly match with the locations of the corresponding buildings in some post-disaster images this happens because the photographs or covering large areas on the map. even slight skewness in the image that lead to the polygon Museum the building from one image download I like to wrap up a talk by comparing two solutions the wrongs we stand with us every two classification of
classification doesn't need prevent images. It doesn't suffer from new alignment of fuel for the next issue is that you just miss my mother which is a semantic segmentation model to be trained over the downside of our approach is that it doesn't missing buildings or is that the buildings that are wiped out by the hurricane damages? It just tells us whether there is damage or not the model developed for XP challenge overcomes the limitations or initiative court, but it's a small
Images discount lead to misalignment issue. We discussed previously. It requires tomorrow's to be trained. He proposed extent of war in order to include more types of Damages in a register such as Fallen trees or degrees 40 planning framework that utilize the aerial image announce. We also considering experiment with other state-of-the-art architectures. At this point, I'd like to thank you for your attention, and thank you. Thank you so much.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.