About the talk
In this session you will build an application security checklist customized for your specific technology needs. The checklist you build can be used by development, operations and/or security teams to improve the application security posture of your applications and minimize the risk of releasing vulnerabilities into production.
Joe Kuemerle is an application security engineer, developer and speaker in the greater New York City area specializing in application security, development, database and application lifecycle topics. He is active in the technical community as well as a speaker at local, regional and national events.View the profile
Why they're welcome to Pilot surgeons in developers how to improve your application security by using checklist. My name is Joe Cameron Lee. I'm a product security engineer. And before we get started, I do want to set a disclaimer that this work is all my own personal work. While I do security work for my employer, not what I'm going to be talking about is relevant to them. Nor have they endorse this in any way or should anything. I say being endorsement of my current employer. Having said that, welcome to this talk. I know that we're doing this virtual and it's going to be a little
bit odd. However, I am in the live chat, alongside the playback of this. So it anytime, if you have comments suggestions feedback like to offer, please feel free to drop it into the chat. I'll be happy to respond. Additionally, there is going to be a live session following this where we'll have much more ability to interact directly. Collaborate on some of the ideas presented in the stock, really love to have. You stick around for that and to have a chance. Interact with you much more personally. With that, we can go ahead and jump right into the talk. Scientist that study problems. Have
classified them into three basic varieties. There are simple problems. There are complicated problems. And there are complex problems. The problems are pretty much what it says in the Box. Something like learning how to bake a cake from a recipe. You have a recipe, you follow the instructions at once you learn it. You basically have a very high likelihood of success, doing the same things, and getting the same steps with the same results, complicated problem is something that's more like sending a rocket to the moon. There's not a simple recipe for off and it requires
multiple people, collaborate and coordinating teams with specialized experience. However, things like the laws of physics are pretty consistent. So that once you've learned how to send one rocket to the moon, you can use repeat the process with similar rockets and expect similar results. Moving on to the complex problem. Something is defined as complex as like raising a child. I'm like rockets that are constructed to be virtually identical. Its child is unique and even more the environment of Any Given child is different from any other environment. If you
have children know that raising a child does provide some experience but the lessons you learned from one child, do not always directly apply to another one. Indeed, the techniques that you may have used for one child, may be completely different and you may be completely ineffective on their brother or sister. And that way building software is defined as a complex activity By definition, most software is bringing something into the world. That wasn't previously there. This means that like a child. There are some common similarities between software
Solutions, but there are also a number of unique circumstances. Building secure application is also something that cannot be completely controlled by a simple set of steps. And it does require active participation by developers as well as those who helped to support the developers developers, freedom to innovate, while providing a usable method to remind them of what they know. And what they need to do is the purpose of building our checklist. We'll start with some simple definitions. I just to make sure that we're all on the same page. A checklist
is basically a set of simple straightforward, easy to assess criteria. Should be well-targeted detailed enough for a user to assess if an item is completed, but also not get bogged down with unnecessary unnecessarily. Overly complex details or items that are not relevant to their current task. There are two major types of checklist. The first is defined as do list or more accurately called called your response. This is what most people commonly picture as a checklist completion process. I need to do list
method. The check was drives the process to step-by-step approach. Use the worst of the checklist perform each action in series, and then recording the results. This method can reduce redundancy, as there's no need for follow-up process. Since the required actions are usually taken during the execution of the style of checklist completion. One of the disadvantages is this method can lead to steps getting skipped or emitted in high distraction environments, very easy for a user to skip a staff. At the execution sequence is interrupted and
not notice the emission until it's too late. While many of our environments are not as distraction, prone as Airline cockpits or operating theaters, I think that in some instances such as building and releasing security fixes in response to an incident that perceived over had of a duelist process may feel like unnecessary overhead for the stakeholders. Second method of checklist completion is called challenge response. It's also better described as challenge, verification response. In this method for using a checklist is a backup procedure,
the user performs, whatever actions they need to take an AR case developers write code. Building software according to whatever process is in quality criteria, you have in place and then they use that checklist to verify that what they have done is in accordance and meets all the checks requirements. This challenge response pattern is most commonly as pattern by major airlines as well as other high-sensitivity environments. Well, one, or both these methods may not work for you. Specifically. They send your
culture, your environment, your requirements and that's okay. The purpose of this talk is to give an overview of different methods, how they've been in an active in the real world and to give you a starting point to take back and to build on, there's not a one-size-fits-all solution checklist, have to take into account, corporate culture development methodologies, how your security teams working interact with development teams as well as your people's preferred work styles. Even in the airline industry with identical equipment. Each airline has their own preferred work
style and thus a unique set of checklist and styles that are suited for how that corporate culture works. It's all so well within the scope of a checklist implementation planned on a hybrid solution where some checklist are doulas and others are challenge response. Cover a little bit of what a checklist is. Now. As I said, a checklist is not intended to be a comprehensive guide. Nor should it replace a well-defined secure coding standard. Comprehensive security, standards detailed process, documentation and strong security, controls are also
essential without any of those. A checklist is essentially useless as the purpose of a checklist is to be a memory aid and to cause you. So to stop and think, before acting a good checklist will raise high-level Wellness of risky situations, and provide a limited set of options, but it's not going to cover all situations. Justice flight checklist rely on training and detailed operational procedures. Application Security check was must also be backed up by good security training and written resources for developers to use. All the check was
in the world will not help. Someone who does not have the necessary background knowledge. Checklist must be usable. If something is too long or complicated. It won't be properly addressed. Human memory is fallible and human attention to detail, especially when working on routine matters that are perceived as mundane can cause items to be easily overlooked especially in the midst complicated activities. Such as preparing to cut someone, open attempting to get 200,000 lb of metal Fuel, and people to fly through the air or building system to allow customers to securely use their credit
cards and purchase things over the internet. Good checklist must be precise. They must be relevant to the situation at hand and describe actions that are both meaningful and actionable to the user. They must be engaging for the user and work as a memory aid. Not a wrote list of steps that just have to be checked off in order to get to the next phase of your work. Just like the Agile development techniques. When you give the developers leeway to innovate a good checklist, will Empower those developers to build, more Secure Solutions.
In order to measure the effectiveness of our checklist. We need to be able to verify the if their effectiveness across multiple Industries, Aviation medicine and construction are some of the major industries with well-documented successes in using checklist to drive a fictive Effectiveness. While we in the security industry are still evolving, our methods of measurement, the effective use of check was will raise communication increase collaboration, and it should result in more secure systems moving forward. Can you ask solutely measure a
reduction and security incidents? That's a definite challenge. Depends on your organization's, use cases and security maturity. This can be easier for some externally facing products with the product security process in place with internally developed and managed systems. It's a lot more difficult to have that sort of absolute metrics. Either way. There's still some uncertainty involved. It's a straight-up reduction in the number of vulnerabilities may or may not be directly attributable. To do use a checklist or any other individual activity.
The aviation industry has been using checklist in various forms, since at least 1935. 1935. There was a new Boeing test bomber that was being prototype than being tested initially. After a smooth take-off, when it's one of its first flight installed, turned on One Wing, crashed and exploded killed two of the five members of the crew, including the pilot, who was also the chief of flight testing the time, so it's arguable that it was not lack of pilot experience that caused the sort of crash. The investigation that occurred afterwards
revealed that there was no mechanical issue. The crash was simply caused by Pilot error. This airplane was more complex than most previous models, including having four engines to manage at this point. The pilot had simply forgotten to release a single locking mechanism on the elevator and Rudder controls. Newspapers at the time reported that the airline was probably too much airplane for any man. One man to fly Boeing nearly went bankrupt as a result of this, when the Army record shows a Douglas airplane instead as their primary bomber choice.
Luckily, the Air corps did purchase a few of these plans to be used as test planes dedicated group of test pilots who are determined to prove that this airplane was flyable. One thing that they did instead of mandating that there needed to be more training because again, the one of the pilots that crashed, the original plan was the head of testing. He had been trained as much as anyone, if not anyone else in the area. So what they came up with was a set of checklist to cover, take off flight landing and taxing. It was a basic list of things that the pilots already knew how to
do, but they could easily be overlooked during complicated but routine set up for a flight. The check was worked the demonstrated that the airplane could affect the flow and successfully, this airplane became the B-17 bomber and it ended up being the third most produced bomber of all time. The aviation industry has used checklist for great Effectiveness over the years. The chart on the left shows plane crashes by year, along with corresponding passenger and crew. That's you can see that after the 1940s even has a number of flights increased. The number of crashes
decreased chart on the right shows, the trend line of United States commercial flight hours since 2018, this current past year as well. Over the last 18 years. Number of hours, spent in Flight continues to increase and yet crashes are rare and noteworthy events. Shimmer, white line, in Madison use of surgical checklist has been increasing the checklist, Manifesto by Atul gawande, which is one of The Inspirations for this talk, describes a process by which doctor Grande pioneered. The concept to check was in medicine. There been studies that show,
consistent, and repeatable Improvement, and surgical outcomes in locations for checklist for use The use of checklist provides noticeable and measurable improvements. Regardless of the resources available to the testing hospital is studies expanded range of Institutions from some of the top-tier institutions and some of the wealthiest countries in the world to underfunded crowded hospitals in developing economies. The youth checklist has resulted in improvements across the board. This is an example
of the World Health Organization. Surgical checklist things to note are the Simplicity of design. The questions that have clear and meaningful answers and the categorization to help people focus on the items that are most relevant to them at any particular phase. Also note that the check was calls out that it is not a comprehensive resource and a customizations by those who use the checklist are actively encouraged. Collaboration. And a feeling of ownership are important factors in successful implementation of a checklist program. What is the use of checklist in development and
application? Security, can provide meaningful Security benefits in the posture of our applications? We're not going to move on to the nuts and bolts of how we can start building a checklist for our organization. There are a number of items that we're going to cover. Will the find a ground rules and objectives for a checklist for the find multiple checklists as no single checklist should ever cover. Everything will then cover the basic questions that are likely to apply to many different areas. And also some situational questions that are more
specific again. I want to remind you that the chat window is there for you. If you have any questions feedback, please feel free to put them in there. I'm happy to answer them as we've gone through this discussion and I'll be sometime at the end as well. The main idea that I want to get across, is it building? A checklist is a collaborative process. A checklist cannot be another set of processes that we inflict on the development team's primary purpose and value checklist is that they contain a collaboration between security knowledge and development processes.
They must be built by direct involvement and open the changes coming from our consumers, the Developers. We'll start our ground Rules by building an effective checklist from the aviation and medical world where our checklist have been shown to work. Well, our ground rules will guide us. And as noted in the previous slide, Our intention is to increase communication and collaboration. Our check was for going to serve a dual purpose. They are explicit. Purpose is, of course, going to ensure that wrote tasks are consistently
implemented that the developers execution, and ongoing revaluation to checklist. Also serves as an important communication bridge between the developers and a Security Professionals who are supporting them. We'll take some lessons from surgery. We're taking time for everyone on the team to introduce themselves and their roles results. In better patient outcomes. I will apply that to application security. If the developers know who their security people are and understand that the goals of we're aiming for, then there can be a larger sense of teamwork and
a willingness to stop and ask about things rather than just plowing ahead with development. Let's take a moment to review. This block of tats scribing what the aviation studies believe their checklist should achieve. I'm not going to read through the whole slide and you can feel free to come back to this later. But essentially, we're going to boil it down, as checklist, must be standardized to a certain extent, and have to at least be consistent. They should serve as an aid to memory and a backup for
processes. The process of working through a checklist. I should be logically ordered and a checklist to provide a framework for both the entry team and cross team communication and collaboration. Our checklist should be designed to allow for any team member, regardless of role, seniority or technical skills to detect and call out an anomalous items. Nobody can know or see everything. Sometimes detecting a potential vulnerability can come from an unexpected quarter. Such as say, a project manager who becomes more aware of data
privacy standards by being exposed to the concept on a checklist. This can really help those who are new to the team. Those are who I saw or experienced developers that she may not have primer development roles to be empowered to notice and call out some vulnerabilities essentially, like, in an operating room where there is a checklist and where that is his team introduction of comfortability with the members of the team non surgeons in general, can feel more comfortable in calling it noticing calling out, potentially risky situations. This is something I think
would greatly benefit and being translated to the application security rum and have that level of trust among the development teams with the security teams. Finally use checklist must have a sense of buy-in and ownership. Those teams need to feel that the checklist items are relevant that they can be effectively executed and evaluated, and they also have to understand and be committed to the purpose behind each of the items on the checklist. I need a relevant or ineffective items on the checklist can lead to minimization of
use or even outright disregard of the checklist. This can give us yet another failed, securing their stuff and even more exposed vulnerabilities. In aviation, they don't have a single checklist and neither. Should we an application security? Just as there are different checklist for different phases, of flight pre-flight. Take off stand light and Landing as well as checklist design for emergency situations. We should also have a number of checklist that are appropriate for various scenarios. The cognitive redundancy provided by checklist, can be a benefit and All Phases
of our application security development lifecycle. Checklist should cover our standard operating procedures and should be able to use by be used by developers as they go about their day-to-day business. Designing writing testing and deploying code Justice. Airline pilots use a checklist to ensure that they do not forget to enable critical setting. Soaring flight developers should use our checklist to have a reference to remind them how to build secure code. Straight and level flight. Still requires monitoring an adjustment standard development. Processes
can integrate check was in there work. Well, with minimal disruption. There are also those more complicated scenarios that require more specialized knowledge. These scenarios can also benefit in checklist development of high-risk components, brought that occasion cartography, or any other high-risk areas, should either have specific steps or even entire checklist for they're covering their specific scenario. Check was can also be used to effectively respond in emergency situations. When there are mechanical issues with an airplane, there numerous
troubleshooting checklist that the pilots can work from similarly. We can build checklist for incident response, breach disclosure and other high-risk high-pressure scenarios having an agreed-upon set of steps that have been thought out ahead of time and shared across an organization. Can help prevent missteps mistakes In the Heat of the Moment. Our checklist shouldn't be something that we impose on developers from the security group. Collaboration is essential that provides both the sense of
ownership over the process as well as Steve feelings at that. To ensure that the items are relevant, and then I'll provide the expected results. In the benefits of checklist is not just going through the items, A major benefit is Improvement in communication, surgical teams that know each other even through a quick introduction of names and titles, just before surgery are more effective and have a better patient outcomes. Its Effectiveness is driven by communication, when you have names to put with face humanizes, everyone in the group increases the sense of ownership
and makes people feel more comfortable with calling out this potential issues. Check was there not a one-size-fits-all solution checklist need to be customized to fit. Your corporate culture. Development processes are to land checklist choices and meet the needs of the software your builder. In order to build a good checklist. You're going to have to collaborate. You're going to have to collaborate with lots of non security people and talk about. Let's talk about some good ways to do just that. We'll start with the major consumers of our check developers.
They need to feel both encouraged to use them as well as empowered to change them as necessary. There's usually a bit of a tension between security groups and developers. We all know that the most secure system is one that's completely isolated and does nothing. Developers on the other hand have different incentives. They get paid based on how many potential security vulnerabilities. Sorry. I mean actual features that they can expose to the world. When building checklist with developers, we should work with them so that we understand how the me and they understand how
every item on the checklist helps their system to be more reliable more available and more stable. Just saying it's more secure to do X is usually not a winning strategy for collaboration. Having checklist that span multiple groups include operations teams can also help with both day-to-day operations, as well as incident response. Rest comes from everywhere, maintaining good communication with people, keeping our systems running day in and day out and be very beneficial checklist or also facilitate
knowledge of people and rolls across groups again leads to increased communication. Increase in provement in things. Such as incident response were, there is a central cross team collaboration. You should check was also provides more effective, communication openings between different security groups. Good working relationships with infrastructure, security, Enterprise security risk management and other security eccentric, teams is enhanced by the knowledge. There is commonly known playbook for everyone to use based upon these checklist.
We even need to collaborate with management as the ones who set priorities and sign the paychecks management needs to have an understanding of their power to cause a Central Security processes to be circumvented management. Should have an understanding of how performing security-related work and be a benefit. The benefits of a more secure code base can be expressed in Waste Management understands. What's one of the things that management best understands Cold Hard Cash? We can demonstrate that more secure systems are more reliable and more reliable systems can lead to more
users were users leads to more Revenue. That's a direct all that management can get on board with and help them to understand that an investment and secure code is an investment, good code and an investment in the bottom line. Has recovered a bit of how to collaborate and build checklist. There are also some antipatterns that we should be aware of so that we can avoid specific problematic areas as we collaborate with the development teams to build these checklist. One of the most significant antipatterns is length. Checklist have to be efficient,
not comprehensive. This is where it helps to have a good understanding of your systems and your major vulnerability categories. You must be ruthless in your process to ensure that only highly relevant, highly probable and highly impactful items are on your checklist. This also helps increase the relevancy of them and therefore the adaptability in use of them. Your checklist need to be simple complexity and ambiguity are counterproductive. Every item on. Your checklist, should be easy to understand the steps to complete that item must be
clear-cut and well-defined. Finally, your check was have to be relevant to give a developer a laundry list of terminology that they don't work with everyday. That's a recipe. For trouble items that are meaningful. Will get attention items that are considered. Irrelevant will get overlooked in minimized. So on the measurement, how do we measure our security output? How do we measure the effectiveness of our checklist? One proxy for this can be developer interaction and relationships with security, having a process in place to collaborate, early with your developers, and to actually have some
measure of that over releases can give you numbers to drive your effective communication. So, the more that you interact with the team, the higher scores could be that something that can be measured and used as a proxy for how interactive you're being with your development teams. Another measure can be security bugs that have been detected over time. And in which phase they are detected catching something and design review is a lot more effective than catching something code review versus having it shipped to production and having a external bug Bounty participant, report it to you
the earlier, you can catch the dirty boner ability. And if you have the ability to track at, which phase those security bugs, are caught, is a really good additional proxy for the effectiveness of your security process. And therefore the effectiveness of having his checklist. Here, we have a few sample checklist items that have come from myself in from previous versions of this talk. You'll notice that that they discover is a variety of common development, scenarios,, security vulnerabilities, and the checklist items and selves questions are
concise. They are at an attempt to be clear. They minimize the amount of security language in those questions, and in the checklist items and the answers are easy to tell what they are and what their completion steps are. This is some really good examples of how checklist items can look, how they can be effective and how they can serve as not. A Long-winded secure development process but just is a quick and simple, reminder to the developers as they're getting ready to commit their coder as a writing their code
that have they encountered the scenario and remind them to do the right thing, security-wise about those two particular pieces of code that they're writing. I have, of course, built a check list in order to help us build checklists. This is again, a list of 10 items that you need to keep in mind as your building checklist. This is something new in our interactive session following this recording, we will use as our corporate area for collaborating on building relevant, checklist items for some of our scenarios and a
checklist. The Stars cover the most important things first, it does provide a clear and meaningful representation of what needs to be done. And it provides that satisfaction of having that check box of here, is the clear well-defined criteria in a definition of done for this particular item, so that we can then move down this list in order. And come up with an effective and useful set of checklist criteria. Against in the collaborative session. After the stock, anyone who's participating that we will collaboratively build checklist,
items are relevant to our particular environments. If you have any feedback that you haven't already put in the chat, now would be a great time. I'm happy to answer any questions. That may come up, we'll have a little bit of extra time at the end of the session and finally, the references slide tape. Please feel free to take this back. Your company's build a process that works for you for your organization for your development team. Collaboration is the key to this. Iterative development is highly important. Something that you get out there. Can test and then build
on is the preferred mechanism just building something once, throwing it out there and saying this is we're going to do it forever and ever is not going to be very adaptable is not going to be very flexible. So I can have those collaboration conversations with your other with your development teams with your other teams within your organization include Management in this, get that buying and get yourself. A good simple straightforward process to remind us developers of all that expensive security training. They've got at the point in time when they're writing code. With that, I want
to thank you for your time today. I appreciate the opportunity to speak to you. Happy to continue this conversation and chat. Happy to continue this after the fact, as well as you can reach out to me on Twitter at anytime. Thank you so much and have a good rest of your r s. A
Buy this talk
Buy this video
Our other topics
With ConferenceCast.tv, you get access to our library of the world's best conference talks.