About the talk
Kelly Shortridge, VP Product Management and Product Strategy, Capsule8 Security Chaos Engineering presents a new approach that harnesses the scientific method and attacker math to form an efficient defensive strategy. This talk will go hands-on with decision trees to postulate attacker strategy,, then explore using these dynamic threat models to craft practical experiments to gain confidence in systems’ resilience -- and to be so ready for incident response that it’s boring.
Thanks so much for having me here today and welcome to the scientific method security, chaos, experimentation, and attacker, math. And Kelly Shortridge. You may know me for the recent O'Reilly reports on Street. Gas, engineering, which you can download for free. I may also know me from the research into the intersection of behavioural economics. And I was most recently, the VP of product management and product strategy, caps light switch in startup based in New York City, providing Linux monitoring and detection or production systems. Actually going to be joining a super
exciting. Cloud computing company is a senior principal product, allergist the summer. Stay tuned. Are we here today? What's the problem is that information security? As it's largely conducted today. I'm going to price Defenders isn't actually a science. Adore. You also is an earth. If I suck his more finger painting, that Masterpiece to thing is we can secure a systems with macaroni and right? We need experiments science. So question here today and what we'll be exploring is a convenience science or trace, it better security outcomes.
Play. We can explore one, particularly promising. Way that I certainly spell switch is decision trees and security chaos engineering. The first we're going to start with security has engineering and how it Blends and ties into the scientific method. We're going to walk through a decision tree case, study will be building one ourselves. Finally. We're going to discuss how to make insulin. Let's get started. About the scientific method and how security is engineering, actually. Harness is it? First things first, how do you sign into the first place? Use the
scientific method, which you should think of more than says, be back Loop rather than a finite sequence of steps. But you're saying that you have to ask a question of reality. I think most of you are probably familiar with Isaac Newton and the whole same case with the Apple, but he had it as far as the question was. Why did the apple fall from the tree in punxsy? I'm head. Why did it fall vertically down words, rather than upwards or sideways? Second step is with that. Question in mind. You developed a hypothesis. Hypothesis is basically a proposed explanation for the question that you had
in mind, which we use as a starting point for investigation, a hypothesis to be some kind of invisible, force, pulling the apple and other falling objects, straight to the ground. Third step is we pursue this investigation by conducting an experiment which allows us to make observations that service as evidence of our reality and Newton's game to the Apple. His experiments required him to actually been to calculus. Which trust me you're probably not going to have to do with security cast engineering. The board is we've been carefully comparing observations from our experiments with
the predictions for my hypotheses. Trying to say is objective as possible. For instance. You could find a range of objects like a better, an apple or a sword. Did you end up falling asleep? The ground doesn't our hypothesis, but maybe we discovered that they fall at different speeds. We finally, you want to report your findings commentary and Analysis of our evidence. We can report and document our findings that use them, then to refine our understanding of reality than the iterate by asking you questions that we could do hypotheses and do experiments. Isaac Newton actually did Justice.
It wasn't like a one-time thing and he discovered gravity, right? He would find his hypotheses over and over and over until he derive. The conclusion is supported by the wealth of evidence and that led to his universal law of gravitation. Do the process really is rightfully revered, across scientific disciplines because it keeps you honest. And also in the mindset of continually revising, your knowledge of reality. We should probably be doing it to, right? And we should do is security Casa engineering, which is also referred to as SCE applies. The scientific method is infosec
our way to think about it. I understanding how our systems are behaving drives good system security and it's really difficult to actually dry those outcomes without this understanding. It recounts engineer increase learning culture with planned, empirical experimentation. Just like really any other type of science. One of the accused uncomfortable truths for some people about security chest and steering is that you intentionally introduce failure because it helps you uncover the system reality, your feels scary for a lot of people, but that's the only
way you can uncover the truth. Ultimately. Of course, we're taking that experiment already but those come after hypotheses. How do we develop those? We have to make assumptions about our reality to inform her experiments. This case our security reality. We need all kind of the investec reality involves attackers and Defenders. I felt really like a conflict scenario. Dragon game theory, lingo, any conflict scenarios, no matter the type. We have to make assumptions better adversary. Is there a core part of our reality? Trace Adkins, you're pretty bad at this pretty bad at understanding
kind of like our opponents perspective. However, a decision trees actually read a novel device studied in the behavioral Game Theory holds that helping humans improve assumptions. You listen to this belief prompting. See some of the other socks already familiar with this thing is a taxi from the ground up but he's okay. He's taking the experimental evidence from a bunch of different study suggests that I thinking about how your opponents will respond to whatever movie about to make,
you're actually going to make a significantly stab. Your choice and more rational Choice decision tree socket. Visualizing articulate. Exactly. This like they think they're kind of like the visual counterpart to delete bubbling. Infosec decision trees, help us navigate attacker math about the text. Now, just like any other projects and attacking pain is expected to generate a positive return on investment. And this attacker our allies quickly referred to, as the attacker math until the nipple. Sagaciously stated all the way back in 2014, Packers have bosses and budgets to. A lot of times.
It is like a 9 to 5:30 job. The understanding the attacker mouth related to your individual systems and services is absolutely invaluable in helping you prioritize. What security controls you need to implement and what your security strategy should be my first place. In the context of future cast, engineering attack her mouth, additionally 4, pints of blueprints for the types of experiments that we should be conducting our systems. So when you think your highest Ry options for the attacker, you're scaring, which actions were more likely to take and dust you eliminate the types of failure
that you should inject intersystems Visa V experiments to test your hypotheses and test your assumptions about what are we doing for? This is actually in an example of a decision tree that I have a recent blog post that coordinate with practice, which would be nice today. It looks complicated, but it's actually, surprisingly straight for it's built. This is another deal that this is how it looks in the O'Reilly Book on a security gas and shading. So Wilson leaders flexibility and how do you make it work? We're going to get there with our own example today, so, Let's
get started. Right? Let's build one together so we can understand how it helps us generate hypoxia. Let's dig into this case study on decision trees. Afraid this isn't treated AR reality. Is that the attacker school is running a Christian minor in whatever cloudhost containers. So I am aware of the popular opinion is that Christian miners are mostly a nuisance, but I think they reflect remote code execution in some of your production systems and they need to be taken way way, way more, seriously. So this really serves as a fantastic example for us all with potentially high impact in our
system, that's not usual for us to explore and generate some hypotheses. The first tranche of your decision tree, really reflects the lowest-cost path for attackers. Like what's the easiest thing for them to do? And I like to very affectionately called this meeting yellow sex Branch. I sent that usually involves the attacker, taking advantage of some sort of, for lack of defense or some sort of implementation of a system and the yellow part. The skinniest is that one of my friends called them and professional attackers. Delightful absolutely try something like showed in or Port scanning if
it works, you know, it's always about working smarter not harder, so lazy can be actually super smart to be jealous of action are really, I guess it's practically an in action here. Is that leaving your duck or suck it publicly exposed. I kind of have to be like that to happen. But in case that's decidedly you look sexier in another form of yellow. So today, I could actually be exposing your kubernetes talking. I can also be discovered for sure. Port scanning orchestrate is actually very tempting way for attackers to access the underlined host. We're not going to cover the
orchestrator branch or Simplicity. Okay. We have yellow SEC, Packers are going to take advantage of that. So once I get access to host, either doctor stuff, it's the easiest thing for them to do to reach the goal, is to schedule a container that includes the Kryptonite. I like to think of this. Instead of like BYOB. Bring your own container option for like, brunch, time for attackers, right, but we can see if we can build this first very, very simplistic, Branch or we have reality. Then we have these hackers using with shodhan or Port scanning
tool and then because we yellow sex rather than actually defended the attacker can find are publicly expose Docker socket. And then from there, they can schedule the round container and then run a crypto Miner in that cloud hosting container, then the attackers with teacher at the French eat some of these colors. Check out more about the formatting. Guess we're not going to cover that today. The obvious mitigation hearing is now. We need to think about like a k, the attackers going to do this. How do we respond? Right? The obvious mitigation is to not publicly exposed to doctor. Socket
has another house. Today, for Saline. What I seen in practice is a lot of Defenders, kind of stuck there with her thinking. I think like, you job, well done, but that's not how reality Works. Attackers have a goal and they're going to escalate their investment is necessary and depending on the value of the gold to them. So after each medication that you assume or you put in place, you always need to think about and ask yourself and your teen. How is the attacker going to respond to this? This case, like we don't publicly expose the doctor socket attacker could respond to a few
different ways. Simply attempting, the feat attacker would escalate to make brewing over day at us. But honestly, that mostly serves to flatter ourselves. Probably not that important. The most part, realistically attacker will consider the next easiest, or at least expensive option. And this case, it's probably getting for Bolivar weather looking like, work dress. Anticipating the attackers response to our response. Response allows us to preemptively respond. This is really the magic of them, something to order thinking and decision trees. So we now have that first realistic
Branch. That's the company. Now, by the second Branch, we have our first mitigation, not exposing the doctor sockets. Kind of the flow from there. You can see, we've introduced this new SunPass do the medication and because we anticipate the attacker will scan for vulnerable webapps next. We can perform phone standing during software development another medication. Safe to say that. Not all bugs will be caught before software is the point spread. So Alaskan generally cash exploitation of known vulnerabilities. So we put that in this Branch as well. I got a big game. Doesn't
ensure we have to consider how the attackers likely to respond to all of this to even when they encounter a seeming dead end, this case. They may switch tactics. Have been trying to find our Cloud use in public repositories. So it doesn't make things a little simpler. You can see about line. And I like teal color, the 3rd grade shows that the attacker would likely skin are get her proposed 480, West clear as your keys and response for medication. You can actually indicate that relationship with a dotted line goes from Ralph up to scanning, add publicly suppose that's usually
illustrate some of these clothes. But once I get those timesheets, that allows them to get into the host of container service, which then let them schedule a container with the crypto Miner and when again, With this Assumption of our adversaries at Hebron and we can think of ways to block up there. Moose in anticipates are alternative facts to this case, skating are towed. We both said things that look like he's our self for the useful as a something like irritation or I am rolls that ensure that Keys. Even if they're compromised only allowed for limited access is the whole Beast.
British thing that we talked about 24 years. Also, as I mentioned, in my 2019, black cat talk with Dr. Nicole for Sprint, billing alerts actually can be a surprising source of security signal. Again, actually helped us off to the fact. That's what's an attacker, scheduling a ton of their own computers via auto-scaling. So she we are getting a little more complexity Regency until this new resulting flow in action and even make some lines mitigation like you're with key rotation dotted if that's an option but you think would work, but maybe you haven't implemented
yet to be able to see your options. The beauty. He was also thinking through all the medications for particular attack, our booth and help you can compare the costs relative to what should be, probably a very similar benefits. Of course. This kind of picture we have here is actually assuming a pretty Rosy view of reality and which are medications, work is anticipated, which is frankly counter, the most complex system thinking, which says that's failure is inevitable. Realistically like an attack, right? Then it came to the host of container service, will actually want more bang for
their Buck, given all the effort they put in. This is a really important points. When you're building your decision trees. We have to think about what's going to allow for greater ongoing value, cuz that's how the attackers going to think. Not just short-term realization of the goal to doesn't mean that attackers are unlike a lot of humans and they can absolutely be myopic and kind of short-term oriented. We need to think about all of their potential options. In the case of a crypto Miner, the ideal scenario for the attackers being able to run the crypto Miner basically 24/7 and persist
across restarts to but even if you don't have anything greatest to do an infrastructure today is an attacker was actually a financial incentive to keep the command-and-control running. Generally attackers are going to want some monetize, their persistence not just like leave it there, especially in cases, where an organised criminal group is frequently called upon by their looks like Hollywood Knock-Knock. Please hand over whatever access you how to add Target organization. Otherwise, we're going to break your knees, something like that, right? So how can be attacker? Cities birth of
requirements and Achieve them. Beverage container is one potential answer for the attacker. Generally are regular containers trapped within resource limits. Of course, the attacker could escape and said they were going to think of it. Really adds up for this container offers a few key benefits for attackers over the regular container visibly that they can use the privilege capabilities itself. The attacker Lloyd, I need something or monitoring systems and the risk of the container being killed or be started to. So if he needs to maintain access to the
Target of organization for you later, princess container helps them maintain that access. And it can also actually served as a canary for whether or not they're being detected and whether or not this good like resting point. That once the attacker used to pillage containers to escape the host as you can see here and outlined until they can create a new system. Danan and asked us some beach by the Christian Mayer. Again resulting in the wind is not going to just roll over and play dead responds right? Where the fenders we supposed to be so we can Implement something like security policies.
Your orchestrator like Cooper. Nettie's. That was not saying 2019 black inventions. I reference Daniel use d. I e, Triad immutability actually, has a security property. This case in a beautiful house, which means it can't be changed or modified after it's spoiled means the attacker can't actually gets disk. So it counts as a nation and the attack of the activity, that the attacker could perform on the host to a visit to actually be caught by some sort of hosts security, monitoring tool. That's looking for something like new files being executed.
It's okay. We don't have any of those medications and all the attackers just like super stressed out there under like 12, longitude, power through this and really, really feeling bad vibes. We haven't made it easy for them. And that's precisely the point here. Are there now, stupid caffeinated and they're going to think like what can we do to get around this final host? Monitoring mitigation that we implemented as Defenders of an investment spots. Are at the attacker. They could deploy a Fila script. A minor that doesn't require, just ask access. This is certainly a fancier but
it's certainly in the realm of capability for like an organised criminal group, and definitely means they haven't made it. Easiest easy for us either. Also liked down and coffees and watches. Actually a very clever medication here that we can borrow from a sarnese monitoring resource usage. This is also why I've been proselytizing the importance of security teams. Caring about availability more than they currently do. Remember, it's part of the ritual CIA Triad since availability signals actually are a really good indicator that something's amiss security buys. It shouldn't just be
considered, like, ox concerns are a story concerns. A compass. So we can take a closer. Look at this part of the tree that we just washed out. So you can see our new medications as far as I can get ability monitoring of host security and see the attacker, responsive finalists for the miners. Then we show our resource usage monitoring. Again, for Cindy close up. Like we do something the attacker response. Will you respond to that? Yeah, before in the attackers day week, maybe even possibly a month, remember attackers aren't otamatone. There are humans that have emotions and he
goes to definitely goes, so you can absolutely frustrating disappoint them and maybe even make them angry horse anyting emotion-driven like they're not just going to give up has been bruised. It's not over yet. This brings us in any decision tree to the creation of the heart attack back. This is the one that requires the most investment on the part of the attacker, to be out of your control, until you've invested medications for the easier wrenches in your decision, tree something with huge dislike colossal, waste of time and budget. Implement
mitigation Forest. Even if it feels right, I usually call this the hardest cost Branch the Otay Ranch, like, oh, they all the way down. Like, OJ James toback style could include things like a stream back. Door is in your supply chain, physically locals, like no taxes data centers compromising it, if you must control clean. So very expensive attacks that he probably thinks you just shouldn't worry about until you covered. Literally everything else. Your case Packers after blood, they want to own all of our Cloud now. So we need to really bring something. What's was ruthless
attack scenario that they'll come up with for them to you, our own all of our cloud. Naturally, it's a maple syrup heist. With black males course that's going to be the hardest cost Branch, right? The Zeppelin detector steals a least amount of maple syrup. Leave it to you to decide whether that's Vermont. Maple syrup on staying out of that piece for vacation home of our organization CFL. Oh wait, we do have a surprise mitigation up. Her sleeves or a yard mow, has a very good boy in their home. So the dog didn't believe it's a p,
zero bork bork sort of alerts and it serves as a very fine form of intrusion detection over smart attackers, right? We could anticipate this. So, maybe we see through her,, since that's a seafoam 18th. And instapage for dogs, very cute. But also has her field, critical intelligence. So they bring a bone with them to distract the dogs. So it works for the plants to maple syrup and see those basement which conveniently fits all of those barrels. Maple syrup. And finally with this a list of Contraband planted. He was property. Attackers, can now blackmail to see if I went to sharing
the organization's. Cloud owner level credentials of them hired Cloud us. If they want to run the crypto Miner, they can do so now. But again, we've now crap that you're the best kind of like she'll kill section here. But just like some of the plants Jared nation-state attacks the camera. Sure our attention and Imagination. Like come on you would invest in mitigation for this purse, right? It's ridiculous to keep that in mind. The next time you hear about some vendor offering witness protection, against the risk, that's largely out of your control. The Felicia's
feeling of control isn't worth sacrificing. Your ability to raise the cost of attack on other branches. I promise. That we can do now to see our decision tree that we built today in full and admire our hard work before moving on to the next phase of science thing. So I can see this beautiful tree and it looks really complex. Right? But we break it down to the steps. You can see it's actually really buildable should build a collaborative way to. I also recommend not having your your decision Tree in PowerPoint like this footage of the PDF. If you can talk about this again relatively
recently in a blog post, but it really helps with navigate bility. If you can like zoom in and pan around cuz he can steal sleep. We have this lovely visualization. Now that documents are a sumption felt like the attacker actions. Also how we can successfully respond to them. Crypto miners. Just one, tackle should be the great decision trees for uncles. Attackers might have against the organization. Even if it's a success would need to only actually attack success could lead to some sort of material or erosion of your organization's value. So, electrical 10 to ask me like how should we
prioritized pastries? So this is an example, prioritization Matrix that some from the O'Reilly Ste book and it's comparing attack her belly with organizational value. So for instance, I slap Mochi database, doesn't matter to attackers, and it doesn't matter to you, don't create a decision, tree for them. But a production revenue-generating circus that's going to be really important to you. Same thing, which is General compute resources. Like, in this case today. Now, we can move on to using science to make incidents boring. I know you're probably
very skeptical at the moment. So Amber my goal like,. Let me look at your Social Security. If you print on ask yourself. The question, like what if instead of like all the stuff we're currently doing, we proactively and purposely the Shaded in Presley to learn about the impacts that they would have in our systems and design, a very graceful automated responses. We have to assume failure and design, listen to expect failure and handle them gracefully. And we can do this with the power of science. Specifically the scientific method. That's what STD does. The number we have our
hypotheses from our decision tree. We created we can now proceed to experimentation. Experimentation. Delete when you think about it, like he's such a rise in new insights and information that was previously Unknown about our reality. These insights for the complete that feedback would be talked about the scientific method and hopefully that is what drives scientific process or progress. So also if you guys are really just essential for continual learning, which is the only way is to enter as we can keep up with the ever-changing context. Right? Constantly hear about what you know,
what security is always evolving. Like this is how you keep up. The security Council engineering introduced the security of servility into your defensive programs through this. Rigorous experimentation that helps illuminate, the security of a system in reality. Not just in theory. The big question is, how do you conceptualize the security cameras experiments? Anyway, really useful? Template for defining experiments is the event of the condition. Ask we are confident that the system will respond with Y. Let's take a look at some of the ones that bubble up for our decision tree today
in the event of a misconfigured doctor socket. You're confident. It will be detected logged and killed. You can see the case study in the security Council. Engineering ebook about canceling her to learn about how. Aaron Michael Arthur conducted experiments. Another one is in the events, an attacker autoskillz their containers for your compliment, a billing others. We generated as a final example, events that occurred reminder payload is downloaded onto a host, your compliment our host security. Monitoring will detect it. So he can be at first is that
experiments. Require careful planning a chapter on how to properly conduct these kind of experiments and ebooks. So, please read it for the sake of time though. We're going to see that you're planning carefully and move on to the next step of science thing. So except for the scientific method where we are now, which is comparing your observations or predictions. Once we've conducted the experiments a scientist, which is what we are now using to drink a sentient life. Experimental results to validate and refine our hypotheses make a decision trees. You want to conduct experiments for
each branch of the decision tree, ideally again, starting with more of those low-hanging fruit. Nothing with his evidence in hand. You should bow to each of the assumptions that you had in the tree. On the way, we can do justice. Conducting a post-mortem to discuss. What did or didn't work as intended? This is true for both the attack and defense. I'd want to examine the experimental results from all facets in order to more deeply understand your systems. Importantly, this post one and a half to be blameless. The finger pointing at humans as the fastest way to kill a reading
culture and your point of decision tree. Instead it's there. So this is my name is really when you should be in the point of the scientific method. Asking a lot of questions. Like, where were your hypotheses cracks? What did you ever look at this? Like maybe one attack step required. Another step in order to meet the attack goal. Maybe didn't anticipate that maybe the policy controls that you thought were being enforced, actually, seem to be taking a nap. What do the analyze your experimental evidence. It's kind of document your findings and it'll read on the experiments in. This is how
you can complete that feedback loop report. Your findings will be useful to update your decision tree. Ideally, with, versioning by incorporating all of your newfound experimental evidence. Honestly, like he likes doing documentation, right but it's super duper essential to stain. A learning culture, number of scientific progress just as a hole with a physics or chemistry or like you the creation of that recent vaccine, like it's always a result of building upon prior knowledge without those prior findings being documented progress, which is a much more limited Even in our world. So
hopefully you get it now. So you did it for you recorded your findings and now you can read on your experiments and where you conduct your experiments that specifically a reflecting the potential security failure actually build new muscle memory for responding to incidents. Repeated practice of these experiments can make incidents actually feel boring and stuff scary and stressful. We all know how much like burnt out and stress. Planes are industry, isn't responders the security Council. Experimentation can transform incidents into problems with known process, for
solving them, which directs the faster and higher quality security outcomes. Under somewhat, ironically up and feel insecure about their ability to handle incidents both in terms of human and systems resilience. So they repeated experimentation with actually boring from security and help us grow confidence in our systems, resilience in the face of failure, and in the resilience of our teams. This cruel collected conference also comes from using our experimental evidence to inform refinements in our system design. That's true, whether that's architecture medications process, these
policies and so forth so we can steal experiments using decision trees are blueprints for a hypotheses. We can continually improve our system safety instead of succumbing to cognitive bias and the Wacom old tactic that we've been doing for decades with a fun when designing, our defensive strategy. In conclusion on our science Journey. Scientific method is basically just super duper legit and security cast engineering, very eagerly copies of homework. After all the scientific method has served us well for centuries. And let's a scientific discoveries Beyond Your Wildest
imaginations. Decision tree specifically help us formulate hypotheses about our security reality and begin that invaluable feedback loop. We found the case study, it can serve as a blueprint for what types of experiments we want to conduct. Security Council engineering, just like other forms of scientific experiments, validator a hypotheses and uncover important truths help this device, our understanding of our system safety, and we can make decisions based on evidence rather than guesswork. What do you think? You can start to make incidents boring after enough? Practice, incidents can
start to feel like live wild. Experimentation was pretty cool. We are confounding disasters, start apply this knowledge to your job today. Next week. I want to identify the three most relevant to tackles your words using that Matrix for its attacker value organizational value. And then you should create a decision Tree in collaboratively for each of those Leah tackles the next 6 months. You can choose a hypothesis and then plan and conduct an experiment for start small and grow from there. Let's begin with only service, dive into some of the stuff around security chance,
engineering to check out the O'Reilly report that I wrote with my co-author Aron Reinhardt, which is available for the lovely price of free. What could be better than that right there. Thank you very much for your time and attention and now I do so some chaos.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.