I'm a programmer professionally and as a hobby - working making software for open data developers and working on open source software of my own creation (https://github.com/JackMc).I'm passionate about Python, open data, and open source.View the profile
About the talk
RailsConf 2019 - Rails Security at Scale by Jack McCracken
This is a sponsored talk by Shopify.
At Shopify we ship code. A lot of it. 1000 PRs a day. This means that our security team can’t reasonably take a look at every change that goes out to Shopify’s core product, let alone the hundreds of other projects deploying every day. Our team has developed some awesome tools and techniques for keeping Rails safe at scale, and we’d like to share them with you.
Alright. Hi everyone. So I'm Jack McCracken. I work at Shopify. And today we're going to talk a little bit about how we managed security and one of the largest rails apps in the world. So first off a little bit about me, they took that photo with a little bit of a low-resolution camera. So, you know, I'm sorry about that but I love puzzles. I love crosswords big into baking. These are the two things I get my mind off software with and most importantly I love security. I love making sure that every single one is shopify's Merchants is secure and like does
not need to worry about their platform security and just get to do what makes them awesome. So first off what is Shopify so Shopify is like a multi-channel e-commerce platform, which is super fancy words for whenever you want to sell something Shopify will let you sell it wherever so if your customers aren't Instagram we still on Instagram if your customers are on Laquita on Facebook Okay, so a little bit contacts on what we're dealing with here. So Shopify the main application deploys about 40 changes to production per day in 2017.
And it has to be at 1 we had 1,900 employees. We had 375,000 merchants and 80,000 people cut for a second. So the the joke that we double every year but I'm going to show you that it's slightly outpaced fat. So this year this is a year-and-a-half after that presentation. We deploy 150 changes to the core product. We have 4,000 employees 101500 which are solely and R&D 800,000 merchants and 170,000 Peak requests per second. Is that are two main customer bases as a security team
the employees the people who need to employ safe changes and the merchants the people who need to use those safe changes. Have both doubled or tripled in the past year-and-a-half. So that kind of leaves me feeling a little bit like this. I'm freaked out. I joined in 2017. When that when those numbers earlier were accurate and at that point I would like oh my God, there's like a thousand people working on this product how the heck in 10 people secure this thing. How can we make sure that every Merchant that uses Shopify had a good experience and doesn't like feel
like we're in skier or experience. God forbid any database problems, so, How do you prioritize these things it turns out it's just straight-up impossible for 10 people to police the actions of 1000 people. It's just not a thing you can do. So you need to beat you need to build it into your culture. You need to make sure that every single the developer at your company is aware that their changes impact 800,000 merchants or a million users or however many people use your application of these are real people used to make sure that those things are embedded in your culture and you need to
make sure that they don't like They don't like forget those things. So other thing we do is we use robots a lot of robots. Hopefully better robots robots. I like to break down what we do in the these three categories. So the first one here is make things safe by default if something looks right. It should be right if something looks like it's going to be secure. It should be secure something looks scary. It should be scary. Second thing make messing up not the end of the world. Nobody wants to be that developer who stuck their being
scammed by some asshole security guy like you messed up and that's just doesn't make anyone want to improve it makes people want to quit and last make security cool. Nobody wants to have again that security guy yelling at them people want Security in their products people care about the product. So why not enable them to make their products secure The first thing by default like I mentioned the point here is to make sure that if someone is doing something it is safe and if it is obvious it's safe. And if it's not then it's probably weird so
How do you keep things safe by default you could set up one of those fancy break Nancy. I things that everybody talks about great man is a great tool but you need an amazing like a big security team to make sure that those changes are relevant to your developers or they will start ignoring them. We had this experience. You need to keep it simple. Make sure that the changes you're making that add process are not necessarily things that impede your developer from doing their job. The first thing I'm going to talk about is the little example of one of the vulnerabilities that we
know we run into on a pretty regular basis. I'm sure most of you are familiar with cross-site scripting but we're going to talk about it just for just for a little bit here. So you can see here pretty standard grb. Template. It has a insertion of the parameter name and has weird method HTML save called on it. So what happens if we go and visit this site we go to good.com. Whatever name Jack yet. Hello Jack. That's exactly what you'd expect interesting thing come from if you try to pass Gabba script in here. So if you do a script alert document. Cookie here,
you will see all the cookies and every time I do one of these every time I show somebody was like, oh my God another scary person popping up an alert box and telling me it's the end of the world. I like to describe this in the sense that anybody who send you this link and you click on it could do anything you can do in that application. So it's outside. Add a product you could change the password you could do anything. And as an attacker, that's a really really juicy Target. So how do we make sure that as a large company? We're not we're not at Target for that kind of thing. so when you
put that into the Dom you actually see that you got hello, and then a literal script tag alert and script tag and That means that you are X allowing. Whoever is inserting code into your or inserting text into your website. You're allowing them to execute code. The problem here is that HTML safe. So if we take that out. We got this. It looks like hello and then a bunch of symbol and you can see the LTE. So that means that insert a literal less than sign and then the GT which means insert a literal greater than sign and this is actually going to show you some
like this weird name, but still better than a cross-site scripting vulnerability. All right. So what the hell is this thing? I started insecurities. I had no experience a year-and-a-half ago and I thought that what this did was Mark the string as like Escape escape the string and it like took out all the bad stuff. And if you call HTML safe on it, you're good to put it into any RV template. It's not safe at all. So So what we sow actually does it take the string and turned into this thing called a safe buffer and a safe buffer inserted into an Erv template will not get escaped
at all. So you won't get that LP thing. You'll get a literal it you'll get it inserted a literal HTML. So what we've done so cops are here. There's a bunch of them already built in Robocop for rails rails last output safety and its job is to literally look for any uses of HCL safe and raw, which is basically the same thing in your code and just slide them. The problem here though is what if you really meant 80ml safe. What if let's say your rendering
markdown and you're sure that whatever came out of here is safe. Okay. So how do you handle that? But we've done to get people away to do that while blacklisting of the HTML States in our code has been given this method a much more appropriate name and our code base dangerous. So the other thing we built it we built this thing called caution tape, but so talkative. It's not an invention of our team, but we co-opted it to effectively just whenever and I'll save in a bar and says hey
documentation on that maybe it's not a great idea and then our team comes along and reviews all it does take somebody for review and comment on a pier in probably 15 minutes with GitHub action. You got GitHub webhook coming in check it against the regex Java doubts about a comment interview request not hard keeping it simple. So you can see here the effectiveness of that. I've done a little bit of redacting for the person's name, but you can see that there was some code with. ACL
safe on it. Our team member mentioned. Hey, there's no way out here. Why don't you just not your tasting all safe? And then the person responsible do I copy pasted it my bad if we're gone and low and other thing was removed from our code base. Another thing that we've built but I really like is the crb Lynn application or you'll be LinkedIn what it does. Is it a little smarter cuz we talked about how inserting it to HTML you pretty much got this handled for you by rails rails will just Escape anything that has passed in the other was like less than %
that it's Safe, obviously, that's not perfect. We can solve cross-site scripting for you. But this will both stop people from putting process cutting into your app and it will allow you to just like deal with it when it happens in production. So the second problem going to talk about today is cross-site request forgery so cross-site request forgery happens when a attacker is able to execute an action on behalf of the user without their knowledge by making them visit another site. So let's say here we have this form and it's on
everyone's favorite website good, so good. Com have an account system unless you specify an account you can edit. Let's say your username and your password on your email everything in here. And this is what the form looks like. You actually see this token here. We'll talk about that in a little bit. But instead of if I were to take that and instead of doing it on the website set up people.com with them called good. Com account / your account ID. I can change your name and password. If you visit my site. I do know how how how does
real handle is so by default theme real handle this for you. The problem is there's a bunch of legitimate use cases where people either need to tweak or completely turn off the settings. The setting actually allows you to just do you see that token in the previous form here. It just like checks whether that session token is equal to the Token in the form and if it is not then it does not allow the request to happen. That's the standard behavior of rails, but If you turn it off. Like a good friend here on Yukon good.com have done using skip before accident graphic
verify authenticity token. That will that request will work and your password will become Hunter 2 and no one wants that so skips before action is another bad pattern that would notice and there's almost always a good way around it. So you can use that most of the time there's a Compadre with exception unless your server is talking to other servers if you're talking to clients, that's unfortunate. That's an option is the safest option hundred percent almost all the time. What we've done is so we build another caution tape bought rule with you to the red jack's looking for skip
before action verify authenticity token and it sends a message that maybe you shouldn't. The location in which you would use why he was an awesome real documentation on this subject. And it request from our forgery protection in Forces Group. So the second thing I want to talk about my second theme from earlier. My slides was messing up should not be the end of the world. I think somebody has a really good way of saying this but I couldn't even like come close to doing God's going to put it up. We don't make mistakes. We make happy accidents
in security little less through but what we What we can do is turn the mistakes into Happy accident so we can take these things have a positive outlook on them. And like I was saying before not get mad at the person who did the thing but get mad at the process get mad at the things that allowed them to get there because obviously this is a failure of a system if it's happening in your code base not a failure of an individual most So what we've done and we built a couple safeguards that allow people to notice when they messed up and quickly
and effectively address it with little to no data exposure. So this is a very common pattern you'll find in the rails at you'll see a controller that controller will have an action that action will then do a find with some parameter and that parameter will then be a user controls. So the problem here is that if the blog word Alexa belong to a user that blog would not necessarily be belonging to that user right? So if you if I went here and let's say I'm user number 5 and I look at blog post number
8 and lock box number 8 doesn't belong to me and I try to edit blog post number 8 that's going to work right? You don't want anybody been able to anybody's blogged. That's just a recipe for disaster. So, how do you handle this? This is the control. So here's what you would look like. What does it look like in a regular app what you would look like if you were using vanilla rails and we built a dam that allows you to Simply add fireballed to the start of that and what that'll do it. It will make it so that your user object has checked every time that you go and you make a request that
involves that user object in it belongs to a relationship. They will check the belongs to check the current user using the new 5.2 current attributes feature, and then it will actually just Pro 500 if the day does not belong to the current user so rather than the background are like, oh we got an error maybe someone should be a 403b an authorized but I think that these things can be solved at a higher level than handle the little bit more gracefully by the involving application and they should just be kind of a safety blanket for if there's a security problem. So how
that would look handling at the higher level in the application would be using your current class and going current user. Blog. Find. We called it the vulnerability that we developed and indirect. Insecure direct object reference. So what that means is you're trying to reference an object via its ID and it is insecure because you're not providing access control on it. You can find the scam open-source a couple months ago at Shopify / active record Fireball. I recommend you install it in your applications soon. It will even have a report only mode so that you can just set
this mode on to your app and you'll get logs instead of blowing up every time that you see one of these unauthorized access has and you'll actually noticed that if you install this on your app and add those Fireball belongs to your test, we quality will increase significantly, we found it in our core application 300 or 400 test for failing just because they didn't have the they didn't have control of like well, we're accessing cup and then we're using chop these data and that would blow up because the staff
and they made our cats with much more consistent unless more like accurate that we are with the real world. So the next thing I'm going to talk about it this thing we both internally called Watchtower again. Open source, but it's not rocket science. This thing literally just go through a big list of domains that you provide to it looks for graphql. It looks for sidekick and it looks for graphical. These free apps. We found have a commonality of being almost always exposed. If you go to most real bats or most unmaintained real that that aren't the core product of a
company you'll find that some of the time they're sidekick will be exposed on the flashlight it cuz it's hard the rails code to make these things into to make you think the Cure and put them behind authentication is not pretty and that's the problem that's safe by default. That's not safe by the fault. So how the heck do you think all these things? A lot of companies have a problem even with knowing what the heck Services exist? So if you like run this app, or if you just run a strep throat have a script that did this it just goes and hits everything one of them extra 200 a problem.
She's a 404 that's fine. That means that the application either doesn't have one or that's not telling you it has 150 a 301 you probably getting redirected to rent authentication all gravy. so the last thing Define spell please meet like we can't make a Like an security proof sandbox for this application because it does so many things. It's impossible to actually like naked completely secure. So how do you handle that what we've done and we've set up a bug Bounty program.
So what these are answer most of your familiar, but it's a way for hackers or researchers to get monetary incentives for hacking on a company. So what this does is gatz good guys on your side when the bad guys are already trying I guarantee you there's already bad guys look on your app. So should you run out in front of your company right now? Maybe the only problem is you need us amatier security team and you need somebody who is able to handle these reports kind of on a full-time basis the thing we recommend if you don't have the resources and a lot of these things can be
implemented by those teams is just setting up a security at your your domain.com email and having a explaining your policy on security. So if you have a page, there's people out there who can help me with this together paid the dish says we won't sue you for hacking on our website and there's a lot of people who will come along and be like, hey, I found the school zone ability for free. I'm going to talk briefly about to vulnerabilities from our by Benny Program. So this first one is here is the most paid out bug bounty on the Shopify program and it was a server-side
request forgery which is just convincing a server to make requests on behalf of you and doing something nasty with that. So let this person is able to do what he used The Exchange app with something we built internally with allowed you to sell your Shopify store to another interested on spinner and it let you just take a screenshot of your store and they'll put the print out onto your listing paid pretty pretty like normal behavior right under the hood and So the combination of that headless browser and the fact that anybody can
customize their Shopify store that contain anything resulted in the fact that you could redirect to this really old obscure Google kubernetes API and just get access to our kubernetes cluster. The cool thing here though is that we were able to get this vulnerability in from a hacker. He gave us an amazing detailed Report with the link to on Twitter later and that report probably saved us millions of dollars. We only paid $25,000. So we were unable we were able to figure out nobody exploded this thing great and then we gave the hacker
reward. so the second one I'm going to talk about you don't have to read the diagram. By the way. I know they're like super long. It's basically just to give you an idea of how complicated is vulnerability is are the second one allowed you to pretend you are confirmed an email when you had not but doesn't sound too bad, but it actually ended up allowing a user to log into any Shopify store again, what the heck this is crazy. So we ended up paying out a decent amount of money for this one, but we probably saved a lot of money and we were able to mitigate both these
vulnerabilities in less than an hour. This one on Christmas Eve or Christmas Day. So the thing to take away from here, is that these bug Bounty programs and vulnerable to do just do your programs allow you to Find vulnerabilities letters impossible to find if you are a D-Link static analysis of everything any of the two of them answered before these are just things this this thing. I think I looked at the number of PR is that were involved in creating this monstrosity of a vulnerability and I think there were three separate PRS
that actually culminated be like weird can circumstance in like the fact that you did 20 requests at the same time that we're doing different things. You could convince the server that you were confirmed when you were not The last thing we're going to hit on it's probably in my opinion the most important as I mentioned. It is impossible. If you got a team of ten working to secure the changes of 1,500 developers that it's not scalable people need to be invested and people of Shopify are everybody every developer of Shopify knows that they're changes are responsible for the
livelihoods of 800,000 individual businesses. And that is a big responsibility. So how do we make sure the people are up-to-date on the latest security and make sure that is not there full-time job cuz it is so how do we make this fun? Everybody wants to be like this guy that's been mine. That's been my experience at least is everybody wants to be a hacker. So if you if you show them how to be a hacker that sounds a lot cooler than security training doesn't it? So, we don't call her workshops mandatory security training editore. We call them
learn to hack. So people come out to these things in droves. We have had 300 Shopify employees attend these things and all you do is we still have on their ability in a nap. So for example, this is what we call a SQL injection. I haven't hit on them in this app in this presentation, but this allowed you to get all the products here and we set up this little simulation of Shopify. We built it over three days and it just lets you like try out these vulnerabilities we demo to you one of them and then you done what you done all us another one after looking for a little
bit and be ghetto prizes for the people who demo this is incredibly effective and you don't need to build a Shopify clone or whatever the hell you're Apostolic alone to do this. You can use something that's already built. So Google made this thing. I'm not going to say I'm not going to I'm not going to go there but basically what this job is is to be vulnerable application on the web. If you go to Google this name that I will but I can't pronounce and you go to the website you can click a button and you'll get an instance of the running on your
on Google Cloud. You don't even have to have it running locally. So what it'll let you do test out a bunch of vulnerabilities Google have documented this incredibly well and the source code is available if I remember correctly so you can go and look at the source code and see how to fix it. The other thing for specifically for this crowd for reals people if a wasp and built the thing called real goat and rails goat its job is to be the most vulnerable in the world. It has every real number you can think of under the sun. The real goat I would recommend just heading this up having your
employees hack at it. Even if you're just a security content developer at your company booked a couple people keep posting you develop. Let's take an hour together to work on this. I don't even see how much we can tear it apart if you can find some really good ones really fast. So another thing we were like, oh my God, this acting workshops are great. How do we level up the security awareness training we decided that we would set up this thing called a Halloween hack this we are we basically work together to make a spooky CTI and let the
CPS worker for the flag competition was was about was finding these little things called Flags a little text strings the end with a garden with a curly brace and you can hack a nap and all kinds of various ways to get these flags and their flag get you points. We were able to have a thing about 30 top of five people attend this free beer free candy event where we talked about. We talked a little bit about a security program and we had about 6:50 challenges set up a team managed to solve all 60 of our challenges in a 3 hour. And they weren't even from security.
These guys who had attended our workshops and had to learn these things. So that was a great feeling even better feeling was when someone hacked our damn leaderboard. So what happened was we had not escaped table and we had not escaped the input. We were putting into the live updating scoreboard scoreboard literally everybody. So that was super awesome was We don't this thing called the Shopify CTF or packing challenge. This thing is basically the same as a hackfest but it runs all year round and it will
have prizes. It has all those kind of good things and people have been engaging so much we had over 50 signups heading towards a hundred and those people have all been working in their spare time getting a coffee whatever they get to spend 15 minutes learning about security. We don't run now we start building changes about vulnerability that happened in Shopify. So if you are able to build something like this where you just take what little later trails around ability set up a little rails app run-on run-on 5.2 and then set it on the internet. There you go. See t f calendar for the
new axon view vulnerability so last thing I have some takeaways for you guys. Little thing that we found are the most important parts of running a security program at least an application security Program scheduling if you're going to build any and you should make sure that the developer is safe while treating them like an adult these people care. As I said a couple of times they care about the people who work who are using your product or they wouldn't be there they care and they want it to be secure. So help them. Don't baby them. The second thing. The only way to scale security
is if you're going automated, there's no way that your security people are even the number of security like superconscious developers that your company is going to scale to the point where it's equal to the number of developer to your company hiring 1530 people to deal with 1500 developers. It's silly because developers care as I said, The third thing is make people want to learn security. So if you package it up, like I was talking about as some dry security guy standing at a Podium telling you about security guilty then no one's going to learn
it. Right? No one's going to want to listen to you. People are going to be checking their watch it. Like how the hell do I get out of here? Whereas if you treat them like adults. Can you tell them? Hey, these are the vulnerabilities that happened in your own application. Then you'll get people's attention. And the last thing is Securities everybody's responsibility if you're working on a product. Take a second to read up on this kind of things take a second to learn about the most recent mail going realities all those kind of good things and that kind of stuff will multiply times over.
Thanks so much for listening. Since this is a sponsor talk. We are hiring Shopify has positions in application security and a bunch of other security-related field along with a crap ton of rails developer positions. So if you're interested in something, we probably do it a child if I feel free to DM me or talk to me outside if you have any questions, or you can actually can put your hand up now have 10 minutes so Alright. Thanks.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.