Braulio is a senior software engineer for the Platform Core team in ActBlue, overseeing payment processing, architecture, and performance optimization.With 6 years in the organization, he’s the third most senior member in the technical team.He has been working with Rails for 9 years and as a developer for 22 years. He wrote his first web application in plain C during the early days of the Internet boom.He holds a MS degree in CS and is originally from Chile.View the profile
About the talk
RailsConf 2019 - From 0.10 to 5.2. The story of a 13 year old Rails app by Braulio Carreno
Small-dollar donations were critical in the last election. The first version of the software that powers ActBlue, the main player in the space, was written in Rails 0.10.
During 13 years, 30K organizations have used the platform to raise $3 billion.
Some lessons we'll be sharing on this presentation: how to scale a monolith to process 1.5M RPM and 250 payments/sec, how to be productive with a 110K line code base, how to minimize the pain of Ruby/Rails upgrades and technical debt.
Intended for beginner to intermediate developers, managers, and anyone interested in building a lasting system.
I'd like to start with a question. How many of you have used? I could blue to make a donation. 1 2 3 I don't see very well for how many you tipped. Fewer how many of you pay $10 or more? I want to see the Hat I can take it from here better. Someone tape here. Nobody's here. Okay, $5. You're the winner. I'm giving away. I did this in our our local Boston RB meeting and someone had donated $100 tipped $100. So I've had it Thursday after I sent the proposal that's why I marked a 14 on top
and my name is Braulio. You can call me brat if that's too difficult to pronounce. I'm with igloo 6 years. I didn't know that Igloo was an early adopter until I met this presentation. I'm originally from Chile in case you're wondering what country accent is from. So in case you don't know what blue they did the gold behind the flu is will know we need money to win elections, but you shouldn't need the support of the wealthy. Blue provides a platform that gives the power to
the people and you can do both things. You can run for office anyone can run for office use in the platform. And anyone can also fun. They are political candidate or complain it was found founded in 2004 as a non-profit. It's independent of political organizations like the DNC the D Triple C. And it's not only about politics. We also help nonprofits. And as of today all the Democratic candidates for president are running using and factories in new rule that says to qualify for the debates. You can either
have 1% on three National polls, or you can fundraise with 65,000 people. So how does they contribution form look like there is an example there for a non-profit. Save animals facing extinction because I donated before my name is and also they have my my card number. So it's as easy as clicking and one of the right buttons and information will be made it what you will need more information to name the address credit card number but it is the same concept
but it's not only about processing payments. We also help with compliance. We have a team of lawyers that we do the filing with a Federal Election Commission or state and local agencies for the person find reasons to have to do anything. We provide tools av-test soup to maximize their conversion rate of the forms. We provide at the sticks and reports for their campaign and Integrations if they want to know right away when it contributions made will send them a notification. Some numbers in the last election cycle. We process
41 million contributions 1.6 billion dollars that's on 39 inch and we helped a 50,000 50,000 organizations. That's for the last 2 years lifetime volume. Now, this is since the beginning of a flu in its accumulated money in billions until today so you can see it took 12 years to raise the first billion then two years to write the second in 1 year to write this Earth. I would also when I came to a glue in the grass, so you have it so you can see the correlation when things really picked up.
Our application is called Indigo of the type of blue eyes. Look the same to me. That's why I have to put the hex code to explain what it is. And this is a piece of archaeology the first comic in rails. I want to prove that. It's it's true, but the bottom is the version of real 0.10 from 2005. Now you have to realize that 14 years is a lot of time I bet some of you were programming in Java or. Net some of you were not programming. I see young people. Maybe you were in high school. The
author of The committee's magdeburg at least he's one of the founders the other found there is someone is Benjamin run. Why did Matt do the first comment? Why do we do software? We do software because there is a business need when in our case we're not a business wearing nonprofit. So we call it an organization. But most of your cases it will be a business but the business is not static that there will be changes and we need to adapt to those things. So the problem is that when you have longer any software,
it would be more difficult to adopt and there will be more lines of coal that thing will be bigger told the goal of the presentation is how we can do this more efficiently how we going to have to change at a low cost and quickly at the same time. We'll have to keep reliability. You have to keep donations coming and we have to provide security. We're saving credit card information, which is very sensitive and also personal information. I have two examples are one example of something that happened recently. We had to implement Apple pay
and we were able to do it very quickly because we used the code. We had four credit cards. Although this is different still the gold was good enough and we understand how how to make the change and we did it very quick. There's another thing we have to a lab. For example, we have the rebates. We have the elections coming the traffic is going to be a lot and we have to be able to handle it. So what are the things that get in the way? What are these challenges for Rolla long-running applications? I put them into groups to the left things getting old
on the right things getting too large and these are not specific to rails if they would happen with any framework. And in fact, they haven't known any software project. Tell Daddy I said I'm going to share with you the lessons dealing with them. I will do it. The first one is the needle depth and easy solution easy solution instead of a better approach that would take longer. So we go with it the easy things in a long-running project you're doing this a lot. You have a lot of things we do
that again. So it's a big problem and the way we do it is with Chief technical feature. Now it doesn't mean I know all of you or thinking I hate this part of the coldest. I really wish you fix it. Then it bothers me every time I see that doesn't mean we have to do it. The only way you can address this is you have to make a case. You have to say why what's the benefit normally the benefit for technical debt will be the Dakotas Mormon trainable can respond to change better. But you have to make an argument and you would have to also
specify how long is going to take and you would have to discuss this with the rest of the organization. There will be a lot of people that would be project managers and everyone of you will have a long list of days that I need to replace all the features and you will have to they will be competing for their resources. We Define priorities and have a short-term plan in a long-term plan for technical did we we address that when traffic is low. Normally that's December after the election or after
end-of-quarter. So I got to two small stories hear about this one is when I came to a blue I saw the code implemented PayPal. No, excuse me. Apple pay The cold wasn't very well structured and I thought I sure I can do something better six years later the Goldie's the same but it's because there was no need and we rarely touched the code. So it's also very complex and only 7% of the payments go through PayPal. So there is no real benefit. The second story is with recurring contribution to the product people came and say
hey when every time we receive the contribution did or not, you have a form giving them the option to make this every month from that point every month was repeat the same contribution the developers thought we did it anyway, and we were wrong people really like it. Sometime later in front of people came back and say now we want to do it weekly. We said we were wrong with this is not going to happen and we were wrong again. Today return contributors very
important. We actually process 30,000 contributions ever day on everyday only in recurring and it has the future has evolved. There have been changes to the way things to be. So every time we have to make a change we are actually paying some of the technical bit with they ask something we realized that we have thinks that we didn't do properly in the beginning. Now we could fix them. So that's a very good tip you hide things are there to a Fisher and in fact, what is way he was technically dead in beginning is not that clear because we didn't know we didn't really
know either but second point on a chase obsolescence What's the problem with obsolescence in software at Opelousas is not about my cold. It's about old versions of gems. For example of the framework that are not my pain. I'll Dae Dae time passes. The problem is that these are found all the problems. I found all the time and you will leave on it all if you don't patch you're not up-to-date the latest version, it sounds obvious, but I know people running race 3.2. So it's a real problem. And how do we address it in in a group? What do you
need to have a good test? Anyway, I'm sure everyone here at the street. But for a great is essential without the test where you won't be able to do anything. Don't keep the minor versions at the last line mentions 523. Demeanor is the second number. You should not go from 50252 n Go from 50251. One spawn online that is the rails to deitz, which are excellent. If you haven't read them definitely you should but for a great they are excellent and they they real guy on
upgrade have Specific Instructions to go from version 5025 one and a different set of instructions to go from 51252. Every time you are great. You should go to the to the guys and they will give you a different set of instructions. They are very good additionally should read the release notes. It has explained formation that might have liked you and I also want the test runs when you should fix the pretation warnings, the the warnings might not be on the standard output. You might have to go to the to
the Lord. And do not release a release the latest patch the Patrick's Day last number for Tinder in this library is 5.3. So we do not release 5.0 if that 5.3 exists. More lessons sometimes or Ruby would provide something in the new release. They provide a new feature that you were taken from a gym. If this happens you should drop the gym and start using the standard Ruby or else feature. You should try to Release changes with without having their butt before they
sometimes they upgrade the release will say they were getting this really exist in the current version. So you should send it right away or they say we are going to extract all this functionality in this gym and the gym can be used right now. So you should do as much as possible before the Opera celebrities and believe those changes to the community actually race and Ruby have half haven't been that difficult the gyms dependencies having a bigger headache The Only Exception was Ruby 187 when we are graded to
1919 introduced in coatings. That's how we do things duration for Lamar. There is a workshop on Thursday about a grading rails if you haven't done an update is recommend going there. It's about 2 hours. The next topic is we will experience increase. Growth and cold base after 14 years, you know what case we have today 35,000 commits 110 lines of source code 124 classes and North County Soccer classes in 25 top body wash. So as anyone here work on a 10 year old cold base.
Do you think it's a mess? So that would be we will have seen this week. There will be powered silicone base where nobody wants to touch because they fear that will break the system. So I like to say that in software you learn to Right before you learn to read and that's why people seem to prefer working on application from scratch at Greenfield. And when when you are working with Legacy, the Legacy has the connotation guard applications going to be replaced. Anyway, so you're doing as isolated small changes only. So to me coming to
a clue it when I came into a seven-year-old app. So it was the first time I would see an application that was old but it wasn't and in fact, I was supposed to understand everything. I was supposed to make changes anywhere and that was a challenge because it was a lot of gold and you're supposed to be reading and understanding everything. So the first thing to a good healthy cold base is good by Kate DiCamillo messages. I like this Source there are many online game. You are at least at the bottom. But if you
Google, please beans commit message. It will be the talk yet. There are seven rules that he he gives showing Kira justice. So there is one line the subject that explains what what exchange is an imperative mode then optionally you have a body separated by a new line that explains the how and the why And in this case, I also have Instagram integrating with the tracking system. So you haven't worked for many years in a gold base. You're not going to appreciate the value of this lesson.
Yeah. It happened to me when I wasn't in a girl after 2 years. I just started reading my own comments. And I I saw all these messages are not very good. I should do something better and I started doing this. So when you ride to get to make message think of your future you your future you will be reading this and we might be very clear to you now, but one year from now, he won't be there. more about healthy gold base We we all agree that a comment in the coals
is a lie waiting to happen but having no comments is as bad as having no, that always a spot is there is a complex algorithm. You should write something explaining how it is because reading the cold with take time to understand. What would what is happening? If there is something counterintuitive you are you should go to the Post earlier. The reader later. My time is important. Sorry. I like the day tomorrow documentation. I believe that if you understand the data model, well you can do that the logic
So I love ER diagrams a complete ER diagram is useless because it's too many things. But if I'm working on lips a payment processing and I have a ER diagram of that piece with pets a v Anthony's is there you when it came to a blue there were no comments. There was no documentation. I think the guys were way smarter than me and that's why I didn't need it but it is better you have it might be good for a team or for which we have them but today we have plenty and
and in fact, we have new developers coming all the time. So it really helps to have this. What happens when now the team is growing? The original team was just the two Founders. We never had a b c money to hire little people when I came in 2012. There were 22 people total in at Blue of them 7 where Technical and of them for where developers the for today. We have plenty 120 people at 34 in the technical team and old M 20 R Developers, but we all know that the number of communications will increase exponentially with the number of people
so when you have so many people there will be a lot more more things necessary to call me Nate you will have a lot to process their standards. You need a lot more to understand how people will be on the same page. So, how did you help with this? You hire the best Developers? They would be more expensive but they will be more productive the other approaches you can develop ourselves, and you. Keep putting them but we think it's not a good idea because there is
evidence that the best developers will be a factor in 10 foot X more productive than the regular developer. There might be a discussion with that factory. Some people saved in time some people complain snow tennis, but it's clear that it's at least twice. They are good ones so laggy for me when I came in 2012. These guys have not figured this out. That's how it got in. Feed them. Well, so they stick around for example, send them to price concrete paid expenses. What salary would compensate good benefits? It's not only about the money. I like to
make 3 words autonomy Mastery and purpose in you give those to the developers. They will be happy and they will stick around. We also do gold review. Everybody else has to be revealed at least by one person if the gold review has more complexity has to be to people but the bullet there is a discussion in the request. So some developers will say why do you do it before why you don't do it that way and in other cases one developer will realize I use that
all the cases that do, you know what we should close this and you should start from scratch. It's better to do in your approach. We don't do their program in the strict sense of the term, but we do work in pairs. Someone more familiar with the code with certain part of the code will work with someone who is less familiar. We do that all the time what else even a senior developer will have a mentor because it will be senior, but they will not be familiar and you develop it was not be familiar with the Cove. So it helps
I'm going to talk about complexity with an example from 2015. I already showed their contribution form from Save animals facing extinction and in the beginning after the form is displayed. There will be a baby step 1 you see they're asking for an optional tape. The donor can say continue. That's no tip or they can choose a different percentage or amount. The problem. We had at the time was if they use her abandoned their the browser them there at that point. The
contribution was not saved. So we had we were supposed to fix that. And don't try to lead the flowchart is not it's no they're too late. It's only there to show you how complex the flow is for the contribution form start Sunday topless intensity bottom, right if they're all the decisions that the contribution for has to make because when you make a donation, you might be giving away tickets for an event. So the confusion for a password for names or there might be a sticker or a t-shirt the Credit Karma Yoga TV say you don't
need to ask for that or you whether what forms of payment due to process you should you offer PayPal or not? Very complicated on top of all of that. It was it is today highly optimized for speed. The most important speed in indigo is how fast you show their contribution form is what most people see and it's very very important. So there is a lot of cool to do to do it that way. So what do we do? What we do we decided at that point that maybe we should
This one is from September of 2009 Obama was talking about Obama care. That's the Affordable Care Act in a joint session of Congress. The reforms the reforms I'm proposing would not apply to those who are here illegally. Not you. Douglas Republican Joe Wilson yelled you lie. Guess you didn't hear. Is it traffic in a glue in contributions per hour in September 9th Aries the first Spike there was no Twitter. So I think that's why wasn't very high because he was in C-SPAN and nobody watch assistant. So the second spy was bigger the same day later on the
news in the big one was actually the next day on September 10th, and there are Beats by Slater. Also, most of the traffic went to the opponent, of course. So that's the first one. The second one is Bernie Sanders in 2016 was racing for the primaries with a blue. He won the state of New Hampshire in February and he gave his victory speech. I'm going to America. Marvel look at the time 9:37. BM can you tell when that night at 9:37? So the top graph is contributions per minute. In a minute we were doing like
2250 and we went to 2630 minute that is about 40 per second. If payment is a credit card being processed and there are multiple calls to external service roads are very expensive action. The second one is requests per minute. We were doing 25000 and we went up to 300 350,000. Is one-week-old feeling the burn. So, this is Lifetime. I'm showing contributions per month. Spikes are all in October of the election year. Except in 2016. They hire point was in March instead of October that
was Sanders last election in October. We processed for 4 and 1/2 million contributions in one month. Not those are per month. I thought for a minute. I'm not going for a minute until every month. Did they highest per per minute washing the TV if I show you? So every time there is there is traffic or there is a big unexpected Spike like in the two videos before those are very valuable lessons. You have to look at all those things and analyze everything every little detail. There will be
processed it that responded well to loathe and others that will not have responded very well in you know, where k for example, we we didn't lose contributions, but we had a backlog we were not processing that processing them as soon as quickly as they would come so we don't you try to fix that. I know that it's time for war from the time. We we had we have a credit card volt that keep stoking and we have Indigo and they have to talk to each other and we realized that the
connections we're being closed on every request. We fix that. Do you remember dates from 2008? So what how do we doing? The main the main thing is architecture. We scale. How do we increase capacity wintrust capacity putting more servers we increase the number of hosts or we proved we put more powerful. We would more Force more powerful computers. So if we have a process that is centralized that something that is coordinating other things and you have to go through that cannot scale ghost. You
cannot duplicate it sold it goes has to be able to run parallel in concurrently. The problem is to write code that weighs more difficult and that's why we avoid it but he's the only way we also do load balancing with you. I think your nose job and I think from his job is when a request for a slow action, like sending an email is received by the web server in the web server would not do reaction because it's too slow. They were several Winstead create a we put the request in a Q and that will be served by a different house. Later.
By the way, this is how we lose how we avoid losing contributions. We do this we use a cashing a lot. We actually you spell cervical fast is a Content delivery Network and it smelled only catching. It's also distributed caching which means that if someone makes a request in La there will be a vastly server close to La responding anybody. If we do the request in Boston, it will be a server from Fast leave close to that location basilikos them pops their points of present. I gave a presentation representation two years
ago that specifically about this because he's a big Topic in case you want to take a look. It's about bike and performance. So lasting if you pick the right framework you pick the right language is going to be much easier to deal with long-running software. So we're very happy. Where rails has been very good to us. We're very grateful to the people behind the tutu project, and that's all I have. I hope you learned something today, and don't forget to read
when there is a request, what's the endpoint? It's the monolith. So the contributions are not created in form of the contribution is created in the database and processed in indigo. The question is what mechanisms mechanisms we have in place to Define how to stay on the current version of rails. Upgrade of bread as fast as possible. I think we're looking all the way sad what things has got going to be with me rails polishes how long it's going to be maintained. So is there is a good margin to make sure that we're not going to be affected by that. That's that's
enough for us. On the seasonality. For example, you are closed today election. We cannot do it. So things like that will play the question is how do you scale? How how do we spell during their Spike when we have if we if we know that it high traffic is coming for samples. We know for the debates. It would be weakened Woodmore servers beforehand. So we treat them ever host some tweaking mean you have to sidekick will update its in the middle of will this synchronous job will be
in the middle. You have to Define define those parameters properly and ready for the database Etc. So you have some you have a notification, you know, you can do it, but if it happens suddenly like we did you lie a moment. There is no way you do it because it was 10 minutes to 2 to put a new server on online. So you don't you don't have it. What what happens in those cases is you will have a backlog and if that is affecting things too much you might decide to stop some
Services you say. Hey, I'm not going to let say I'm not going to take more requests for reports. I'm not going to take more requests for logins. Nobody can log in things like that. You you have to depend on the case. We have to second one second one time. Thank you so much.
Buy this talk
Access to all the recordings of the event
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.