Events Add an event Speakers Talks Collections
 
GitOps Summit 2021
June 22, 2021, Online, USA
GitOps Summit 2021
Request Q&A
GitOps Summit 2021
From the conference
GitOps Summit 2021
Request Q&A
Video
ReleaseOps: GitOps for the People - Lian Li, Container Solutions
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Add to favorites
75
I like 0
I dislike 0
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
  • Description
  • Transcript
  • Discussion

About the talk

ReleaseOps: GitOps for the People - Lian Li, Container Solutions

MoneyBank Inc. is a fintech enterprise that recently made the jump to K8s and GitOps to cope with the shift of demands from cranking out features towards stability and scalability. Yet, even with a fully automated CICD and shiny new microservices, features still take weeks to be released. As teams keep waiting on each other, frustrations, resentment, and mistrust grow.

MoneyBank’s situation is typical for organizations with enterprise processes and startup mindsets. When faced with problems, the urge is often to move fast and automate them away. However, the cultural and regulatory structures to support these changes are not in the scope of said automation. One more piece is missing to address the needs of non-technical stakeholders within the ever-changing CICD landscape.

In this talk, we will attempt to automate the non-automatable with ReleaseOps: GitOps for the people.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

About speaker

Lian Li
Engineering Manager & Cloud Native Engineer at Container Solutions BV
Share

Hey everyone. Thanks for coming, but it came from anywhere. But anyhow, thanks for tuning in to my talk. Today, I will discuss a case study with you which is based on one or two. I won't name any names, but I think some of the things are we talkin about will sound familiar to a lot of you and for the rest. I hope you have a vivid imagination. First things first. Who even am I? My name is Mandy and I'm engineering manager at Container Store. We are a consultancy focused on programmable infrastructure,

South Nature's, Transformations and song. I've been in Texas for a decade about the kids first as back, and then from there and then. I also dabbled in Daedra data engineering data science profile, and I moved to Amsterdam about three years ago to join C&S. So what kind of signs am I speaking about Evan engineer? I had experiences with small startups with just a handful of people to really pick up or since I started working with us and highly regulated

sectors. Like the one that I will introduce and this company that I'm going to be speaking about doesn't exist. It is a stand and an amalgamation of couple of a couple of similar experiences that I've made at someone and they just asked for the problems I've encountered and how we solve them. So, let's Jump Right In this time. I called it is called money bank and I guess can kind of already, maybe guess what kind of sector this organization operates and so money bank

is a huge Enterprise with tech department of around 200 Engineers that include software engineering operations and off-site E. Also, as is typical with Enterprises, there are hierarchies. And to some extent their roles and responsibilities are not always clear to everyone. There's a c-level, there's multiple layers of middle management just as you would probably know it. And finally, as you might have guessed Money Bank is operating in a highly regulated sector infant. It definitely has some.

Some companies that you can imagine. Within the engineering department. So now I'm excluding office at you from that. There are proteins, which develop individual products and features examples of software used by or online banking software used science. There are 17 which provide internal services for the team's most notably this with operations as they provide the service and Patch Farms to run the Run. And finally, we have specialized teeth. They asked us to see the teachers and internal

Consultants, but they do not directly contribute to the product type. So the security team, which provides the best practices and guidelines for the developers are an example of certainty. There's one team that's kind of kind of doesn't entirely said which sqa or in between a service and a specialist team because on one hand they provide us with quality kind of 28 as a service, but they also the best practices which they then share with all the apps. So when our team first met with money

Bank, their biggest challenge was it that every was slow and they identified this you already, which was that for a lot of Emmanuel steps involved. But already mentioned to edit their somewhat of a service team somewhat not. So they provided as I sent you a as a service and the consequences of this was what day was doing was a complete Black Box to anyone outside. No one really knew Powell. They were testing how well they were proven changes and they had their own way of doing things. They have their own independent and then over to and from to

a were often the Warriors. I have to be done manually. Spelling of test environment, spoke to a also has to be done manually. DC individuals groups, had to be modified, and then triggered manually and environment belongs to which you are. So no, develop was allowed to do anything on test but also station requires a manual work to get ready because for example, some data and you to be loaded into metal, Santa Ana menu, And finally, cuz of compliance the final check before

things go, too, but I had to be done manually by all the major stakeholder in a cyclist in a big meeting. So we spoke to the developers with the managers. We spoke to the operations team for about a week after we found, we kind of understood the problems at this level. We went in and started to throw on some suggestions. So fuck shit. We figured that we needed to get them to automate all the test. There was no way around it and you to standardize, how to approve and how to, you know, how to do the hanovers. Because there was no way around

that using cognitive load and works out in the end of the way. To quickly spinner test environment. We suggested to move to container orchestration, but they chose to host their own in-house. So this way, they could guarantee that all data and traffic was kept on premise in accordance with European data protection laws. And finally to he's auditing and to give everyone a security blanket. We suggested to go with get UPS as the CI CD flavor. Came to an agreement.

They really liked our suggestions and hired us to help move the services to Open start with new get us that form that we were supposed to help built. Understand the changes that need to be made. Let's consider the starting position. So this is the situation that we encounter when we first spoke to Money Bank. A release or change, would always start with a product on a requesting. A new feature. They will speak directly to the death or open a ticket and a ticketing system, the deaths would pick up that breakfast implemented and do some local rudimentary testing

sounded. NYX, the dust would emerge at Aerotek or changes into One released branch, and this would be the release package. The release manager represented. By the third line. Here is a specific role that is sometimes taken over by the death of team leads, or sometimes Phillip Edwin. You depended on the structure of the artifacts and then deploy them to the development environment where the Destin can do some integration tests between the services. And finally, when everyone was happy, the release manner would approve changes in form to a

lot of changes to their testing by me again in their own way, run their tests and then approve the changes. But you ain't standing from the release managers to the release manager. Now can trigger the deployment of staging. And then inform the PO to do their own tests and give final approval. Finally, the entire change to Lynn all references to commit hashes artist. And so I sent you the operations team with, then deploy other changes to production. Sometimes things go wrong here because despite all the rigorous testing

production, staging and tests fundamentally different from each other in the way that they were maintained and provisioned. Mainly, because of convenience. Let's take a look at this Middle Lane first. These are the things that the release manager does. The stuff that's blue. Sheer is already in itself, automated but required some trigger a mango tree during the test and approval. Step is someone automated for some service is not father, and the deployment of staging is kind of its of automated, but because the environment,

the stage environment is a shared One, release manager can use an automated process to apply, but they still need to do some leg work after some consideration here in there and make sure that all services are still talking to each other. So the first thing we try to do is to automate the entire middle. And so the role of the release manager could be taken over completely by the system. Would start again with appeal. Requesting a feature with the implementing and testing but now any Mercer the release Branch would trigger a dubstep. At, at this

with and start the building of a Docker container. Once the deployment of the development environment was automatically trigger through get-ups. So we would create a pull request for a Canton Repository, which would automatically be merged and then we would run our alternator test results has passed to any would-be informed. The new release of tested over, go with the flow of the point estimate and then to give their approval. All they have to do was to approve. A pull request that had also been automatically created to deploy the changes to staging.

And that's what that happen automatically again through get abs. Once deployed, the PO would receive again, automatically a notification that the changes are not ready to be tested on staging and the rest of the process stays the same. What's great about this is that we started to abstract infrastructure in such a way that automated Apartments through the ghettos at from, We're technically possible for all inbox. What's the weather in this process? The deployments were some of the departments were done manually. Let's take a look at the architecture.

Everything starts with the application. Republic coming from there. We have a plan that two things. What does more than two things? The artifact artifact and push them into an artifact repository in the Nexus that lived outside of her costumes. After all the artifacts were pleased, we done with submittal change. The consequent pository started with death. So we had one Rapunzel story for each environment, for each namespace, a business unit. This repository would trigger an Argo City, instance to

deploy the resources on to the Custer and we have this for each customer. We had one. Or was it in Sims 4? And this August to deal with cheap monitor in the repository and Supply automatic. See if any changes for the text on the main branch. And technically this would work for all environments up until production. The actual setup was a bit more complex in this but it should. Hopefully give you an idea how the flow kind of look like. No, the thing is when you look again at this delivery flow, you can see the deployment prod is still being done manually by the obscene. The reason

why it wasn't meant to be here was because they asked him where the owners of the most sacred place the product back. So they were personally responsible for what happens to their life hurt people. Stay while you have to sign it with your blood three times before you let anything go to clutch. To address that we come to the same thing as a staging. We automatically created a request with the changes between Providence agent and the ox team simply need to approve that toe request. And when did it end March that? It would

automatically trigger a deployment to Argo City in the same way as we just stating. This took a lot of time and responsibilities of the plane and interesting me enough, we run an experiment and we tested me to text lucero's and trot instead of more which had always been management. Biggest concern. So next we took a look at the Q, a witch on their own infrastructure and has their own deployment development culture structure and way of doing things. That was something that could cause a lot of friction between the product teams and Q. A I already talked about this a little bit to

a didn't really belong in the normal spot shop the engineering team and they will always somewhere kind of in-between, but still, they were very crucial to delivery process. Answer a lot of discussions and Designs. We figured that the ideal solution would actually be a truly radical change. So the first have to get by in from the customers and they said were Radical changes fine. As long as it pays for itself in the end. So we suggested that we want to eliminate you as completely as a team and also has its own land in this process.

So how does work obviously I'm not suggesting to or we were not suggesting to eliminate QA quality assurance that I lie. But instead we suggested embedded into the development development teams with Patchouli cross functional. The team member responsible for acute stroke, you a would write the test with already during implementation and still keep in touch with the form of to a true community of practice, but not all the way that they work was completely transparent to everyone. And it was standardized across the entire organization. In this process, everything stays stays the

same up until when the debts are done with their preliminary tests on the development. The test environment now belongs to the death teams, which meant they were finally able to do away with all the extra stuff for QA, and couldn't stand standardized, their deployment slow because the defense of all the environment The tests were run and approve automatically and fuss because the death teams on everything here. Now, they were soaked role of the process. And once the automated test passed, it will automatically trigger the deployment of staging and the

rest of us law stays the same. At this point, we were with money banks for about a year. Open speak to take out and developers were much more productive. Now that they could focus on development and emerging strategies and confidence in the system group. You use it often enough and doesn't say this afternoon, you believe new past 11. But there was one thing still bugging the engineers and they had United and picked someone you to blame the slow Department of fraud. So everything was for fast, but in front of

broad changes with Tyler. Why I think so, it's true. That manual steps are some of these stats are still menu in this process, which will never be automated though because they are by Nature, a manual tasks. At least for now at least before the robot Uprising when they ask. But if you look at these other steps, I think there's still something we can do about it. One of the main reasons why the PO a table, still need to do manual approvals is because they need to stay with a board and high octave

of stakeholders about intention. Do we need to communicate a downtime? What are the risks? You could be a do we need to prepare for maybe in time breakdown service? On the surface. This also seems like yesterday in the nature Manuel. Could we still felt that there was something missing. There was a lot of cognitive load over the South was not necessary here. And then one of us for interview, one of our stakeholders, who was a critical person pointing out a prayer request is just not enough business. People cannot

read that. And I wanted to make a joke and Brookside kind of cheekily it, but they can reach our tickets, right? So, check out. Then we looked at each other, like, thinking, what I'm thinking. Apparent was born. So, I'm pretty sure that they were other smarter people that came up with the wedding before me, but I came up with a catchy name to Great idea of release of us was born. It's not a new idea. What a release means for the business with water released mean for the platform.

I've moved some of the stimulants around to better illustrate. What happened to take a look at the second Slimline here. This is our newly introduced shiny release of Salvation layer and all it does is represent the state of a driver issue. We would start again with the pr requesting a feature. But this time they don't message to Jess about it. They instead create a driver issue. Outlining the feature request acceptance criteria and the release get an identifiable ID, CID. The best Ultimate, informed and implement the request, they had comments. When they

make design decisions, explained the reasoning. And when they're done, they move the issue to the next stage of switches in deployment that automatic emergency speech a branch into the release Branch. So we have to only allow development on the branches that have been created for the issue and those issues related to higher-level releases. Some of you have possibly been wanting a bit of a Snicker habitable, bitbucket and Chara. And I get why, but honestly, Allison, deliver solid product and the integrated very well. We'll go from there. We can now

go to the below swim Lane. As soon as we marched, the release Branch to Maine, which triggers the entire process as before, until we get to the point where the changes are deployed to Staton. So instead of directly apply now, that's the issue stages to in review and then a science issue to the PO. This only work with us, keeping a mapping of the service, and the respective PO on the clusters of conflict map and a pipeline could grab this information and send it to driver. The PO knows as soon as I got the notification that they can notice the

changes that were listed in the starter issue on staging and give their approval in the ticket. Now, all issues that are in review, automatically, trigger the Ops Team. They are the people who owned the approve column. So they are the only ones who can truly move a ticket to the approved called, and they can only do it if the PO has already approved it. Finally, when when a ticket is approved. This would trigger the deployments to project through the magic of collapse. So this is what the final design looks like. Everything

starts with a driver issue, which is tied to the application, repositories for the cherubim wants. The bronze is merged into the main branch a pipeline disfigured to build and store artifacts into the artificial pository. And at this time, we had no split out a cluster, specifically, for managing, managing the Sky City manager, and managers the other Argos, but I don't want to go to the street from the topic at hand. We replaced Jenkins as a pipeline Engine with tekton pipeline,

which at this point has been integrated into open shift as openshift Pipelines. Same as with a Jenkins happen before, or after pushing the artifacts Nexus. We would then publish of changes to the country, repositories to trigger a deployment just us before we start with F. Unfortunately, before we had the chance to fully Implement a vision Money Bank for the budget for this was at the beginning of the pandemic. So organizations try to keep that money tied and R&B, you know, after the

first thing to be able to build a integration of Dora and the bucket before I left as a POC. And from what I hear, they kept with the general idea, but made a couple of text with changes. We were with money banks for a total of 2 years and transformed them from an organization that when using Docker swarm and applying maybe once every 7 to 14 days to open chest and applying daily. I want to make one more point with the street from Kelsey Hightower. He says I am convinced the majority of people managing infrastructure. Just once

a s on it. The only requirement that has to be built by them. I think there's a lot of Truth in that. And since I left money, bang, I've seen the rise of the internal developer platform. And I think that is a great. I think it makes a little sense. You should always do, you know, you'd rather than do something yourself. You something that's already there by people who specify who specifically your deal involving the thing. But if you need something custom for yourself, and the best way to do it was to build a platform yourself. And I think what I talked about today, released,

as a parent is, a great condition is kind of a Natural Evolution to also invite non-technical stakeholders into the process to make collaboration smoother and easier. And that's all the more rewarding. This is the obligatory slide that we are hiring. If you live or want to move to any of these amazing places, then you can do that. But also, if you live anywhere on Earth, we also now a foot with remote. If you're interested, you can talk to me or check out our website. So, thank you for listening. And again, if you want to reach

out to me to learn anything more about the company, she has embraced for you to get in touch. Hi, can you all hear me? Or just let me know if she can't. Thanks Tracy. So this is interesting cuz I don't have anyone else to talk to son. Just got to ramble to myself a little bit. But thanks everyone. I'm glad you liked it. I was the first time I've actually seen my talk kind of played back to me in full and started of chips and things that I'm probably going to cut out for the next time. So I will, I saw a couple questions in the

chat is all so cute. And I don't know if there's like an order to, which I should get two. But the first question I saw was by Paul. Do I understand if the security of production has degenerated to the security of jaira, and I asked Paul to elaborate a bit. Is it was not sure if I fully understood what he meant that as engineer as we built automation, which is the cure, but if Jared and secure than Productions. Yes. Agreed. So the one thing, the the srur up steam, they have to approve the changes.

Jireh is jira. So there's that but then of course if there's a malicious actor who can How do you say I faked being a nap? So I sorry person. Then. Yes, that would be an issue and a super short habit that you was there. So we were not responsible for setting up a gyro gyro the ticketing system. That was part part of a different apartment. It was self-hosted though. So I guess there was a pretty small, a text surface, but generally I would agree to that at that statement. So I hope it answered your question and then I'm going to go to the

Q&A. So that was a question by Joe trip to my mind. The quality of the inbound feature request is crucial to this process. How did the product an engineering course? Get a line on what was needed to be effective. Okay, cool. So, that if you are asking a question just from like, the product management at what point of view? I don't know if you haven't heard about the safe framework. So this was kind of the, I guess people try to marry waterfall with HR practices of a scrum. So then out came, do you say frame-work where you basically

said you have 10 weeks. I plan is, and that in the p.i., You decide on a high-level objectives, and then those are brought into the teams, and then they do regular scrums, and release, plantings from planning, and that screw this type Landings. Where they were like 300 people or something, they weren't there, but it was a huge kind of Greenfield project where before we had this universe of us were mainly. And this new thing was completely new. So we built this new platform

with open chef. And on top of that was the, the get upset. If I found that we have helped build. So yeah, that's how they worked. Was it affected effective. So the next question in the Q&A is by sudeep. I hope I'm pronouncing that correctly. It would be good to see some demo of house that you are going. You're doing the release of the Nexen. Yeah, that would be good. Wasn't it? Unfortunately, since Money Bank is a huge corporation and the old one, actually, from the nothing we were doing could be Open

Source, Gas, lot of times. So I don't have any code that I can show you right now, but I am planning to work on like a little, a demo for the for this may be using some open sores. A ticketing system. Also may be using get a guitar. So yeah, unfortunately. No demo at the moment, but I'm working on it. Okay, so I'm just going through the Q&A right now because I'm your cousins coming in. If you have questions, put them in there easier to follow up in the shed. Discussions by Alicia in regards to automating. Pee ours. Is it

compatible with good flow? I guess you can make it compatible, but I would not recommend. So I think there was we had this little conversation Brian and myself about Release branches and I guess if I understand it correctly, it's mainly around reading those release branches and you can do that actually had some teams that kind of insisted on doing it slow. And it is possible because you're repulsed or your deployment or does not necessarily lift need to listen to actually, at this would be the integration part. So if you have a pipeline, you could

just listen to that release branch and then use that as a trigger to deploy or changes, but I think the point of this is to make things easy and visible and Rita. I don't think that gets low helps with that. It's kind of the The attitude of emergent when you mean it. So you have to be. Absolutely sure. You that you cannot have the safety blanket at other than you do your test. Do you have to tell trust the system? And you don't need release process because you're released is kind of met all the information about this

somewhere else. So that was kind of the idea, but you can make it happen if that's what needs to happen and you can't do that. What is the next question is from Paul? What ruling frame with that? You adopt for your automation? What do you use the same set again? So I'm not sure which part of the automation, you exactly mean that we had with buckets as the source repositories and the configuration and then we used basically just HDPE caused mainly, and if we ever

needed the real pipeline in any way we use text on pipelines. So the plants which is now fully integrated an open shift so that openshift pipelines. Other than that, just, you know, if we could use some kind of hooked to to do an issue, if you request it with that. And I think I have to end the session in the next minute. So if you have anymore questions, feel free to reach out to me on Twitter at 7:42 or unlinked and Yandy, and I'm also going to hang out in the network section. Probably, so thanks for coming. And yeah,

I hope that, you know, you learn something or at least could laugh about some stuff, right? So have a great rest of the conference.

Cackle comments for the website

Buy this talk

Access to the talk “ReleaseOps: GitOps for the People - Lian Li, Container Solutions”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Ticket

Get access to all videos “GitOps Summit 2021”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Ticket

Interested in topic “IT & Technology”?

You might be interested in videos from this event

November 9 - 17, 2020
Online
50
35
future of ux, behavioral science, design engineering, design systems, design thinking process, new product, partnership, product design, the global experience summit 2020, ux research

Similar talks

Dan Lorenc
Software Engineer at Google
+ 3 speakers
Dan Garfield
Co-founder and Chief Open Source Officer at Codefresh
+ 3 speakers
Tracy Ragan
CEO at DeployHub
+ 3 speakers
Cornelia Davis
Senior Director of Technology at Pivotal
+ 3 speakers
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Henrik Blixt
Principal Product Manager at Intuit
+ 1 speaker
Jesse Suen
Co-Founder & CTO at Intuit
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Chanwit Kaewkasi
Developer Experience at Weaveworks
+ 1 speaker
David Aronchick
Partner, Program Manager at Azure Innovations in the Office of the CTO at Microsoft
+ 1 speaker
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Buy this video

Video
Access to the talk “ReleaseOps: GitOps for the People - Lian Li, Container Solutions”
Available
In cart
Free
Free
Free
Free
Free
Free
Free
Free
Free
Free

Conference Cast

With ConferenceCast.tv, you get access to our library of the world's best conference talks.

Conference Cast
915 conferences
36609 speakers
14033 hours of content