Seth Vargo is a Developer Advocate at Google. Previously he worked at HashiCorp, Chef Software, CustomInk, and a few Pittsburgh-based startups. He is the author of Learning Chef and is passionate about reducing inequality in technology. When he is not writing, working on open source, teaching, or speaking at conferences, Seth enjoys spending time with his friends and advising non-profits.View the profile
About the talk
In the past, you’ve heard Seth share stories of Secrets Management in serverless. That has all been leading up to this moment - Google has a Secret Manager! In this zero-slides talk, Seth demonstrates the new best practices for managing secrets in serverless environments with Google Secret Manager.
Speaker: Seth Vargo
Google Cloud Next ’20: OnAir → https://goo.gle/next2020
Subscribe to the GCP Channel → https://goo.gle/GCP
product: Secret Manager; fullname: Seth Vargo;
event: Google Cloud Next 2020; re_ty: Publish;
Hi everyone and welcome to secrets and serverless 2.0. I'm Seth and I meant developer Advocate at Google cloud in the past, I spoke about strategies for managing secrets and service environments but a lot has changed since then. So let's Dive Right In if you seen the earlier versions of this session, you're already familiar with this application architecture. But if you haven't let's run through it very quickly. Suppose we have a web application to track the number of visitors via a hit counter like a 1990 style website at its finest. We only want to pay while our site
is being accessed. So we're going to leverage Google Cloud run and his ability to scale 204 this application. But because our application can scale up and down. We need to persist our visitor counter in some external persistent datastore like Reddit, but our application needs to authenticate and communicate with redis. So, that begs the question, how do we securely provide the retta's password to our serverless application? That's the journey that will be going on together today. What would it? Three unique an independent approaches to managing
secrets in serverless environments. The first is buying a plain text environment variables that encrypted environment variables. And then finally with a centralized Secrets management solution, Let's jump right into the first plane text environment, variables, plaintext environment variables are by far the simplest architecture. We create a redis password and we just pay it directly into our server was application by an environment variable. Let's take a look at a demo of how this works. Alright, let's take a look at our first example. We're going to deploy an application to cabron that
puts secrets and environment variables in plain text. Let's take a look at what this looks like. What we're going to do is we're going to look at our deployment script which I wrote to just clean things up a little bit. We're collecting the IP address of a ride assistance and we're getting the rent as password again in plain text. Then we're going to build our application as a container. Push it to a Docker registry and to play it. And you'll also notice that were using the rest password Here, VI an environment variable to solve run, Let's go ahead and quit and let's deploy our
application. So this is going to do is this is going to build a container. Push a container to our Google container registry and then it's going to ask Cloud run to deploy that container with those environment variable or application will read those environment variables at start time and deployment. So we are pushing to container registry now And then, we're waiting for our Cloud run to pull that image and Route traffic to it at. Once is done, we will get a URL or we can visit or service.
Great going to go ahead and open up this URL in a browser. Great. Here it is. And as you can see, this is a very basic website that has a head counter. That's backed by rest. Every time I visit the site, I increase the hit counter, I can reset the counter. But what happens if someone breaks the site. So you'll notice the reset counter method actually included the couch that I can that too. And I can set this to a negative number and the site continues to function. What happens if I said it to something like a fruit?
Well then, my whole application crashes, I were presented as a very nice helpful. Debugging paid is very similar to what you would see in like rails or Django or any popular web framework. We get a bunch of information about the request but we also get the environment and you'll notice in this environment are right as host and a red, his password are available in. Plain text. Anyone with access to this application, you can visit this URL right now in your browser. You would have access to this run as host and this Reddit password. Hopefully, this makes it clear why plaintext
environment variables aren't a great idea. Let's go back over to the slides. Hopefully, that made it really clear that you should not store secrets in plain text environment variables. Not only do you open yourself up two classifications of vulnerabilities, it's really not a best practice. That really only leaves the other two options encrypted environment variables or a centralized Secrets management solution. Let's take a look and encrypted environment variables. So instead of taking our plane text secret and putting it in an environment variable weekend instead and
crip that secret first, what that looks like is a human or a machine takes the Playtex secret and they give it to a key management system like Google call Camas Google. Call Camas in Crypts that plain Text data and produces what we call ciphertext for the encrypted Secret. We then provide this encrypted secret as an environment variable to our application. This way, the environment variable has an encrypted secret, not a plain text Secret. When are applications boots, it communicates with Google Cloud cam has to decrypt the value and then just a plain
text but that plain text only exist in memory, it doesn't exist in the environment given to the serverless application. Let's go ahead and take a look at a demo. Great or second example is going to leverage Google Cloud cam s to encrypt the secret value, before we put it in an environment variable. Then when our application boots it will decrease the value using clock a mess and Only Store The decrypted Secret in plain text in memory. So if someone gains access the application or the environment of the application, they would only see this ciphertext. The encrypted secret. Let's take a
look at what that looks like. The first thing we have to do is we have to update our application? I've already written some help or code that makes this easy. So we're going to leverage this. Kam SD script method that accept the ciphertext or the encrypted secret via the reddest pass environment variable. And then we actually have to encrypt our secret. Let's go ahead and do that. We're going to run in crib string and this is going to encrypt the string that we give it which in this example is the right as password. With a Google Cloud
kanaskie. I'm going to copy this to my clipboard. And then we also have to update our deployment script because instead of passing in the plaintext secret, we now need to pass in this Cipher text or this encrypted secret notice that were still passing. It is an environment variable but we're passing in is the ciphertext. Now, let's go ahead and actually deploy the application. I got this is going to build and push our container, and then we're going to wait for cloud, run to deploy this new revision. And then, after the
revision is deployed, the application should behave. Exactly the same because we only decrypt, the secret, at start time and we only eat the additional cost if you cramping that secret one time, let's go ahead and jump back over. See you notice that the application is still airing because we haven't fixed the bad value in Reddit for the reddest password is now that ciphertext and because we will talk to him as his guard. I am identity and access management. You cannot connect to this red as host using the threat of password because you cannot decrypt it unless you have
access to the call kanaskie. No, this is great but there are some disadvantages to this approach which will talk about in a second. Great Western Bank over to the slides. So as you can see, encrypted environment variables do protect against the immediate threat and there were great starting point. But there are some drawbacks the biggest of which is once you deploy your service application with an encrypted secret, you lose centralized auditing and visibility into how my secrets being used. If that's a requirement for your organization or security practices, we need to
take a look at the third option, which is a centralized Secrets management solution. In this model, the application request, its Secrets directly from a secret management system. So as a secret admirer or a developer, I put my secret into the centralized storage system. And as an application, I request that secret directly from the storage system. Got centralized Storage storage system is responsible for encryption decryption, logging and monitoring all in one place. On Google Chrome. Our first party Secrets management solution is
secret manager. We aim to be that centralized Secret store to provide you, that value of auditing and logging and visibility Google secret manager is a new fully-managed gcp products. That provides a secure and convenient method for storing API, key, certificates, passwords and other sensitive data that you might need in your applications and services. Secrets rarely differ across Cobb regions. For example, your twilio API key is unlikely to change. If you move your app from us East one to Europe last two, for this reason, secret names in Google secret manager, our project
global resources, The resource you are right in this example, refers to the secret name to my secret in a project named my project even though that secret name is global the underlying secret, the actual API key is still store Regional. For example, in this line, you can see that it stored in u.s. East one, u.s. West one and you're up, north one. But how does the secret data end up in those regions? Handles at all we do this with replication policies and we have 2 replication policies for Google secret manager.
For customers who don't really have a preference where their secret payloads are stored. You should choose an automatic replication policy with this policy. There's no configuration and Google besides the best place to put your secret, but if you're a customer that has a a requirement that your data exists, only in certain regions, you should choose a user manage replication policy and pick the specific region or regions in which, you want the secret to be stored guardless of, which policy you choose though. Google Cloud handles the replication of that. Secret across all of the selected.
Regents you don't have to run a Con Funk Shun or kick off a nightly Cron job. Google is the first Public Library by her to offer offer a fully managed secret replication across regions. Now I have to admit, I've been lying up until this point, I keep talking about Secrets but I'm actually talking about secret version that's because all secrets in secret manager or version. If I default the secret is actually just a very thin wrapper around a logical collection of virgins. The secret version is what actually stores the payload data and State. In
your applications, you can pin to a specific version or you can use the latest Alias to pick up the most recent. Highest number version. Note that latest is a floating Alias. So if a newer version is created latest automatically points to that new version. SheiKra manager uses an API first design. So that means that anything you can do by clicking buttons in the cloud console. Or buy running commands in your terminal by a g. Cloud is also possible in your favorite programming language. In fact, we have samples in seven different
languages that allow you to call and execute secret manager directly from your service applications. You can do so in Ruby or python Java. Net. Or Peach Vigo and no Js. Before moving on, you might be a little bit confused about secret manager, and key manager, we talked earlier, and then the second example, we looked at leverage, Google Cloud, cam apps. What is the difference between Google Cloud kmsn? Google secret picture. Google call K. M S is a cryptographic Oracle system.
Nobody including yourself or Google can extract the keys in that system. Google call Kenneth provide an API that does high level operations like encrypt, decrypt sign and verify using those keys. Secret manager. On the other hand, is a data storage system that saves and allows you to access secret material. So well, they both involve an encryption and decryption. Secret manager is Morty storage solution. Where is cam cam? S is more of a crypto Oracle. Let's take a look at the different user Journeys. Tell do, you understand the difference
between these two products with Google, can us in order to persist a secret? We have to create an encryption key grip, the secret with that key and then store the resulting ciphertext somewhere a filesystem a database or in memory. As we saw in the second example we have to store it in an environment variable. Google call. K Masters not for the secret. That's your responsibility. It also doesn't keep track of how the secret is used after. It's been encrypted because Google Quad Campus is concerned about the encryption keys. Not the actual data is encrypted or decrypted. With
secret manager, when you persist a secret, what secret manager, you give it a name like for the API key and you give it the value that you want the store. Secret manager, handles, the encryption and decryption automatically as well as the replication across regions for availability. What do you want that? Secret back in with Google card, you have to provide Google, Camus, the ciphertext. And then Google, Kamas will decrypt, that ciphertext and return the plaintext value. Assuming you have the proper authorization to do so. To retrieve a secret, what secret manager? You just access the
secret directly. By name example to leave API. Key is your authorized to access. The secret secret manager will return the secret and it's mine. Let's go ahead and take a look at using Google secret manager in action. Our third approach is to use, Google Cloud. Secret manager is Google secret manager actually stores, the secret material, and you only need a reference to retrieve it. Let's take a look at what this looks like, again, just like before we need to update our application. So instead of pulling the secret from Kenneth, where is Dad going to pull the secret from
Secret manager, using this access secret function, and like I said before this coat is all open source on GitHub and I'll be at Lincoln the size and in the show notes, so you can always take a look later. Let's go ahead and save and now we need to actually create the secret. I've written a little script called create secret that just Autumn Asus for us, you'll notice that uses a G-Code commands to create the secret. The secret is named my secret and it has a value of super-secret and then we're granting. I am permissions on the secret to our Cobra and service account because Secrets or
deny by default. So we have to Grant access Let's go ahead and run this command to go ahead and create a secret. And it looks like it failed because I already created it. Let me fix that real fast. This is why you never do live demos. Something's always bound to break. So I'm just delete it. Another tab over here. Let's try that again. Will go ahead and create a secret. And great. It's exceeded. You're the nearest, it is creating version one that's because secret manager secrets are automatically version 2. But we're
just using the latest here as one example, And then lastly, we need to update our deployment script. So instead of having this big long password, from Claude Canna, we had such as passing the reference to our secret, which is my secret We'll save this and deploy the application. This will need to rebuild the container because we change the code and I'll push it to the container registry and then it will deploy at the Cobra and exact. Same URL, just creating a new revision. So we're just waiting for traffic to route at this point.
Great. And it's done. And if we jump back over to our browser and refresh. The page, you'll notice that are read his password is still my secret. It sits or sorry. If you noticed that, it's now the reference to my secret instead of the plaintext password, and secret managers guarded by identity and access management store. Application is retrieving this value, at boot time, storing the plaintext only in memory. And if even this Depot page were to be presented, an attacker, would not be able to get this plane text secret because they don't have access to secret manager. And we can see that, you
know, the application is totally functional great, go back over to the slides. Today's Journey looked at three, distinct patterns, play the environment variables which no one is going to use encrypted environment variable, and a centralized secret storage solution with Google secret manager. In the past. Centralized Secrets management have required up front and continued operational investment with Google secret manager U of a fully managed Secrets management solution right on Google Cloud. You're also expanding Integrations with other Google, products in the
future. You won't even have to do this environment. Variable danced, with Cobra norqain functions are other Google, products because we'll have first-party Integrations to directly retrieve your secret at runtime, Now, if you come the earlier versions of the stalk, you might be familiar with fiberglass. If you're not familiar where glasses and open source tool for doing Secrets Management on Google, it predates, secret manager and largely influenced. The design of the first party product. We have to that wear glasses, a fully interoperable solution with Google secret manager. We
actually decided that way from the onset. If you're already using burglass for your server list egrets, migrate a secret manager is super easy. There's a new building command in a bird box. Burglass migraine that takes the name of the storage bucket, where your secrets are stored and automatically migrates those from Google Cloud Storage to secret manager, keeping their existing names and permissions. Then in your application, simply change the burglass reference to the SM reference, which is short for secret manager. And redeploy your application.
That's actually it burglass is completely interoperable with secret manager and you don't have to change your workflow to use it. Best of all you can mix and match, you can have some stickers that come from her glass and some secrets to come from Secret manager, all using the same unified API. Thank you so much for coming on this second secret, since our religious Journey with me. Hopefully, we've made it. Clear the difference, Secrets Management Solutions on Google cloud, and how you can best secure your service secrets in your applications.
Buy this talk
Buy this video
With ConferenceCast.tv, you get access to our library of the world's best conference talks.