Securely Storing Environment Variables in Gae with App.Yaml

Securely storing environment variables in GAE with app.yaml

If it's sensitive data, you should not store it in source code as it will be checked into source control. The wrong people (inside or outside your organization) may find it there. Also, your development environment probably uses different config values from your production environment. If these values are stored in code, you will have to run different code in development and production, which is messy and bad practice.

In my projects, I put config data in the datastore using this class:

from google.appengine.ext import ndb

class Settings(ndb.Model):
  name = ndb.StringProperty()
  value = ndb.StringProperty()

  @staticmethod
  def get(name):
    NOT_SET_VALUE = "NOT SET"
    retval = Settings.query(Settings.name == name).get()
    if not retval:
      retval = Settings()
      retval.name = name
      retval.value = NOT_SET_VALUE
      retval.put()
    if retval.value == NOT_SET_VALUE:
      raise Exception(('Setting %s not found in the database. A placeholder ' +
        'record has been created. Go to the Developers Console for your app ' +
        'in App Engine, look up the Settings record with name=%s and enter ' +
        'its value in that record\'s value field.') % (name, name))
    return retval.value

Your application would do this to get a value:

API_KEY = Settings.get('API_KEY')

If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.

I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!

The code above uses the ndb library which uses memcache and the datastore under the hood, so it's fast.

Update:

jelder asked for how to find the Datastore values in the App Engine console and set them. Here is how:

Go to https://console.cloud.google.com/datastore/
Select your project at the top of the page if it's not already selected.
In the Kind dropdown box, select Settings.
If you ran the code above, your keys will show up. They will all have the value NOT SET. Click each one and set its value.

Hope this helps!

Your settings, created by the Settings class

Click to edit

Enter the real value and save

Does google app engine support environment variables?

Environment variables can be defined in your application's app.yaml

An example for a python/php/(maybe go?) app. Java uses a different format.

env_variables:
  MY_ENV_VAR: 'some value here'

https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Defining_environment_variables

You can set these values during your CI process as well if you need to by programmatically appending them to your app.yaml before deploying.

Where to store environment variables in App Engine for CI/CD Pipeline?

For those looking, here is how I solved this problem.

I followed the steps outlined in this blog post.

Basically we set variables in the .yaml file, which we then compile into an .env file during the build process. We can set what the value of those variables is via Cloud Build configuration so we can restrict access to them and have them hidden.

Add environment variable in app.yaml file during Google Build

When you run gcloud app deploy, the deployment process won't take the cloudbuild.yaml file into account and will deploy your app along with your unpopulated app.yaml file.

To run a custom build step, you'll need to create a cloudbuild.yaml file as you did, define your custom build step and then add a build step to run the deploy command. That'd be something like this:

steps:
 - name: gcr.io/cloud-builders/gsutil
   args:
   [
    "cp",
    "gs://<path to bucket>/app.yaml",
    "app.yaml",
   ]
 - name: 'gcr.io/cloud-builders/gcloud'
   args: ['app', 'deploy']

You'll then run the build by issuing the following command (in the same directory where you'd have run the gcloud app deploy one):

gcloud builds submit --config cloudbuild.yaml .

This will:

Upload the current directory to the Cloud Build instance
run the gsutil command from within that directory on the CB instance to retrieve the app.yaml file populated with your environment variables
deploy your code to App Engine from the Cloud Build instance

Securely Storing Environment Variables in Gae with App.Yaml