Setting Up Private Github Access with Aws Elastic Beanstalk and Ruby Container

Setting up private Github access with AWS Elastic Beanstalk and Ruby container

After a good day of effort, I finally enabled use of my organization's private GitHub repos with Elastic Beanstalk by just using a .config file. I am using Python and pip, but it should also work for other package installers on EB.

rhetonik's ssh-agent+ssh-add approach did not work for me at all, so I elected to set up an ssh configuration file instead.

Here is my .ebextensions/3-pip-install-from-github.config file:

files:
"/root/.ssh/config":
owner: root
group: root
mode: "000600"
content: |
Host github.com
User git
Hostname github.com
IdentityFile /root/.ssh/github

commands:
01-command:
command: sudo ssh-keyscan -H github.com >> /root/.ssh/known_hosts
02-command:
command: sudo chmod 644 /root/.ssh/known_hosts
03-command:
command: sudo aws s3 cp s3://bucket-with-your-github-ssh-key/github /root/.ssh
04-command:
command: sudo chmod 600 /root/.ssh/github

Rough instructions:

  • Set up an S3 bucket accessible by your EB instance. Inside of that bucket, store the SSH key allowing access to the GitHub repository you want to access via pip, npm, bundle, etc. Use sudo aws s3 cp to copy that key onto your EB instance on deploy. sudo is necessary because EB scripts use root and not ec2-user.

  • This ebextensions config file also creates 2 files on your EB instance. /root/.ssh/config tells ssh (invoked by pip and git) to use the key you copied from S3. Storing the output of ssh-keyscan -H github.com into /root/.ssh/known_hosts will pre-verify that ssh on your EB instance is actually communicating with GitHub to avoid MITM attacks. This is better than disabling StrictHostKeyChecking in /root/.ssh/config.

Here is my requirements.txt file for pip:

Beaker==1.7.0
Flask==0.10.1
Jinja2==2.7.3
MarkupSafe==0.23
# [...]
git+ssh://git@github.com/myorganization/myprivaterepo.git@0.0.142

While running eb-deploy, you can tail -f /var/log/eb-activity.log to make sure everything runs smoothly.

Configuring Elastic Beanstalk for SSH access to private git repo using Amazon Linux 2 hooks

Summary

Assume we have defined the following Elastic Beanstalk environment properties, and both the bitbucket public key file and our private key file have been uploaded to the specified S3 bucket:

S3_BUCKET_NAME="my_bucket"
REPO_HOST_NAME="bitbucket.org"
REPO_HOST_PUBLIC_KEY_NAME="bitbucket_public_key"
REPO_PRIVATE_KEY_NAME="my_private_key"

The configuration can then be accomplished using this hook in .platform/hooks/prebuild:

#!/bin/bash

# git is required to install our python packages directly from bitbucket
yum -y install git

# file paths (platform hooks are executed as root)
SSH_KNOWN_HOSTS_FILE="/root/.ssh/known_hosts"
SSH_CONFIG_FILE="/root/.ssh/config"
PRIVATE_KEY_FILE="/root/.ssh/$REPO_PRIVATE_KEY_NAME"

# remove any existing (stale) keys for our host from the known_hosts file
[ -f $SSH_KNOWN_HOSTS_FILE ] && ssh-keygen -R $REPO_HOST_NAME

# read the (fresh) host key from S3 file and append to known_hosts file
aws s3 cp "s3://$S3_BUCKET_NAME/$REPO_HOST_PUBLIC_KEY_NAME" - >> $SSH_KNOWN_HOSTS_FILE

# copy our private key from S3 to our instance
aws s3 cp "s3://$S3_BUCKET_NAME/$REPO_PRIVATE_KEY_NAME" $PRIVATE_KEY_FILE

# create an ssh config file to point to the private key file
tee $SSH_CONFIG_FILE <<HERE
Host $REPO_HOST_NAME
User git
Hostname $REPO_HOST_NAME
IdentityFile $PRIVATE_KEY_FILE
HERE

# file permissions must be restricted
chmod 600 $SSH_CONFIG_FILE
chmod 600 $PRIVATE_KEY_FILE

Note this file requires execution permission (chmod +x <file path>).

Detailed explanation

Read on for a detailed rationale.

Git

To access a git repository, our Elastic Beanstalk environment will need to have git installed.
This can be done in a platform hook using yum (-y assumes "yes" to every question):

yum -y install git

SSH keys

To set up an SSH connection between our Elastic Beanstalk (EB) instance and e.g. a bitbucket repository, we need three SSH keys:

  • The public key for bitbucket.org, to verify that we are connecting to a trusted host.

    To obtain the public key for bitbucket.org, in a suitable format for known_hosts, we can use ssh-keyscan.
    To be on the safe side, we should verify this key using a "trusted" source.
    In our case the best we can do is compare the public key fingerprint with the "official" one published on the bitbucket (or github) website.
    The fingerprint can be calculated from the public key using ssh-keygen e.g.

    ssh-keyscan -t rsa bitbucket.org | ssh-keygen -lf -
  • The private key and public key for our repository.

    A key pair, consisting of private and public key, can be generated using ssh-keygen.
    The private key must be kept secret, the public key must be copied to the list of "access keys" for the bitbucket repository, as described in the bitbucket docs.
    Note that it is most convenient to create a key pair without passphrase, otherwise our script will need to handle the passphrase as well.

Storing the keys on AWS

The public bitbucket host key and our private repo key need to be available in the EB environment during deployment.
The private key is secret, so it should not be stored in the source code, nor should it be otherwise version controlled.

The most convenient option would be to store the key values as EB environment properties (i.e. environment variables), because these are readily available during deployment.
In principle, this can be done, e.g. using base64 encoding to store the multiline private key in a single line environment property.
However, the total size of all EB environment property keys and values combined is limited to a mere 4096 bytes, which basically precludes this option.

An alternative is to store the key files in a secure private bucket on AWS S3.
The documentation describes how to set up an IAM role that grants access to your S3 bucket for the EC2 instance. The documentation does provide a configuration example, but this uses .ebextensions and does not apply to .platform hooks.

In short, we can create a basic S3 bucket with default settings ("block public access" enabled, no custom permissions), and upload the SSH key files to that bucket.
Then, using the AWS IAM web console, select the aws-elasticbeanstalk-ec2-role (or, preferably, create a custom role), and attach the AmazonS3ReadOnlyAccess policy.

During deployment to Elastic Beanstalk, we can use .platform hooks to download the key files from the S3 bucket to the EC2 instance using the aws cli.

To test connectivity between EC2 and S3, we could use eb ssh to connect to the EC2 instance, followed by, for example, aws s3 ls s3://<bucket name> to list bucket contents.

Updating known_hosts

To indicate that bitbucket.org is a trusted host, its public key needs to be added to the known_hosts file on our instance.
In our platform hook script, we remove any existing public keys for the host, in case they are stale, and replace them by the current key from our file on S3:

SSH_KNOWN_HOSTS_FILE="/root/.ssh/known_hosts"
[ -f $SSH_KNOWN_HOSTS_FILE ] && ssh-keygen -R $REPO_HOST_NAME
aws s3 cp "s3://$S3_BUCKET_NAME/$REPO_HOST_PUBLIC_KEY_NAME" - >> $SSH_KNOWN_HOSTS_FILE

Specifying the private key

The private key can be downloaded from S3 as follows, and we need to restrict the file permissions:

PRIVATE_KEY_FILE="/root/.ssh/$REPO_PRIVATE_KEY_NAME"
aws s3 cp "s3://$S3_BUCKET_NAME/$REPO_PRIVATE_KEY_NAME" $PRIVATE_KEY_FILE
chmod 600 $PRIVATE_KEY_FILE

An SSH configuration file is also required to point to the private key:

tee $SSH_CONFIG_FILE <<HERE
Host $REPO_HOST_NAME
User git
Hostname $REPO_HOST_NAME
IdentityFile $PRIVATE_KEY_FILE
HERE
chmod 600 $SSH_CONFIG_FILE

Again, file permissions must be restricted.

The final script is shown in the summary at the top.
This script could be stored e.g. as .platform/hooks/prebuild/01_configure_bitbucket_ssh.sh in the project folder.

Hooks and confighooks

Note that Amazon Linux 2 uses .platform/hooks, for normal deployments, and .platform/confighooks, for configuration deployments.
Often, identical scripts need to be used in both cases.
To prevent duplication of code, our .platform/confighooks/prebuild/01_configure_bitbucket_ssh.sh could look like this:

#!/bin/bash
source "/var/app/current/.platform/hooks/prebuild/01_configure_bitbucket_ssh.sh"

Note that the application code ends up in /var/app/current on the instance.

Setting up SSH keys for github private repo access on Elastic Beanstalk

After a full day's struggle and finally stumbling over this answer to a very similar question I had previously missed, it turns out the correct place to put ssh keys in order to be picked up by git on EB is in /root/.ssh, not /tmp/.ssh, not /home/ec2-user/.ssh.

My final configuration (assuming there's a private SSH key located in a S3 bucket at <my-bucket>/github-eb-key, and the corresponding public key is registered with a github user having access to the repo(s)), using an AMI configured as 64bit Amazon Linux 2016.09 v3.3.0 running Node.js, and with the following in .ebextensions/01_ssh_setup.config:

Resources: 
AWSEBAutoScalingGroup:
Metadata:
? "AWS::CloudFormation::Authentication"
:
S3Auth:
buckets:
- <my-bucket>
roleName:
? "Fn::GetOptionSetting"
:
DefaultValue: aws-elasticbeanstalk-ec2-role
Namespace: "aws:asg:launchconfiguration"
OptionName: IamInstanceProfile
type: s3
files:
/root/.ssh/github-eb-key:
authentication: S3Auth
mode: "000600"
owner: root
group: root
source: "https://s3-eu-west-1.amazonaws.com/<my-bucket>/github-eb-key"
/root/.ssh/config:
mode: "000600"
owner: root
group: root
content: |
Host github.com
IdentityFile /root/.ssh/github-eb-key
IdentitiesOnly yes
UserKnownHostsFile=/dev/null
StrictHostKeyChecking no

Rails - AWS (Elastic Beanstalk) deployment, Error Command 'git clone'

Ok I solved this by installing git on my EC2 instance. Just ssh into your instance and run the following:

sudo yum install git

and that should fix the problem.

Issue Deploying Rails 6.1 to AWS Elastic Beanstalk

Did you check if your line-ending format is either Windows (CRLF or \r\n) or *NIX (LF alone or \n)?

If your source control is git, both options are available.
And if you develop on Windows, or if you moved your code between platforms without configuring git config core.autocrlf, your sources files could have both these different formats and that can confuse text interpreters.

Have you checked also if AWS has any recommended Rails version? Often, cloud providers will lag on Ruby and most other languages/frameworks if you are using anything else except a VM.

So Another option is to deploy Rails on a VM (AWS or GCP or Azure or Digital Ocean etc ..) and in this case you will have full control of everything and you can run any version of Rails but it will require a bit more options to open ingress ports and assign static IP addr, fundamentally Virtual Machine initial config.

Reference for git line ending config:

https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings

About Windows CRLF:

https://www.hanselman.com/blog/carriage-returns-and-line-feeds-will-ultimately-bite-you-some-git-tips

Permission issues with elastic beanstalk deployment via git aws.push and IAM user

It sounds like you should use an existing IAM Policy instead of trying to build up a Policy as you run into issues.

To do this, go to the IAM Console and then go to the user that you are using.

Click on the User and then you'll see all the info relating to that User. Now go to "Attach User Policy". It'll bring up a window that lets you select a Policy Template, use the Policy Generator, or set a Custom Policy.

We're just going to use the Policy Template feature. Next scroll in the list until you find "AWS Elastic Beanstalk" in the list. There will be two templates:

  • AWS Elastic Beanstalk Full Access
  • AWS Elastic Beanstalk Read Only Access

Each has a description so you can read that for each. Basically the Full Access template is what you will want for using the eb cli which provides git aws.push.

Select it and then click "Apply Policy".

Try running git aws.push again and you shouldn't have issues if you have everything configured properly to use that user.



Related Topics



Leave a reply



Submit