PostgreSQL on Elastic Beanstalk (Amazon Linux 2)
The following works:
packages:
yum:
amazon-linux-extras: []
commands:
01_postgres_activate:
command: sudo amazon-linux-extras enable postgresql10
02_postgres_install:
command: sudo yum install -y postgresql-devel
Elastic Beanstalk unable to install packages with postgresql
Indeed there is no postgresql96-devel
, assuming Amazon Linux 2 (AL2).
In the AL2, there is only postgresql-devel
(9.2). However, if 9.2 is not suited, then you can install 9.6 using amazon-linux-extras
amazon-linux-extras install postgresql9.6
To execute that command in EB in an automated manner, you can run it as a in Commands section of your .ebextensions
:
The commands run before the application and web server are set up and the application version file is extracted.
How to Install Postgresql 11 in Amazon Linux AMI?
The key was the PGDG is no longer available to Amazon Linux's yum
since 9.3 so the individual pieces must be installed.
# Remove old Postgres
yum remove -y postgresql postgresql-server
# Install Postgres 11
yum install -y https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-6-x86_64/postgresql11-libs-11.4-1PGDG.rhel6.x86_64.rpm
yum install -y https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-6-x86_64/postgresql11-11.4-1PGDG.rhel6.x86_64.rpm
yum install -y https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-6-x86_64/postgresql11-server-11.4-1PGDG.rhel6.x86_64.rpm
[edit]
Replace the 11.4
in each link above with any version you need available at https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-6-x86_64/
Deploy Postgres11 to Elastic Beanstalk - Requires /etc/redhat-release
I have the same problem with PostgreSQL 10 for a Django 2.1 project on AWS Elastic Beanstalk.
The issue was introduced last week around 17 Apr 2019 to make sure the operating system is an actual Red Hat release (which Amazon Linux is not). I found some details on the PostgreSQL mailing list:
"Amazon Linux support was removed years ago actually. I just made sure that our
repo file reflects that."
(BUG #15768: Removed rpms and now require /etc/redhat-release)
One poster on the mailing list suggested the following fix:
"We have temporarily mitigated the issue by using rpm and explicitly ignoring the repositories dependencies, but that seems like a band-aid fix for the real problem which is that dependency shouldn’t be there."
Personally, I've done the same as you, Scott, I've simply reverted to the PostgreSQL 9.6 client packages that AWS supplies directly. As long as Django and psycopg support that release, this works fine.
However, the long-term fix is for AWS to finally provide platforms with Amazon Linux 2...
I'm having a problem with an elastic bean stalk setup on AWS linux 2. It's a laravel App with Postgres as the DB. Password auth failed for user
If your DB is a Managed PostgreSQL on AWS RDS
Then you may want to reset password of the postgres
user.
As per RDS manual
To modify the master user password, follow these steps:
- Open the Amazon RDS console.
- Select Databases.
- Select the RDS DB instance, and then choose Modify.
Note: If you use Aurora, expand the cluster, and choose the instance that you want to modify. Then, choose Modify.- Enter the master user password you want to use in the New Master Password field.
Note: The password change is asynchronous, and applies as soon as possible. This change ignores the Apply Immediately setting.- Choose Continue, and then choose Modify DB Instance.
The Status field for your RDS DB instance on the RDS dashboard changes to resetting-master-credentials. When the modification is complete, the Status column changes to Available. You can also modify an RDS DB instance using the Amazon RDS API or the AWS Command Line Interface (AWS CLI).
If your DB is self-deployed to the AWS EC2
instance
Then you may want to enable password-based authentication of the postgres
user. It's disabled by default for new DB instances.
- Find the
pg_hba.conf
file
To find the pg_hba.conf file on your server, you can look in the PostgreSQL configuration directory.
E.g.:
find /etc/postgresql/ -name pg_hba.conf
- Then open found file in your
${EDITOR}
- To allow md5 password authentication for any connections coming from the local 192.168.0.0/24 network, add a line like this:
# TYPE DATABASE USER ADDRESS METHOD OPTIONS
host all all 192.0.0.0/24 md5
- And then restart PostgreSQL:
sudo pgctl restart
AWS Elastic Beanstalk failed to install psycopg2 using requirements.txt Git Pip
I could solve the error. I have to change psycopg2
by psycopg2-binary
as it was suggested by the AWS logs:
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
This issue has to be with the particular configuration of the libraries and the specific Linux machines used in AWS.
Upgrading Elastic Beanstalk environment from AWS Linux 1 to AWS Linux 2
Differences in Configuration
AWS Linux 2 has changed a lot of how elastic beanstalk works and how it is configured. Regardless of whether you are doing an in-place upgrade or spinning up a new environment, here is a list of things that will be different to run through before making the upgrade. Most of the items here are things that are different in Elastic Beanstalk config that live in .ebextensions
.
There are differences in sub-package dependencies between Python 3.6 and 3.8. You should test your requirements file on Python 3.8 and make sure it's compatible, especially if you use a generated
requirements.txt
.AWS Linux 2 no longer allows you to write Apache config using a
file
directive in.ebextensions
. These modifications now need to live in.platform/httpd/conf
.The virtual environment is no longer active while running container_commands. Any container commands that use your code need to have
source $PYTHONPATH/activate
run first.Generated files now get wiped on config changes, so commands like django's
collectstatic
need to get moved to hooks.Postgres client is no longer available normally though yum. To install it, you need to do:
packages:
yum:
amazon-linux-extras: []
commands:
01_postgres_activate:
command: sudo amazon-linux-extras enable postgresql10
02_postgres_install:
command: sudo yum install -y postgresql-develApache is no longer the default web server (it is Nginx). To continue using it, you need to specify that as an option on your environment, such as:
option_settings:
aws:elasticbeanstalk:environment:proxy:
ProxyServer: apacheModwsgi has been replaced with Gunicorn. Any modwsgi customizations you have will no longer work, and the WSGI path has a different format:
option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: config.wsgi:applicationStatic files config has a different format:
option_settings:
aws:elasticbeanstalk:environment:proxy:staticfiles:
/static: staticfilesYou will get opted-in to advanced health reporting. Adding an Elastic Beanstalk health check is strongly recommended:
option_settings:
aws:elasticbeanstalk:application:
Application Healthcheck URL: /health-check/The application is now run on port 8000 on the server via Gunicorn and Apache/Nginx are just proxying requests to Gunicorn. This matters if you are doing apache customizations such as encrypting traffic between the load balancers and applications servers.
Apache is now run through systemctl rather than supervisord. If you are trying to restart Apache, the command is now
sudo systemctl restart httpd
If you want to load your environment variables when sshed into the server, you need to parse them differently:
The environment variables live in a different place and have a different format. To get access to them when SSHed in, you need to add
jq: []
to your yum installs. Then, either run the following commands or add them to the bashrc of the server (using afile
directive in.ebextensions
) to load environment variables and activate the python virtual environment:source <(/opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "export \(.key)=\"\(.value)\""')
source $PYTHONPATH/activate
cd /var/app/current
Upgrading by Launching a New Environment
To take this upgrade path, you need to not be using the Elastic Beanstalk data tier (i.e. you launched your RDS instance yourself, rather than through Elastic Beanstalk).
Create a code branch with your AWS Linux 2 config
Launch a new Elastic Beanstalk environment on AWS Linux 2.
Copy the environment variables from your previous environment.
Allow access to your database from the new environment (add the new environment's server security group as the target of an ingress rule on the database's security group)
Set up SSL on the new environment.
Deploy the AWS Linux 2 code branch to the new environment.
Test this new environment, ignoring browser certificate warnings (or set up a temporary DNS entry to test it).
Switch the DNS entry to point to your new environment or use AWS's CNAME swap feature on the two environemnts.
After your new environment has been running without problems for sufficient time, terminate your old environment.
Upgrading In-Place
There is a way to do the upgrade in-place, though there will be a few minutes where your site says "502 Bad Gateway". In order to do this, you need EB config that is compatible with both the AWS Linux 1 and AWS Linux 2 environments.
For Python, you can do this with a small Flask app and a four part deploy.
Part 1: deploy placeholder app that is compatible with both platforms
Add
flask
to yourrequirements.txt
(if it's not already there).Delete all files in
.ebextensions
Make
.ebextensions/01.config
:option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: wsgi_shim.pymake
wsgi_shim.py
:from flask import Flask
application = Flask(__name__)
@application.route("/")
@application.route("/<path:path>/")
def hello_world(path=None):
return "This site is currently down for maintenance"[If using load balancer to application server encryption, change the load balancer to send all traffic to the server via HTTP.]
eb deploy
Part 2: upgrade platform to AWS Linux 2
[If you have any static routes configured in elastic beanstalk delete them.]
Upgrade your eb environment
# Get list of solution stacks
aws elasticbeanstalk list-available-solution-stacks --output=json --query 'SolutionStacks' --region us-east-1
# Use one of the above options here
aws elasticbeanstalk update-environment --environment-name <ENV_NAME> --solution-stack-name "64bit Amazon Linux 2 v3.3.7 running Python 3.8"
Part 3: deploy your main application to AWS Linux 2
Replace
.ebextensions/01.config
with your new AWS Linux 2 config.Add
.platform/httpd/conf.d/ssl_rewrite.conf
:RewriteEngine On
<If "-n '%{HTTP:X-Forwarded-Proto}' && %{HTTP:X-Forwarded-Proto} != 'https'">
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]eb deploy
Part 4: cleanup
[If using load balancer to application server encryption, change the load balancer back to sending traffic to the server via HTTPS.]
Delete
wsgi_shim.py
and remove flask fromrequirements.txt
(unless it's a flask project).eb deploy
Related Topics
Why Do I Get "Suspended (Tty Output)" in One Terminal But Not in Others
How to Install a Rpm Package and Its Dependencies Offline
How to Toggle Cr/Lf in Gnu Screen
How Is Stack Size of Linux Process Related to Pthread, Fork and Exec
Evaluating Smi (System Management Interrupt) Latency on Linux-Centos/Intel MAChine
How Is the Init Process Started in the Linux Kernel
See Socket Options on Existing Sockets Created by Other Apps
System Wide Keyboard Hook on X Under Linux
How to Check Fips 140-2 Support in Openssl
Could Not Load Shared Library Symbols for Linux-Vdso.So.1. While Debugging
Sed Help: Matching and Replacing a Literal "\N" (Not the Newline)
Add Blank Line After Every Result in Grep
Merging Through Fuzzy Matching of Variables in R