How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)
(Jun 2021) This post is over a decade old, and so an updated answer is warranted now.
- Python 3: list 3P libraries in
requirements.txt
along with any desired version#s; they'll be automatically installed by Google upon deployment. (This is the same technique used if you decide to migrate your app to Google Cloud Functions or Cloud Run.) - Python 2 without built-in 3P libraries (regular 3P libraries):
- Create
requirements.txt
as above - Install/self-bundle/copy them locally, say to
lib
, viapip install -t lib -r requirements.txt
- Create
appengine_config.py
as shown in step 5 on this page
- Python 2 with built-in 3P libraries (special set of 3P libraries):
- All listed 3P libraries linked above are "built-in," meaning they're available on App Engine servers so you don't have to copy/self-bundle them w/your app (like in #2 above)
- It suffices to list them with an available version in the
libraries:
section of yourapp.yaml
like this - (Don't put built-in libraries in
requirements.txt
nor usepip install
to install them locally unless you want to self-bundle because, say if you need a newer version of the built-in library.) - Create
appengine_config.py
like the above.
If you have a Python 2 app with both built-in and non-built-in 3P libraries, use the techniques in both #2 and #3 above (built-in libraries in app.yaml
and non-built-in libraries in requirements.txt
and run the pip install
cmd above). One of the improvements in the second generation runtimes like Python 3 is that all these games with 3P libraries go away magically (see #1 above).
Example: Flask
Flask is a 3rd-party micro web framework, and it's an interesting case for this specific question. For Python 3, they all go into requirements.txt
, so you'd just add flask
to that file, and you're done. (Just deploy from there.)
For Python 2, it's even more interesting because it's a built-in library. Unfortunately, the version on App Engine servers is 0.12
. Who wants to use that when we're at/beyond 2.0.3
now?!? So instead of putting it in app.yaml
like other built-in libraries, you'd pretend the built-in version doesn't exist and put it in requirements.txt
then run pip2 install -t lib -r requirements.txt
to bundle/vendor it with your application code. (However, the final version for Python 2 is 1.1.4
, so that's what gets installed.)
How to include third party Python libraries in Google App Engine?
Actually I think this answer fits better here.
If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.
In order to include manually any other library you have to have them inside the directory where the app.yaml
lives. So for example if you have the following structure:
hello
├── libs
│ └── bs4
├── hello.py
└── app.yaml
then in your hello.py
you have to put these two lines in the beginning of the file:
import sys
sys.path.insert(0, 'libs')
After doing that you'll be able to use any 3rd party library that you're going to put in that libs
directory.
For example:
from bs4 import BeautifulSoup
Importing 3rd Party Libraries in GAE (not manually) using Pip or another tool
If you are already using pip and virtualenv, you need to create symlinks from ./lib/python2.7/site-packages/"libxxx"
to your GAE project directory (where .
is the root of your virtualenv directory, and libxxx
is the name of the 3rd party library you're willing to install).
If there are symbolic links in your GAE project, then appcfg.py
will resolve these links when publishing your app on Google's infrastructure.
For example, on Mac OS X, to install HTTPlib on GAE, I did the following:
ln -s ~/Projets/myproject/lib/python2.7/site-packages/httplib2/ ~/Projets/myproject/src/packages/libs/httplib2/
After that, provided your code is somewhere inside ~/Projets/myproject/src
, you can use the library using: from packages.libs import httplib2
.
Google App Engine Python, virtualenv and mimetypes
This is described in Issue 4339 for GAE. Here's how to fix it:
- Download patch from this Issue comment: patch
- Move the patch to google_appengine/google/appengine/tools/
- Change your working directory to the same path as above
- Type: patch -p0 < dev_appserver.patch
How to use virtualenv with Google App Engine SDK on Mac OS X 10.6
It's an issue 4339 with the GAE SDK, it's confirmed and there are two slightly different patches available in the bug entry that make it work.
What happens is dev_appserver.py
sets up a restricted python environment by disallowing access to any non-system-python modules and it does that by calculating the system python folder from the location of the os
module. In a virtualenv instance the os.py
gets symlinked into the virtualenv but gets compiled straight into virtualenv, and this is the path that dev_appserver uses, effectively blocking access to any module from the system python library that is not linked by virtualend, which is most of them. The solution is to "bless" both paths.
Install third party libs to be available from within and without the Google App Engine dev server
I follow a variation of the same steps but with
$ ln -s {virtualenv}/lib/python2.7/site-packages lib
This way a pip install in the virtualenv automatically goes to the lib directory as well.
Every pip install would then be available to the virtualenv's python and to the dev_appserver without supplying the target folder to make testing things bearable. Eg.:
$ pip install sqlalchemy
Related Topics
How to Implement the Softmax Function in Python
Matplotlib Legend Markers Only Once
Importing from a Relative Path in Python
Getting Only Element from a Single-Element List in Python
How to Transpose Dataframe in Pandas Without Index
How to Get the Current Time in Milliseconds in Python
Regular Expressions: Search in List
Store Mouse Click Event Coordinates with Matplotlib
Intercepting Stdout of a Subprocess While It Is Running
Ssl Insecureplatform Error When Using Requests Package
Extract a Page from a PDF as a Jpeg
Plotting 3D Polygons in Python-Matplotlib
How to Mark a Portion of a Text Widget as Readonly
Make Requests Using Python Over Tor
Angles Between Two N-Dimensional Vectors in Python
Why Is Looping Over Range() in Python Faster Than Using a While Loop