How to Manage Third-Party Python Libraries with Google App Engine? (Virtualenv? Pip)

How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)

(Jun 2021) This post is over a decade old, and so an updated answer is warranted now.

  1. Python 3: list 3P libraries in requirements.txt along with any desired version#s; they'll be automatically installed by Google upon deployment. (This is the same technique used if you decide to migrate your app to Google Cloud Functions or Cloud Run.)
  2. Python 2 without built-in 3P libraries (regular 3P libraries):
  • Create requirements.txt as above
  • Install/self-bundle/copy them locally, say to lib, via pip install -t lib -r requirements.txt
  • Create appengine_config.py as shown in step 5 on this page

  1. Python 2 with built-in 3P libraries (special set of 3P libraries):
  • All listed 3P libraries linked above are "built-in," meaning they're available on App Engine servers so you don't have to copy/self-bundle them w/your app (like in #2 above)
  • It suffices to list them with an available version in the libraries: section of your app.yaml like this
  • (Don't put built-in libraries in requirements.txt nor use pip install to install them locally unless you want to self-bundle because, say if you need a newer version of the built-in library.)
  • Create appengine_config.py like the above.

If you have a Python 2 app with both built-in and non-built-in 3P libraries, use the techniques in both #2 and #3 above (built-in libraries in app.yaml and non-built-in libraries in requirements.txt and run the pip install cmd above). One of the improvements in the second generation runtimes like Python 3 is that all these games with 3P libraries go away magically (see #1 above).

Example: Flask

Flask is a 3rd-party micro web framework, and it's an interesting case for this specific question. For Python 3, they all go into requirements.txt, so you'd just add flask to that file, and you're done. (Just deploy from there.)

For Python 2, it's even more interesting because it's a built-in library. Unfortunately, the version on App Engine servers is 0.12. Who wants to use that when we're at/beyond 2.0.3 now?!? So instead of putting it in app.yaml like other built-in libraries, you'd pretend the built-in version doesn't exist and put it in requirements.txt then run pip2 install -t lib -r requirements.txt to bundle/vendor it with your application code. (However, the final version for Python 2 is 1.1.4, so that's what gets installed.)

How to include third party Python libraries in Google App Engine?

Actually I think this answer fits better here.

If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.

In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:

hello
├── libs
│ └── bs4
├── hello.py
└── app.yaml

then in your hello.py you have to put these two lines in the beginning of the file:

import sys
sys.path.insert(0, 'libs')

After doing that you'll be able to use any 3rd party library that you're going to put in that libs directory.

For example:

from bs4 import BeautifulSoup

Importing 3rd Party Libraries in GAE (not manually) using Pip or another tool

If you are already using pip and virtualenv, you need to create symlinks from ./lib/python2.7/site-packages/"libxxx" to your GAE project directory (where . is the root of your virtualenv directory, and libxxx is the name of the 3rd party library you're willing to install).

If there are symbolic links in your GAE project, then appcfg.py will resolve these links when publishing your app on Google's infrastructure.

For example, on Mac OS X, to install HTTPlib on GAE, I did the following:

ln -s ~/Projets/myproject/lib/python2.7/site-packages/httplib2/ ~/Projets/myproject/src/packages/libs/httplib2/

After that, provided your code is somewhere inside ~/Projets/myproject/src, you can use the library using: from packages.libs import httplib2.

Google App Engine Python, virtualenv and mimetypes

This is described in Issue 4339 for GAE. Here's how to fix it:

  1. Download patch from this Issue comment: patch
  2. Move the patch to google_appengine/google/appengine/tools/
  3. Change your working directory to the same path as above
  4. Type: patch -p0 < dev_appserver.patch

How to use virtualenv with Google App Engine SDK on Mac OS X 10.6

It's an issue 4339 with the GAE SDK, it's confirmed and there are two slightly different patches available in the bug entry that make it work.

What happens is dev_appserver.py sets up a restricted python environment by disallowing access to any non-system-python modules and it does that by calculating the system python folder from the location of the os module. In a virtualenv instance the os.py gets symlinked into the virtualenv but gets compiled straight into virtualenv, and this is the path that dev_appserver uses, effectively blocking access to any module from the system python library that is not linked by virtualend, which is most of them. The solution is to "bless" both paths.

Install third party libs to be available from within and without the Google App Engine dev server

I follow a variation of the same steps but with

$ ln -s {virtualenv}/lib/python2.7/site-packages lib

This way a pip install in the virtualenv automatically goes to the lib directory as well.

Every pip install would then be available to the virtualenv's python and to the dev_appserver without supplying the target folder to make testing things bearable. Eg.:

$ pip install sqlalchemy


Related Topics



Leave a reply



Submit