Using Moviepy, Scipy and Numpy in Amazon Lambda

Using moviepy, scipy and numpy in amazon lambda

With the help of all posts in this thread here is a solution for the records:

To get this to work you'll need to:

1. start a EC2 instance with at least 2GO RAM (to be able to compile NumPy & SciPy)

2. Install the needed dependencies

   sudo yum -y update
   sudo yum -y upgrade
   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install blas --enablerepo=epel
   sudo yum -y install lapack --enablerepo=epel
   sudo yum -y install Cython --enablerepo=epel
   sudo yum install python27-devel python27-pip gcc
   virtualenv ~/env
   source ~/env/bin/activate
   pip install scipy
   pip install numpy
   pip install moviepy
   

3. Copy to your locale machine all the content of the directories (except _markerlib, pip*, pkg_resources, setuptools* and easyinstall*) in a stack folder:

   • home/ec2-user/env/lib/python2.7/dist-packages
   • home/ec2-user/env/lib64/python2.7/dist-packages

4. get all required shared libraries from you EC2instance:

   • libatlas.so.3
   • libf77blas.so.3
   • liblapack.so.3
   • libptf77blas.so.3
   • libcblas.so.3
   • libgfortran.so.3
   • libptcblas.so.3
   • libquadmath.so.0

5. Put them in a lib subfolder of the stack folder

6. imageio is a dependency of moviepy, you'll need to download some binary version of its dependencies: libfreeimage and of ffmpeg; they can be found here. Put them at the root of your stack folder and rename libfreeimage-3.16.0-linux64.soto libfreeimage.so

7. You should now have a stack folder containing:

   • all python dependencies at root
   • all shared libraries in a lib subfolder
   • ffmpeg binary at root
   • libfreeimage.so at root

8. Zip this folder: zip -r9 stack.zip . -x ".*" -x "*/.*"

9. Use the following lambda_function.py as an entry point for your lambda

   from __future__ import print_function
   
   import os
   import subprocess
   
   SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
   LIB_DIR = os.path.join(SCRIPT_DIR, 'lib')
   FFMPEG_BINARY = os.path.join(SCRIPT_DIR, 'ffmpeg')
   
   def lambda_handler(event, context):
       command = 'LD_LIBRARY_PATH={} IMAGEIO_FFMPEG_EXE={} python movie_maker.py'.format(
           LIB_DIR,
           FFMPEG_BINARY,
       )
       try:
           output = subprocess.check_output(command, shell=True)
           print(output)
       except subprocess.CalledProcessError as e:
           print(e.output)
   

10. write a movie_maker.pyscript that depends on moviepy, numpy, ...

11. add those to script to your stack.zip file zip -r9 lambda.zip *.py

12. upload the zip to S3 and use it as a source for your lambda

You can also download the stack.zip here.

Installing numpy on Mac to work on AWS Lambda

Building on @MarkB's comment, it would not be possible to build numpy on a Mac to use on AWS Lambda. So why do some packages work and others don't?

Python extension modules, as explained on Mark Nunnikhoven's blog here, are

written in C or C++ that can either extend python or call C or C++
libraries.

Since these modules are compiled specific to the system you're on, and AWS Lambda is a Linux environment, you'll need to install any extension modules on a Linux environment.

Related Topics