I use Pipenv to bundle Python libraries but you cannot run pipenv install on Lambda environment. There was also a gotcha when I tried to use psycopg2, so in this post, I’m going to clearify how I successfully connected to RDS.

(Note: We decided not to go with Serverless because we were in the middle of migrating from AWS to GCP and things around deployment would be totally different in case you use it.)

To simplify things, let’s say we have just one file called main.py. If you have other files, of course you need to copy them into the deployed directory.

Python Pipenv package

When you need libraries from PyPI, you want to make a zip file and that zip file needs to include those PyPI libraries. As I mentioned, I use Pipenv and this is how you can do it.

# Create a directory that has all the necessary libraries.
$ mkdir deploy
# Make requirements.txt
$ pipenv lock -r > requirements.txt
# Install the requirements into the "deploy" directory
$ pip3 install -r requirements.txt --no-deps -t deploy

And now you have libraries from Pipfile into deploy directory.

Gotcha regarding directory structure when you deploy in zip format

So AWS Lambda requires your main python file be at the root level when you unzip the deployed zip file. This is what I did first.

# Let's assume our package name is "main" here
$ cp main.py deploy
$ zip -r deploy.zip deploy

But I kept seeing the error when I deployed the file, that says my main package is missing.

The issue is when you unzip this file, actually main.py belongs to the second top level like this.

$ cp -r deploy.zip /tmp
$ cd /tmp
$ unzip deploy.zip
$ cd deploy
$ ls deploy
deploy
$ ls deploy/deploy
main.py ... ...
# So there is another level of directory!!

So the right way to zip the package is,

$ cp main.py deploy
$ cd deploy
$ zip -r deploy.zip *

Maybe it was clear to you already, but I don’t zip files so often and figuring out the issue took me a while.

psycopg2

So psycopg2 needs to be compiled on Amazon Linux OS on which lambda functions run. You cannot compile it on the fly on Lambda environment, so you have to deploy the compiled one. The solution is to use this library awslambda-psycopg2. You can simply download the library and put the whole necessary directory into the deploy directory as README suggests, UNLESS you need SSL/TLS option on Postgres connection.
When you need SSL/TLS connection, you have to compile pyscopg2 yourself on Amazon Linux Image.

I will update step by step command when I have a change later but this is the procedures you need to take.

  1. Run docker container from this image on your local
  2. Install the “build essentials” for the OS
  3. Install the version of Python you want and on the container. (I had to compile from source).
  4. Download regular psycopg2
  5. Install postgres you need, FOLLOWING THIS SSL INSTRUCTION
  6. Build psycopg2 by following this README
  7. Do docker cp and put the generated directory into our deploy directory. The name needs to be psycopg2.

So they are the gotchas I have seen when I deployed Python functions to AWS Lambda.

Upload script sample

This is not limited to this particular case but when I work on the lambda functions, I found the work flow becomes repetitive. I think it’s useful to have a simple script that does the zip/deployment. My script looks something like this:

#!/bin/bash
pushd deploy
  rm deploy.zip
  zip -r deploy.zip * 
  aws s3 cp deploy.zip s3://my-bucket-path/deploy.zip
popd

You can run it in the directory right above the deploy directory.