Serverless Python profiler for AWS Lambda using AWS X-Ray

Last week at AWS re:Invest 2017, I decided to give a try to AWS X-Ray, the AWS Distributed tracing system. This service has been announced a year ago at AWS re:Invest 2016 as a Preview. At that time, the tracing SDK was available for Node.js, Java and C# .NET. The AWS X-Ray service is GA (General Availability) since April 2017 and the AWS X-Ray Python SDK is available since August 2017.

As a first try at AWS X-Ray, I created a small Flask app that is deployed on AWS with Zappa. To enable AWS X-Ray on the Lambda function, you just have to add the Zappa setting "xray_tracing": true to the zappa_settings.json.

{
    "test": {
        "app_function": "app.app",
        "aws_region": "us-east-1",
        "memory_size": 128,
        "project_name": "aws-xray-example",
        "runtime": "python3.6",
        "s3_bucket": "aws-xray-example",
        "xray_tracing": true
    }
}

AWS X-Ray Daemon

If you are using AWS X-Ray SDK on a non-lambda function, you first need to start the AWS X-Ray daemon. Then, when using the SDK, the tracing functions will automatically send UDP packets to that daemon. This is an asynchronous fire and forget pattern. By using this pattern, if the AWS X-Ray daemon or the AWS X-Ray API are down, this will not delay your application. This allows you to use AWS X-Ray from anywhere (on AWS, another Cloud or your own servers) without service degradation but keep in mind that if you have packet loss between the daemon and your application, you might lost some traces.

Once X-Ray is enabled on AWS Lambda (through Zappa settings or manually), the AWS X-Ray daemon will automatically start. The daemon is located in the Lambda image and is using 16 MB of memory (or 3 percent, which one is greater). This will automatically send traces to AWS X-Ray console but you will not have any details on your Python application.

Initial application

You will find below a simple Hello World Flask application.

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello world!', 200

To deploy this application on AWS, save the code above in app.py and the Zappa settings in zappa_settings.json. Then, run:

### Install all required librairies ###
$ pip install pipenv
$ pipenv install aws_xray_sdk flask zappa

### To deploy on AWS: ###
$ pipenv shell
$ zappa deploy test

### To undeploy from AWS: ###
$ pipenv shell
$ zappa undeploy test

Once you access the function, you will have a first trace. In this application trace, you will see the lambda runtime and your function.

Manual traces

Now that you can send traces to X-Ray, it's time to add code to your application to send more information.

from aws_xray_sdk.core import xray_recorder

@app.route('/manual_traces')
def manual_traces():
    xray_recorder.begin_subsegment('manual_traces')
    sleep(0.1)
    xray_recorder.begin_subsegment('inside trace')
    sleep(0.3)
    xray_recorder.end_subsegment()
    xray_recorder.end_subsegment()
    return 'Manual traces completed', 200

Note: AWS X-Ray for Lambda only supports Subsegment because the Segment is the Lambda itself. With this limitation, some metadata are not available (more details in Issue #2). For more details on Segment and Subsegment, check out AWS X-Ray Concepts.

Decorator-based subsegment

In the SDK, there is a native way to decorate your functions to capture them as Subsegment.

@xray_recorder.capture('my_function')
def my_function():
    sleep(0.2)

@app.route('/decorator')
def decorator():
    my_function()
    my_function()
    return 'Decorator-based subsegment', 200

Patch third-party libraries

To add tracing inside third-party libraries, it's very easy to patch them with the SDK patching commands.

import boto3
import botocore
import requests
import sqlite3

from aws_xray_sdk.core import patch_all

patch_all()

For more details on third-party libraries patching read Patching Libraries to Instrument Downstream Calls in the AWS Developer Guide.

Conclusion

AWS X-Ray Distributed tracing system allows you to trace and profile your applications. This is useful for both native or serverless applications. You can find more information in the AWS X-Ray Developer Guide.

The AWS X-Ray Python SDK is Opensource and available on Github.

This is my first codementor post. Don't hesitate if you have any questions or comments on this post.