Deployment

In this section, you will find information on how to run Palladium-based application with another web server instead of Flask’s built in solution, and how to benchmark your service. Additionally, you will find information on how to use provided scripts to automatically generate Docker images of your service and how to deploy such Docker images using Mesos / Marathon.

Web server installation

Palladium uses HTTP to respond to prediction requests. Through use of the WSGI protocol (via Flask), Palladium can be used together with a variety of web servers.

For convenience, a web server is included for development purposes. To start the built-in web server, use the pld-devserver command.

For production use, you probably want to use something faster and more robust. Many options are listed in the Flask deployment docs. If you follow any of these instructions, be aware that the Flask app in Palladium is available as palladium.wsgi:app. So here’s how you would start an Palladium prediction server using gunicorn:

export PALLADIUM_CONFIG=/path/to/myconfig.py
gunicorn palladium.wsgi:app

An example configuration to use nginx to proxy requests to gunicorn is also available. It can be used without modification for our example and has to be made available in the /etc/nginx/sites-enabled/ folder and is active after a restart of nginx. For convenience it is reprinted here:

server {
    listen 80;

    server_name _;

    access_log  /var/log/nginx/access.log;
    error_log  /var/log/nginx/error.log;

    location / {
        proxy_pass         http://127.0.0.1:8000/;
        proxy_redirect     off;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    }
}

Note

In previous versions (Palladium 1.0.1 and older), the Flask app was accessed via palladium.server:app. This was changed in order to initialize the configuration during start-up.

Benchmarking the service with Apache Benchmark

In order to benchmark the response time of a service, existing tools like Apache Benchmark (ab) or siege can be used. They can be installed using install packages, e.g., for Ubuntu with apt-get install apache2-utils and sudo apt-get install siege.

If a web server with the Iris predict service is running (either using the built-in pld-devserver or a faster solution as described in Web server installation), the ab benchmarking can be run as follows:

ab -n 1000 -c 10 "http://localhost:5000/predict?
                  sepal%20length=5.2&sepal%20width=3.5&
                  petal%20length=1.5&petal%20width=0.2"

In this ab call, it is assumed that the web server is available at port 5000 of localhost and 1000 requests with 10 concurrent requests at a time are sent to the web server. The output provides a number of statistics about response times of the calls performed.

Note

If there is an error in the sample request used, response times might be suspiciously low. If very low response times occur, it might be worth manually checking the corresponding response of used request URL.

Note

ab does not allow to use different URLs in a benchmark run. If different URLs are important for benchmarking, either siege or a multiple URL patch for ab could be used

Building a Docker image with your Palladium application

Building the Palladium base image

Here’s instructions on how to build the Palladium base image. This isn’t usually necessary, as you’ll probably want to just use the released base images for Palladium and add your application on top, see Building a Palladium app image.

A Dockerfile is available in the directory addons/docker/palladium_base_image for building a base image. You can download the file here: Dockerfile.

Run docker build in your terminal:

sudo docker build -t myname/palladium-base:1.1.0.1 .

A Docker image with the name myname/palladium-base:1.1.0.1 should now be created. You can check this with:

sudo docker images

Building a Palladium app image

Palladium has support for quickly building a Docker image to run your own application based on the Palladium base image. The Palladium base image can be pulled from Docker Hub as follows:

docker pull ottogroup/palladium-base

As an example, let’s build a Docker image for the Iris example that’s included in the source. We’ll use the Palladium base image for version 1.0, and we’ll name our own image my-palladium-app. Thus, we invoke pld-dockerize like so:

pld-dockerize palladium-src/examples/iris ottogroup/palladium-base:1.1.0.1 myname/my-palladium-app:1.0

This command will in fact create two images: one that’s called my-palladium-app, another one that’s called my-palladium-app-predict. The latter extends the former by adding calls to automatically fit your model and start a web server.

By default pld-dockerize will create the Dockerfile files and create the Docker containers. You may want to create the Dockerfile files only using the -d flag, and then modify files Dockerfile-app and Dockerfile-predict according to your needs.

Your application’s folder (examples/iris in this case) should look like this:

.
|--- config.py
|--- setup.py (optional)
|--- requirements.txt (optional)
'--- python_packages (optional)
     |--- package1.tar.gz
     |--- package2.tar.gz
     '--- ...

You may put additional requirements as shown into a python_packages subdirectory.

To test your image you can:

  1. Create app images using pld-dockerize as shown above.

  2. Run the “predict” image (e.g., my-palladium-app-predict if you used my-palladium-app to create the image), and map the Docker container’s port 8000 to a local port (e.g., 8001):

    sudo docker run -d -p 8001:8000 my-palladium-app-predict
    
  3. Your application should be up and running now. You should be able to access this URL: http://localhost:8001/alive

Setup Palladium with Mesos / Marathon and Docker

This section describes how to setup Mesos / Marathon with a containerized Palladium application. If you have not built a docker image with your Palladium application yet, you can follow the instructions that are provided in the Building a docker image with your Palladium application section.

For the installation of Mesos and Marathon you can follow the guide on Mesosphere. If you want to try it out locally first, we recommend to set up a single node Mesosphere cluster. Before adding a new application to Marathon you need to make sure that the Mesos slaves and Marathon are configured properly to work with Docker. To do so, follow the steps as described in the Marathon documentation.

An easy way to add a new application to Marathon is to use its REST API. For this task you need a json file which contains the relevant information for Marathon. A basic example of the json file could look like this:

{
    "id": "<app_name>",
    "container": {
        "docker": {
            "image": "<owner/palladium-app-name:version>",
            "network": "BRIDGE",
            "parameters": [
                {"key": "link", "value":"<some_container_to_link>"}
            ],
            "portMappings": [
                { "containerPort": 8000, "hostPort": 0, "servicePort": 9000,
                  "protocol": "tcp" }
            ]
        },
        "type": "DOCKER",
        "volumes": [
            {
                "containerPath": "/path/in/your/container",
                "hostPath": "/host/path",
                "mode": "RO"
            }
        ]
    },
    "cpus": 0.2,
    "mem": 256.0,
    "instances": 3,
    "healthChecks": [
        {
            "protocol": "HTTP",
            "portIndex": 0,
            "path": "/alive",
            "gracePeriodSeconds": 5,
            "intervalSeconds": 20,
            "maxConsecutiveFailures": 3
        }
    ],
    "upgradeStrategy": {
        "minimumHealthCapacity": 0.5
    }
}

You have to replace the Docker image name, port number (currently set to 8000) and - if there is any dependency - specify links to other containers. If you have a Docker image of the Iris service available (named user/palladium-iris-predict:0.1), you can use this file:

{
  "id": "palladium-iris",
    "container": {
        "docker": {
            "image": "user/palladium-iris-predict:0.1",
            "network": "BRIDGE",
            "parameters": [
            ],
            "portMappings": [
                { "containerPort": 8000, "hostPort": 0, "servicePort": 9000,
                  "protocol": "tcp" }
            ]
        },
        "type": "DOCKER",
        "volumes": [
         ]
    },
    "cpus": 0.2,
    "mem": 256.0,
    "instances": 3,
    "healthChecks": [
        {
            "protocol": "HTTP",
            "portIndex": 0,
            "path": "/alive",
            "gracePeriodSeconds": 5,
            "intervalSeconds": 20,
            "maxConsecutiveFailures": 3
        }
    ],
    "upgradeStrategy": {
        "minimumHealthCapacity": 0.5
    }
}

Now you can send the json application file to Marathon via POST (assuming Marathon is available at localhost:8080:

curl -X POST -H "Content-Type: application/json" localhost:8080/v2/apps
     -d @<path-to-json-file>

You can see the status of your Palladium service instances using the Marathon web user interface (available at http://localhost:8080 if you run the single node installation mentioned above) and can scale the number of instances as desired. Marathon keeps track of the Palladium instances. If a service instance breaks down, a new one will be started automatically.

Authorization

Sometimes you will want the Palladium web service’s entry points /predict and /alive to be secured by OAuth2 or similar. Defining predict_decorators and alive_decorators in the Palladium configuration file allows you to put any decorators in place to check authentication.

Let us first consider an example where you want to use HTTP Basic Auth to guard the entry points. Consider this code taken from the Flask snippets repository:

# file: mybasicauth.py

from functools import wraps
from flask import request, Response


def check_auth(username, password):
    """This function is called to check if a username /
    password combination is valid.
    """
    return username == 'admin' and password == 'secret'

def authenticate():
    """Sends a 401 response that enables basic auth"""
    return Response(
    'Could not verify your access level for that URL.\n'
    'You have to login with proper credentials', 401,
    {'WWW-Authenticate': 'Basic realm="Login Required"'})

def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = request.authorization
        if not auth or not check_auth(auth.username, auth.password):
            return authenticate()
        return f(*args, **kwargs)
    return decorated

The requires_auth can now be used to decorate Flask views to guard them with basic authentication. Palladium allows us to add decorators to the /predict and /alive views that it defines itself. To do this, we only need to add this bit to the Palladium configuration file:

'predict_decorators': [
    'mybasicauth.requires_auth',
    ],

'alive_decorators': [
    'mybasicauth.requires_auth',
    ],

Of course, alternatively, you could set up your mod_wsgi server to take care of authentication.

Using Flask-OAuthlib to guard the two views using OAuth2 follows the same pattern. We will configure and use the flask_oauthlib.provider.OAuth2Provider for security. In our own package, we might have an instance of OAuth2Provider and a require_oauth decorator defined thus:

# file: myoauth.py

from flask_oauthlib.provider import OAuth2Provider
from palladium.server import app


oauth = OAuth2Provider(app)

# more setup code here... see Flask-OAuthlib

require_oauth = oauth.require_oauth('myrealm')

Alternatively, to get more decoupling from Palladium’s Flask app, you can use the following snippet inside your Palladium configuration and assign the Flask app to OAuth2Provider at application startup:

'oauth_init_app': {
    '__factory__': 'myoauth.oauth.init_app',
    'app': 'palladium.server.app',
    },

Now, to guard, /predict and /alive with the previously defined require_oauth, add this to your configuration:

'predict_decorators': [
    'myoauth.require_oauth'
    ],

'alive_decorators': [
    'myoauth.require_oauth'
    ],