Hello World (Python)

Xilinx Inference Server’s Python API allows you to start the server and send requests to it from Python. This example walks you through how to start the server and use Python to send requests to a simple model. The complete script used here is available: examples/python/hello_world_rest.py.

Import the library

We need to bring in the Xilinx Inference Server Python library. The library’s source code is in src/python and it gets installed in the dev container on startup.

import proteus

Create our client and server objects

We assume that the server will be running locally with the default HTTP port and pass that to the client. In this example, we’ll be using REST to communicate to the server so we create a RestClient object.

server = proteus.Server()
client = proteus.RestClient("127.0.0.1:8998")

Is Xilinx Inference Server already running?

If the server is already started externally, we don’t want to start it again. So, we attempt to check if the server is live. If this fails, we start the server ourselves. Either way, the client will attempt to communicate to the server at http://localhost:8998.

try:
    start_server = not client.server_live()
except proteus.ConnectionError:
    start_server = True
if start_server:
    server.start(quiet=True)
    client.wait_until_live()

Load a worker

Inference requests in Xilinx Inference Server are made to workers. Workers are started as threads in Xilinx Inference Server and have a defined lifecycle. Before making an inference request to a worker, it must first be loaded. Loading a worker returns an identifier that the client should use for future operations.

This worker, Echo, is a simple example worker that accepts integers as input, adds one to the inputs and returns the sum. We’ll use it to demonstrate the Python flow.

response = client.load("Echo")
assert not response.error, response.error_msg
worker_name = response.html

while not client.model_ready(worker_name):
    pass

Inference

Once the worker is ready, we can make an inference request to it. We construct a request that contains five integers and send it to Xilinx Inference Server. The NumericalInferenceRequest class is a helper class that simplifies creating a request in the right format.

data = [3, 1, 4, 1, 5]
request = proteus.NumericalInferenceRequest(data)
response = client.infer(worker_name, request)

Validate the response

Now, we want to check what our response is. Here, we can simplify our checks because we already know what we expect to receive. So we check is that the number of outputs match the number of inputs we used. We also check that each output only has one index and is one more than the corresponding input since we know that’s what the Echo worker does.

assert not response.error, response.error_msg
assert len(response.outputs) == len(data)
for index, output in enumerate(response.outputs):
    assert len(output.data) == 1
    assert output.data[0] == data[index] + 1

Clean up

Workers that are loaded in Xilinx Inference Server will persist until the server shuts down or they’re explicitly unloaded. While it’s not shown here, the Python API provides an unload() method for this purpose. Finally, if we started the server from Python, we shut it down before finishing. If there are any loaded workers at this time, they will be cleaned up before shutdown.

if start_server:
    server.stop()
    client.wait_until_stop()