Hello World - Python¶

AMD Inference Server’s Python API allows you to start the server and send requests to it from Python. This example walks you through how to start the server and use Python to send requests to a simple model. The complete script used here is available in the repository.

Import the library¶

You can install the Python client library with pip:

$ pip install amdinfer

In the development container, the Python library is automatically built and installed as part of the CMake build process.

import numpy as np

import amdinfer

Create our client and server objects¶

This example assumes that the server will be running locally with the default HTTP port. Since you’ll be using HTTP/REST to communicate with the server, you should create an HttpClient using the address of the server as the argument.

client = amdinfer.HttpClient("http://127.0.0.1:8998")

Is AMD Inference Server already running?¶

If the server is already started externally, you don’t want to start it again. You can see if the server is live using the client object. If it’s not live, then you can start the server from Python by instantiating the Server object. Depending on what protocol you’re using to communicate with the server, that protocol may need to be started on the server. The server will remain active as long as the server object stays in scope.

start_server = not client.serverLive()
if start_server:
    print("No server detected. Starting locally...")
    server = amdinfer.Server()
    server.startHttp(8998)
amdinfer.waitUntilServerReady(client)

Load a worker¶

Inference requests in AMD Inference Server are made to workers. Workers are started as threads in AMD Inference Server and have a defined lifecycle. Before making an inference request to a worker, it must first be loaded. Loading a worker returns an endpoint that the client should use for future operations.

This simple worker, “cplusplus”, is a worker that accepts a C++-based “model” which is specified in the parameters. This model, “echo” is a simple test model that accepts an integer as input, adds one to it and returns the sum.

parameters = amdinfer.ParameterMap()
parameters.put("model", "echo")
endpoint = client.workerLoad(client, "cplusplus", parameters)
amdinfer.waitUntilModelReady(client, endpoint)

Inference¶

Once the worker is ready, you can make an inference request to it. To do so, first construct a request. We construct a request that contains an integer and send it to the server.

data = 3
request = make_request(data)
response = client.modelInfer(endpoint, request)

To make a request, you need to create, at minimum, a amdinfer.InferenceRequest object and add amdinfer.InferenceRequestInput objects to it. Each input object represents an input tensor for the request and has a number of metadata attributes associated with it such as a name, datatype, and shape. This format is based on KServe’s v2 specification. For images, there’s also a helper method called amdinfer.ImageInferenceRequest that you can use to create requests. It’s used in the ResNet50 Python examples.

def make_request(data):
    """
    Make a request containing an integer

    Args:
        data (int): Data to send

    Returns:
        amdinfer.InferenceRequest: Request
    """
    request = amdinfer.InferenceRequest()
    # each request has one or more input tensors, depending on the worker/model that's going to process it
    input_0 = amdinfer.InferenceRequestInput()
    input_0.name = "input0"
    input_0.setUint32Data(np.array([data], np.uint32))
    input_0.datatype = amdinfer.DataType.UINT32
    input_0.shape = [1]
    request.addInputTensor(input_0)
    return request

Validate the response¶

After making the inference, you should check what the response was. First, make sure the inference didn’t fail by checking if the response is erroneous. Then, you can get the outputs and examine them. The format and contents of the output data will depend on the worker and model used for inference. In this case, the worker returns a single output tensor back with one integer that should be one larger than what was sent. The assertions here check these cases.

assert not response.isError(), response.getError()
outputs = response.getOutputs()
assert len(outputs) == 1
for output in outputs:
    recv_data = output.getUint32Data()
    assert len(recv_data) == 1
    assert recv_data[0] == data + 1

Clean up¶

Workers that are loaded in AMD Inference Server will persist until the server shuts down or they’re explicitly unloaded. As the script ends, the Server object’s destructor will clean up any active workers on the server and it will shut down

client.workerUnload(endpoint)

Next steps¶

This example demonstrates the complete process from starting the server, creating a client, loading a worker, and making an inference. Depending on your use case, you may only be performing a subset of these actions but this provides an overview of what’s happening behind the scenes. You can look at more sophisticated examples for another overview of the steps associated with making inferences.