Class InferenceRequest

Class Documentation

class InferenceRequest

Creates an inference request object based on KServe’s V2 spec that is used to communicate between workers.

Public Functions

InferenceRequest() = default
void setCallback(Callback &&callback)

Sets the request’s callback function used by the last worker to respond back to the client.

Parameters

callback – a function pointer that accepts a InferenceResponse object

void runCallback(const InferenceResponse &response)

Runs the request’s callback function.

Parameters

response – the response data

void runCallbackOnce(const InferenceResponse &response)

Runs the request’s callback function and clear it after. This prevents calling the callback multiple times. If this function is called again, it’s a no-op.

Parameters

response – the response data

void runCallbackError(std::string_view error_msg)

Runs the request’s callback function with an error response. The callback function is not cleared.

Parameters

error_msg – error message to send back to the client

void addInputTensor(void *data, const std::vector<uint64_t> &shape, DataType data_type, const std::string &name = "")

Constructs and adds a new input tensor to this request.

Parameters

  • data – pointer to data to add

  • shape – shape of the data

  • data_type – the datatype of the data

  • name – the name of the input tensor

void addInputTensor(InferenceRequestInput input)

Adds a new input tensor to this request.

Parameters

input – an existing InferenceRequestInput object

void addOutputTensor(const InferenceRequestOutput &output)

Adds a new output tensor to this request.

Parameters

output – an existing InferenceRequestOutput object

const std::vector<InferenceRequestInput> &getInputs() const

Gets a vector of all the input request objects.

size_t getInputSize() const

Get the number of input request objects.

const std::vector<InferenceRequestOutput> &getOutputs() const

Gets a vector of the requested output information.

inline const std::string &getID() const

Gets the ID associated with this request.

Returns

std::string

inline void setID(std::string_view id)

Sets the ID associated with this request.

Parameters

id – ID to set

inline RequestParameters *getParameters() const

Get a pointer to the request’s parameters.

inline void setParameters(RequestParametersPtr parameters)

Sets the parameters for the request.

Parameters

parameters – pointer to the parameters