Class Batcher

Inheritance Relationships

Derived Types

Class Documentation

class Batcher

The base batcher implementation defines the basic structure of how batchers behave in amdinfer. The run() method is purely virtual and must be implemented by the child classes.

Subclassed by amdinfer::HardBatcher, amdinfer::SoftBatcher

Public Functions

Batcher()

Construct a new Batcher object.

The C++ Interface class encapsulates incoming requests from the C++ API to the batcher.

explicit Batcher(RequestParameters *parameters)
Batcher(const Batcher &batcher)

Construct a new Batcher object.

Parameters

name – the endpoint corresponding to the batcher’s worker group copy constructor

Batcher &operator=(const Batcher&) = delete

Copy assignment constructor.

Batcher(Batcher &&other) = delete

Move constructor.

Batcher &operator=(Batcher &&other) = delete

Move assignment constructor.

virtual ~Batcher() = default

Destructor.

void start(WorkerInfo *worker)

Start the batcher.

Parameters

worker –

void setBatchSize(size_t batch_size)

Set the batch size for the batcher.

Parameters

batch_size – target batch size

void setName(const std::string &name)

Set the name of the batcher (i.e. the batcher’s worker group endpoint)

Parameters

name – the endpoint

std::string getName() const

Get the batcher’s worker group name.

BlockingQueue<InterfacePtr> *getInputQueue()

Get the batcher’s input queue (used to enqueue new requests)

BatchPtrQueue *getOutputQueue()

Get the batcher’s output queue (used to push batches to the worker group)

void run(WorkerInfo *worker)
BatcherStatus getStatus()
void enqueue(InterfacePtr request)

Enqueue a new request to the batcher.

Parameters

request –

void end()

End the batcher.

Protected Attributes

size_t batch_size_ = 1
std::shared_ptr<BlockingQueue<InterfacePtr>> input_queue_
std::shared_ptr<BatchPtrQueue> output_queue_
std::thread thread_
std::string model_
RequestParameters parameters_