VVAS Meta Data Structures

This section covers various meta data structures defined by VVAS.

VVAS Inference Metadata

The VVAS inference metadata (GstInferenceMeta) object holds information about the metadata produced by the machine learning (ML) inference acceleration software libraries. This metadata structure stores the inferred data generated at multiple levels of ML operation into a single structure in a hierarchical way. This data structure combines, and stores metadata generated by detection and classification models.

The GStreamer plug-ins can set and get inference metadata from the GstBuffer by using the gst_buffer_add_meta () API and gst_buffer_get_meta () API, respectively.

GstInferenceMeta

GstInferenceMeta is the root node of the inference metadata. GstInferencePrediction hosts the actual inference metadata.

/**
* Implements the placeholder for inference information.
*/
typedef struct _GstInferenceMeta GstInferenceMeta;
struct _GstInferenceMeta
{
GstMeta meta;
GstInferencePrediction *prediction;
gchar *stream_id;
};

GstInferencePrediction

This structure represents the inference data generated from the ML operation. BBox represents the bounding box around the detected object from detection model. The classification parameter of this structure stores a list of classification information associated with the detected object. This structure also stores the results of the next level of ML operation performed (typically in cascaded network based ML operation).

typedef struct _VvasColorMetadata {
  uint8_t red;
  uint8_t green;
  uint8_t blue;
  uint8_t alpha;
} VvasColorMetadata;

struct _BoundingBox
{
  gint x;
  gint y;
  guint width;
  guint height;
  VvasColorMetadata box_color;
};

typedef struct _BoundingBox BoundingBox;

/**
* GstInferencePrediction:
* @prediction_id: unique id for this specific prediction
* @enabled: flag indicating whether or not this prediction should be
* used for further inference
* @bbox: the BoundingBox for this specific prediction
* @classifications: a linked list of GstInfereferenceClassification
* associated to this prediction
* @predictions: a n-ary tree of child predictions within this
* specific prediction. It is recommended to access the tree
* directly, but to use this module's API to interact with the
* children.
* @sub_buffer: A buffer created from the main buffer.
* @bbox_scaled: bbox co-ordinates scaled to root node resolution or not
* @segmentation: contains output of segmentation model wrapped in GstBuffer
* @obj_track_label: This is currently unused, kept for future use
* Abstraction that represents a prediction
*/
struct _GstInferencePrediction
{
   /*<private>*/
   GstMiniObject base;
   GMutex mutex;
   /*<public>*/
   guint64 prediction_id;
   gboolean enabled;
   BoundingBox bbox;
   GList * classifications;
   GNode * predictions;
   GstBuffer *sub_buffer;
   gboolean bbox_scaled; /* bbox co-ordinates scaled to root node resolution or not */
   Segmentation segmentation;
   gchar *obj_track_label;
   /* for future extension */
   void * reserved_1;
   void * reserved_2;
   void * reserved_3;
   void * reserved_4;
   void * reserved_5;
};

GstInferenceClassification

This structure stores the results of the ML operation by the classification network.

/**
* GstInferenceClassification:
* @classification_id: a unique id associated to this classification
* @class_id: the numerical id associated to the assigned class
* @class_prob: the resulting probability of the assigned
* class. Typically, between 0 and 1
* @class_label: the label associated to this class or NULL if not
* available
* @num_classes: the amount of classes of the entire prediction
* @probabilities: the entire array of probabilities of the prediction
* @labels: the entire array of labels of the prediction or NULL if
* not available
*/
typedef struct _GstInferenceClassification GstInferenceClassification;
struct _GstInferenceClassification
{
   /*<private>*/
   GstMiniObject base;
   GMutex mutex;
   /*<public>*/
   guint64 classification_id;
   gint class_id;
   gdouble class_prob;
   gchar *class_label;
   gint num_classes;
   gdouble *probabilities;
   gchar **labels;
   VvasColorMetadata label_color;
};

VVAS Overlay Metadata

VVAS overlay metadata structure hold the information of geometric shapes and text need to be overlaid on video frames. VVAS overlay plugin parses the overlay metadata structures to overlay information on the frames. An intermediate plugin is required for converting metadata generated from upstream plugins like infer, segmentation or optical flow plugins to overlay metadata for displaying information on frames. Currently supported structures in gstvvasoverlaymeta are rectangles, text, lines, arrows, circles and polygons. Maximum number of structures of any geometric shape or text that can be draw on frames are 16 (VVAS_MAX_OVERLAY_DATA 16). For displaying text, text need to be display must be copied into the text structure.

The GStreamer plug-ins can set and get overlay metadata from the GstBuffer by using the gst_buffer_add_meta () API and gst_buffer_get_meta () API, respectively.

GstOverlayMeta

GstOverlayMeta structure stores the information of different geometric structures and text.

typedef enum
{
  AT_START,
  AT_END,
  BOTH_ENDS,
} vvas_arrow_direction;

struct _vvas_pt
{
  int x;
  int y;
};

struct _vvas_font_params
{
  int font_num;
  float font_size;
  VvasColorMetadata font_color;
};

struct _vvas_rect_params
{
  vvas_pt offset;
  int width;
  int height;
  int thickness;
  VvasColorMetadata rect_color;
  int apply_bg_color;
  VvasColorMetadata bg_color;
};

struct _vvas_text_params
{
  vvas_pt offset;
  char disp_text[VVAS_MAX_TEXT_SIZE];
  int bottom_left_origin;
  vvas_font_params text_font;
  int apply_bg_color;
  VvasColorMetadata bg_color;
};

struct _vvas_line_params
{
  vvas_pt start_pt;
  vvas_pt end_pt;
  int thickness;
  VvasColorMetadata line_color;
};

struct _vvas_arrow_params
{
  vvas_pt start_pt;
  vvas_pt end_pt;
  vvas_arrow_direction arrow_direction;
  int thickness;
  float tipLength;
  VvasColorMetadata line_color;
};

struct _vvas_circle_params
{
  vvas_pt center_pt;
  int radius;
  int thickness;
  VvasColorMetadata circle_color;
};

struct _vvas_polygon_params
{
  vvas_pt poly_pts[VVAS_MAX_OVERLAY_POLYGON_POINTS];
  int num_pts;
  int thickness;
  VvasColorMetadata poly_color;
};

/**
* GstVvasOverlayMeta:
* @num_rects: number of bounding boxes
* @num_text: number of text boxes
* @num_lines: number of lines
* @num_arrows: number of arrows
* @num_circles: number of circles
* @num_polys: number of polygons
* @vvas_rect_params: structure for holding rectangles information
* @vvas_text_params: structure for holding text information
* @vvas_line_params: structure for holding lines information
* @vvas_arrow_params: structure for holding arrows information
* @vvas_circle_params: structure for holding circles information
* @vvas_polygon_params: structure for holding polygons information
*/

struct _GstVvasOverlayMeta {
  GstMeta meta;
  int num_rects;
  int num_text;
  int num_lines;
  int num_arrows;
  int num_circles;
  int num_polys;

  vvas_rect_params rects[VVAS_MAX_OVERLAY_DATA];
  vvas_text_params text[VVAS_MAX_OVERLAY_DATA];
  vvas_line_params lines[VVAS_MAX_OVERLAY_DATA];
  vvas_arrow_params arrows[VVAS_MAX_OVERLAY_DATA];
  vvas_circle_params circles[VVAS_MAX_OVERLAY_DATA];
  vvas_polygon_params polygons[VVAS_MAX_OVERLAY_DATA];
};

VVAS Opticalflow Metadata

VVAS optical flow metadata structure hold the information of motion of frame in x and y direction and object motion information. VVAS optical flow plugin set the optical flow meta data of frame. This metadata structure also supports storing of motion information in object level for further analysis by downstream plugins.

The GStreamer plug-ins can set and get optical flow metadata from the GstBuffer by using the gst_buffer_add_meta () API and gst_buffer_get_meta () API, respectively.VVAS optical flow metadata structure hold the information of motion of frame in x and y direction and object motion information. VVAS optical flow plugin set the optical flow meta data of frame. This metadata structure also supports storing of motion information in object level for further analysis by downstream plugins.

The GStreamer plug-ins can set and get optical flow metadata from the GstBuffer by using the gst_buffer_add_meta () API and gst_buffer_get_meta () API, respectively.

GstOptflowMeta

GstOptflowMeta stores the information of optical flow of frames and object motion information.

struct _vvas_obj_motinfo
{
  float mean_x_displ;
  float mean_y_displ;
  float angle;
  float dist;
  char dirc_name[DIR_NAME_SZ];
  BoundingBox bbox;
};


/**
* GstVvasOverlayMeta:
* @num_objs: number of objects with motion information
* @obj_mot_infos: list of objects
* @x_displ: pointer to motion data of frame in x-direction
* @y_displ: pointer to motion data of frame in y-directiont
*/
struct _GstVvasOFMeta
{
  GstMeta meta;

  guint num_objs;
  GList *obj_mot_infos;

  GstBuffer *x_displ;
  GstBuffer *y_displ;
};