Host Runtime (`aie.utils.hostruntime`)¶

Host-side helpers for selecting a device, allocating NPU-accessible tensors, running compiled designs, and sharing common CLI / argument plumbing across programming examples.

Import the public surface from aie.utils.hostruntime (or via the aie.utils / iron re-exports where noted). The symbols below are rendered from the Python package under python/utils/hostruntime/ using the same mkdocstrings path layout as the other Python API pages (utils.*, not the installed aie.utils.* import name).

Runtime abstractions¶

Abstract runtime types that concrete backends (for example XRT) implement.

HostRuntimeError ¶

Bases: Exception

Error raised when a NPU kernel encounters an error during runtime operations.

KernelHandle ¶

Bases: ABC

Abstract representation that represents a kernel already registered/loaded with a runtime.

KernelResult ¶

KernelResult(
    npu_time: int, trace_config: TraceConfig | None = None
)

Bases: ABC

A wrapper around data produced as the result of running a kernel

Initialize the KernelResult.

Parameters:

Name	Type	Description	Default
`npu_time`	`int`	The execution time on the NPU in nanoseconds.	required
`trace_config`	`TraceConfig \| None`	Configuration for tracing. Defaults to None.	`None`

Source code in python/utils/hostruntime/hostruntime.py

def __init__(
    self,
    npu_time: int,
    trace_config: TraceConfig | None = None,
):
    """
    Initialize the KernelResult.

    Args:
        npu_time (int): The execution time on the NPU in nanoseconds.
        trace_config (TraceConfig | None, optional): Configuration for tracing. Defaults to None.
    """
    self._npu_time = npu_time
    self._trace_config = trace_config

npu_time `property` ¶

npu_time: int

Get the NPU execution time.

Returns:

Name	Type	Description
`int`	`int`	The execution time in nanoseconds.

trace_config `property` ¶

trace_config: TraceConfig | None

Get the trace configuration.

Returns:

Type	Description
`TraceConfig \| None`	TraceConfig \| None: The trace configuration if available, else None.

has_trace ¶

has_trace() -> bool

Check if trace data is available.

Returns:

Name	Type	Description
`bool`	`bool`	True if trace configuration is present, False otherwise.

Source code in python/utils/hostruntime/hostruntime.py

def has_trace(self) -> bool:
    """
    Check if trace data is available.

    Returns:
        bool: True if trace configuration is present, False otherwise.
    """
    return not (self._trace_config is None)

is_success `abstractmethod` ¶

is_success() -> bool

Check if the kernel execution was successful.

Returns:

Name	Type	Description
`bool`	`bool`	True if successful, False otherwise.

Source code in python/utils/hostruntime/hostruntime.py

@abstractmethod
def is_success(self) -> bool:
    """
    Check if the kernel execution was successful.

    Returns:
        bool: True if successful, False otherwise.
    """
    pass

HostRuntime ¶

Bases: ABC

An abstract class for a generic host runtime

check_device_consistency ¶

check_device_consistency()

Check if the overridden device is loadable on the runtime device.

A 1- or N-column variant of a generation (e.g. NPU1Col1) is loadable on a wider device of the same generation (e.g. a 4-column NPU1), so we accept any override whose arch matches and whose column count is <= the runtime device's column count.

Source code in python/utils/hostruntime/hostruntime.py

def check_device_consistency(self):
    """
    Check if the overridden device is loadable on the runtime device.

    A 1- or N-column variant of a generation (e.g. NPU1Col1) is loadable
    on a wider device of the same generation (e.g. a 4-column NPU1), so we
    accept any override whose arch matches and whose column count is <=
    the runtime device's column count.
    """
    assert __package__ is not None
    mod = sys.modules[__package__]
    override = getattr(mod, "_CURRENT_DEVICE", None)
    if override is None:
        return
    runtime_device = self.device()
    try:
        same_arch = override.arch == runtime_device.arch
        fits = override.cols <= runtime_device.cols
    except AttributeError:
        same_arch = fits = False
    if not (same_arch and fits):
        raise RuntimeError(
            f"Overridden device {override} is not loadable on runtime "
            f"device {runtime_device}"
        )

cleanup ¶

cleanup() -> None

Release any cached device/runtime resources held by this runtime.

Base implementation is a no-op: a plain runtime holds nothing to release. Caching runtimes override this to free hardware contexts, loaded executables, instruction buffers, etc. Safe to call even if the runtime never ran anything.

Source code in python/utils/hostruntime/hostruntime.py

def cleanup(self) -> None:
    """Release any cached device/runtime resources held by this runtime.

    Base implementation is a no-op: a plain runtime holds nothing to
    release. Caching runtimes override this to free hardware contexts,
    loaded executables, instruction buffers, etc. Safe to call even if the
    runtime never ran anything.
    """
    return

evict_context ¶

evict_context(xclbin_path: Path) -> None

Drop any cached device context associated with xclbin_path.

Recovery hook invoked after the driver rejects a submit against a stale context (e.g. an XRT IOCTL EINVAL) so the next load rebuilds a fresh context. Base implementation is a no-op for runtimes that keep no evictable context cache.

Source code in python/utils/hostruntime/hostruntime.py

def evict_context(self, xclbin_path: Path) -> None:
    """Drop any cached device context associated with ``xclbin_path``.

    Recovery hook invoked after the driver rejects a submit against a stale
    context (e.g. an XRT IOCTL EINVAL) so the next ``load`` rebuilds a fresh
    context. Base implementation is a no-op for runtimes that keep no
    evictable context cache.
    """
    return

load `abstractmethod` ¶

load(npu_kernel: NPUKernel, **kwargs) -> KernelHandle

Load an NPU kernel into the runtime.

Parameters:

Name	Type	Description	Default
`npu_kernel`	`NPUKernel`	The NPU kernel to load.	required
`**kwargs`		Additional arguments for loading.	`{}`

Returns:

Name	Type	Description
`KernelHandle`	`KernelHandle`	A handle to the loaded kernel.

Source code in python/utils/hostruntime/hostruntime.py

@abstractmethod
def load(self, npu_kernel: NPUKernel, **kwargs) -> KernelHandle:
    """
    Load an NPU kernel into the runtime.

    Args:
        npu_kernel (NPUKernel): The NPU kernel to load.
        **kwargs: Additional arguments for loading.

    Returns:
        KernelHandle: A handle to the loaded kernel.
    """
    pass

run `abstractmethod` ¶

run(
    kernel_handle: KernelHandle,
    args,
    trace_config: TraceConfig | None = None,
    fail_on_error: bool = True,
    only_if_loaded=False,
    **kwargs
) -> KernelResult

Run a loaded kernel.

Parameters:

Name	Type	Description	Default
`kernel_handle`	`KernelHandle`	The handle to the loaded kernel.	required
`args`		Arguments to pass to the kernel.	required
`trace_config`	`TraceConfig \| None`	Configuration for tracing. Defaults to None.	`None`
`fail_on_error`	`bool`	Whether to raise an exception on kernel failure. Defaults to True.	`True`
`only_if_loaded`	`bool`	If True, only run if already loaded. Defaults to False.	`False`
`**kwargs`		Additional arguments.	`{}`

Returns:

Name	Type	Description
`KernelResult`	`KernelResult`	The result of the kernel execution.

Source code in python/utils/hostruntime/hostruntime.py

@abstractmethod
def run(
    self,
    kernel_handle: KernelHandle,
    args,
    trace_config: TraceConfig | None = None,
    fail_on_error: bool = True,
    only_if_loaded=False,
    **kwargs,
) -> KernelResult:
    """
    Run a loaded kernel.

    Args:
        kernel_handle (KernelHandle): The handle to the loaded kernel.
        args: Arguments to pass to the kernel.
        trace_config (TraceConfig | None, optional): Configuration for tracing. Defaults to None.
        fail_on_error (bool, optional): Whether to raise an exception on kernel failure. Defaults to True.
        only_if_loaded (bool, optional): If True, only run if already loaded. Defaults to False.
        **kwargs: Additional arguments.

    Returns:
        KernelResult: The result of the kernel execution.
    """
    pass

load_and_run ¶

load_and_run(
    npu_kernel: NPUKernel, run_args: list, **kwargs
) -> tuple[KernelHandle, KernelResult]

Load and run an NPU kernel.

Parameters:

Name	Type	Description	Default
`npu_kernel`	`NPUKernel`	The NPU kernel to load and run.	required
`run_args`	`list`	Arguments to pass to the kernel.	required
`**kwargs`		Additional arguments passed to load.	`{}`

Returns:

Type	Description
`tuple[KernelHandle, KernelResult]`	tuple[KernelHandle, KernelResult]: A tuple containing the kernel handle and the execution result.

Source code in python/utils/hostruntime/hostruntime.py

def load_and_run(
    self,
    npu_kernel: NPUKernel,
    run_args: list,
    **kwargs,
) -> tuple[KernelHandle, KernelResult]:
    """
    Load and run an NPU kernel.

    Args:
        npu_kernel (NPUKernel): The NPU kernel to load and run.
        run_args (list): Arguments to pass to the kernel.
        **kwargs: Additional arguments passed to load.

    Returns:
        tuple[KernelHandle, KernelResult]: A tuple containing the kernel handle and the execution result.
    """
    trace_config = npu_kernel.trace_config
    handle = self.load(npu_kernel, **kwargs)
    if trace_config:
        if trace_config.reuse_output_buffer and len(run_args) > 0:
            trace_config.last_tensor_shape = run_args[-1].shape
            trace_config.last_tensor_dtype = np.dtype(run_args[-1].dtype)
        self.prepare_args_for_trace(run_args, trace_config)

        # Passing a trace_config to a design that never called enable_trace
        # means the lowering appended no trace operand, yet the host just
        # appended a trace buffer above. The extra buffer has no matching
        # runtime_sequence operand and would run with an empty trace (or,
        # before the firmware-ABI floor over-declared kernels.json, segfault
        # in XRT argument setup). Compare against the design's true operand
        # count -- floor-independent, unlike the kernels.json boN slot count.
        num_host_bos = npu_kernel.num_host_bos
        if num_host_bos is not None and len(run_args) > num_host_bos:
            raise HostRuntimeError(
                f"A trace_config was supplied but the compiled design has "
                f"{num_host_bos} host buffer argument(s), while running with "
                f"a trace buffer requires {len(run_args)}. The design must "
                f"call enable_trace(...) so trace lowering appends a trace "
                f"buffer operand; otherwise the trace buffer has nowhere to "
                f"land."
            )

    ret = self.run(handle, list(run_args), trace_config=trace_config)

    if trace_config:
        trace_buffer, ctrl_buffer = self.extract_trace_from_args(
            run_args, trace_config
        )
        self.process_trace(trace_buffer, ctrl_buffer, trace_config)

    return handle, ret

device `abstractmethod` ¶

device() -> Device

Get the device associated with this runtime.

Returns:

Name	Type	Description
`Device`	`Device`	The device object.

Source code in python/utils/hostruntime/hostruntime.py

@abstractmethod
def device(self) -> "Device":
    """
    Get the device associated with this runtime.

    Returns:
        Device: The device object.
    """
    pass

read_insts_binary `classmethod` ¶

read_insts_binary(insts_path: Path)

Reads instructions from a binary file.

Parameters:

Name	Type	Description	Default
`insts_path`	`Path`	Path to the binary instruction file.	required

Returns:

Type	Description
	np.ndarray: Array of uint32 instructions.

Source code in python/utils/hostruntime/hostruntime.py

@classmethod
def read_insts_binary(cls, insts_path: Path):
    """
    Reads instructions from a binary file.

    Args:
        insts_path (Path): Path to the binary instruction file.

    Returns:
        np.ndarray: Array of uint32 instructions.
    """
    with open(insts_path, "rb") as f:
        data = f.read()
    # Interpret the binary data as an array of uint32 values.
    return np.frombuffer(data, dtype=np.uint32)

read_insts `classmethod` ¶

read_insts(insts_path: Path)

Reads instructions from the given file.

If the file extension is .bin, uses binary read. If the file extension is .txt, uses sequence (text) read.

Parameters:

Name	Type	Description	Default
`insts_path`	`Path`	Path to the instruction file.	required

Returns:

Type	Description
	np.ndarray: Array of instructions.

Raises:

Type	Description
`HostRuntimeError`	If the file extension is not supported.

Source code in python/utils/hostruntime/hostruntime.py

@classmethod
def read_insts(cls, insts_path: Path):
    """
    Reads instructions from the given file.

    If the file extension is .bin, uses binary read.
    If the file extension is .txt, uses sequence (text) read.

    Args:
        insts_path (Path): Path to the instruction file.

    Returns:
        np.ndarray: Array of instructions.

    Raises:
        HostRuntimeError: If the file extension is not supported.
    """
    ext = insts_path.suffix.lower()
    if ext == ".bin":
        return cls.read_insts_binary(insts_path)
    else:
        raise HostRuntimeError(
            "Unsupported file extension for instruction file: expected .bin"
        )

prepare_args_for_trace `classmethod` ¶

prepare_args_for_trace(
    args: list[Tensor], trace_config: TraceConfig
) -> list[Tensor]

Prepare arguments for tracing by appending necessary buffers.

Parameters:

Name	Type	Description	Default
`args`	`list[Tensor]`	List of input/output tensors.	required
`trace_config`	`TraceConfig`	Trace configuration.	required

Returns:

Type	Description
`list[Tensor]`	list[Tensor]: The updated list of tensors with trace buffers appended.

Source code in python/utils/hostruntime/hostruntime.py

@classmethod
def prepare_args_for_trace(
    cls, args: list[Tensor], trace_config: TraceConfig
) -> list[Tensor]:
    """
    Prepare arguments for tracing by appending necessary buffers.

    Args:
        args (list[Tensor]): List of input/output tensors.
        trace_config (TraceConfig): Trace configuration.

    Returns:
        list[Tensor]: The updated list of tensors with trace buffers appended.
    """
    if trace_config.reuse_output_buffer:
        # Trace data is written into the tail of the last output buffer.
        # Extend that buffer by the trace size; no new host buffer is added.
        out_size = trace_config.trace_size
        if len(args) > 0:
            out_size += args[-1].nbytes
            # TODO: should really copy previous contents of output into this buffer...? What if it's in/out?
            args[-1] = tensor((out_size,), dtype=np.uint8)
        else:
            out = tensor((out_size,), dtype=np.uint8)
            args.append(out)
    else:
        # Dedicated trace buffer: trace lowering appended one trailing
        # argument to the runtime_sequence, so the host appends exactly one
        # trailing buffer here. The trace buffer lands at index len(args),
        # which matches the appended argument's index by construction -- no
        # positional padding is needed.
        if trace_config.enable_ctrl_pkts:
            # write ctrl packets
            ctrl_pkts = [
                create_ctrl_pkt(1, 0, 0x32004),  # core status
                create_ctrl_pkt(1, 0, 0x340D8),  # trace status
            ]
            # Pad to 8 words
            ctrl_pkts += [0] * (8 - len(ctrl_pkts))

            header = tensor(np.array(ctrl_pkts, dtype=np.uint32))
            args.append(header)

        # Allocate extra space for control packets if enabled
        alloc_size = trace_config.trace_size
        if trace_config.enable_ctrl_pkts:
            alloc_size = trace_config.trace_size * 4

        trace_buff = tensor((alloc_size,), dtype=np.uint8)
        args.append(trace_buff)
    return args

extract_trace_from_args `classmethod` ¶

extract_trace_from_args(
    args: list[Tensor], trace_config: TraceConfig
) -> tuple[ndarray, ndarray | None]

Extract trace and control buffers from the arguments.

Parameters:

Name	Type	Description	Default
`args`	`list[Tensor]`	List of tensors used in execution.	required
`trace_config`	`TraceConfig`	Trace configuration.	required

Returns:

Type	Description
`tuple[ndarray, ndarray \| None]`	tuple[np.ndarray, np.ndarray \| None]: A tuple containing the trace buffer and optionally the control buffer.

Source code in python/utils/hostruntime/hostruntime.py

@classmethod
def extract_trace_from_args(
    cls, args: list[Tensor], trace_config: TraceConfig
) -> tuple[np.ndarray, np.ndarray | None]:
    """
    Extract trace and control buffers from the arguments.

    Args:
        args (list[Tensor]): List of tensors used in execution.
        trace_config (TraceConfig): Trace configuration.

    Returns:
        tuple[np.ndarray, np.ndarray | None]: A tuple containing the trace buffer and optionally the control buffer.
    """
    trace_buff = None
    ctrl_buff = None

    if trace_config.reuse_output_buffer:
        prefix, trace_buff = cls._extract_prefix(
            args[-1], trace_config.last_tensor_shape, trace_config.last_tensor_dtype
        )
        args[-1] = prefix  # pyright: ignore[reportCallIssue, reportArgumentType]
    else:
        # The trace position is always last.
        trace_buff = args[-1].numpy()

    if trace_config.enable_ctrl_pkts:
        trace_buff, ctrl_buff = cls._extract_prefix(
            trace_buff, trace_config.trace_size, np.dtype(np.uint8)
        )
    trace_buff = trace_buff.view(np.uint32).reshape(
        trace_config.trace_size // np.dtype(np.uint32).itemsize
    )
    return trace_buff, ctrl_buff

process_trace `classmethod` ¶

process_trace(
    trace_buffer, ctrl_buffer, trace_config, verbosity=0
)

Process the trace buffer and control buffer.

Parameters:

Name	Type	Description	Default
`trace_buffer`	`ndarray`	The trace data buffer.	required
`ctrl_buffer`	`ndarray`	The control packet buffer.	required
`trace_config`	`TraceConfig`	Trace configuration.	required
`verbosity`	`int`	Verbosity level. Defaults to 0.	`0`

Source code in python/utils/hostruntime/hostruntime.py

@classmethod
def process_trace(cls, trace_buffer, ctrl_buffer, trace_config, verbosity=0):
    """
    Process the trace buffer and control buffer.

    Args:
        trace_buffer (np.ndarray): The trace data buffer.
        ctrl_buffer (np.ndarray): The control packet buffer.
        trace_config (TraceConfig): Trace configuration.
        verbosity (int, optional): Verbosity level. Defaults to 0.
    """
    logger.debug("trace_buffer shape: %s", trace_buffer.shape)
    logger.debug("trace_buffer dtype: %s", trace_buffer.dtype)
    trace_config.write_trace(trace_buffer)

    if trace_config.enable_ctrl_pkts:
        logger.debug("ctrl_buffer shape: %s", ctrl_buffer.shape)
        logger.debug("ctrl_buffer dtype: %s", ctrl_buffer.dtype)
        logger.debug("ctrl buffer: %s", [hex(d) for d in ctrl_buffer])
        for i in range(ctrl_buffer.size // 2):
            col, row, pkt_type, pkt_id = extract_tile(ctrl_buffer[i * 2])
            overflow = True if (ctrl_buffer[i * 2 + 1] >> 8) == 3 else False
            if overflow:
                logger.warning(
                    "Trace overflow detected in tile(%d,%d). Trace results may be invalid.",
                    row,
                    col,
                )

verify_results `classmethod` ¶

verify_results(io_args, refs=None, verbosity=0)

Verify the results of the kernel execution against reference data.

Parameters:

Name	Type	Description	Default
`io_args`	`list[Tensor]`	List of input/output tensors.	required
`refs`	`dict \| None`	Dictionary mapping index to reference numpy array. Defaults to None (empty dict).	`None`
`verbosity`	`int`	Verbosity level. Defaults to 0.	`0`

Returns:

Name	Type	Description
`int`		Number of errors found.

Raises:

Type	Description
`HostRuntimeError`	If a reference index is out of bounds.

Source code in python/utils/hostruntime/hostruntime.py

@classmethod
def verify_results(cls, io_args, refs=None, verbosity=0):
    """
    Verify the results of the kernel execution against reference data.

    Args:
        io_args (list[Tensor]): List of input/output tensors.
        refs (dict | None, optional): Dictionary mapping index to reference numpy array. Defaults to None (empty dict).
        verbosity (int, optional): Verbosity level. Defaults to 0.

    Returns:
        int: Number of errors found.

    Raises:
        HostRuntimeError: If a reference index is out of bounds.
    """
    if refs is None:
        refs = {}
    errors = 0
    if verbosity >= 1:
        logger.info("Verifying results ...")

    for idx, ref in refs.items():
        if idx >= len(io_args):
            raise HostRuntimeError(
                f"Error: Reference index {idx} out of bounds for {len(io_args)} IO buffers"
            )
        io_args[idx].to("cpu")
        o = io_args[idx].numpy()
        e = bfloat16_safe_allclose(ref.dtype, ref, o)
        errors += np.size(e) - np.count_nonzero(e)
    return errors

run_test ¶

run_test(
    npu_kernel,
    io_args,
    ref,
    verify: bool = True,
    verbosity: int = 0,
) -> int

Run a test for the given NPU kernel.

Parameters:

Name	Type	Description	Default
`npu_kernel`	`NPUKernel`	The NPU kernel to test.	required
`io_args`	`list[Tensor]`	List of input/output tensors.	required
`ref`	`dict`	Reference data for verification.	required
`verify`	`bool`	Whether to verify results. Defaults to True.	`True`
`verbosity`	`int`	Verbosity level. Defaults to 0.	`0`

Returns:

Name	Type	Description
`int`	`int`	0 if successful, 1 otherwise.

Source code in python/utils/hostruntime/hostruntime.py

def run_test(
    self,
    npu_kernel,
    io_args,
    ref,
    verify: bool = True,
    verbosity: int = 0,
) -> int:
    """
    Run a test for the given NPU kernel.

    Args:
        npu_kernel (NPUKernel): The NPU kernel to test.
        io_args (list[Tensor]): List of input/output tensors.
        ref (dict): Reference data for verification.
        verify (bool, optional): Whether to verify results. Defaults to True.
        verbosity (int, optional): Verbosity level. Defaults to 0.

    Returns:
        int: 0 if successful, 1 otherwise.
    """
    kernel_handle = self.load(npu_kernel)
    trace_config = npu_kernel.trace_config

    # Ensure io_args is a list
    if not isinstance(io_args, list):
        io_args = [io_args] if io_args else []

    buffers = io_args
    last_out = buffers[-1] if buffers else None

    if trace_config:
        trace_config.last_tensor_shape = last_out.shape if last_out else None
        trace_config.last_tensor_dtype = last_out.dtype if last_out else None
        self.prepare_args_for_trace(buffers, trace_config)

    ret = self.run(kernel_handle, buffers)

    if verbosity >= 1:
        logger.info("npu_time: %s us", ret.npu_time / 1000.0)

    if trace_config:
        trace_buffer, ctrl_buffer = self.extract_trace_from_args(
            buffers, trace_config
        )
        self.process_trace(trace_buffer, ctrl_buffer, trace_config, verbosity)

    errors = 0
    if verify:
        errors = self.verify_results(io_args, ref, verbosity)

    if not errors:
        return 0
    else:
        logger.error("Error count: %d", errors)
        logger.error("Failed.")
        return 1

Tensor and device utilities¶

Device selection, the abstract Tensor type, and numerical helpers used when comparing host results.

Host runtime utilities: device selection, tensor allocation, and numerical helpers.

Tensor ¶

Tensor(
    shape_or_data, dtype: DTypeLike = uint32, device="npu"
)

Bases: ABC

Tensor object backed by NPU or CPU memory.

The class provides common tensor operations such as creation, filling with values, and accessing data.

Initialize the tensor.

Parameters:

Name	Type	Description	Default
`shape_or_data`	`tuple or array - like`	If a tuple, creates a new tensor with the given shape and dtype. If array-like, wraps the data into a tensor with optional dtype casting.	required
`dtype`	`dtype`	Data type of the tensor. Defaults to np.uint32.	`uint32`
`device`	`str`	Device string identifier (e.g., 'npu', 'cpu'). Defaults to 'npu'.	`'npu'`

Source code in python/utils/hostruntime/tensor_class.py

def __init__(self, shape_or_data, dtype: npt.DTypeLike = np.uint32, device="npu"):
    """
    Initialize the tensor.

    Args:
        shape_or_data (tuple or array-like):
            - If a tuple, creates a new tensor with the given shape and dtype.
            - If array-like, wraps the data into a tensor with optional dtype casting.
        dtype (np.dtype, optional): Data type of the tensor. Defaults to np.uint32.
        device (str, optional): Device string identifier (e.g., 'npu', 'cpu'). Defaults to 'npu'.
    """
    if device not in self.__class__.DEVICES:
        raise ValueError(f"Unsupported device: {device}")
    self.device = device
    self.dtype = dtype

data `abstractmethod` `property` ¶

data: ndarray

Subclasses must implement a data property.

Returns:

Type	Description
`ndarray`	np.ndarray: The underlying data of the tensor.

shape `abstractmethod` `property` ¶

shape: tuple[int, ...]

Subclasses must implement a shape property.

Returns:

Name	Type	Description
`tuple`	`tuple[int, ...]`	The shape of the tensor.

nbytes `cached` `property` ¶

nbytes: int

Number of bytes consumed by elements in the tensor

element_size `cached` `property` ¶

element_size: int

Number of bytes per element

to ¶

to(target_device: str)

Moves the tensor to a specified target device.

Parameters:

Name	Type	Description	Default
`target_device`	`str`	The target device.	required

Returns:

Type	Description
	The tensor object on the target device.

Source code in python/utils/hostruntime/tensor_class.py

def to(self, target_device: str):
    """
    Moves the tensor to a specified target device.

    Args:
        target_device (str): The target device.

    Returns:
       The tensor object on the target device.
    """
    if target_device == self.device:
        # nothing to do
        pass
    elif target_device == "npu":
        self._sync_to_device()
        self.device = "npu"
    elif target_device == "cpu":
        self._sync_from_device()
        self.device = "cpu"
    else:
        raise ValueError(f"Unknown device '{target_device}'")
    return self

numpy ¶

numpy()

Returns a NumPy view of the tensor data on host memory.

This method ensures that data is first synchronized from the device (e.g., NPU) to the host before returning the array.

Returns:

Type	Description
	np.ndarray: The tensor's data as a NumPy array.

Note: For NPU tensors, this method causes implicit data synchronization from device to host to ensure the returned array reflects the current device state.

Source code in python/utils/hostruntime/tensor_class.py

def numpy(self):
    """
    Returns a NumPy view of the tensor data on host memory.

    This method ensures that data is first synchronized from the device
    (e.g., NPU) to the host before returning the array.

    Returns:
        np.ndarray: The tensor's data as a NumPy array.

    Note: For NPU tensors, this method causes implicit data synchronization from device to host
    to ensure the returned array reflects the current device state.
    """
    if self.device == "npu":
        self._sync_from_device()
    return self.data

to_torch ¶

to_torch()

Returns a torch tensor sharing the data in this tensor if possible.

Syncs from device first if the tensor is on the NPU.

Returns:

Type	Description
	torch.Tensor: A torch tensor containing the data.

Raises:

Type	Description
`ImportError`	If torch is not installed.

Source code in python/utils/hostruntime/tensor_class.py

def to_torch(self):
    """
    Returns a torch tensor sharing the data in this tensor if possible.

    Syncs from device first if the tensor is on the NPU.

    Returns:
        torch.Tensor: A torch tensor containing the data.

    Raises:
        ImportError: If torch is not installed.
    """
    return _array_to_torch(self.numpy())

torch_view ¶

torch_view()

Returns a torch tensor sharing this buffer's host memory without syncing from device.

Unlike to_torch(), this does NOT sync from the NPU first. Marks the buffer as CPU-resident so that a subsequent .to("npu") call (or the NPU operator's implicit sync) will push the written data to device. Use this on write paths where the caller is about to overwrite the buffer contents.

Returns:

Type	Description
	torch.Tensor: A zero-copy torch tensor view of the host-side buffer.

Raises:

Type	Description
`ImportError`	If torch is not installed.

Source code in python/utils/hostruntime/tensor_class.py

def torch_view(self):
    """
    Returns a torch tensor sharing this buffer's host memory without syncing from device.

    Unlike to_torch(), this does NOT sync from the NPU first. Marks the buffer as
    CPU-resident so that a subsequent .to("npu") call (or the NPU operator's implicit
    sync) will push the written data to device. Use this on write paths where the
    caller is about to overwrite the buffer contents.

    Returns:
        torch.Tensor: A zero-copy torch tensor view of the host-side buffer.

    Raises:
        ImportError: If torch is not installed.
    """
    self.device = "cpu"  # mark dirty so next to("npu") will actually sync
    return _array_to_torch(self.data)

from_torch `classmethod` ¶

from_torch(torch_tensor, device=None, **kwargs)

Returns a tensor with a copy of the data in the torch_tensor.

Parameters:

Name	Type	Description	Default
`torch_tensor`	`Tensor`	The source torch tensor.	required
`device`	`str`	The target device. Defaults to None.	`None`
`**kwargs`		Additional arguments for tensor creation.	`{}`

Returns:

Name	Type	Description
`Tensor`		A new tensor containing the data from the torch tensor.

Raises:

Type	Description
`ImportError`	If torch is not installed.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def from_torch(cls, torch_tensor, device=None, **kwargs):
    """
    Returns a tensor with a copy of the data in the torch_tensor.

    Args:
        torch_tensor (torch.Tensor): The source torch tensor.
        device (str, optional): The target device. Defaults to None.
        **kwargs: Additional arguments for tensor creation.

    Returns:
        Tensor: A new tensor containing the data from the torch tensor.

    Raises:
        ImportError: If torch is not installed.
    """
    import torch  # pyright: ignore[reportMissingImports]
    from ml_dtypes import bfloat16

    # Detach (to drop grad) and ensure on CPU
    t = torch_tensor.detach()
    if t.device.type != "cpu":
        t = t.cpu()
    # Ensure contiguous for safe view operations
    if not t.is_contiguous():
        t = t.contiguous()

    if t.dtype == torch.bfloat16:
        # View the same memory as int16, then as NumPy bfloat16
        # This avoids numeric conversion and extra passes over memory.
        u16_np = t.view(torch.uint16).numpy()  # shares memory
        np_array = u16_np.view(bfloat16)  # reinterpret
    else:
        np_array = t.numpy()

    return cls(
        np_array,
        dtype=np_array.dtype,
        device=device or cls.DEFAULT_DEVICE,
        **kwargs,
    )

fill_ ¶

fill_(value)

Fills the tensor with a scalar value (in-place operation).

Parameters:

Name	Type	Description	Default
`value`		The scalar value to fill the tensor with.	required

Note: For NPU tensors, this method syncs the filled data to device after modification.

Source code in python/utils/hostruntime/tensor_class.py

def fill_(self, value):
    """
    Fills the tensor with a scalar value (in-place operation).

    Args:
        value: The scalar value to fill the tensor with.

    Note: For NPU tensors, this method syncs the filled data to device after modification.
    """
    self.data.fill(value)
    if self.device == "npu":
        self._sync_to_device()

numel ¶

numel()

Calculates the number of elements in the tensor.

Returns:

Name	Type	Description
`int`		The total number of elements in the tensor.

Source code in python/utils/hostruntime/tensor_class.py

def numel(self):
    """
    Calculates the number of elements in the tensor.

    Returns:
        int: The total number of elements in the tensor.
    """
    return int(np.prod(self.shape))

ones `classmethod` ¶

ones(*size, out=None, dtype=None, device=None, **kwargs)

Returns a tensor filled with ones, with shape defined by size.

Parameters:

Name	Type	Description	Default
`*size`	`int...`	Shape of the tensor, passed as separate ints or a single tuple/list.	`()`
`out`	`Tensor`	Optional output tensor to write into.	`None`
`dtype`	`dtype`	Desired dtype. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`**kwargs`		Additional keyword args.	`{}`

Returns:

Name	Type	Description
`Tensor`		A one-filled tensor.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def ones(cls, *size, out=None, dtype=None, device=None, **kwargs):
    """
    Returns a tensor filled with ones, with shape defined by size.

    Args:
        *size (int...): Shape of the tensor, passed as separate ints or a single tuple/list.
        out (Tensor, optional): Optional output tensor to write into.
        dtype (np.dtype, optional): Desired dtype. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        **kwargs: Additional keyword args.

    Returns:
        Tensor: A one-filled tensor.
    """
    t = cls.__check_or_create(*size, out=out, dtype=dtype, device=device, **kwargs)
    t.fill_(1)
    return t

zeros `classmethod` ¶

zeros(*size, out=None, dtype=None, device=None, **kwargs)

Returns a tensor filled with zeros, with shape defined by size.

Parameters:

Name	Type	Description	Default
`*size`	`int...`	Shape of the tensor, passed as separate ints or a single tuple/list.	`()`
`out`	`Tensor`	Optional output tensor to write into.	`None`
`dtype`	`dtype`	Desired dtype. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`**kwargs`		Additional keyword args.	`{}`

Returns:

Name	Type	Description
`Tensor`		A zero-filled tensor.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def zeros(cls, *size, out=None, dtype=None, device=None, **kwargs):
    """
    Returns a tensor filled with zeros, with shape defined by size.

    Args:
        *size (int...): Shape of the tensor, passed as separate ints or a single tuple/list.
        out (Tensor, optional): Optional output tensor to write into.
        dtype (np.dtype, optional): Desired dtype. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        **kwargs: Additional keyword args.

    Returns:
        Tensor: A zero-filled tensor.
    """
    t = cls.__check_or_create(*size, out=out, dtype=dtype, device=device, **kwargs)
    t.fill_(0)
    return t

full `classmethod` ¶

full(
    size,
    fill_value,
    *,
    out=None,
    dtype=None,
    device=None,
    **kwargs
)

Returns a tensor of shape size filled with fill_value.

Parameters:

Name	Type	Description	Default
`size`	`int or tuple/list of int`	Shape of the returned tensor.	required
`fill_value`	`scalar`	Value to fill the tensor with.	required
`out`	`Tensor`	Optional output tensor to write into.	`None`
`dtype`	`dtype`	Desired dtype. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`**kwargs`		Additional keyword args.	`{}`

Returns:

Name	Type	Description
`Tensor`		A tensor filled with `fill_value`.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def full(cls, size, fill_value, *, out=None, dtype=None, device=None, **kwargs):
    """
    Returns a tensor of shape `size` filled with `fill_value`.

    Args:
        size (int or tuple/list of int): Shape of the returned tensor.
        fill_value (scalar): Value to fill the tensor with.
        out (Tensor, optional): Optional output tensor to write into.
        dtype (np.dtype, optional): Desired dtype. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        **kwargs: Additional keyword args.

    Returns:
        Tensor: A tensor filled with `fill_value`.
    """
    t = cls.__check_or_create(size, out=out, dtype=dtype, device=device, **kwargs)
    t.fill_(fill_value)
    return t

randint `classmethod` ¶

randint(
    low,
    high,
    size,
    *,
    out=None,
    dtype=None,
    device=None,
    generator=None,
    **kwargs
)

Returns a tensor filled with random integers uniformly sampled from [low, high).

Parameters:

Name	Type	Description	Default
`low`	`int`	Lowest integer to be drawn (inclusive).	required
`high`	`int`	One above the highest integer to be drawn (exclusive).	required
`size`	`tuple`	Shape of the returned tensor.	required
`out`	`Tensor`	Optional tensor to write the result into.	`None`
`dtype`	`dtype`	Data type. Defaults to np.int64.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`generator`	`Generator`	Source RNG for reproducibility. If None, uses np.random module-level state.	`None`
`**kwargs`		Additional arguments passed to the constructor.	`{}`

Returns:

Name	Type	Description
`Tensor`		A tensor with random integers.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def randint(
    cls,
    low,
    high,
    size,
    *,
    out=None,
    dtype=None,
    device=None,
    generator=None,
    **kwargs,
):
    """
    Returns a tensor filled with random integers uniformly sampled from [low, high).

    Args:
        low (int): Lowest integer to be drawn (inclusive).
        high (int): One above the highest integer to be drawn (exclusive).
        size (tuple): Shape of the returned tensor.
        out (Tensor, optional): Optional tensor to write the result into.
        dtype (np.dtype, optional): Data type. Defaults to np.int64.
        device (str, optional): Target device. Defaults to 'npu'.
        generator (np.random.Generator, optional): Source RNG for reproducibility.
            If None, uses np.random module-level state.
        **kwargs: Additional arguments passed to the constructor.

    Returns:
        Tensor: A tensor with random integers.
    """
    dtype = dtype or np.int64
    device = device or cls.DEFAULT_DEVICE

    t = cls.__check_or_create(size, out=out, dtype=dtype, device=device, **kwargs)
    if generator is not None:
        random_val = generator.integers(low, high, size=size, dtype=dtype)
    else:
        random_val = np.random.randint(low, high, size=size, dtype=dtype)
    if size == ():
        t.data.fill(random_val)
    else:
        t.data[:] = random_val
    if device == "npu":
        t._sync_to_device()
    return t

rand `classmethod` ¶

rand(
    *size,
    out=None,
    dtype=None,
    device=None,
    generator=None,
    **kwargs
)

Returns a tensor filled with random numbers from a uniform distribution on [0, 1).

Parameters:

Name	Type	Description	Default
`*size`	`int...`	Variable number of integers or a single tuple defining the shape.	`()`
`out`	`Tensor`	Output tensor to write into.	`None`
`dtype`	`dtype`	Desired data type. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`generator`	`Generator`	Source RNG for reproducibility. If None, uses np.random module-level state.	`None`
`**kwargs`		Additional arguments passed to constructor.	`{}`

Returns:

Name	Type	Description
`Tensor`		A tensor with random values in [0, 1).

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def rand(cls, *size, out=None, dtype=None, device=None, generator=None, **kwargs):
    """
    Returns a tensor filled with random numbers from a uniform distribution on [0, 1).

    Args:
        *size (int...): Variable number of integers or a single tuple defining the shape.
        out (Tensor, optional): Output tensor to write into.
        dtype (np.dtype, optional): Desired data type. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        generator (np.random.Generator, optional): Source RNG for reproducibility.
            If None, uses np.random module-level state.
        **kwargs: Additional arguments passed to constructor.

    Returns:
        Tensor: A tensor with random values in [0, 1).
    """
    if not size:
        raise ValueError("rand() received no arguments")
    dtype = dtype or np.float32
    device = device or cls.DEFAULT_DEVICE

    t = cls.__check_or_create(*size, out=out, dtype=dtype, device=device, **kwargs)
    if generator is not None:
        random_val = generator.uniform(0.0, 1.0, size=t.shape).astype(dtype)
    else:
        random_val = np.random.uniform(0.0, 1.0, size=t.shape).astype(dtype)
    # Ensure values are < 1.0 for low-precision types
    is_bfloat16 = False
    try:
        from ml_dtypes import bfloat16

        if dtype == bfloat16:
            is_bfloat16 = True
    except ImportError:
        pass

    if np.issubdtype(dtype, np.floating) or is_bfloat16:
        max_val = np.nextafter(dtype(1.0), dtype(0.0))
        random_val = np.clip(random_val, 0.0, max_val)

    if t.shape == ():
        t.data.fill(random_val)
    else:
        t.data[:] = random_val
    if device == "npu":
        t._sync_to_device()
    return t

arange `classmethod` ¶

arange(
    start=0,
    end=None,
    step=1,
    *,
    shape=None,
    out=None,
    dtype=None,
    device=None,
    **kwargs
)

Returns a tensor with values from the interval [start, end) with spacing step.

Parameters:

Name	Type	Description	Default
`start`	`number`	Start of interval. Defaults to 0.	`0`
`end`	`number`	End of interval (exclusive). Required if only one argument is given.	`None`
`step`	`number`	Gap between elements. Defaults to 1.	`1`
`shape`	`tuple`	If given, reshape the 1-D sequence to this shape. `prod(shape)` must equal the length of the generated range.	`None`
`dtype`	`dtype`	Desired output data type. Inferred if not provided.	`None`
`out`	`Tensor`	Optional tensor to write output to (must match shape and dtype).	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`

Returns:

Name	Type	Description
`Tensor`		Tensor containing the sequence (1-D by default, or `shape` if given).

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def arange(
    cls,
    start=0,
    end=None,
    step=1,
    *,
    shape=None,
    out=None,
    dtype=None,
    device=None,
    **kwargs,
):
    """
    Returns a tensor with values from the interval [start, end) with spacing `step`.

    Args:
        start (number): Start of interval. Defaults to 0.
        end (number): End of interval (exclusive). Required if only one argument is given.
        step (number): Gap between elements. Defaults to 1.
        shape (tuple, optional): If given, reshape the 1-D sequence to this shape.
            `prod(shape)` must equal the length of the generated range.
        dtype (np.dtype, optional): Desired output data type. Inferred if not provided.
        out (Tensor, optional): Optional tensor to write output to (must match shape and dtype).
        device (str, optional): Target device. Defaults to 'npu'.

    Returns:
        Tensor: Tensor containing the sequence (1-D by default, or `shape` if given).
    """

    if end is None:
        start, end = 0, start

    if dtype is None:
        if any(isinstance(x, float) for x in (start, end, step)):
            dtype = np.float32
        else:
            dtype = np.int64

    device = device or cls.DEFAULT_DEVICE

    data = np.arange(start, end, step, dtype=dtype)

    if shape is not None:
        shape = tuple(shape)
        if int(np.prod(shape)) != data.size:
            raise ValueError(
                f"iron.arange: shape={shape} (prod={int(np.prod(shape))}) does "
                f"not match generated range size {data.size}"
            )
        data = data.reshape(shape)
    else:
        shape = (data.size,)

    if out is not None:
        if out.shape != shape or out.dtype != dtype or out.device != device:
            raise ValueError(
                "Provided `out` tensor must match shape, dtype, and device"
            )
        out.data[...] = data
        if device == "npu":
            out._sync_to_device()
        return out

    t = cls(shape, dtype=dtype, device=device, **kwargs)
    t.data[...] = data
    if device == "npu":
        t._sync_to_device()
    return t

zeros_like `classmethod` ¶

zeros_like(other, dtype=None, device=None, **kwargs)

Creates a new tensor with the same shape as other, filled with zeros.

Parameters:

Name	Type	Description	Default
`other`	`Tensor`	The reference tensor to copy shape from.	required
`dtype`	`dtype`	Data type of the new tensor. Defaults to other's dtype.	`None`
`device`	`str`	Target device. Defaults to other's device.	`None`
`**kwargs`		Additional keyword arguments forwarded to the constructor.	`{}`

Returns:

Name	Type	Description
`Tensor`		A new zero-filled tensor with the same shape.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def zeros_like(cls, other, dtype=None, device=None, **kwargs):
    """
    Creates a new tensor with the same shape as `other`, filled with zeros.

    Args:
        other (Tensor): The reference tensor to copy shape from.
        dtype (np.dtype, optional): Data type of the new tensor. Defaults to other's dtype.
        device (str, optional): Target device. Defaults to other's device.
        **kwargs: Additional keyword arguments forwarded to the constructor.

    Returns:
        Tensor: A new zero-filled tensor with the same shape.
    """
    dtype = dtype or other.dtype
    device = device or other.device
    t = cls(other.shape, dtype=dtype, device=device, **kwargs)
    t.data.fill(0)

    if device == "npu":
        t._sync_to_device()

    return t

set_current_device ¶

set_current_device(device: Device)

Set the current device.

Parameters:

Name	Type	Description	Default
`device`	`Device`	The device to set as current.	required

Source code in python/utils/hostruntime/__init__.py

def set_current_device(device: "Device"):
    """
    Set the current device.

    Args:
        device (Device): The device to set as current.
    """
    global _CURRENT_DEVICE
    _CURRENT_DEVICE = device

bfloat16_safe_allclose ¶

bfloat16_safe_allclose(dtype, arr1, arr2)

Check if two arrays are element-wise equal within a tolerance, handling bfloat16 safely.

Parameters:

Name	Description	Default
`dtype`	The data type of the arrays.	required
`arr1`	First input array.	required
`arr2`	Second input array.	required

Returns:

Name	Type	Description
`bool`		True if the arrays are equal within tolerance, False otherwise.

Source code in python/utils/hostruntime/__init__.py

def bfloat16_safe_allclose(dtype, arr1, arr2):
    """
    Check if two arrays are element-wise equal within a tolerance, handling bfloat16 safely.

    Args:
        dtype: The data type of the arrays.
        arr1: First input array.
        arr2: Second input array.

    Returns:
        bool: True if the arrays are equal within tolerance, False otherwise.
    """
    if dtype == bfloat16:
        if isinstance(arr1, Tensor):
            arr1 = np.array(arr1, dtype=np.float16)
        else:
            arr1 = arr1.astype(np.float16)
        if isinstance(arr2, Tensor):
            arr2 = np.array(arr2, dtype=np.float16)
        else:
            arr2 = arr2.astype(np.float16)
    return np.allclose(arr1, arr2)

Tensor ¶

Tensor(
    shape_or_data, dtype: DTypeLike = uint32, device="npu"
)

Bases: ABC

Tensor object backed by NPU or CPU memory.

The class provides common tensor operations such as creation, filling with values, and accessing data.

Initialize the tensor.

Parameters:

Name	Type	Description	Default
`shape_or_data`	`tuple or array - like`	If a tuple, creates a new tensor with the given shape and dtype. If array-like, wraps the data into a tensor with optional dtype casting.	required
`dtype`	`dtype`	Data type of the tensor. Defaults to np.uint32.	`uint32`
`device`	`str`	Device string identifier (e.g., 'npu', 'cpu'). Defaults to 'npu'.	`'npu'`

Source code in python/utils/hostruntime/tensor_class.py

def __init__(self, shape_or_data, dtype: npt.DTypeLike = np.uint32, device="npu"):
    """
    Initialize the tensor.

    Args:
        shape_or_data (tuple or array-like):
            - If a tuple, creates a new tensor with the given shape and dtype.
            - If array-like, wraps the data into a tensor with optional dtype casting.
        dtype (np.dtype, optional): Data type of the tensor. Defaults to np.uint32.
        device (str, optional): Device string identifier (e.g., 'npu', 'cpu'). Defaults to 'npu'.
    """
    if device not in self.__class__.DEVICES:
        raise ValueError(f"Unsupported device: {device}")
    self.device = device
    self.dtype = dtype

data `abstractmethod` `property` ¶

data: ndarray

Subclasses must implement a data property.

Returns:

Type	Description
`ndarray`	np.ndarray: The underlying data of the tensor.

shape `abstractmethod` `property` ¶

shape: tuple[int, ...]

Subclasses must implement a shape property.

Returns:

Name	Type	Description
`tuple`	`tuple[int, ...]`	The shape of the tensor.

nbytes `cached` `property` ¶

nbytes: int

Number of bytes consumed by elements in the tensor

element_size `cached` `property` ¶

element_size: int

Number of bytes per element

to ¶

to(target_device: str)

Moves the tensor to a specified target device.

Parameters:

Name	Type	Description	Default
`target_device`	`str`	The target device.	required

Returns:

Type	Description
	The tensor object on the target device.

Source code in python/utils/hostruntime/tensor_class.py

def to(self, target_device: str):
    """
    Moves the tensor to a specified target device.

    Args:
        target_device (str): The target device.

    Returns:
       The tensor object on the target device.
    """
    if target_device == self.device:
        # nothing to do
        pass
    elif target_device == "npu":
        self._sync_to_device()
        self.device = "npu"
    elif target_device == "cpu":
        self._sync_from_device()
        self.device = "cpu"
    else:
        raise ValueError(f"Unknown device '{target_device}'")
    return self

numpy ¶

numpy()

Returns a NumPy view of the tensor data on host memory.

This method ensures that data is first synchronized from the device (e.g., NPU) to the host before returning the array.

Returns:

Type	Description
	np.ndarray: The tensor's data as a NumPy array.

Note: For NPU tensors, this method causes implicit data synchronization from device to host to ensure the returned array reflects the current device state.

Source code in python/utils/hostruntime/tensor_class.py

def numpy(self):
    """
    Returns a NumPy view of the tensor data on host memory.

    This method ensures that data is first synchronized from the device
    (e.g., NPU) to the host before returning the array.

    Returns:
        np.ndarray: The tensor's data as a NumPy array.

    Note: For NPU tensors, this method causes implicit data synchronization from device to host
    to ensure the returned array reflects the current device state.
    """
    if self.device == "npu":
        self._sync_from_device()
    return self.data

to_torch ¶

to_torch()

Returns a torch tensor sharing the data in this tensor if possible.

Syncs from device first if the tensor is on the NPU.

Returns:

Type	Description
	torch.Tensor: A torch tensor containing the data.

Raises:

Type	Description
`ImportError`	If torch is not installed.

Source code in python/utils/hostruntime/tensor_class.py

def to_torch(self):
    """
    Returns a torch tensor sharing the data in this tensor if possible.

    Syncs from device first if the tensor is on the NPU.

    Returns:
        torch.Tensor: A torch tensor containing the data.

    Raises:
        ImportError: If torch is not installed.
    """
    return _array_to_torch(self.numpy())

torch_view ¶

torch_view()

Returns a torch tensor sharing this buffer's host memory without syncing from device.

Unlike to_torch(), this does NOT sync from the NPU first. Marks the buffer as CPU-resident so that a subsequent .to("npu") call (or the NPU operator's implicit sync) will push the written data to device. Use this on write paths where the caller is about to overwrite the buffer contents.

Returns:

Type	Description
	torch.Tensor: A zero-copy torch tensor view of the host-side buffer.

Raises:

Type	Description
`ImportError`	If torch is not installed.

Source code in python/utils/hostruntime/tensor_class.py

def torch_view(self):
    """
    Returns a torch tensor sharing this buffer's host memory without syncing from device.

    Unlike to_torch(), this does NOT sync from the NPU first. Marks the buffer as
    CPU-resident so that a subsequent .to("npu") call (or the NPU operator's implicit
    sync) will push the written data to device. Use this on write paths where the
    caller is about to overwrite the buffer contents.

    Returns:
        torch.Tensor: A zero-copy torch tensor view of the host-side buffer.

    Raises:
        ImportError: If torch is not installed.
    """
    self.device = "cpu"  # mark dirty so next to("npu") will actually sync
    return _array_to_torch(self.data)

from_torch `classmethod` ¶

from_torch(torch_tensor, device=None, **kwargs)

Returns a tensor with a copy of the data in the torch_tensor.

Parameters:

Name	Type	Description	Default
`torch_tensor`	`Tensor`	The source torch tensor.	required
`device`	`str`	The target device. Defaults to None.	`None`
`**kwargs`		Additional arguments for tensor creation.	`{}`

Returns:

Name	Type	Description
`Tensor`		A new tensor containing the data from the torch tensor.

Raises:

Type	Description
`ImportError`	If torch is not installed.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def from_torch(cls, torch_tensor, device=None, **kwargs):
    """
    Returns a tensor with a copy of the data in the torch_tensor.

    Args:
        torch_tensor (torch.Tensor): The source torch tensor.
        device (str, optional): The target device. Defaults to None.
        **kwargs: Additional arguments for tensor creation.

    Returns:
        Tensor: A new tensor containing the data from the torch tensor.

    Raises:
        ImportError: If torch is not installed.
    """
    import torch  # pyright: ignore[reportMissingImports]
    from ml_dtypes import bfloat16

    # Detach (to drop grad) and ensure on CPU
    t = torch_tensor.detach()
    if t.device.type != "cpu":
        t = t.cpu()
    # Ensure contiguous for safe view operations
    if not t.is_contiguous():
        t = t.contiguous()

    if t.dtype == torch.bfloat16:
        # View the same memory as int16, then as NumPy bfloat16
        # This avoids numeric conversion and extra passes over memory.
        u16_np = t.view(torch.uint16).numpy()  # shares memory
        np_array = u16_np.view(bfloat16)  # reinterpret
    else:
        np_array = t.numpy()

    return cls(
        np_array,
        dtype=np_array.dtype,
        device=device or cls.DEFAULT_DEVICE,
        **kwargs,
    )

fill_ ¶

fill_(value)

Fills the tensor with a scalar value (in-place operation).

Parameters:

Name	Type	Description	Default
`value`		The scalar value to fill the tensor with.	required

Note: For NPU tensors, this method syncs the filled data to device after modification.

Source code in python/utils/hostruntime/tensor_class.py

def fill_(self, value):
    """
    Fills the tensor with a scalar value (in-place operation).

    Args:
        value: The scalar value to fill the tensor with.

    Note: For NPU tensors, this method syncs the filled data to device after modification.
    """
    self.data.fill(value)
    if self.device == "npu":
        self._sync_to_device()

numel ¶

numel()

Calculates the number of elements in the tensor.

Returns:

Name	Type	Description
`int`		The total number of elements in the tensor.

Source code in python/utils/hostruntime/tensor_class.py

def numel(self):
    """
    Calculates the number of elements in the tensor.

    Returns:
        int: The total number of elements in the tensor.
    """
    return int(np.prod(self.shape))

ones `classmethod` ¶

ones(*size, out=None, dtype=None, device=None, **kwargs)

Returns a tensor filled with ones, with shape defined by size.

Parameters:

Name	Type	Description	Default
`*size`	`int...`	Shape of the tensor, passed as separate ints or a single tuple/list.	`()`
`out`	`Tensor`	Optional output tensor to write into.	`None`
`dtype`	`dtype`	Desired dtype. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`**kwargs`		Additional keyword args.	`{}`

Returns:

Name	Type	Description
`Tensor`		A one-filled tensor.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def ones(cls, *size, out=None, dtype=None, device=None, **kwargs):
    """
    Returns a tensor filled with ones, with shape defined by size.

    Args:
        *size (int...): Shape of the tensor, passed as separate ints or a single tuple/list.
        out (Tensor, optional): Optional output tensor to write into.
        dtype (np.dtype, optional): Desired dtype. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        **kwargs: Additional keyword args.

    Returns:
        Tensor: A one-filled tensor.
    """
    t = cls.__check_or_create(*size, out=out, dtype=dtype, device=device, **kwargs)
    t.fill_(1)
    return t

zeros `classmethod` ¶

zeros(*size, out=None, dtype=None, device=None, **kwargs)

Returns a tensor filled with zeros, with shape defined by size.

Parameters:

Name	Type	Description	Default
`*size`	`int...`	Shape of the tensor, passed as separate ints or a single tuple/list.	`()`
`out`	`Tensor`	Optional output tensor to write into.	`None`
`dtype`	`dtype`	Desired dtype. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`**kwargs`		Additional keyword args.	`{}`

Returns:

Name	Type	Description
`Tensor`		A zero-filled tensor.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def zeros(cls, *size, out=None, dtype=None, device=None, **kwargs):
    """
    Returns a tensor filled with zeros, with shape defined by size.

    Args:
        *size (int...): Shape of the tensor, passed as separate ints or a single tuple/list.
        out (Tensor, optional): Optional output tensor to write into.
        dtype (np.dtype, optional): Desired dtype. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        **kwargs: Additional keyword args.

    Returns:
        Tensor: A zero-filled tensor.
    """
    t = cls.__check_or_create(*size, out=out, dtype=dtype, device=device, **kwargs)
    t.fill_(0)
    return t

full `classmethod` ¶

full(
    size,
    fill_value,
    *,
    out=None,
    dtype=None,
    device=None,
    **kwargs
)

Returns a tensor of shape size filled with fill_value.

Parameters:

Name	Type	Description	Default
`size`	`int or tuple/list of int`	Shape of the returned tensor.	required
`fill_value`	`scalar`	Value to fill the tensor with.	required
`out`	`Tensor`	Optional output tensor to write into.	`None`
`dtype`	`dtype`	Desired dtype. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`**kwargs`		Additional keyword args.	`{}`

Returns:

Name	Type	Description
`Tensor`		A tensor filled with `fill_value`.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def full(cls, size, fill_value, *, out=None, dtype=None, device=None, **kwargs):
    """
    Returns a tensor of shape `size` filled with `fill_value`.

    Args:
        size (int or tuple/list of int): Shape of the returned tensor.
        fill_value (scalar): Value to fill the tensor with.
        out (Tensor, optional): Optional output tensor to write into.
        dtype (np.dtype, optional): Desired dtype. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        **kwargs: Additional keyword args.

    Returns:
        Tensor: A tensor filled with `fill_value`.
    """
    t = cls.__check_or_create(size, out=out, dtype=dtype, device=device, **kwargs)
    t.fill_(fill_value)
    return t

randint `classmethod` ¶

randint(
    low,
    high,
    size,
    *,
    out=None,
    dtype=None,
    device=None,
    generator=None,
    **kwargs
)

Returns a tensor filled with random integers uniformly sampled from [low, high).

Parameters:

Name	Type	Description	Default
`low`	`int`	Lowest integer to be drawn (inclusive).	required
`high`	`int`	One above the highest integer to be drawn (exclusive).	required
`size`	`tuple`	Shape of the returned tensor.	required
`out`	`Tensor`	Optional tensor to write the result into.	`None`
`dtype`	`dtype`	Data type. Defaults to np.int64.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`generator`	`Generator`	Source RNG for reproducibility. If None, uses np.random module-level state.	`None`
`**kwargs`		Additional arguments passed to the constructor.	`{}`

Returns:

Name	Type	Description
`Tensor`		A tensor with random integers.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def randint(
    cls,
    low,
    high,
    size,
    *,
    out=None,
    dtype=None,
    device=None,
    generator=None,
    **kwargs,
):
    """
    Returns a tensor filled with random integers uniformly sampled from [low, high).

    Args:
        low (int): Lowest integer to be drawn (inclusive).
        high (int): One above the highest integer to be drawn (exclusive).
        size (tuple): Shape of the returned tensor.
        out (Tensor, optional): Optional tensor to write the result into.
        dtype (np.dtype, optional): Data type. Defaults to np.int64.
        device (str, optional): Target device. Defaults to 'npu'.
        generator (np.random.Generator, optional): Source RNG for reproducibility.
            If None, uses np.random module-level state.
        **kwargs: Additional arguments passed to the constructor.

    Returns:
        Tensor: A tensor with random integers.
    """
    dtype = dtype or np.int64
    device = device or cls.DEFAULT_DEVICE

    t = cls.__check_or_create(size, out=out, dtype=dtype, device=device, **kwargs)
    if generator is not None:
        random_val = generator.integers(low, high, size=size, dtype=dtype)
    else:
        random_val = np.random.randint(low, high, size=size, dtype=dtype)
    if size == ():
        t.data.fill(random_val)
    else:
        t.data[:] = random_val
    if device == "npu":
        t._sync_to_device()
    return t

rand `classmethod` ¶

rand(
    *size,
    out=None,
    dtype=None,
    device=None,
    generator=None,
    **kwargs
)

Returns a tensor filled with random numbers from a uniform distribution on [0, 1).

Parameters:

Name	Type	Description	Default
`*size`	`int...`	Variable number of integers or a single tuple defining the shape.	`()`
`out`	`Tensor`	Output tensor to write into.	`None`
`dtype`	`dtype`	Desired data type. Defaults to np.float32.	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`
`generator`	`Generator`	Source RNG for reproducibility. If None, uses np.random module-level state.	`None`
`**kwargs`		Additional arguments passed to constructor.	`{}`

Returns:

Name	Type	Description
`Tensor`		A tensor with random values in [0, 1).

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def rand(cls, *size, out=None, dtype=None, device=None, generator=None, **kwargs):
    """
    Returns a tensor filled with random numbers from a uniform distribution on [0, 1).

    Args:
        *size (int...): Variable number of integers or a single tuple defining the shape.
        out (Tensor, optional): Output tensor to write into.
        dtype (np.dtype, optional): Desired data type. Defaults to np.float32.
        device (str, optional): Target device. Defaults to 'npu'.
        generator (np.random.Generator, optional): Source RNG for reproducibility.
            If None, uses np.random module-level state.
        **kwargs: Additional arguments passed to constructor.

    Returns:
        Tensor: A tensor with random values in [0, 1).
    """
    if not size:
        raise ValueError("rand() received no arguments")
    dtype = dtype or np.float32
    device = device or cls.DEFAULT_DEVICE

    t = cls.__check_or_create(*size, out=out, dtype=dtype, device=device, **kwargs)
    if generator is not None:
        random_val = generator.uniform(0.0, 1.0, size=t.shape).astype(dtype)
    else:
        random_val = np.random.uniform(0.0, 1.0, size=t.shape).astype(dtype)
    # Ensure values are < 1.0 for low-precision types
    is_bfloat16 = False
    try:
        from ml_dtypes import bfloat16

        if dtype == bfloat16:
            is_bfloat16 = True
    except ImportError:
        pass

    if np.issubdtype(dtype, np.floating) or is_bfloat16:
        max_val = np.nextafter(dtype(1.0), dtype(0.0))
        random_val = np.clip(random_val, 0.0, max_val)

    if t.shape == ():
        t.data.fill(random_val)
    else:
        t.data[:] = random_val
    if device == "npu":
        t._sync_to_device()
    return t

arange `classmethod` ¶

arange(
    start=0,
    end=None,
    step=1,
    *,
    shape=None,
    out=None,
    dtype=None,
    device=None,
    **kwargs
)

Returns a tensor with values from the interval [start, end) with spacing step.

Parameters:

Name	Type	Description	Default
`start`	`number`	Start of interval. Defaults to 0.	`0`
`end`	`number`	End of interval (exclusive). Required if only one argument is given.	`None`
`step`	`number`	Gap between elements. Defaults to 1.	`1`
`shape`	`tuple`	If given, reshape the 1-D sequence to this shape. `prod(shape)` must equal the length of the generated range.	`None`
`dtype`	`dtype`	Desired output data type. Inferred if not provided.	`None`
`out`	`Tensor`	Optional tensor to write output to (must match shape and dtype).	`None`
`device`	`str`	Target device. Defaults to 'npu'.	`None`

Returns:

Name	Type	Description
`Tensor`		Tensor containing the sequence (1-D by default, or `shape` if given).

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def arange(
    cls,
    start=0,
    end=None,
    step=1,
    *,
    shape=None,
    out=None,
    dtype=None,
    device=None,
    **kwargs,
):
    """
    Returns a tensor with values from the interval [start, end) with spacing `step`.

    Args:
        start (number): Start of interval. Defaults to 0.
        end (number): End of interval (exclusive). Required if only one argument is given.
        step (number): Gap between elements. Defaults to 1.
        shape (tuple, optional): If given, reshape the 1-D sequence to this shape.
            `prod(shape)` must equal the length of the generated range.
        dtype (np.dtype, optional): Desired output data type. Inferred if not provided.
        out (Tensor, optional): Optional tensor to write output to (must match shape and dtype).
        device (str, optional): Target device. Defaults to 'npu'.

    Returns:
        Tensor: Tensor containing the sequence (1-D by default, or `shape` if given).
    """

    if end is None:
        start, end = 0, start

    if dtype is None:
        if any(isinstance(x, float) for x in (start, end, step)):
            dtype = np.float32
        else:
            dtype = np.int64

    device = device or cls.DEFAULT_DEVICE

    data = np.arange(start, end, step, dtype=dtype)

    if shape is not None:
        shape = tuple(shape)
        if int(np.prod(shape)) != data.size:
            raise ValueError(
                f"iron.arange: shape={shape} (prod={int(np.prod(shape))}) does "
                f"not match generated range size {data.size}"
            )
        data = data.reshape(shape)
    else:
        shape = (data.size,)

    if out is not None:
        if out.shape != shape or out.dtype != dtype or out.device != device:
            raise ValueError(
                "Provided `out` tensor must match shape, dtype, and device"
            )
        out.data[...] = data
        if device == "npu":
            out._sync_to_device()
        return out

    t = cls(shape, dtype=dtype, device=device, **kwargs)
    t.data[...] = data
    if device == "npu":
        t._sync_to_device()
    return t

zeros_like `classmethod` ¶

zeros_like(other, dtype=None, device=None, **kwargs)

Creates a new tensor with the same shape as other, filled with zeros.

Parameters:

Name	Type	Description	Default
`other`	`Tensor`	The reference tensor to copy shape from.	required
`dtype`	`dtype`	Data type of the new tensor. Defaults to other's dtype.	`None`
`device`	`str`	Target device. Defaults to other's device.	`None`
`**kwargs`		Additional keyword arguments forwarded to the constructor.	`{}`

Returns:

Name	Type	Description
`Tensor`		A new zero-filled tensor with the same shape.

Source code in python/utils/hostruntime/tensor_class.py

@classmethod
def zeros_like(cls, other, dtype=None, device=None, **kwargs):
    """
    Creates a new tensor with the same shape as `other`, filled with zeros.

    Args:
        other (Tensor): The reference tensor to copy shape from.
        dtype (np.dtype, optional): Data type of the new tensor. Defaults to other's dtype.
        device (str, optional): Target device. Defaults to other's device.
        **kwargs: Additional keyword arguments forwarded to the constructor.

    Returns:
        Tensor: A new zero-filled tensor with the same shape.
    """
    dtype = dtype or other.dtype
    device = device or other.device
    t = cls(other.shape, dtype=dtype, device=device, **kwargs)
    t.data.fill(0)

    if device == "npu":
        t._sync_to_device()

    return t

Shared CLI/argument helpers¶

Reusable argparse flag groups and the standard design CLI dispatcher used by the programming examples.

Reusable argparse flag groups shared by the programming examples.

Almost every basic/ design repeats the same handful of CLI flags: -d/--dev (device selector), --xclbin-path/--insts-path (compile-only output paths), optionally --elf-path (xrt::elf testbench) and --emit-mlir (aiecc / vck5000 path), and the benchmark/trace pair -w/--warmup / -i/--iters / -t/--trace_size.

The helpers in this module mutate an existing argparse.ArgumentParser (so each design can keep its own design-specific flags and prog).

add_compile_args ¶

add_compile_args(
    parser: ArgumentParser,
    *,
    with_dev: bool = True,
    dev_choices: tuple[str, ...] = DEFAULT_DEV_CHOICES,
    default_dev: str | None = None,
    short_dev: str | None = "-d",
    with_elf: bool = False,
    with_full_elf: bool = False,
    with_pdi: bool = False,
    with_emit_mlir: bool = False
) -> None

Add the standard compile-mode flags.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	Parser to mutate.	required
`with_dev`	`bool`	If True (default), add `-d/--dev` (or just `--dev` when `short_dev=None`).	`True`
`dev_choices`	`tuple[str, ...]`	Allowed device-name strings (default `("npu", "npu2")`). Pass e.g. `("npu", "npu2", "xcvc1902")` for designs that also accept the VCK5000 target.	`DEFAULT_DEV_CHOICES`
`default_dev`	`str \| None`	Fixed target used when `--dev` is omitted. `None` (the default) selects the attached runtime device for run and local compile-only modes. `--emit-mlir` requires an explicit target.	`None`
`short_dev`	`str \| None`	Short-option for `--dev`. `None` to skip the short opt (matmul designs use `--dev` only because `-d` is already taken by something else).	`'-d'`
`with_elf`	`bool`	When True, also add `--elf-path` (for testbenches that load the insts as an `xrt::elf` module instead of a raw `insts.bin`).	`False`
`with_full_elf`	`bool`	When True, also add `--full-elf-path` (compile-only mode: write a single self-contained full ELF instead of an xclbin + insts pair).	`False`
`with_pdi`	`bool`	When True, also add `--pdi-path` (write the Programmable Device Image to a chosen path in compile-only mode).	`False`
`with_emit_mlir`	`bool`	When True, also add `--emit-mlir` (action `store_true`) — for designs that have a vck5000 / aiecc print-MLIR path next to the @iron.jit NPU path.	`False`

Source code in python/utils/hostruntime/argparse.py

def add_compile_args(
    parser: argparse.ArgumentParser,
    *,
    with_dev: bool = True,
    dev_choices: tuple[str, ...] = DEFAULT_DEV_CHOICES,
    default_dev: str | None = None,
    short_dev: str | None = "-d",
    with_elf: bool = False,
    with_full_elf: bool = False,
    with_pdi: bool = False,
    with_emit_mlir: bool = False,
) -> None:
    """Add the standard compile-mode flags.

    Args:
        parser: Parser to mutate.
        with_dev: If True (default), add ``-d/--dev`` (or just ``--dev``
            when ``short_dev=None``).
        dev_choices: Allowed device-name strings (default
            ``("npu", "npu2")``).  Pass e.g. ``("npu", "npu2", "xcvc1902")``
            for designs that also accept the VCK5000 target.
        default_dev: Fixed target used when ``--dev`` is omitted. ``None``
            (the default) selects the attached runtime device for run and
            local compile-only modes. ``--emit-mlir`` requires an explicit
            target.
        short_dev: Short-option for ``--dev``.  ``None`` to skip the short
            opt (matmul designs use ``--dev`` only because ``-d`` is
            already taken by something else).
        with_elf: When True, also add ``--elf-path`` (for testbenches
            that load the insts as an ``xrt::elf`` module instead of a
            raw ``insts.bin``).
        with_full_elf: When True, also add ``--full-elf-path`` (compile-only
            mode: write a single self-contained full ELF instead of an
            xclbin + insts pair).
        with_pdi: When True, also add ``--pdi-path`` (write the
            Programmable Device Image to a chosen path in compile-only
            mode).
        with_emit_mlir: When True, also add ``--emit-mlir`` (action
            ``store_true``) — for designs that have a vck5000 / aiecc
            print-MLIR path next to the @iron.jit NPU path.
    """
    if with_dev:
        names = ("--dev",) if short_dev is None else (short_dev, "--dev")
        if default_dev is None:
            dev_help = (
                "target device family (auto-detected for run and local "
                "compilation; required for --emit-mlir)"
            )
        else:
            dev_help = "target device family (default: %(default)s)"
        parser.add_argument(
            *names,
            type=str,
            choices=list(dev_choices),
            default=default_dev,
            help=dev_help,
        )
    if with_emit_mlir:
        parser.add_argument(
            "--emit-mlir",
            action="store_true",
            help="print the resolved MLIR module to stdout (requires --dev)",
        )
    parser.add_argument(
        "--xclbin-path",
        type=str,
        default=None,
        help="compile-only mode: write the xclbin here (pairs with --insts-path)",
    )
    parser.add_argument(
        "--insts-path",
        type=str,
        default=None,
        help="compile-only mode: write the instruction binary here (pairs with --xclbin-path)",
    )
    if with_elf:
        parser.add_argument(
            "--elf-path",
            type=str,
            default=None,
            help="optional ELF-wrapped insts (for the test.cpp xrt::elf flow)",
        )
    if with_full_elf:
        parser.add_argument(
            "--full-elf-path",
            type=str,
            default=None,
            help="compile-only mode: write a single self-contained full ELF "
            "(PDIs + control code) here instead of an xclbin + insts pair",
        )
    if with_pdi:
        parser.add_argument(
            "--pdi-path",
            type=str,
            default=None,
            help="compile-only mode: write the PDI here (requires --xclbin-path + --insts-path)",
        )

add_benchmark_args ¶

add_benchmark_args(
    parser: ArgumentParser,
    *,
    default_warmup: int = 2,
    default_iters: int = 5
) -> None

Add the standard benchmark flags: -w/--warmup and -i/--iters.

Source code in python/utils/hostruntime/argparse.py

def add_benchmark_args(
    parser: argparse.ArgumentParser,
    *,
    default_warmup: int = 2,
    default_iters: int = 5,
) -> None:
    """Add the standard benchmark flags: ``-w/--warmup`` and ``-i/--iters``."""
    parser.add_argument(
        "-w",
        "--warmup",
        type=int,
        default=default_warmup,
        help="benchmark warmup iterations (excluded from timings; default: %(default)s)",
    )
    parser.add_argument(
        "-i",
        "--iters",
        type=int,
        default=default_iters,
        help="benchmark timed iterations (default: %(default)s)",
    )

add_trace_arg ¶

add_trace_arg(
    parser: ArgumentParser,
    *,
    with_short: bool = True,
    default: int = 0
) -> None

Add the standard --trace_size flag.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	Parser to mutate.	required
`with_short`	`bool`	When True (default), exposes `-t` as a short opt. Set False for designs whose `-t` is already taken (e.g. matmul).	`True`
`default`	`int`	Default trace size in bytes (`0` disables tracing).	`0`

Source code in python/utils/hostruntime/argparse.py

def add_trace_arg(
    parser: argparse.ArgumentParser,
    *,
    with_short: bool = True,
    default: int = 0,
) -> None:
    """Add the standard ``--trace_size`` flag.

    Args:
        parser: Parser to mutate.
        with_short: When True (default), exposes ``-t`` as a short opt.
            Set False for designs whose ``-t`` is already taken (e.g.
            matmul).
        default: Default trace size in bytes (``0`` disables tracing).
    """
    names = ("-t", "--trace_size") if with_short else ("--trace_size",)
    parser.add_argument(
        *names,
        type=int,
        default=default,
        help="hardware trace buffer size in bytes (0 disables tracing; default: %(default)s)",
    )

add_runtime_args ¶

add_runtime_args(
    parser: ArgumentParser,
    *,
    with_io_sizes: bool = False,
    with_benchmark: bool = False
) -> None

Add the standard runtime / test-harness flags.

Pairs with :func:add_compile_args: add_compile_args covers the write-side flags (--xclbin-path, --insts-path) used by JIT designs, while this helper covers the read-side flags (--xclbin, --instr) used by test.py-style host harnesses that load a pre-compiled xclbin and run it on the NPU.

Parameters:

Name	Type	Description	Default
`parser`	`ArgumentParser`	Parser to mutate.	required
`with_io_sizes`	`bool`	When True, adds `--in1-size` / `--in2-size` / `--out-size` (bytes, `int`) for designs whose Makefile drives buffer sizes from the test harness.	`False`
`with_benchmark`	`bool`	When True, also calls :func:`add_benchmark_args` (adds `-i/--iters` and `-w/--warmup`). Off by default because most correctness-test harnesses do not benchmark.	`False`

Adds (always): --xclbin, --instr, -k/--kernel, -v/--verbosity, --verify/--no-verify, --trace-file, --ddr-id, --enable-ctrl-pkts; and via :func:add_trace_arg, -t/--trace_size.

Source code in python/utils/hostruntime/argparse.py

def add_runtime_args(
    parser: argparse.ArgumentParser,
    *,
    with_io_sizes: bool = False,
    with_benchmark: bool = False,
) -> None:
    """Add the standard runtime / test-harness flags.

    Pairs with :func:`add_compile_args`: ``add_compile_args`` covers the
    write-side flags (``--xclbin-path``, ``--insts-path``) used by JIT
    designs, while this helper covers the read-side flags (``--xclbin``,
    ``--instr``) used by ``test.py``-style host harnesses that load a
    pre-compiled xclbin and run it on the NPU.

    Args:
        parser: Parser to mutate.
        with_io_sizes: When True, adds ``--in1-size`` / ``--in2-size`` /
            ``--out-size`` (bytes, ``int``) for designs whose Makefile
            drives buffer sizes from the test harness.
        with_benchmark: When True, also calls :func:`add_benchmark_args`
            (adds ``-i/--iters`` and ``-w/--warmup``).  Off by default
            because most correctness-test harnesses do not benchmark.

    Adds (always): ``--xclbin``, ``--instr``, ``-k/--kernel``,
    ``-v/--verbosity``, ``--verify``/``--no-verify``, ``--trace-file``,
    ``--ddr-id``, ``--enable-ctrl-pkts``; and via :func:`add_trace_arg`,
    ``-t/--trace_size``.
    """
    parser.add_argument(
        "--xclbin",
        type=str,
        required=True,
        help="path to the pre-compiled xclbin to load",
    )
    parser.add_argument(
        "--instr",
        type=str,
        default="instr.txt",
        help="path of file containing userspace instructions sent to the NPU (default: %(default)s)",
    )
    parser.add_argument(
        "-k",
        "--kernel",
        type=str,
        default="MLIR_AIE",
        help="kernel name in the xclbin (default: %(default)s)",
    )
    parser.add_argument(
        "-v",
        "--verbosity",
        type=int,
        default=0,
        help="verbosity level (default: %(default)s)",
    )
    parser.add_argument(
        "--verify",
        action=argparse.BooleanOptionalAction,
        default=True,
        help="verify the NPU output against the reference (default: %(default)s)",
    )
    add_trace_arg(parser)
    parser.add_argument(
        "--trace-file",
        type=str,
        default="trace.txt",
        help="where to store trace output (default: %(default)s)",
    )
    parser.add_argument(
        "--ddr-id",
        type=int,
        default=4,
        help="DDR buffer index for trace (0-4, or -1 to append after last tensor; default: %(default)s)",
    )
    parser.add_argument(
        "--enable-ctrl-pkts",
        action="store_true",
        help="enable control packets",
    )
    if with_io_sizes:
        parser.add_argument(
            "--in1-size",
            type=int,
            default=0,
            help="input 1 buffer size in bytes (default: %(default)s)",
        )
        parser.add_argument(
            "--in2-size",
            type=int,
            default=0,
            help="input 2 buffer size in bytes (default: %(default)s)",
        )
        parser.add_argument(
            "--out-size",
            type=int,
            default=0,
            help="output buffer size in bytes (default: %(default)s)",
        )
    if with_benchmark:
        add_benchmark_args(parser)

device_from_args ¶

device_from_args(
    args,
    *,
    dev_attr: str = "dev",
    n_cols: "int | None | str" = "auto"
)

Resolve a parsed-argparse namespace to a :class:aie.iron.device.Device.

Collapses the boilerplate variants the example suite used to repeat across ~30 sites::

from_name(args.dev, n_cols=None)   # "use all cols"
from_name(args.dev, n_cols=1)      # "single col"

into one helper with an explicit n_cols parameter.

Parameters:

Name	Type	Description	Default
`args`		An `argparse.Namespace` with a `.dev` attribute (or whatever `dev_attr` names). When `n_cols="auto"` and `args` also has an `.n_cols` attribute, that attribute overrides the default.	required
`dev_attr`	`str`	Name of the device-string attribute on `args`. Defaults to `"dev"`.	`'dev'`
`n_cols`	`'int \| None \| str'`	Column-count selector. `"auto"` (default) means single column on every NPU family — the dominant pattern across the example suite and the safest "I haven't thought about this yet" choice. Pass an explicit `int` (`2`, `4`, ...) or `None` (full width) to override. An `.n_cols` attribute on `args` (e.g. from a future CLI flag) takes precedence over the default when `n_cols="auto"`.	`'auto'`

Returns:

Type	Description
	Device \| None: The selected device, or `None` when the CLI leaves
	device selection to the runtime.

Source code in python/utils/hostruntime/argparse.py

def device_from_args(
    args,
    *,
    dev_attr: str = "dev",
    n_cols: "int | None | str" = "auto",
):
    """Resolve a parsed-argparse namespace to a :class:`aie.iron.device.Device`.

    Collapses the boilerplate variants the example suite used to repeat
    across ~30 sites::

        from_name(args.dev, n_cols=None)   # "use all cols"
        from_name(args.dev, n_cols=1)      # "single col"

    into one helper with an explicit ``n_cols`` parameter.

    Args:
        args: An ``argparse.Namespace`` with a ``.dev`` attribute (or
            whatever ``dev_attr`` names).  When ``n_cols="auto"`` and
            ``args`` also has an ``.n_cols`` attribute, that attribute
            overrides the default.
        dev_attr: Name of the device-string attribute on ``args``.
            Defaults to ``"dev"``.
        n_cols: Column-count selector.  ``"auto"`` (default) means
            single column on every NPU family — the dominant pattern
            across the example suite and the safest "I haven't thought
            about this yet" choice.  Pass an explicit ``int`` (``2``,
            ``4``, ...) or ``None`` (full width) to override.  An
            ``.n_cols`` attribute on ``args`` (e.g. from a future CLI
            flag) takes precedence over the default when
            ``n_cols="auto"``.

    Returns:
        Device | None: The selected device, or ``None`` when the CLI leaves
        device selection to the runtime.
    """
    dev = getattr(args, dev_attr)
    if dev is None:
        return None

    # Lazy import: argparse.py is otherwise pure-stdlib and avoids the
    # aie.iron import edge until a caller actually needs to resolve a Device.
    from aie.iron.device import from_name

    resolved_cols: int | None
    if n_cols == "auto":
        resolved_cols = getattr(args, "n_cols", None)
        if resolved_cols is None:
            resolved_cols = 1
    elif isinstance(n_cols, str):
        raise ValueError(f"n_cols must be an int, None, or 'auto'; got {n_cols!r}")
    else:
        resolved_cols = n_cols
    return from_name(dev, n_cols=resolved_cols)

run_design_cli — standard 3-mode dispatcher for IRON design CLIs.

Almost every basic/ design's main() is the same skeleton:

def main():
    opts = _make_argparser().parse_args()
    # optional: _validate(opts)
    if opts.emit_mlir:
        print(design.specialize(**_compile_kwargs(opts)).as_mlir()); return
    if opts.xclbin_path:
        _compile_only(opts)
        return
    _run_and_verify(opts)

…where _compile_only always does the same --insts-path check + device binding + design.specialize(**kw).compile( xclbin_path=opts.xclbin_path, inst_path=opts.insts_path [, elf_path=opts.elf_path][, pdi_path=opts.pdi_path]).

This module wraps that skeleton so each design just declares the two pieces it actually owns (compile kwargs, the verify body) and lets the dispatcher do the branching + the boilerplate around it. An optional emit_mlir= callback covers the rare design whose generator needs real iron.tensor instances at MLIR-gen time.

run_design_cli ¶

run_design_cli(
    design,
    opts,
    *,
    compile_kwargs: (
        Mapping[str, Any]
        | Callable[[Any], Mapping[str, Any]]
    ),
    run_and_verify: Callable[[Any], None] | None = None,
    device: _DeviceArg | None = None,
    emit_mlir: Callable[[Any], None] | None = None,
    validate: Callable[[Any], None] | None = None
) -> None

Standard 3-mode CLI dispatcher for basic/ designs.

The standard branch tree (in order):

Bind an explicit --dev target or concrete device argument, or detect the attached runtime device for run and local compile-only modes.
If validate is given, call validate(opts).
If opts.emit_mlir is True, then:
- If an emit_mlir callback was supplied, call emit_mlir(opts).
- Otherwise print design.specialize(**compile_kwargs).as_mlir() — the right answer for almost every design (no tensor args needed when shapes come from CompileTime[T] params).
Then return.
If opts.xclbin_path or opts.full_elf_path is set (compile-only):
- For full_elf_path: call design.specialize(**compile_kwargs) .compile(full_elf_path=opts.full_elf_path) — a single self-contained ELF, no xclbin/insts.
- Otherwise refuse if opts.insts_path is unset (sys.exit with the standard message), then call design.specialize(**compile_kwargs).compile( xclbin_path=opts.xclbin_path, inst_path=opts.insts_path, [elf_path=opts.elf_path]).
Otherwise, call run_and_verify(opts).

Parameters:

Name	Type	Description	Default
`design`		The `@iron.jit`-decorated design (a `CallableDesign`).	required
`opts`		Parsed `argparse.Namespace` — must expose at minimum `xclbin_path` / `insts_path` (the standard `add_compile_args` flags). `dev` may be `None` to request automatic runtime selection. `emit_mlir` and `elf_path` are read if present.	required
`compile_kwargs`	`Mapping[str, Any] \| Callable[[Any], Mapping[str, Any]]`	Either a dict OR a callable that takes `opts` and returns the kwargs dict to pass to `design.specialize()`. Callable form is convenient for the typical `_compile_kwargs(opts)` helper most designs already have.	required
`run_and_verify`	`Callable[[Any], None] \| None`	Callable invoked in the default (NPU run + numpy verify) branch. Takes `opts`, returns nothing — exits non-zero on failure (e.g. via `assert_pass`). Optional: ml/ designs whose verification lives in a C++ test harness omit this; reaching the run branch without it exits with a clear "no python run path" message.	`None`
`device`	`_DeviceArg \| None`	Optional iron `Device` instance OR callable `opts -> Device`. An explicit `opts.dev` is resolved through this value, or through `from_name(opts.dev)` when omitted. When `opts.dev` is `None`, the dispatcher detects the attached NPU family, updates `opts.dev`, and then invokes a callable selector so it can retain its declared column profile. Pass a `Device` instance to pin a target independently of the CLI.	`None`
`emit_mlir`	`Callable[[Any], None] \| None`	Optional callable for the `--emit-mlir` branch. If `opts.emit_mlir` is True but this is None, the dispatcher falls back to printing `design.specialize(**compile_kwargs).as_mlir()` — sufficient for any design whose generator reads its shapes from `CompileTime[T]` params (i.e. doesn't read shape off the passed-in tensor). Pass an explicit callable when the generator needs real `iron.tensor` instances at MLIR-gen time.	`None`
`validate`	`Callable[[Any], None] \| None`	Optional callable invoked after target selection — e.g. for shape / arg consistency checks that should fire in all modes.	`None`

Source code in python/utils/hostruntime/cli.py

def run_design_cli(
    design,
    opts,
    *,
    compile_kwargs: Mapping[str, Any] | Callable[[Any], Mapping[str, Any]],
    run_and_verify: Callable[[Any], None] | None = None,
    device: _DeviceArg | None = None,
    emit_mlir: Callable[[Any], None] | None = None,
    validate: Callable[[Any], None] | None = None,
) -> None:
    """Standard 3-mode CLI dispatcher for basic/ designs.

    The standard branch tree (in order):

      1. Bind an explicit ``--dev`` target or concrete ``device`` argument,
         or detect the attached runtime device for run and local compile-only
         modes.
      2. If ``validate`` is given, call ``validate(opts)``.
      3. If ``opts.emit_mlir`` is True, then:

         * If an ``emit_mlir`` callback was supplied, call ``emit_mlir(opts)``.
         * Otherwise print ``design.specialize(**compile_kwargs).as_mlir()``
           — the right answer for almost every design (no tensor args needed
           when shapes come from ``CompileTime[T]`` params).

         Then return.

      4. If ``opts.xclbin_path`` or ``opts.full_elf_path`` is set (compile-only):

         * For ``full_elf_path``: call ``design.specialize(**compile_kwargs)
           .compile(full_elf_path=opts.full_elf_path)`` — a single
           self-contained ELF, no xclbin/insts.
         * Otherwise refuse if ``opts.insts_path`` is unset (``sys.exit`` with
           the standard message), then call
           ``design.specialize(**compile_kwargs).compile(
           xclbin_path=opts.xclbin_path, inst_path=opts.insts_path,
           [elf_path=opts.elf_path])``.

      5. Otherwise, call ``run_and_verify(opts)``.

    Args:
        design: The ``@iron.jit``-decorated design (a ``CallableDesign``).
        opts: Parsed ``argparse.Namespace`` — must expose at minimum
            ``xclbin_path`` / ``insts_path`` (the standard
            ``add_compile_args`` flags). ``dev`` may be ``None`` to request
            automatic runtime selection. ``emit_mlir`` and ``elf_path`` are
            read if present.
        compile_kwargs: Either a dict OR a callable that takes ``opts``
            and returns the kwargs dict to pass to
            ``design.specialize()``.  Callable form is convenient for
            the typical ``_compile_kwargs(opts)`` helper most designs
            already have.
        run_and_verify: Callable invoked in the default (NPU
            run + numpy verify) branch.  Takes ``opts``, returns nothing
            — exits non-zero on failure (e.g. via ``assert_pass``).
            Optional: ml/ designs whose verification lives in a C++ test
            harness omit this; reaching the run branch without it exits
            with a clear "no python run path" message.
        device: Optional iron ``Device`` instance OR callable
            ``opts -> Device``. An explicit ``opts.dev`` is resolved through
            this value, or through ``from_name(opts.dev)`` when omitted. When
            ``opts.dev`` is ``None``, the dispatcher detects the attached NPU
            family, updates ``opts.dev``, and then invokes a callable selector
            so it can retain its declared column profile. Pass a ``Device``
            instance to pin a target independently of the CLI.
        emit_mlir: Optional callable for the ``--emit-mlir`` branch.
            If ``opts.emit_mlir`` is True but this is None, the dispatcher
            falls back to printing
            ``design.specialize(**compile_kwargs).as_mlir()`` — sufficient
            for any design whose generator reads its shapes from
            ``CompileTime[T]`` params (i.e. doesn't read shape off the
            passed-in tensor).  Pass an explicit callable when the
            generator needs real ``iron.tensor`` instances at MLIR-gen
            time.
        validate: Optional callable invoked after target selection — e.g.
            for shape / arg consistency checks that should fire in all modes.
    """
    # Late imports keep this module cheap to import and avoid circular imports
    # until a design actually enters the dispatcher.
    from aie.utils.hostruntime import set_current_device

    emit_mlir_requested = getattr(opts, "emit_mlir", False)
    full_elf_path = getattr(opts, "full_elf_path", None)
    compile_only_requested = (
        getattr(opts, "xclbin_path", None) is not None or full_elf_path is not None
    )
    requested_dev = getattr(opts, "dev", None)

    if getattr(opts, "xclbin_path", None) is not None and not getattr(
        opts, "insts_path", None
    ):
        sys.exit("--xclbin-path requires --insts-path (must be set together)")

    if full_elf_path is not None and getattr(opts, "xclbin_path", None) is not None:
        sys.exit(
            "--full-elf-path and --xclbin-path/--insts-path are mutually exclusive"
        )

    if requested_dev is None:
        has_concrete_device = device is not None and not callable(device)
        if emit_mlir_requested and not has_concrete_device:
            sys.exit(
                "run_design_cli: --emit-mlir requires an explicit target; "
                "pass --dev."
            )

        if device is None and not hasattr(opts, "dev"):
            raise ValueError(
                "run_design_cli: device=None requires opts to expose a "
                "'dev' attribute (the standard add_compile_args flag). "
                "Pass device=<Device or callable> explicitly otherwise."
            )

        if device is not None and not callable(device):
            resolved_device = device
        else:
            mode = "compile" if compile_only_requested else "run"
            _detect_runtime_target(opts, mode=mode)
            if device is None:
                from aie.iron.device import from_name

                resolved_device = from_name(opts.dev)
            else:
                resolved_device = _resolve_device(device, opts)
        set_current_device(resolved_device)
    else:
        if device is None:
            from aie.iron.device import from_name

            resolved_device = from_name(requested_dev)
        else:
            resolved_device = _resolve_device(device, opts)
        set_current_device(resolved_device)

    if validate is not None:
        validate(opts)

    if emit_mlir_requested:
        if emit_mlir is not None:
            emit_mlir(opts)
        else:
            kwargs = _resolve(compile_kwargs, opts)
            print(design.specialize(**kwargs).as_mlir())
        return

    if compile_only_requested:
        kwargs = _resolve(compile_kwargs, opts)
        spec = design.specialize(**kwargs)
        if full_elf_path is not None:
            # Full ELF is self-contained: no xclbin/insts pair.
            spec.compile(full_elf_path=full_elf_path)
            return
        compile_opts = dict(xclbin_path=opts.xclbin_path, inst_path=opts.insts_path)
        elf_path = getattr(opts, "elf_path", None)
        if elf_path is not None:
            compile_opts["elf_path"] = elf_path
        pdi_path = getattr(opts, "pdi_path", None)
        if pdi_path is not None:
            compile_opts["pdi_path"] = pdi_path
        spec.compile(**compile_opts)
        return

    if run_and_verify is None:
        sys.exit(
            "run_design_cli: no run_and_verify callback was provided — this "
            "design only supports the compile-only path (pass "
            "--xclbin-path + --insts-path) or --emit-mlir."
        )
    run_and_verify(opts)

XRT runtime implementation¶

Concrete XRT-backed runtime, tensor, and scratchpad types. These modules import pyxrt, so they are only importable in environments where XRT / PyXRT is installed. MkDocs CI does not install XRT, so these symbols are summarized here (same approach as the non-importable JIT helpers on the IRON page) with links to source. On a machine with PyXRT available they can also be rendered with mkdocstrings, for example:

::: utils.hostruntime.xrtruntime.hostruntime
    options:
      show_root_heading: false

Symbol	Module	Summary
`XRTKernelHandle`	`xrtruntime.hostruntime`	Handle for a loaded XRT kernel.
`XRTKernelResult`	`xrtruntime.hostruntime`	Result wrapper for a PyXRT kernel run.
`XRTHostRuntime`	`xrtruntime.hostruntime`	Singleton manager for AIE XRT resources.
`CachedXRTKernelHandle`	`xrtruntime.hostruntime`	Cached handle for a loaded XRT kernel.
`CachedXRTRuntime`	`xrtruntime.hostruntime`	Cached `XRTHostRuntime` that reuses contexts for the same xclbin.
`XRTTensor`	`xrtruntime.tensor`	Tensor backed by NPU/CPU-accessible memory managed with PyXRT.
`ParameterScratchpad`	`xrtruntime.parameter_scratchpad`	Write named runtime parameters to the NPU scratchpad buffer.

Host Runtime (aie.utils.hostruntime)¶

Runtime abstractions¶

HostRuntimeError ¶

KernelHandle ¶

KernelResult ¶

npu_time property ¶

trace_config property ¶

has_trace ¶

is_success abstractmethod ¶

HostRuntime ¶

check_device_consistency ¶

cleanup ¶

evict_context ¶

load abstractmethod ¶

run abstractmethod ¶

load_and_run ¶

device abstractmethod ¶

read_insts_binary classmethod ¶

read_insts classmethod ¶

prepare_args_for_trace classmethod ¶

extract_trace_from_args classmethod ¶

process_trace classmethod ¶

verify_results classmethod ¶

run_test ¶

Tensor and device utilities¶

Tensor ¶

data abstractmethod property ¶

shape abstractmethod property ¶

nbytes cached property ¶

element_size cached property ¶

to ¶

numpy ¶

to_torch ¶

torch_view ¶

from_torch classmethod ¶

fill_ ¶

numel ¶

ones classmethod ¶

zeros classmethod ¶

full classmethod ¶

randint classmethod ¶

rand classmethod ¶

arange classmethod ¶

zeros_like classmethod ¶

set_current_device ¶

bfloat16_safe_allclose ¶

Tensor ¶

data abstractmethod property ¶

shape abstractmethod property ¶

nbytes cached property ¶

element_size cached property ¶

to ¶

numpy ¶

to_torch ¶

torch_view ¶

from_torch classmethod ¶

fill_ ¶

numel ¶

ones classmethod ¶

zeros classmethod ¶

full classmethod ¶

randint classmethod ¶

rand classmethod ¶

arange classmethod ¶

zeros_like classmethod ¶

Shared CLI/argument helpers¶

add_compile_args ¶

add_benchmark_args ¶

add_trace_arg ¶

add_runtime_args ¶

device_from_args ¶

run_design_cli ¶

XRT runtime implementation¶

Host Runtime (`aie.utils.hostruntime`)¶

npu_time `property` ¶

trace_config `property` ¶

is_success `abstractmethod` ¶

load `abstractmethod` ¶

run `abstractmethod` ¶

device `abstractmethod` ¶

read_insts_binary `classmethod` ¶

read_insts `classmethod` ¶

prepare_args_for_trace `classmethod` ¶

extract_trace_from_args `classmethod` ¶

process_trace `classmethod` ¶

verify_results `classmethod` ¶

data `abstractmethod` `property` ¶

shape `abstractmethod` `property` ¶

nbytes `cached` `property` ¶

element_size `cached` `property` ¶

from_torch `classmethod` ¶

ones `classmethod` ¶

zeros `classmethod` ¶

full `classmethod` ¶

randint `classmethod` ¶

rand `classmethod` ¶

arange `classmethod` ¶

zeros_like `classmethod` ¶

data `abstractmethod` `property` ¶

shape `abstractmethod` `property` ¶

nbytes `cached` `property` ¶

element_size `cached` `property` ¶

from_torch `classmethod` ¶

ones `classmethod` ¶

zeros `classmethod` ¶

full `classmethod` ¶

randint `classmethod` ¶

rand `classmethod` ¶

arange `classmethod` ¶

zeros_like `classmethod` ¶