Skip to content

Engine execution and comparison API

The following names are available directly from tensoracle.

Build declarations

InputShape dataclass

The minimum, preferred, and maximum dimensions of one input.

minimum instance-attribute

optimum instance-attribute

maximum instance-attribute

__init__(minimum, optimum, maximum)

__post_init__()

fixed(shape) classmethod

Declare one fixed input shape.

ranged(minimum, optimum, maximum) classmethod

Declare the three shapes of one TensorRT optimization profile.

BuilderSettings dataclass

TensorRT builder options that affect the serialized engine.

workspace_limit = 1 << 30 class-attribute instance-attribute

optimization_level = 3 class-attribute instance-attribute

tf32 = False class-attribute instance-attribute

__init__(workspace_limit=1 << 30, optimization_level=3, tf32=False)

EngineBuild dataclass

One engine's named input shapes and builder settings.

inputs instance-attribute

settings = field(default_factory=BuilderSettings) class-attribute instance-attribute

__init__(inputs, settings=BuilderSettings())

build_serialized_engine(graph, declaration, *, plugins=(), logger=None)

Register plugins before calling graph and serializing its engine.

ShapeTensorOutputWarning

Bases: UserWarning

Shape tensor outputs cannot be handled by tensoracle execution units.

Saved engines

TensorSpec dataclass

Expected dtype, build-time dimensions, and optional location and format.

dtype instance-attribute

shape instance-attribute

location = None class-attribute instance-attribute

format = None class-attribute instance-attribute

__init__(dtype, shape, location=None, format=None)

EngineIO dataclass

Expected named inputs and outputs of a TensorRT engine.

inputs instance-attribute

outputs instance-attribute

__init__(inputs, outputs)

EngineHandle dataclass

Keep the deserialized engine and its owning runtime alive together.

engine instance-attribute

runtime instance-attribute

path instance-attribute

_shape_tensor_outputs = dataclass_field(default=(), repr=False) class-attribute instance-attribute

__init__(engine, runtime, path, _shape_tensor_outputs=())

create_execution()

Create an independent execution context with explicit close semantics.

open_engine(graph, declaration, *, name, graph_version, weights, io, plugins=(), force_rebuild=False, logger=None)

Register plugins and build or load an engine for the supplied conditions.

EngineCacheError

Bases: ValueError

A stored engine cannot be used for this declaration.

CorruptEngineError

Bases: EngineCacheError

A stored engine is incomplete or corrupt.

EngineConditionError

Bases: EngineCacheError

A stored engine belongs to different build conditions.

EngineIOError

Bases: EngineCacheError

The engine's input or output signature differs from the declaration.

Engine execution

Call EngineHandle.create_execution() after open_engine. Each unit owns an independent context; the caller owns device buffers and streams and confirms completion before closing it.

EngineExecution

One independent TensorRT context retained with its checked engine handle.

Use handle.create_execution() or construct this type with an EngineHandle. The caller owns buffers, streams, their ordering, and completion. This object neither copies data nor synchronizes. Finish outstanding work before changing shapes or streams, closing, or reusing the object from another thread.

_handle = handle instance-attribute

_names = tuple(engine.get_tensor_name(i) for i in range(engine.num_io_tensors)) instance-attribute

_ranges = {} instance-attribute

_context = engine.create_execution_context() instance-attribute

_ready = not self._ranges instance-attribute

__init__(handle)

set_input_shapes(shapes)

Set every ranged input together, validating all shapes before TensorRT.

Fixed inputs, including single-point profiles, are initialized at creation and omitted here. A TensorRT failure invalidates the setting until a complete subsequent call succeeds; partial settings are not rolled back.

get_tensor_shape(name)

Read a concrete input or output shape after all input shapes are ready.

get_tensor_dtype(name)

Read a tensor's TensorRT dtype for allocating a caller-owned buffer.

enqueue(addresses, *, stream)

Enqueue all tensor addresses on the caller's stream without waiting.

Addresses must be positive integers, and the stream must be a nonnegative integer (0 is the default stream). Bool is rejected. This method cannot check buffer dtype, dimensions, layout, size, lifetime, or device selection.

close()

Release context before handle without synchronization; repeated calls do nothing.

The caller must confirm that all enqueued work has completed before closing.

__enter__()

__exit__(exc_type, exc_value, traceback)

_require_open()

_require_ready()

_check_tensor(name)

_check_names(values, names, field) staticmethod

_set_shapes(shapes, *, creating=False)

Plugin loading

PluginLibraryError

Bases: ValueError

A native plugin library cannot be registered for this process.

Output comparison

OutputComparison dataclass

Measurements for a candidate output against its reference output.

Float measurements are None for integer or boolean arrays and when either float array contains a non-finite value. Tolerance measurements are also None when the caller has not supplied both tolerances. Mismatch measurements apply only to integer and boolean arrays.

candidate_nonfinite_count instance-attribute

reference_nonfinite_count instance-attribute

max_absolute_error = None class-attribute instance-attribute

relative_l2_error = None class-attribute instance-attribute

cosine_similarity = None class-attribute instance-attribute

outside_tolerance_count = None class-attribute instance-attribute

outside_tolerance_fraction = None class-attribute instance-attribute

mismatch_count = None class-attribute instance-attribute

mismatch_fraction = None class-attribute instance-attribute

__init__(candidate_nonfinite_count, reference_nonfinite_count, max_absolute_error=None, relative_l2_error=None, cosine_similarity=None, outside_tolerance_count=None, outside_tolerance_fraction=None, mismatch_count=None, mismatch_fraction=None)

compare_outputs(candidate, reference, *, atol=None, rtol=None)

Measure one candidate output against the corresponding reference output.

Both inputs must be nonempty NumPy arrays of equal shape and the same kind: floating point, integer, or boolean. Floating point inputs are converted to float64 before arithmetic. Tolerances must be finite, nonnegative, and supplied together. This function reports measurements; the caller chooses the limits that constitute an acceptable output.