Performance¶

teeny's performance model rests on one idea: fold everything that is known at compile time, carry only what is genuinely dynamic. A strides<...> layout with known strides is an empty base (EBO) — it costs zero bytes and its offsets fold to compile-time constants. You pay, in footprint and in lost optimization, only for the axes whose extent or stride is truly a runtime value.

This page is about where that cost lands — especially for dynamic strides, which a kernel carries by value — and how to keep it off the hot path.

Footprint: what a by-value view actually carries¶

A view is passed to a kernel by value (it is trivially copyable — a pointer plus the dynamic parts of its geometry). sizeof the view is what you carry in registers / kernel argument space:

Two things fall out immediately:

• A fully-static view is a bare pointer. Bake known dims/strides into the type (shape<3,3>, strides<...>) and the geometry vanishes.
• ccontiguous never stores strides — it derives them from the shape. So even for a dynamic shape, a contiguous view is 24 bytes lighter than a dynamic_strides one of the same rank. Reach for dynamic_strides only when the strides are genuinely arbitrary (a transpose gap, an external DLPack layout).

On the GPU this footprint is the dominant cost: a rank-3 dynamic_strides view is ~7 registers per thread, per view; a kernel juggling several fields can lose occupancy to it. This is the cost your instinct should flag.

The per-element hot path is not the problem¶

A natural worry: "runtime strides are extra weight carried on every element access." For a well-structured loop, the compiler removes it. Loop-invariant strides are hoisted once and the address is strength-reduced to a pointer increment. The same for i: d(i) += a(i) loop, compiled three ways at -O2, has an identical three-instruction body:

No imul, no per-element stride reload in any of them. This works because operator() is _TNY_API inline and the rank is constexpr, so the offset unrolls and the strides are plainly loop-invariant to the optimizer. In a simple march, dynamic strides cost nothing per element.

Where dynamic strides genuinely cost¶

1. Register pressure / GPU occupancy — the footprint above. The real, measurable cost, and the reason to prefer static strides and 32-bit indexing (below).
2. Vectorization — an unknown stride can never be proven contiguous, so it can't SIMD. For contiguous ops teeny takes a linear fast path that auto-vectorizes in every case except one (below): an out-of-place op writes a fresh, non-aliasing result (__restrict__ destination), and an in-place scalar or unary op (a *= 2, a.exp_()) is a single-array read-modify-write with no second pointer — so both vectorize. The lone exception is an in-place op with a tensor rhs (a.add_(b)): b may alias or overlap the destination (a.add_(a)), so the compiler must assume overlap and stays scalar — restricting there would be UB. (Measured: at -O3, the fresh/single-array loops emit packed SIMD; the may-alias tensor case stays scalar.)
3. ND random-access gather — the one place loop-invariant hoisting can't help. An interpolation/resampling gather (reading a window of neighbours around each arbitrary sample point) computes base + Σ idxₖ·strideₖ at scattered points, not a linear march. With dynamic spatial strides that is N runtime multiply-adds per gather; with static strides they fold to compile-time constants. Keep the spatial strides static (below) and the gather folds.
4. Index width — dynamic offset math in int64 is slower and uses more registers than int32, especially on the device. reindex/dispatch_index narrow a boundary view to 32-bit wherever that is lossless (below).

How to keep it fast — the fold in practice¶

teeny is built to route around the above; the idioms:

• Bake known geometry into the type. local<double, shape<3,3>>, strides<9,3,1> — zero footprint, compile-time offsets.
• Views preserve static strides. Slicing, peel, permute, flip, unsqueeze/squeeze, and reshape all fold their output strides to compile-time constants where the source is static — so a static source stays static through a chain of views (see Views & structure).
• Recover static inner dims at the boundary. recast(shape<-1,3,3>{}) re-types a runtime (n,3,3) view so the 3s fold, without a copy and preserving the source strides.
• Peel the batch into the pointer. peel_front<Nbatch> bakes the batch offset into each sub-view's data handle, so the inner kernel sees only the (usually static) spatial + channel strides — the batch strides never enter the inner loop (see Dispatch & the anyrank boundary).
• Dispatch runtime shape to static kernels. dispatch_value / dispatch_rank instantiate a static-shape kernel from a runtime spatial rank, so the hot code is compiled against a folded shape and strides.

Rule of thumb for a hot kernel: the inner loop should run over a view whose strides are static (or contiguous). If it doesn't, recast / dispatch at the launch boundary until it does; carry the dynamic strides only in the outer/batch loop, where they hoist for free.

The fast paths in detail¶

Three optimizations do most of the work behind the advice above. Each is described here as it behaves today — including where it deliberately stops.

32-bit offset math¶

t.reindex<int32_t>() (giving a shape32 view) is a no-copy, layout-preserving retype of the offset index width: same pointer, same layout kind, extents and dynamic strides narrowed. Narrowing a dynamic boundary view halves its by-value footprint (rank-2: 40 → 24 B) and runs its offset arithmetic in 32-bit — the biggest single device win on this page.

dispatch_index(v, f) is the launch-site spelling: it instantiates the kernel for both widths and picks the narrowed one when v.index_fits<int32_t>() — which checks both halves of what narrowing touches: every element offset must fit the target width, and so must every axis's size (the shape narrows too, and it is what your loops count up to). dispatch_rank<narrow_index>(at, f) fuses that choice into the anyrank rank dispatch. Both are opt-in per launch site — narrowing doubles the instantiations, so teeny never does it behind your back.

The rank-erased carrier carries the same pair, so a CUDA boundary can narrow once, before the launch, and still keep the batch idiom (peel_front<-Sr>, whose whole point is that ndim stays runtime):

if (at.index_fits<int32_t>()) launch(at.reindex<int32_t>());
else                          launch(at);

Every cell peeled off the narrowed carrier is int32-indexed, and the carrier's own inline shape/stride store halves (MaxRank × 2 × 8 B → × 4 B) — which matters when the carrier itself is a kernel parameter. See Dispatch. This is a device optimization: on the CPU it measures neutral, so it stays opt-in there too.

Mixed-width operands broaden rather than narrow: a + b over an int32-indexed and an int64-indexed view takes the wider operand's index type (and, where the two also disagree in signedness, a signed type wide enough for both). That is lossless, and it avoids truncating the wide operand's strides down to a narrow result width.

Vectorized elementwise ops¶

Contiguous elementwise ops take a linear fast path in place of the per-element mixed-radix decode, and that flat loop auto-vectorizes. Two flavours, by whether a second array is in play:

• Out-of-place (a + b, a * 2, exp(a), a < b) writes a freshly allocated result, so the flat for (i) c[i] = op(a[i], …) marks the destination __restrict__. That is safe precisely because the destination is fresh; the sources are left un-restricted, so a + a stays correct.
• In-place scalar / unary / iota_ / fill_ (a *= 2, a.add_(1), a.exp_(), a.iota_(…)) is a single-array read-modify-write — one pointer, nothing to alias, so it vectorizes with no __restrict__ at all. The scalar and unary ops are order-independent, so they take the fast path over any dense view (is_dense() — C-order, F-order, or permuted; a transposed in-place op vectorizes too); iota_ is order-dependent and so needs exact C-contiguity. An atomic scalar (a.atomic_add_(x)) keeps the general decode path.

One case stays scalar by design: an in-place op with a tensor rhs (a.add_(b)). b may alias or overlap the destination — a.add_(a) is a legal call — so neither a restrict (undefined behaviour) nor a plain loop (where the compiler must assume overlap) can safely vectorize it. A broadcast or strided operand likewise falls back to the decode: correct, just not vectorized.

Codegen check (-O3 -S, g++ and clang++): the fast-path write loops emit packed addpd, and the may-alias tensor case emits scalar addsd.

Unrolled small static shapes¶

When both dot/sqdist operands are static-shaped (their extents match exactly at compile time — each static_asserts that) and both C-contiguous, one shared linear index addresses the same logical element in both, so there is no per-step decode at all. Unrolling over the compile-time element count then emits straight-line code with no loop back-edge. Axis reductions over a fully static shape take the same unrolled path. So a small fixed-size dot — a cross-channel dot in a positive-definite solve, a fixed-size stencil tap accumulation — pays no loop overhead it doesn't need.

The unroll falls back to the ordinary decode as soon as either operand is dynamic, is not C-contiguous (a strided or permuted view, or teeny's own strides<...> layout), or the two disagree in layout. The results are identical either way.

A deliberate limit: the unroll is capped at small shapes. Both unrolled paths emit one step per element, so a large static shape costs compile time rather than buying run time — clang refuses outright past 256 elements (its default -fbracket-depth expression-nesting limit), and g++ takes about a minute on a shape<64,64> reduction. A fully static shape of more than TNY_MAX_STATIC_UNROLL (256) elements therefore takes the ordinary runtime-decode path, exactly as a dynamic one does — same results, no unroll. -DTNY_MAX_STATIC_UNROLL=64 trades more of the unroll away for faster compiles; raising it past 256 breaks clang.