2805 Bowers Ave, Santa Clara, CA 95051 | 408-730-2275
research@colfax-intl.com

Accelerating Staple Networks for Normalizing Flows in Lattice Quantum Chromodynamics

Frank Lin1, Ryan Abbott2, Mathis Gerdes3,4, Daniel C. Hackett5, Gurtej Kanwar6, Fernando Romero-López7, Phiala E. Shanahan3,4

1Colfax Research2Columbia University3Massachusetts Institute of Technology4Institute for AI and Fundamental Interactions5Institute for Nuclear Theory6University of Edinburgh7University of Bern

Introduction

Machine learning offers a promising route to accelerating lattice quantum chromodynamics (QCD) simulations. In particular, normalizing flows can speed up Monte Carlo sampling in lattice QCD [1, 2] (also see [3] for an introductory tutorial), but there is a high computational cost to running these flow models at a useful resolution. This post describes our use of CuTe DSL kernels to accelerate the staple network, a core component of our approach to implementing these flows.

Staple networks involve many small 3×3 complex double-precision matrix multiplications in a stencil pattern on a 4D lattice, a workload unlike those commonly found in LLMs. On an NVIDIA H200 NVL, our CuTe DSL kernels accelerated hidden layers of the forward pass of these networks by 700–1200x over a reference PyTorch implementation and 9–11x over a previous fused Triton implementation. With 4 or 8 channels per layer, they are also 2.0–3.4x faster than a Triton kernel to which we ported, with AI assistance, our CuTe DSL optimizations (where expressible by Triton).

In this post we briefly introduce some physics background for the problem, describe the mathematical operations involved in a layer of a staple network, and then discuss our kernel design along with additional optimizations targeted towards a larger number of channels.

Background on lattice QCD and the staple network

Quantum chromodynamics is the theory of the strong nuclear force—the force that binds quarks together to create protons and neutrons and that binds those protons and neutrons together into the nuclei of every atom. As the name implies, the strong nuclear force is strong: pulling apart protons and neutrons into their individual quark and gluon pieces is only possible by smashing them together at extremely high energies, typically only available in environments like the Large Hadron Collider at CERN. What this means in practice is that to calculate strong nuclear force effects—such as the binding energy of a proton—typical pen-and-paper methods that rely on weak-coupling approximations break down. Instead, lattice QCD discretizes spacetime into a four-dimensional grid, on which these quantities can be computed with large-scale numerical simulations.

The numerical simulations need to be sufficiently high-resolution to make controlled predictions, but these simulations get very expensive as the resolution is increased. One route to cheaper simulations is machine-learned samplers, in particular normalizing flows [1, 2]. Stripping away the complexity of the problem, the basic setup involves neural networks that act on four-dimensional grids (“lattices”) with one 3 \times 3 complex-valued matrix U_\mu(x), a link, assigned to each edge (x, x+\hat{\mu}) of the grid, where \mu \in \{0,1,2,3\} specifies the direction. (In terms of the physics, these links represent the state of the gluon field.) More generally, within the neural networks used, each edge may carry a stack of links with an additional channel index.

Figure 1: Simplified 3-dimensional illustration of a unit cubic region inside a lattice. Here each edge has 4 channels.

In the architectures of [2, 4], each coupling layer contains a staple network: a series of layers that mix these stacks of links with specific products of neighboring links while preserving gauge equivariance [5]. This looks much like a convolutional network, except that each “pixel” is a 3×3 complex matrix.

Two implementations of the staple network already exist: one built from standard PyTorch operations and used as a reference, and the other built from fused Triton kernels, written with AI assistance, which is 80–100x faster than the PyTorch reference. We briefly discuss both implementations in a later section.

Ultimately, the aim is to combine a staple network with local “output heads” that return a local transformation field over the links of the lattice. These fields are optimized for transport between lattice ensembles at different action parameters [4] and for related noise-reduction methods [6, 7]. A recent development has also been to divide the evaluation of the staple network into local patches centered around each output link, which allows the total memory requirements of the model to be decoupled from the lattice geometry, by streaming the local staple network and output head evaluations in small batches. This target use case motivates some aspects of the mathematical description of the staple layer below and also specific optimizations discussed later in this post.

The staple network

Each staple network has several layers, with the output staples tensor of each layer being the input of the next layer.

The staple layer we accelerate generalizes the gauge-equivariant convolution of [4] (Eq. 14 there), which combines each link with the features S_1 and S_2 (see Eq. (4) below). This generalization includes multiple channels, trace features S_3 and S_4, separate weights for left and right staples, and RMS normalization.

Each layer in a staple network has input and output staples tensors U, V and a tensor of weights W. All tensors are complex double-precision and row-major (right-fastest) in this post, so to describe their layouts we only describe their shapes.

We use the following symbols:

Table 1

SymbolMeaningTypical value(s)
Bbatch size64, 256
n_{in}number of in-channels1 to 8
n_{out}number of out-channels1 to 8
n_{aug}number of distinct “augmented staple features”25 to 200
N_dnumber of (spacetime) dimensions4
N_cnumber of colors3
Klocal spacetime patch width5

The shapes of the tensors are the following:

Table 2

TensorShape
U(B, n_{in}, N_d, K, K, K, K, N_c, N_c)
W(n_{out}, n_{aug}) = (n_{out}, (25, n_{in}))
V(B, n_{out}, N_d, K, K, K, K, N_c, N_c)

For a fixed batch index b < B, in-channel c_i < n_{in}, lattice site x < K^4, direction \mu < N_d, the slice U[b, c_i, mu, x, :] is a 3×3 complex128 input link, or “color” matrix. The slice U[b, c_i, :] is a lattice of links, and each site x has one link in each of the four spacetime directions. We similarly view V[b, c_o, :] as a lattice of output links for out-channel c_o < n_{out}. We omit b from notation for simplicity.

Each output link of V at some site x and out-channel c_o shall be computed as a linear combination of n_{aug} features derived from all input channels of some input links in a neighborhood around x. In particular, n_{aug} = (8(N_d-1) + 1)\, n_{in}. We may further view W[:, (:24, :)] in the shape (n_{out}, (3,8, n_{in})).

Let

\begin{equation} V^{\prime}_{\mu,c_o}(x) = \sum_{c_i = 0}^{n_{in}-1} \left( W_{c_o, (24, c_i)} U_{\mu, c_i}(x) + \sum_{\substack{\nu = 0 \\ \nu \ne \mu}}^{N_d -1} \sum_{j=0}^{7} W_{c_o, (\nu^{\prime},j,c_i)}S_{\mu, \nu, c_i, j}(x) \right). \tag{1} \end{equation}

Here \nu^{\prime} \in \{0, 1, 2\} is the index of \nu among the three directions orthogonal to \mu.

Then

\begin{equation} V_{\mu,c_o}(x) = g_{c_o}\, V^{\prime}_{\mu,c_o}(x)/\operatorname{RMS}_\mu(x) \tag{2} \end{equation}

where g_{c_o} is a norm gain factor and

\begin{equation} \operatorname{RMS}_\mu(x) = \sqrt{\tfrac{1}{N_c\, n_{out}}\sum_{c_o = 0}^{n_{out}-1}\sum_{i,j=0,0}^{2,2} \Big|[V^{\prime}_{\mu,c_o}(x)]_{i,j}\Big|^2} \tag{3} \end{equation}

i.e., RMS depends on site and direction and is computed over all entries and all out-channels. The factor of 1/N_c (instead of 1/N_c^2) gives a result of 1 in Eq. (3) for matrices in SU(3).

We next describe the 8 features S_{\mu, \nu, c_i, j}(x), each a 3×3 complex128 matrix. Fix an in-channel c_i; we omit it from the notation for simplicity. We also fix a pair of orthogonal directions \mu, \nu \ne \mu and a site x. Write M = U_\mu(x). In the (\mu,\nu)-plane, we can form two “staples”, each built from three neighboring links:

Table 3

StapleOperands
left = A_L B_L C_L^\daggerA_L = U_\nu(x+\hat\mu-\hat\nu),
B_L = U_\mu(x-\hat\nu),
C_L = U_\nu(x-\hat\nu)
right = A_R^\dagger B_R C_RA_R = U_\nu(x+\hat\mu),
B_R = U_\mu(x+\hat\nu),
C_R = U_\nu(x)
Figure 2: Left and right staples in a (μ, ν)-plane. Each staple is a path of three links forming a “staple”-like shape. Adjoints are applied when traversing in the positive direction along an axis.

For each staple P \in \{\text{left}, \text{right}\} we also derive three additional features:

\begin{equation} \begin{aligned} S_1 &= P, & S_2 &= \tfrac{1}{8}\, M P^\dagger M, \\ S_3 &= \tfrac{1}{64}\, t_P\, M, & S_4 &= \tfrac{1}{64}\, \overline{t_P}\, M . \end{aligned} \tag{4} \end{equation}

where t_P = \operatorname{tr}(P^\dagger M). We thus have our 8 features. Note that each of S_1, S_2 is a product of matrices while each of S_3, S_4 is a scalar multiple of M.

P^\dagger M can be thought of as a kind of plaquette and S_2 a Wilson loop.

Each patch has open boundary conditions (OBC), meaning any link addressed outside the lattice is out of bounds (OOB) and is treated as the zero matrix.

The number of channels in each layer across the network has the form 1 \to H \to \cdots \to H \to \operatorname{out}_H, where H, \operatorname{out}_H are usually powers of 2 and \operatorname{out}_H \ge H. This means that the first layer has n_{in} = 1, n_{out} = H, the last layer has n_{in} = H, n_{out} = \operatorname{out}_H, and all the other layers (called “hidden layers”; typically there are at least 4 of them) have n_{in} = n_{out} = H. Usually H \in \{2, 4, 8\} and larger H is preferred for better expressivity.

Our target devices are NVIDIA’s H200 NVL and SXM GPUs. The runtimes we report throughout this post are on the NVL.

Previous implementations

The PyTorch reference has two main bottlenecks, one computational and one in memory movement:

  • Unspecialized cuBLAS heuristics dispatch each 3×3×3 complex matmul (we use m×n×k as a shorthand for an (m, k) \times (k, n) matmul) into a tensor core matmul tile of 32×16×64, thus executing ~1200x as many computations as a 3×3×3 matmul needs. Writing an “elementwise” 3×3×3 matmul function directly in PyTorch, as shown below, avoids this cost and gives an 18–25x speedup.
def mm(a, b):
    return (a[..., :, :, None] * b[..., None, :, :]).sum(-2)
  • Unfused operations. As seen in the staple layer definition above, each layer needs to perform a series of many operations to obtain each output link. TorchInductor (as of Torch 2.13.0+cu130) does not support fusion of operations with complex dtypes, so the PyTorch implementation ends up materializing intermediate data to HBM (high-bandwidth memory, the GPU’s main memory) many times, making each layer highly bandwidth-bound. The Triton kernel, which does fuse operations, remains 3–6x faster than the PyTorch implementation even with elementwise matmuls.

The Triton implementation pads each 3×3×3 matmul into 4×4×4 executed cooperatively across a warp, adding overhead in two ways:

  • There is an inflated number of FP64 instructions relative to the ideal, both by the padded tile and because each output element’s reduction axis is split across threads, which prevents the compiler from fusing addition and multiplication instructions into FMAs.
  • A large number of shuffle instructions, even larger than the number of FP64 instructions, is needed to communicate within each warp.

We return to discussing improvements of the Triton implementation in the summary.

Basic kernel design

This section describes the thread-per-site kernel, which we use for layers with n_{in} \ne n_{out} (the first and last layers of the network). It is also the basis of the ring kernel we introduce later for optimizing hidden layers.

Work parallelization

We launch one thread per site, and each thread is responsible for computing the output links in all directions and out-channels at its site. We arrange threads in a CTA (cooperative thread array, i.e., thread block) and CTAs in the grid linearly across the batch and sites.

We illustrate the structure of a thread’s work, when translated directly from Eqs. (1) and (4), with the following pseudocode:

# gmem tensors U,V,W with shapes as in Table 2.
tidx, _, _ = cute.arch.thread_idx() # only the first dimension is nontrivial
bidx, _, _ = cute.arch.block_idx()
bdim, _, _ = cute.arch.block_dim()
idx = bidx * bdim + tidx
sW = cooperative_load(W)  # shared across all threads in CTA                     
syncthreads()

Vol = K**4
b, x = idx // Vol, idx % Vol  # one thread per (batch_id, site)

for mu in range(Nd):
    acc[0:n_out] = 0  # zero the n_out 3×3 accumulators in registers
    for ci in range(n_in):
        M = load(U[b, ci, mu, x])  # reused across nu, left, right
        acc[:] += sW[:, 24, ci] * M  # accumulate residual term
        for nu in range(Nd):
            if nu == mu:
                continue
            nu_ord = nu if nu < mu else nu - 1 
            for i in (left, right):          
                A, B, C = load_operands(U, b, ci, x, mu, nu, i)  # OOB links read as 0
                # i==left: P = A @ B @ C^H; i==right: P = A^H @ B @ C
                P  = compute_staple(A, B, C, i) 
                PM = P^H @ M
                S  = (P, M@PM/8, tr(PM)*M/64, conj(tr(PM))*M/64)
                for co in range(n_out):
                    o = 0 if i==left else 4
                    acc[co] += sum(sW[co, nu_ord, k + o, ci] * S[k] for k in range(4))
    for co in range(n_out):
        store(V[b, co, mu, x], acc[co])

M, A, B, C, P, PM, and each element of acc and S are 3×3 complex128 matrices. In the CuTe DSL kernel they are FP64 tensors of shape (3,3,2), because complex dtypes are not natively supported. nu_ord is the index of nu among the three directions orthogonal to mu and is used to index into sW.

The above pseudocode computes V^{\prime} of Eq. (1) in a naive fashion. Our kernel design modifies this algorithm with several optimizations.

FP64 arithmetic

We do not use tensor cores but rather FP64 CUDA cores to do all computations, including the 3×3×3 matmuls. FP64 tensor cores are not used because they provide only 2x the throughput of FP64 CUDA cores but require rigid shapes that are poorly tiled by the small, odd-sized 3×3×3 matmuls with little operand reuse. For similar tiling reasons we ruled out using an Ozaki scheme with lower-precision tensor cores. Performing each matmul with a single thread also avoids the costs of tile padding and intra-warp communications seen in the Triton baseline. We manually write out the 3×3 complex matmuls, and fix at compile time whether to take the adjoint of each operand.

Fusing DMUL and DADD into DFMA. A naive source code implementation for complex multiplication computes each complex product in full and then adds it to the running sum, which prevents the compiler from fusing many double-precision multiplies (DMUL) and adds (DADD) into fused multiply-add (DFMA) instructions needed for high performance:

# naive form
# (c_re, c_im) = entry (i, j) of C = A @ B
for k in range(3):
    a_re, a_im = A[i][k]
    b_re, b_im = B[k][j]
    p_re = a_re * b_re - a_im * b_im    # DMUL + DFMA
    p_im = a_re * b_im + a_im * b_re    # DMUL + DFMA
    if k == 0:
        c_re, c_im = p_re, p_im
    else:
        c_re = c_re + p_re              # DADD
        c_im = c_im + p_im              # DADD

The compiler can fuse a_re*b_re - a_im*b_im into a DMUL and one DFMA, but when k>0, the subsequent c_re = c_re + p_re is another standalone DADD, so each complex multiply-accumulate costs 2 DMUL + 2 DFMA + 2 DADD.

Instead, when k=1,2 we thread the accumulator through every real multiply, resulting in 4 DFMAs:

# DFMA form
# returns c + a*b in 4 DFMAs
def zfma(c, a, b):  
    (c_re, c_im), (a_re, a_im), (b_re, b_im) = c, a, b
    c_re = a_re * b_re + c_re           # DFMA
    c_re = c_re - a_im * b_im           # DFMA
    c_im = a_re * b_im + c_im           # DFMA
    c_im = a_im * b_re + c_im           # DFMA
    return c_re, c_im

# (c_re, c_im) = entry (i, j) of C = A @ B
for k in range(3):
    a_re, a_im = A[i][k]
    b_re, b_im = B[k][j]
    if k == 0:
        c_re = a_re * b_re - a_im * b_im    # DMUL + DFMA
        c_im = a_re * b_im + a_im * b_re    # DMUL + DFMA
    else:
        c_re, c_im = zfma((c_re, c_im), (a_re, a_im), (b_re, b_im))

Table 4

VersionInstructions per matmul term (k=0)Instructions per matmul term (k=1,2)Total instructions per 3×3 matmul
naive form2 DMUL + 2 DFMA = 42 DMUL + 2 DFMA + 2 DADD = 69 × (4 + 2 × 6) = 144
DFMA form2 DMUL + 2 DFMA = 44 DFMA = 49 × (4 + 2 × 4) = 108

Compute common factors only once. In Eq. (4), there are several factors that appear more than once. In particular, we only need to compute P once, then use it to compute P^\dagger M, then use that to compute S_2 and t_P.

Additionally, for a fixed (\mu, c_i), S_3, S_4, and the residual term (W_{c_o, (24, c_i)} U_{\mu, c_i}(x)) are all scalar multiples of M, so it suffices to accumulate only the scale factor, and only scale M once at the end of each c_i-iteration.

The above optimizations improve performance by reducing computation, but they incur greater register pressure. The second optimization in particular needs 4 extra registers per out-channel persistently live.

Memory access and data reuse

Each thread loads all of its own staple operands and relies on L2 and L1 caching for data reuse. Threads in a CTA cooperatively load W once into shared memory (SMEM). W is independent of batch index, site, and \mu.

For each in-channel’s contribution to an output link, a thread needs up to a total of 19 input links (depending on how many of those are OOB): 3 for each of left and right staples, for each of the three \nu directions, in addition to the thread’s own link M. Conversely, across neighboring sites, each input link is also loaded up to 19 times. But explicit SMEM reuse of input links is constrained by capacity and tiling overhead. For example, storing all input links of a particular batch index requires 5^4 \cdot 3^2 \cdot 4 \cdot 16\,\text{B} = 360{,}000 B per channel, exceeding the 228 KiB of SMEM available to a streaming multiprocessor (SM) of the H200. Furthermore, the odd length of K makes it awkward to tile the patch in a regular way, and any such tiling further incurs the cost of large halos relative to each tile due to the 4D nature of the patch.

Coalesced loads. The row-major memory layouts for U and V as shown in Table 2 are used by the rest of the model and are thus a natural assumption for the staple network as well. However, they present an obstacle to coalesced loads. Threads in a warp own neighboring output links, so they load neighboring input links (ignoring OBC for now). But each link is 9 \cdot 128 bits wide while each thread’s load instruction is only 128 bits, so neighboring threads load data with a gap of 8 \cdot 128 bits. Thus, each 32 B sector only serves a single thread’s 16 B load, leading to 50% wasted data.

We instead rearrange U to a “structure of arrays” (SoA) layout, (B, n_{in}, N_d, N_c, N_c, K, K, K, K), so that data varies fastest by site (in contrast, the original layout can be referred to as “array of structures” (AoS)). In this new layout, a warp’s load instruction reads a contiguous block of 32 128-bit words, up to small inefficiencies at patch boundaries and from sector misalignment that we quantify and address later. This layout permutation only needs to be done once per staple network call and each layer can consume and produce in the SoA layout, so this data movement cost is amortized over several staple layers and typically less than 1% of total runtime.

Figure 3: Loading link data in AoS vs SoA layouts. Squares of the same color are loaded by a warp in a single instruction. In AoS, each 32 B sector serves only one thread’s 16 B load, so a load fetches twice the data it uses. SoA solves this problem with each load being contiguous in memory.

On-chip resource usage

Although input links are not explicitly reused across threads through shared memory, we nevertheless use SMEM to stage input links M, A, B, C, with the SM80 instruction cp.async.ca, in a thread-local way, primarily as a means of alleviating register pressure. There is a tradeoff, however: increasing SMEM allocations reduces space available for L1 reuse of input links. We use .ca instead of .cg to allow for L1 cache hits.

At the beginning of each (\mu, c_i) loop, each thread loads M and keeps it on SMEM to be reused as \{\text{left}, \text{right}\} and \nu vary. This uses 9 \cdot 16 \cdot 32 = 4608 B per warp. We also allocate one buffer for each of A, B, and C, so each warp needs 18{,}432 B of SMEM. Due to register pressure, we are limited to 8 warps/CTA using a total of 144 KiB of SMEM. Staging W amounts to another 16 \cdot 25\, n_{in} n_{out} = 400\, n_{in} n_{out} B per CTA, which is small until n_{in} n_{out} is large. This leaves 92 KiB of the 256 KiB L1/SMEM for L1 (factoring in carveout granularity).

Compared to direct global-memory loads (LDG) of A,B,C, cp.async allows for greater flexibility as registers do not need to be live while the data is in flight or waiting to be used pending other dependencies.

Register pressure. A complex128 3×3 matrix takes 36 registers, and a matmul needs 3 such matrices. We also need one such matrix for the persistent accumulator for the output link for each output channel. The compiler also reorders ops to better hide FP64 latency, further increasing register pressure. Therefore register use is near the limit of 255, capping occupancy at 8 warps per SM.

For each \mu, each thread keeps accumulators for its output link in each channel live on registers across all \nu loops, which we found to produce better performance than keeping accumulators on SMEM (at H=1,2).

Loops across \mu,\nu and in-channels are not unrolled; otherwise, performance is reduced due to both register spills and instruction cache misses.

Fused RMS normalization

We fuse RMS normalization into the epilogue. From Eq. (3), each thread holds exactly the accumulators needed to compute its own RMS factor, so no cross-thread communication is needed. The kernel with fused RMS, at H = 2, has a 53% and 72% speedup at B = 64 and 256, respectively, compared to the alternative of using PyTorch RMS normalization after the kernel.

We illustrate the structure of a thread’s work in our kernel with the following pseudocode:

# cooperative load of W as before
# b,x, zfma as before
# U has SoA shape (B, n_in, Nd, Nc, Nc, K, K, K, K) 

def zaxpy_3x3(S, w, M):
    for i in range(3):
        for j in range(3):
            S[i,j] = zfma(S[i,j], w, M[i,j])
    return S

for mu in range(Nd):
    acc[0:n_out] = 0  # zero the n_out-many 3×3 accumulators on registers
    for ci in range(n_in):
        cp_async(sM, U[b, ci, mu, :, :, x])           
        m_scale[0:n_out] = sW[:, 24, ci]  # only track the per-co scale factor 
        for nu in range(Nd):
            if nu == mu:
                continue
            nu_ord = nu if nu < mu else nu - 1
            for i in (left, right):
                # thread-local SMEM staging of operands; no syncthreads or syncwarp required 
                cp_async(sA, sB, sC, operands(U, b, ci, x, mu, nu, i))    
                cp_async_wait()
                # compute P, PM, S2 using mostly DFMAs (omitted)
                t  = tr(PM)                        
                w_idx_start, w_idx_end  = (0,4) if i==left else (4,8)
                for co in range(n_out):
                    w1, w2, w3, w4 = sW[co, nu_ord, w_idx_start:w_idx_end, ci]
                    acc[co] = zaxpy_3x3(acc[co], w1, P)  # Accumulate S1 = P and S2
                    acc[co] = zaxpy_3x3(acc[co], w2, S2)  
                    # Accumulate only scale factors of S3, S4
                    m_scale[co] += (w3 * t + w4 * conj(t)) / 64   
        for co in range(n_out):
            # scale M once at the end of each (mu, ci)-iteration
            acc[co] = zaxpy_3x3(acc[co], m_scale[co], sM)  
    # thread-local fused RMS epilogue 
    e = 0
    for co in range(n_out):
        for i in range(3):
            for j in range(3):
                e += |acc[co][i,j]|**2          
    rms = sqrt(e / (n_out * 3))                 # Eq. (3)
    store(R[b, mu, x], rms)                     # saved for the backward pass
    for co in range(n_out):
        s = gain[co] / rms                      # one divide per out-channel
        store(V[b, co, mu, :,:, x], s * acc[co])  # V also has SoA shape

sA, sB, sC, sM are all 3×3 complex128 SMEM tensors.

S_3 and S_4 are not materialized; only t_P is computed.

R is an auxiliary tensor of RMS factors used for the backward pass, which we do not discuss in this post.

We compute (gain/rms) * acc instead of gain * (acc/rms) so that division is done once per out-channel instead of per 3×3 matrix entry.

Arithmetic intensity and FP64 latency

FP64 op count. Let n_{in} = n_{out} = 1. Per staple P (left or right), counting a DFMA as one instruction:

Table 5

StepDMULDFMADADDTotal
A^\dagger B (or A B)1890108
(A^\dagger B) C (or (A B) C^\dagger)1890108
P^\dagger M, t_P18904112
M (P^\dagger M)1890108
accumulate W_1 S_1 into acc3636
accumulate W_2 S_2 into acc23638
fold W_3 t_P + W_4 \overline{t_P} into m_scale4812
per staple784404522

The four 3×3 matmuls and the trace are 84% of the instructions, with applying weights and accumulation accounting for the rest. Each thread runs the staple body 2 \times (N_d-1) = 6 times per \mu (left and right for each orthogonal \nu), then scales M once by m_scale (36 DFMA + 2 DADD), and does this for all N_d = 4 output directions:

Table 6

RegionExecutions per threadDMULDFMADADDTotal
staple body241872105609612528
M scale at end of (\mu, c_i)41448152
per thread (per site), unfused18721070410412680
fused RMS epilogue (sum of squares, sqrt, 2 divides, gain scale)496144~250

Counting a DFMA as two flops, that is about 23 kflops per site for all four output links. The fused RMS epilogue costs about 60 FP64 instructions per output direction, including a short sequence of multiplies and fused multiply-adds seeded by one special-function instruction for each square root or division.

Data movement. Suppose first that all data only needs to be loaded once from HBM. Then most of each thread’s memory movement is to load its own four input links, store four output links, and store four RMS norms, totaling 4 \cdot 144 + 4 \cdot 144 + 4 \cdot 8 = 1184 B. The ratio of flops to bytes is thus about 20, which is well above the FP64/HBM ridge point of 7 flops/B for the H200 SXM or 6.3 flops/B for the NVL.

But each input link is loaded up to 19 times, so each thread requests about 9.3 KiB of data (assuming we skip roughly 20% of input loads due to being OOB), which amounts to a flops/byte ratio of about 2.5 if all data came from HBM, well below the H200 ridge points. In practice, whether the kernel may be compute- or memory-bound will depend on L2 and L1 reuse.

Nsight Compute (NCU) shows that the situation is more complicated than just arithmetic intensity considerations (all profiles in this post are at B=256). At H=1, it shows FP64 utilization of only 49%, still more than HBM (13%) and L2 (37%). The most-utilized memory unit is L1/shared memory, at 71.8%. However, the profiler also shows that short scoreboard stalls account for less than 5% of stall reasons; the largest warp stall reason is actually “Stall Wait”, at 2.12 cycles/instruction, due to waiting for a previous FP64 op to finish. So the kernel is bound by FP64 latency, which is not sufficiently hidden by instruction-level parallelism (ILP) or occupancy (partially addressed in a later section).

Scaling to 4 and 8 channels

Using the thread-per-site kernel, we observed the following runtimes for a hidden layer:

Table 7

HB=256 runtime (ms)B=64 runtime (ms)
20.6430.256
42.0540.765
813.103.823

Runtime from 4 to 8 channels grew 5.0–6.4x, yet FP64 operations can be calculated to have grown only 2.9x (in Eq. (1), the S features, each requiring several 3×3×3 matmuls (Eq. (4)), depend only on the in-channel and so scale with n_{in}, while the weighted sums scale with n_{in} n_{out}.).

The cause of this performance degradation at 8 channels is register spillage due to keeping the output accumulators of every out-channel live on-chip.

Moving accumulators onto SMEM did not help performance. Let n_reg_acc \le n_{out} be the number of out-channels whose accumulators are kept on registers (the rest are kept on SMEM). We observed that at n_reg_acc = 2 there is already slight register spillage (though the kernel is still performant at H=2), and setting n_reg_acc > 2 at n_{out} > 2 results in significant performance degradation due to register spillage.

On the other hand, at H = 8, if we let n_reg_acc = 2, then we need 10 \cdot 4608 = 46{,}080 B per warp for SMEM. Along with the weights tensor requiring 64 \cdot 400 = 25600 B, our occupancy goes down to 4 warps/SM, which also decreases performance by exacerbating FP64 latency stalls.

To optimize performance for hidden layers with H = 4, 8 (and, with the refinements in the next two subsections, also at H = 2), we use a channel-parallel kernel, where a group of H threads cooperates to compute the H channels of each site. Each thread holds accumulators for only one out-channel. A thread computes features at one in-channel, accumulates, and repeatedly participates in a shuffle to accumulate its computed features into all other out-channels.

This kernel, which we call the ring kernel (for the ring-like rotation of accumulators in each group), is illustrated in pseudocode below (refinements will be introduced in the next two subsections):

# bdim,bidx,Vol,sW,U,V, zaxpy_3x3 as before
# n = n_in = n_out

def rotate(acc, src_lane):
    for i in range(3):
        for j in range(3):
            c_re, c_im = acc[i, j]
            acc[i, j] = (cute.arch.shuffle_sync(c_re, src_lane), cute.arch.shuffle_sync(c_im, src_lane))

lane_idx = cute.arch.lane_idx()
warp_idx = cute.arch.warp_idx()

sites_per_warp = 32 // n
group_pos = lane_idx // sites_per_warp      # this lane's in-channel, and its "home" out-channel
site_in_warp = lane_idx % sites_per_warp
src_lane = (lane_idx + sites_per_warp) % 32  # this lane's rotation neighbor 
sites_per_block = bdim // n
idx = bidx * sites_per_block + warp_idx * sites_per_warp + site_in_warp  # global site index
b, x = idx // Vol, idx % Vol      

for mu in range(Nd):
    acc = zeros(3,3)  # zero the single 3×3 accumulator on registers
    cp_async(sM, U[b, group_pos, mu, :, :, x])           
    m_scale[0:n_out] = sW[:, 24, group_pos]   
    for nu in range(Nd):
        if nu == mu:
            continue
        nu_ord = nu if nu < mu else nu - 1
        for i in (left, right):
            cp_async(sA, sB, sC, operands(U, b, group_pos, x, mu, nu, i))    
            cp_async_wait()
            # compute P, PM, S2 (omitted)
            t  = tr(PM)                        
            w_idx_start, w_idx_end  = (0,4) if i==left else (4,8)
            for k in range(n_out):
                co = (group_pos + k) % n  # which out-channel this lane holds now
                w1, w2, w3, w4 = sW[co, nu_ord, w_idx_start:w_idx_end, group_pos]
                acc = zaxpy_3x3(acc, w1, P)  # Accumulate S1 = P and S2
                acc = zaxpy_3x3(acc, w2, S2)  
                rotate(acc, src_lane)
                # scale factors do not participate in shuffle
                m_scale[co] += (w3 * t + w4 * conj(t)) / 64   
    # scale M once at the end of each mu-iteration
    for k in range(n_out):
        co = (group_pos + k) % n 
        acc = zaxpy_3x3(acc, m_scale[co], sM)  
        rotate(acc, src_lane)
    # fused RMS normalization with shuffle_sync_bfly (omitted)
    store(V[b, group_pos, mu, :,:, x], acc) 

Some consequences of channel-parallelism:

  • A thread with linear index tidx now works on a site x whose linear index is about tidx / n. The number of waves is also multiplied by a factor of n and thus wave quantization effects are diminished.
  • The mainloop no longer loops over in-channels.
  • As each RMS normalization factor is calculated over all out-channels, threads in a group need to cooperate with shuffle_sync_bfly. Details omitted.

Scale factors cannot be rotated across a group, because m_scale scales sM, which is of a fixed in-channel, but any scale factor that is rotated across a group would accumulate values belonging to varying in-channels, leading to incorrect computation. This is in contrast to an accumulator of a fixed out-channel, which needs to accumulate from all in-channels. Thus each thread still needs 4n registers for scale factors.

The ring kernel’s largest gain is at H = 8, reducing runtime from 13.10 to 4.80 ms at B = 256 (2.7x) and from 3.82 to 1.23 ms at B = 64 (3.1x). Furthermore, runtime now grows 2.9–3.1x from H = 4 to H = 8, close to the 2.9x growth in FP64 ops (see Table 8).

At B = 64, the thread-per-site kernel launches (64 \cdot 625)/(256 \cdot 132) \approx 1.18 waves on the H200, resulting in a large wave-quantization effect. As mentioned earlier, channel-parallelism multiplies the number of waves, which is why we see a larger speedup at B = 64.

Memory alignment and channel-fastest layout

In an NCU profile for the ring kernel at H = 8, we saw 20–25% excess sector accesses for most LDGSTS instructions (the assembly instruction that cp.async compiles to). Even at H = 1 (where the ring kernel coincides with the thread-per-site kernel), NCU still shows 8–12% excess at most LDGSTS instructions, despite the SoA layout facilitating memory coalescing. There are two causes for the observed excesses, which we discuss at H=1 as an example:

  • Memory misalignment. Data from L2 and HBM is aligned at 32 B sector granularity, but due to K being odd, about half of all warp-level 512 B instructions start “mid-sector” and so need to fetch 17 sectors instead of 16. For example, each “color-entry plane” U[b, c_i, mu, i, j, :] holds 625 \cdot 16 B of data, an odd multiple of 16 B. So consecutive such planes alternate between starting on a sector boundary and starting mid-sector. Loads of neighboring links A, B, C similarly will start at an odd multiple of 16 B half the time.
Figure 4: A warp’s load starts and ends mid-sector.
  • OBC excess sectors. Since K = 5, when loading a neighboring link in the innermost (fastest-varying in memory) spacetime direction, about a fifth of these neighboring links are OOB and zero-filled. At the warp level, these OOB sites result in isolated 16 B predicated-off “holes” in its 512 B of data, so no whole sector can be skipped, and a warp still ends up requesting 16 or 17 sectors instead of the ideal 12.5 or 13 sectors’ worth of data.
Figure 5: Due to OBC when loading neighboring links, the leftmost column of data is not needed, but it must be fetched anyway in a site-fastest layout.

With the ring kernel at H = 8 for example, each warp loads 8 separate 64 B blocks of data, each of which is susceptible to the above issues, which is why we see even larger excess percentages.

For the ring kernel at even H, we completely eliminate these excesses by switching the layouts of U and V to be most contiguous in the channel mode, e.g. U having shape (B, N_d, N_c, N_c, K, K, K, K, n_{in}). With this “channel-fastest” layout (the previous SoA layout will be called “site-fastest” for contrast), loads and stores of U and V always shift or predicate-off in multiples of 16H bytes, so no sector is half-used.

The channel-fastest layout speeds up the ring kernel by 8% at H = 8 and about 3% at H = 2, 4. The main reason for the small speedup is that the largest source of warp stalls is still Stall Wait, e.g. at 1.57 cycles/instruction at H=8.

Instruction reordering

zaxpy_3x3 as written computes the nine entries of an accumulator one entry at a time, and an entry has two dependent DFMAs for each of its real and imaginary components. In this kernel, which runs close to the register limit, the PTX-to-assembly-code compiler (ptxas) interleaved only the two components of an entry but not instructions for different entries, leaving dependent DFMAs only two instructions apart.

We instead rewrite accumulation to process one row at a time in a breadth-first fashion, issuing each DFMA step for all of a row’s three entries before the next step, so that dependent instructions are at least three apart. We further increase overlap by warp-shuffling each finished row before starting on the next.

Together, these changes brought Stall Wait down from 1.57 to 1.31 cycles per instruction, raised FP64 pipe utilization from about 55% to 59%, and gave a further speedup of about 7.5% at H = 8, 3.5% at H=4, and 1.5–2% at H=2.

Summary of runtime progressions

The table below collects the forward pass runtimes of our kernel for a hidden layer (n_{in} = n_{out} = H) at B = 256 and B = 64 after each stage described above, on the H200 NVL (using Triton’s do_bench, running 100 ms of warmup and repeat each, with a minimum of 30 repeat iterations for slower implementations, and returning the median). We also report runtimes of the PyTorch reference and with cuBLAS calls replaced by direct “elementwise” 3×3×3 matmuls, as well as the Triton baseline and after porting our CuTe DSL ideas to it, where expressible by Triton.

We restructured the Triton kernel so that each thread owned a site and executed its own 3×3×3 matmuls, similar to our CuTe DSL kernels. We also added SoA layouts and computed common factors only once. These optimizations improved performance greatly at H = 2, to a lesser extent at H = 4, and by very little at H = 8 due to being overshadowed by large register spills. Channel parallelism does not have a straightforward port to Triton, so we added out-channel chunking at H = 4, 8 as an alternative method for dealing with the register bottleneck. This chunking has each thread only holding and accumulating into two out-channels at a time, thus multiplying staple computations by a factor of H/2. This nevertheless gives a large improvement for the Triton kernel at H = 8 and a modest improvement at H = 4, noting that the share of FP64 ops in the algorithm due to 3×3×3 matmuls shrinks as H increases. This new kernel we call improved Triton.

Table 8 (runtimes in ms)

StageB=256
H=2
B=256
H=4
B=256
H=8
B=64
H=2
B=64
H=4
B=64
H=8
PyTorch reference73514662928177365734
PyTorch, elementwise 3×3×3 matmul33.261.21169.9217.731.9
Triton baseline7.04516.2136.661.7824.1279.425
Improved Triton0.7152.84510.970.2590.9463.558
CuTe DSL + PyTorch RMS1.1052.788— †0.3920.967— †
+ fused RMS epilogue0.6432.05413.100.2560.7653.823
+ channel-parallel ring0.6481.5594.7970.2050.4211.225
+ channel-fastest layout0.6321.5074.4330.1990.4081.129
+ accumulation/shuffle interleaving0.6211.4584.1230.1960.3951.052
speedup over Triton baseline11.3x11.1x8.9x9.1x10.5x9.0x
speedup over improved Triton1.15x1.95x2.66x1.32x2.39x3.38x

† Omitted; not tuned for this configuration.

At H = 2, B = 64 the CuTe DSL kernel suffers from significant wave quantization effects, so its speedup over the Triton baseline is smaller than at H = 2, B = 256.

Accounting for the aforementioned wave quantization, we generally see a smaller speedup over the Triton baseline at H = 8. This is because the baseline kernel’s matmuls are more inflated (see baseline diagnosis) than weighted accumulations (which are only inflated by 4×4 padding, and which start to dominate FP64 ops at H=8).

On the other hand, CuTe DSL’s speedup over improved Triton grows with H. This is because with inefficiencies from padding gone, register pressure is the dominant bottleneck at larger H, and Triton does not expose methods we use to handle this pressure such as explicit SMEM staging and the ring kernel’s accumulator shuffles.

Directions for further work

  • Stall Wait is still the dominant stall reason, so further instruction reordering could help improve performance. An alternative way to deal with Stall Wait would be increasing warp occupancy, but this would be difficult without substantial kernel redesign as occupancy is bottlenecked by available on-chip memory resources. Furthermore, shared memory bandwidth may become saturated before FP64 pipes do—at H = 8 the ring kernel’s L1 throughput is already at ~72% of peak, compared to ~59% FP64 utilization.
  • Supporting larger H such as H=16 could further improve model expressivity. At these values of H in a hidden layer, weighted accumulations further dominate FP64 ops, and the weights tensor itself is 100 KiB at H = 16, which would not practically fit in SMEM alongside our other current SMEM allocations.
  • We currently zero-fill OOB links and compute on them anyway. Noting that e.g. all left-staple features are zero if any of A, B, C is zero, it can be calculated that about 36% of staple computations are currently on identically zero data. These redundant computations could be pruned, but we would need to be careful of intra-warp divergence as threads in a warp may not all be on a boundary site.
  • In the current model, per batch, a pass through the network extracts only the link in one direction at the center site of each local patch. Currently we compute all output links in each layer, so there is a large amount of wasted computation, especially in the last few layers. If at each layer we’re able to compute only those links needed for the final output link, we could reduce computations by 50% or more, depending on the number of layers in the network.

Conclusion

In this post we have described and analyzed kernel optimizations for hidden layers of staple networks. These optimizations address bottlenecks due to high-precision data types, small 3×3×3 complex matrix multiplications, and a 4D stencil. Our CuTe DSL kernels use no SM90 features and only one SM80 instruction, cp.async; instead, their performance comes from register and SMEM management, work decomposition, operation fusion, and transaction alignment.

We note that this post describes only the forward pass of the staple network. For the backward pass we also have a CuTe DSL kernel that achieves roughly an order-of-magnitude speedup over the baseline Triton kernel’s backward pass. The backward kernel uses most of the optimizations we’ve described in this post, but also has additional challenges. In a future post we may describe optimizations focused on the backward kernel.

References

  1. K. Cranmer, G. Kanwar, S. Racanière, D. J. Rezende, and P. E. Shanahan, “Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics,” Nature Reviews Physics 5, 526–535 (2023), doi:10.1038/s42254-023-00616-w.
  2. R. Abbott, D. Boyda, D. C. Hackett, G. Kanwar, F. Romero-López, P. E. Shanahan, and J. M. Urban, “Progress in normalizing flows for 4d gauge theories,” PoS LATTICE2024, 066 (2025), doi:10.22323/1.466.0066, arXiv:2502.00263.
  3. M. S. Albergo, D. Boyda, D. C. Hackett, G. Kanwar, K. Cranmer, S. Racanière, D. J. Rezende, and P. E. Shanahan, “Introduction to normalizing flows for lattice field theory,” arXiv:2101.08176 (2021).
  4. R. Abbott, A. Botev, D. Boyda, D. C. Hackett, G. Kanwar, S. Racanière, D. J. Rezende, F. Romero-López, P. E. Shanahan, and J. M. Urban, “Applications of flow models to the generation of correlated lattice QCD ensembles,” Phys. Rev. D 109, 094514 (2024), doi:10.1103/PhysRevD.109.094514, arXiv:2401.10874.
  5. D. Boyda, G. Kanwar, S. Racanière, D. J. Rezende, M. S. Albergo, K. Cranmer, D. C. Hackett, and P. E. Shanahan, “Sampling using SU(N) gauge equivariant flows,” Phys. Rev. D 103, 074504 (2021), doi:10.1103/PhysRevD.103.074504, arXiv:2008.05456.
  6. R. Abbott, D. Boyda, Y. Fu, D. C. Hackett, G. Kanwar, F. Romero-López, P. E. Shanahan, and J. M. Urban, “Variance reduction in lattice QCD observables via normalizing flows,” Phys. Rev. D 114, 014513 (2026), doi:10.1103/5xns-7qmj, arXiv:2603.02984.
  7. R. Abbott, Y. Fu, D. C. Hackett, G. Kanwar, F. Romero-López, and P. E. Shanahan, “Learning the generating functional for variance reduction in lattice QCD,” arXiv:2606.15986 (2026).

Discover more from Colfax Research

Subscribe to get the latest posts sent to your email.

Posted

in

, ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *