Resonetic Lab | AI Systems Code Review

For the matrix A = 1e-8 * I , the pseudoinverse is 1e8 * I . PyTorch PR #200536 reports that its MPS backend instead returned an all-zero matrix. Solving A X = I through torch.linalg.lstsq returned zeros as well, while the CPU results contained the expected 100000000 diagonal. The failure came from PyTorch's Metal SVD kernel. It calculated nonzero singular values, then discarded their corresponding vector directions during normalization because the values were smaller than a fixed machine-epsilon threshold. The patch changes that normalization condition from sigma > eps to sigma > 0.0f . It also adds tests that check orthonormality, reconstruction, pseudoinverses, and least-squares solutions at a small input scale. The useful distinction is between the size of a matrix and whether its information has become numerically insignificant. This example has four equal singular values and condition number one. Scaling an identity matrix down did not make it rank-deficient. The re...

NVIDIA TensorRT-LLM PR #20030 fixes a pipeline-parallel deadlock in which a message had been posted, but no thread remained able to drive it to completion. A pipeline rank sent schedule and sample-state messages with nonblocking MPI operations and continued into GPU work. In the failing UCX configuration, a rendezvous-protocol send could remain incomplete until the sending process entered MPI again. TensorRT-LLM already had a Python progress pump that periodically called MPI_Iprobe . That pump still needed the Python global interpreter lock. On GB300, the executor thread could enter a native CUDA call while retaining the GIL. The CUDA call then waited for an in-flight NCCL operation. The next pipeline rank was waiting for the MPI schedule message before it would post the matching NCCL receive. The resulting cycle was: sending rank: MPI schedule send still needs progress ↓ executor holds the GIL inside a CUDA call ↓ CUDA call waits for an NCCL receive on the next rank n...

OpenXLA PR #49428 fixes a command-buffer update bug that appeared when one GPU custom-call thunk was recorded more than once into the same unrolled loop. Each loop iteration created a distinct set of command-buffer nodes with its own operand and result addresses. The custom call’s record state kept only one sequence of FFI command handles. Every new RecordCreate replaced the previous sequence. When XLA later updated the command buffer with new buffer allocations, every loop iteration was given the command sequence from the last recorded iteration. The final iteration was updated repeatedly. Earlier iterations retained stale buffer pointers. The patch replaces the single stored sequence with a map: recorded sink command → full FFI command sequence for that recording An update can now recover the commands belonging to the exact loop iteration being updated. Command-buffer creation and update have different jobs XLA’s GPU command-buffer path has two relevant phases. During creation, a co...

OpenXLA PR #49368 fixes a specialized GPU TopK kernel whose comparator did not implement the ordering promised by the source operation. The old comparator was: return lhs.key == rhs.key ? lhs.idx < rhs.idx : lhs.key > rhs.key; For ordinary finite values, this usually behaves like a descending stable sort. For NaNs, neither equality nor greater-than establishes an order. For signed zero, equality removes the distinction between +0 and -0 . That was enough to make the bespoke GPU kernel disagree with XLA’s sort-based TopK path. The same HLO operation could therefore produce different values and indices depending on whether its shape crossed the threshold for the optimized kernel. The patch bitcasts float32 and bfloat16 keys to unsigned integers, transforms those bits into a monotonic total-order key, and compares the integer keys instead. One shape boundary changed the meaning of TopK XLA has more than one way to implement TopK. For smaller or unsupported configurations, it can ...

ONNX Runtime PR #32807 fixes a CUDA Plugin Execution Provider allocator whose ownership boundary was smaller than the lifetime of the addresses captured from it. The factory kept one BFC device arena for each GPU and returned that arena to every allocator request: environment shared allocator session A session B other sessions on the same GPU A session could capture a CUDA graph while its intermediate tensors occupied chunks from that arena. After capture, the session released those chunks through the normal allocator path. The captured graph still contained the same device addresses and used them during every replay. Because the arena was shared, another session could receive one of those apparently free chunks and write unrelated data into it. The first graph would later replay against an address whose contents or ownership had changed. The failure did not require an invalid pointer. Both sessions could hold valid allocations while silently interpreting the same storage as different ...

OpenXLA PR #49441 fixes an algebraic rewrite that treated cropping as negative padding and then added it to a consumer’s window padding. That arithmetic preserved output shapes. It did not preserve values. The failing graph had this form: input → pad with a negative edge amount → reduce-window or convolution with positive window padding In XLA, a negative edge amount on pad removes elements from the tensor. It is a crop. The simplifier merged the two operations by adding their padding numbers. If the pad cropped one element and the consumer added one padding element on the same side, the amounts canceled to zero. The compiler then used the original input element where the source program required a padding value. The program remained valid and executable. It silently computed a different function. Positive padding can be merged by addition For ordinary nonnegative padding, the rewrite is natural. Suppose an input is padded with one zero on the left: [x0, x1, x2] ↓ [0, x0, x1, x...