Triton PR #11265: How a Python Callback Deadlocked ROCm Startup

Triton PR #11265 fixes a startup deadlock caused by calling back into Python while glibc held its dynamic-loader lock.

The AMD backend searched the process for an already loaded HIP runtime by calling dl_iterate_phdr() through ctypes.

ctypes.CDLL released Python’s global interpreter lock for the native call. glibc then acquired its loader lock and invoked a ctypes.CFUNCTYPE callback for each loaded object. That callback was written in Python, so entering it required the GIL again.

A second thread could simultaneously hold the GIL while hipModuleLoadData() entered HIP/HSA and waited for the same loader lock.

The two threads formed a lock cycle:

library-lookup thread:
dynamic-loader lock → waits for GIL

kernel-loading thread:
GIL → waits for dynamic-loader lock

Neither thread could proceed.

The patch moves the entire loaded-library walk into Triton’s existing native AMD extension. The Python binding releases the GIL before calling dl_iterate_phdr(), and the callback performs only native, non-allocating work while the loader lock is held.

Why Triton searches the loaded-library list

Triton needs a path to libamdhip64.so when it initializes its AMD driver helper.

The process may already contain a HIP runtime loaded by PyTorch or another component. Reusing that mapped library helps avoid selecting a different ROCm installation and mixing incompatible runtime components inside one process.

The lookup participates in a larger precedence order:

explicit TRITON_LIBHIP_PATH
        ↓
TheRock SDK consistency check
        ↓
already loaded HIP runtime
        ↓
Triton-local library
        ↓
PyTorch-packaged library
        ↓
LD_LIBRARY_PATH, HIP_PATH, hipconfig, ROCM_PATH
        ↓
ldconfig and common installation paths

PR #11265 does not redesign that policy.

It replaces only the mechanism used to inspect the libraries already mapped into the process.

The old callback crossed the Python boundary under a foreign lock

The previous implementation lived entirely in driver.py.

It loaded dl_iterate_phdr from libc.so.6, declared a partial dl_phdr_info structure, created a Python callback with ctypes.CFUNCTYPE, and passed that callback into glibc.

The callback performed several Python-level operations:

read dlpi_name through ctypes
        ↓
decode the filesystem path
        ↓
construct pathlib.Path
        ↓
extract the basename
        ↓
check whether it contains the requested library name
        ↓
copy the matching path into a Python-owned buffer

Those operations are ordinary in isolation.

The dangerous part was where they ran.

dl_iterate_phdr() protects the loaded-object list while it invokes callbacks. The pull request’s GDB trace identified the held glibc lock as _dl_load_write_lock.

The callback closure then entered CPython through PyGILState_Ensure while that loader lock was still owned by the current thread.

The lookup therefore established this lock order:

loader lock
        ↓
GIL

The other thread acquired the same locks in reverse

The reported workload loaded AOT-cached GPU binaries in parallel.

Another Python thread called the native HIP module-loading path. The public trace shows that thread inside hipModuleLoadData() and CodeObjectReaderImpl::SetMemory() while still owning the GIL.

HIP/HSA then called into dynamic-loader machinery and waited for _dl_load_write_lock.

Its order was:

GIL
        ↓
loader lock

That is the reverse of the lookup thread.

A deadlock requires more than two contended locks. It requires a cycle in ownership and waiting:

Thread A owns loader lock
Thread A waits for GIL

Thread B owns GIL
Thread B waits for loader lock

The PR’s GDB capture reported the loader mutex owner and both native stacks, while py-spy --native showed the same pair of blocked paths in both tensor-parallel workers.

Why the hang was intermittent

The lock cycle depended on timing.

The lookup thread had to enter dl_iterate_phdr() and reach its Python callback while another thread was loading a HIP code object and still held the GIL.

If either operation finished before the other acquired its first lock, startup completed normally.

The author observed the problem during concurrent warm-cache loading in a vLLM workload using an AMD Mistral-7B FP8-KV configuration. Several consecutive runs could succeed before one hung. A Qwen3.5-9B workload did not reproduce it in the reported testing.

That does not make the lock order safe. It means the scheduler, cache contents, module set, and allocator timing determined whether the cycle became visible on a particular launch.

The user-facing symptom was a process that stopped during startup rather than a Python exception explaining the conflict.

The callback is now native from start to finish

The patch adds findLoadedLibrary() to the build-time triton._C.libtriton.amd extension.

Its Linux implementation passes a C++ callback directly to dl_iterate_phdr().

While the loader lock is held, the callback:

reads the dlpi_name pointer
        ↓
skips empty paths
        ↓
finds the final slash with strrchr
        ↓
searches only the basename with strstr
        ↓
measures the path into a fixed limit
        ↓
copies the bytes into stack-owned storage
        ↓
returns 1 to stop at the first match

It does not call Python or nanobind.

It does not construct a std::string, allocate memory, or throw while glibc is invoking it. The callback is marked noexcept, and its result storage is a fixed structure supplied through the opaque callback pointer.

The more convenient C++ and Python objects are created only after dl_iterate_phdr() returns.

Releasing the GIL removes the other risky edge

The nanobind wrapper enters from Python with the GIL held, then creates:

py::gil_scoped_release release;

before calling the native lookup.

That means the lookup thread does not hold the GIL while it waits to acquire or traverse under the dynamic-loader lock.

The patch therefore removes both problematic relationships from this path:

old callback:
loader lock → Python callback → GIL

new callback:
loader lock → native callback only

and:

old Python entry:
foreign call releases GIL, but callback needs it again under loader lock

new Python entry:
GIL released for the complete native walk

The other HIP-loading thread may still enter loader code while holding the GIL. The repaired lookup no longer creates the opposite loader-lock-to-GIL dependency needed to close the cycle.

Only after the library walk is complete does the binding reacquire Python execution and convert the returned path with PyUnicode_DecodeFSDefault.

The patch preserves the lookup’s visible policy

The original Python callback compared the requested name with Path(decoded_path).name.

The native callback likewise extracts the basename before applying the substring match. This avoids finding a library merely because one of its parent directories contains the requested text.

The first matching loaded object still wins because the callback returns a nonzero value immediately.

A miss still becomes Python None, and non-Linux callers still return None before entering the native function.

The environment override, TheRock consistency rule, and later filesystem-search order remain unchanged.

One defensive edge differs. The old code copied at most 4,096 path bytes and could return a truncated path. The new callback records an overlong path and raises only after dl_iterate_phdr() has released the loader lock.

That is a bounded behavior change and avoids both silent truncation and exception construction inside the callback.

Why the helper was added to the existing extension

An alternative would have been to compile a small native callback helper at runtime.

Triton already builds the AMD Python extension, so this PR adds dylib_utils.cc to that target instead.

This avoids making HIP runtime discovery depend on:

a compiler being available during startup
concurrent helper compilation
another cached extension
loading another shared object while diagnosing loader contention
or a fallback to a different runtime when helper compilation fails

The broader AMD driver still compiles its normal launcher helper where designed. The library-discovery callback itself no longer needs a new runtime compilation step.

The committed tests validate lookup behavior, not the deadlock

The new test file contains two Linux-only cases.

The positive case searches for libc.so, verifies that a path is returned, checks that the basename contains the requested text, and confirms that the path exists.

The negative case searches for an invented library name and expects None.

These tests protect:

binding availability
loaded-object traversal
basename matching
path conversion
first-order success and miss behavior

They do not construct the two-thread lock inversion.

The committed suite does not hold the GIL in one thread while forcing HIP to wait on the loader lock, nor does it verify repeated concurrent AOT-cache startup. It also does not directly cover the 4,096-byte error, TheRock conflict handling, or parallel calls to the new binding.

The author separately reports that the native binding found the mapped HIP runtime, Triton selected the same runtime already used by the process, and the original warm-cache vLLM workload no longer hung.

That runtime validation is meaningful evidence, but it is author-reported rather than a deterministic deadlock regression checked into the repository.

Review preserved the old matching semantics

Public review asked why the native callback extracted the basename instead of searching the complete path.

The author retained the basename operation because it matches the old Python behavior and avoids directory-name false positives.

Review also requested a separate header, a path-derived include guard, and a clearer source filename. The final patch separates the Python-agnostic findLoadedLibrary() declaration from the nanobind initialization function and builds the new source into the AMD plugin.

The final revision received public approval before merge.

What the patch establishes

The visible code, traces, and tests support this sequence:

Python ctypes called dl_iterate_phdr
        ↓
glibc invoked a Python callback under its loader lock
        ↓
that callback waited for the GIL
        ↓
a concurrent HIP module load held the GIL and waited for the loader lock
        ↓
startup could deadlock

The merged implementation changes the sequence to:

release the GIL
        ↓
walk loaded objects with a native callback
        ↓
copy only primitive data while the loader lock is held
        ↓
return from dl_iterate_phdr
        ↓
reacquire Python and construct the result object

The code-review verdict is:

PASS — the merged patch removes the lock inversion identified by the public traces while preserving the ordinary HIP-library selection behavior.

The concurrency contract restored by the change

A callback inherits the locks held by the code that invokes it.

Crossing into a higher-level runtime from that callback can silently add locks that the native API never documents in its function signature.

The safe boundary is:

enter foreign locked region
        ↓
perform bounded native work only
        ↓
leave the foreign locked region
        ↓
reenter Python and allocate rich objects