NVIDIA TensorRT-LLM PR #20030: How the GIL Stopped MPI Progress and Deadlocked PP4
NVIDIA TensorRT-LLM PR #20030 fixes a pipeline-parallel deadlock in which a message had been posted, but no thread remained able to drive it to completion.
A pipeline rank sent schedule and sample-state messages with nonblocking MPI operations and continued into GPU work. In the failing UCX configuration, a rendezvous-protocol send could remain incomplete until the sending process entered MPI again.
TensorRT-LLM already had a Python progress pump that periodically called MPI_Iprobe.
That pump still needed the Python global interpreter lock.
On GB300, the executor thread could enter a native CUDA call while retaining the GIL. The CUDA call then waited for an in-flight NCCL operation. The next pipeline rank was waiting for the MPI schedule message before it would post the matching NCCL receive.
The resulting cycle was:
sending rank:
MPI schedule send still needs progress
↓
executor holds the GIL inside a CUDA call
↓
CUDA call waits for an NCCL receive on the next rank
next rank:
waits for the MPI schedule message
↓
does not reach the matching NCCL receive
Python progress pump:
could drive MPI
↓
cannot acquire the GILThe patch adds a native C++ thread that periodically calls MPI_Iprobe without entering Python. Once MPI progress resumes, the schedule message can arrive, the next rank can post its NCCL receive, and the stalled CUDA operation can continue.
Posting a nonblocking send did not finish the transfer
The relevant host-side messages were sent asynchronously.
A successful isend call establishes the request. It does not mean that the peer has already received the payload or that the sender no longer needs to participate in progress.
For the UCX rendezvous path described in the pull request, the transfer could remain pending until the sending process called an MPI routine again.
That distinction matters in a pipeline.
A rank can logically move past the send:
post schedule message
↓
start later GPU preparationwhile the communication system still needs another MPI entry point to complete the first step.
The deadlock appeared because the program had no guaranteed MPI entry point independent of the executor thread and the GIL.
A Python progress thread was not independent enough
PR #19643 had already added a progress pump to the sample-state relay thread.
That relay thread polled its Python queue with a timeout and called a nonmatching MPI_Iprobe between polls. Under ordinary Python scheduling, it could keep the process entering MPI while the main executor was busy.
The new failure crossed a different boundary.
The executor entered a native operation such as:
cuKernelGetFunction
or
cuKernelSetAttributeduring lazy CUDA kernel setup. In the reported GB300 traces, that native operation held the GIL while waiting for an in-flight NCCL send.
A second Python thread cannot run Python code while another thread retains the GIL, even when that second thread exists specifically to make communication progress.
The earlier pump was useful when the executor was blocked but Python scheduling continued. It could not help when the block itself prevented all other Python threads from running.
The fix therefore did not replace the relay pump. It added a progress source below the Python runtime.
The deadlock linked two communication systems
The reported cycle involved MPI control traffic and NCCL GPU communication.
One rank needed MPI to deliver the next schedule message.
The receiving rank needed that schedule before it reached the operation that posted an NCCL receive.
The sender was already waiting inside CUDA for the corresponding NCCL operation.
Neither MPI nor NCCL was independently corrupted.
The deadlock came from the order in which progress in one system enabled progress in the other:
MPI schedule delivery
↓
receiver advances its pipeline schedule
↓
receiver posts NCCL receive
↓
sender's NCCL operation completes
↓
sender leaves the GIL-holding CUDA callWithout the first transition, the remaining transitions never occurred.
The replacement thread never enters Python
The new MpiProgressThread is implemented in TensorRT-LLM’s native MPI utilities.
Its worker loop:
waits for the configured interval
↓
releases its internal mutex
↓
calls MPI_Iprobe
↓
increments a probe counter
↓
returns to the timed waitThe thread is named mpi_progress on supported POSIX systems.
It does not call Python, nanobind, or a Python callback from its worker loop.
Python constructs it by passing:
the communicator's Fortran handle
the interval in microseconds
the reserved probe tagThe native binding converts the handle with MPI_Comm_f2c and starts the C++ thread.
The probe tag is deliberately unused
The worker probes:
source = MPI_ANY_SOURCE
tag = PPCommTag.MPI_PROGRESS_PROBETensorRT-LLM reserves that tag and does not use it for normal traffic.
The expected result is therefore a miss whose purpose is to enter MPI and give the implementation an opportunity to advance outstanding operations.
The operation remains a plain MPI_Iprobe.
The source documentation explicitly rejects MPI_Mprobe and MPI_Improbe for this role because matched-probe operations can claim a message and interfere with the application’s own probe-then-receive sequences.
MPI_Iprobe reports whether a match exists without removing the message.
The regression sends a payload on the ordinary SAMPLE_STATE tag while the native thread continues probing the reserved tag. The payload must remain available to the intended receiver.
Concurrent MPI calls require MPI_THREAD_MULTIPLE
The new thread calls MPI while other TensorRT-LLM threads can also enter MPI.
That requires the implementation to provide:
MPI_THREAD_MULTIPLEThe Python startup path checks MPI.Query_thread().
The C++ constructor checks again with MPI_Query_thread().
If the provided level is lower, TensorRT-LLM logs an error and leaves the progress thread disabled.
The patch does not pretend that periodically probing from a second native thread is safe under MPI_THREAD_SERIALIZED or a lower level.
This is an availability boundary: on an MPI installation without MPI_THREAD_MULTIPLE, the original class of deadlock can still occur.
The interval trades CPU activity for progress latency
The environment variable is:
TLLM_PP_MPI_PROGRESS_INTERVAL_USIts default is:
5,000 microsecondsA value at or below zero disables the thread.
A malformed value or a value above one second falls back to the default.
The author reports a measurable CPU cost on the throttled GB300 cluster:
| Probe interval | Median CPU use per rank |
|---|---|
| 5 ms | 12.7% of one core |
| 20 ms | 4.4% of one core |
| 1 s | 0.24% of one core |
Longer intervals reduced polling activity.
They also increased the reported execution time in that environment:
5 ms:
66 s median
20 ms:
72.5 s median
30 ms to 1 s:
75 to 76.5 s medianThe reported GSM8K result did not change across those interval tests.
The default is therefore a workload-specific correctness and latency choice, not a claim that five milliseconds is universally optimal.
Shutdown is part of the correctness fix
A progress thread must stop before the communicator or MPI runtime disappears.
PyExecutor starts the thread only when:
pipeline parallel size > 1
and
MPI is enabledNormal shutdown stops the native thread after the pipeline relay thread has exited.
The hang-detected early-return path also stops it before the hang handler proceeds with MPI teardown.
That second path was added during review. Without it, the new worker could continue calling MPI_Iprobe while another shutdown path aborted or finalized MPI.
An atexit callback provides a fallback for process-exit paths that do not call normal shutdown.
TensorRT-LLM stores the exact bound thread.stop object used for registration so it can later unregister the same callable.
The nanobind stop() method releases the GIL while joining the native thread.
stop() is designed to be idempotent
The C++ object uses two separate mutexes.
One protects the stop flag and condition variable.
Another serializes join() calls.
This permits several callers to request shutdown concurrently without two threads trying to join the same worker at the same time.
The regression exercises four concurrent stop() callers over twenty repetitions. Each caller has a bounded join deadline so a synchronization error fails the test instead of hanging the test process.
The test also verifies that:
probing occurs before stop
probing ceases after stop
a second stop call is harmless
is_running becomes falseTests cover lifecycle and message preservation
The new CPU-side test file covers three layers.
Native worker behavior
It checks:
periodic probing
probe count
positive interval validation
idempotent stop
concurrent stop
preservation of messages on another tagTests that require native MPI concurrency are skipped unless MPI provides MPI_THREAD_MULTIPLE.
Configuration
The interval parser is tested with:
unset value
ordinary integer
zero
negative value
malformed text
floating-point text
the maximum accepted value
a value above the maximum
an integer larger than int64Executor integration
A focused executor double verifies:
pp_size = 2 starts and retains the native thread
normal shutdown stops it
hang-detected shutdown stops it
the registered atexit callable is unregistered by identity
disabled interval starts no thread
insufficient MPI thread support starts no thread
pp_size = 1 starts no threadThe integration waiver for the reported GB300 fp8kv=False PP4 case is also removed. The adjacent fp8kv=True case remains waived.
The workload evidence supports the progress mechanism
Before the earlier Python pump was merged, the author reports:
native progress thread:
0 hangs in 120 runs
no native progress thread:
8 hangs in 98 runsThe reported one-sided Fisher exact probability is approximately 0.0014.
On a later base that already included the Python pump, the deterministic reproducer still hung in three of three runs without the native worker and passed in six of six runs with it.
Four full runs at the default interval also passed, with one running mpi_progress thread per rank.
These results support the claim that the native thread closes a gap left by the Python pump.
They do not prove that every pipeline-parallel hang is caused by missing MPI progress.
The committed tests do not reconstruct the full cycle
The unit suite does not reproduce all of these conditions in one test:
UCX rendezvous send
GIL retained by lazy CUDA loading
in-flight NCCL operation
next rank waiting for an MPI schedule
four pipeline stagesThe deterministic and full-workload reproductions are author-reported.
The committed tests instead protect the mechanism introduced by the patch:
the native thread really probes
it does not consume unrelated traffic
it starts only under supported conditions
it stops on every covered teardown pathThe public PR also reports 34 focused unit tests passing, but the new unit file was not listed separately in the inspected integration test lists.
The progress contract restored by the patch
A communication system that needs periodic entry cannot depend exclusively on threads governed by the runtime that may be blocked.
The corrected relationship is:
post asynchronous pipeline messages
↓
maintain one GIL-independent MPI progress source
↓
deliver the control message
↓
allow the next rank to post dependent NCCL work
↓
complete the blocked GPU operation
↓
stop the progress source before MPI teardownA progress thread is useful only when it can run independently of the resource causing the stall.
NVIDIA TensorRT-LLM PR #20030 moves the critical progress loop below the Python GIL and closes the reported PP4 deadlock.
Sources
https://github.com/NVIDIA/TensorRT-LLM/pull/20030
https://github.com/NVIDIA/TensorRT-LLM/commit/a77de9e3bfa7b0e885e4da6d637d5f38c2e78b61