OpenXLA #47530 — Why a Rendezvous Key Had to Be Deleted After It Signaled Completion

OpenXLA PJRT Cross-Host Transfer, Part 3 of 3 — Ephemeral completion receipts, stale-key collisions, and why cleanup runs after promise publication

The first two articles in this series examined how OpenXLA changed PJRT’s GPU cross-host receive path.

The old receiver waited for one rendezvous key after another:

Get(Key 0)
        ↓
Get(Key 1)
        ↓
Get(Key 2)

Even when several GPU transfers progressed independently, their completion signals were observed through one blocking loop.

No individual destination buffer received its sequencing event until the complete wait returned.

PR #47530 changed that structure to:

one AsyncGet per buffer

one PromiseOnce per buffer

one Future per buffer

one sequencing event per buffer

Part 2 examined how successful completion, lookup failure, and cancellation compete for one terminal PromiseOnce status.

One lifecycle question still remains.

What happens to the rendezvous key after it has served its purpose?

Before the patch, the receiver waited until the key existed.

It did not remove the key from the key-value store in the shown completion path.

A transfer could finish while its completion marker remained behind indefinitely.

PR #47530 added an explicit cleanup sequence:

AsyncGet completes
        ↓
publish the terminal status to PromiseOnce
        ↓
delete the rendezvous key

The PR describes that deletion as protection against:

resource leaks

and

key collisions

This is not merely cosmetic cleanup.

A rendezvous key is an ephemeral completion receipt.

Once a receive buffer has consumed it, leaving the key in the shared coordination namespace can preserve control-plane state longer than the transfer it represented.

A later lookup that encounters the stale key may interpret an old completion marker as evidence about a new operation.

A completion signal is correct only when its identity and lifetime belong to the same transfer.

The key represents completion, not the transferred data

For every destination buffer, the receiver generates a descriptor containing:

buffer index

destination allocation handle

offset inside that allocation

rendezvous key

The key is shaped approximately like:

pjrt_gpu_remote_recv_<transfer_id>_<buffer_index>

The transfer_id is generated for the receive batch, and the buffer index distinguishes the buffers within that batch.

The sender imports the destination allocation, copies the source data into that remote memory, and sets the rendezvous key only after the GPU stream reaches a host callback.

Remote GPU copy
        ↓
stream completion callback
        ↓
kv_store->Set(key, "")

The key’s value is an empty string because the payload does not travel through the key-value store.

Tensor data
→ GPU transfer path

Completion receipt
→ key-value-store path

The key has one job:

Tell the receiver that one destination buffer’s transfer
has reached its terminal completion signal.

A completion receipt should have a bounded lifetime

The natural lifecycle is:

Key name created
        ↓
AsyncGet registered
        ↓
sender or cancellation path sets the key
        ↓
receiver observes the key
        ↓
buffer future reaches a terminal state
        ↓
key is retired

The old path stopped one step earlier:

Key observed

→ transfer status consumed

→ key remains in store

That leaves a mismatch.

Transfer lifecycle
→ finished

Completion-marker lifecycle
→ still active

The patch closes that gap.

The old blocking wait did not retire the keys

Before PR #47530, CrossHostRecvState::Wait() iterated through the key vector and called blocking Get().

Conceptually:

for (const std::string& key : rendezvous_keys) {
    kv_store->Get(
        key,
        InfiniteDuration()
    );
}

Once the full loop completed, it returned the cancellation-status vector.

There was no corresponding:

kv_store->Delete(key);

inside the shown wait path.

The key remained stored after it had already been used as a completion signal.

In the in-memory store, callback completion does not erase the data entry

OpenXLA’s in-memory key-value store separates two internal structures.

data_
→ stored key-value entries

callbacks_
→ listeners waiting for keys

When Put() stores a key, the implementation:

writes the value into data_

then

notifies callbacks registered for that key

After notification, the waiting callback list is erased.

The data entry itself remains.

Callback lifecycle
→ completed and removed

Key-value entry
→ still present

Only Delete() removes the entry from the store’s data map.

This distinction defines the resource leak precisely.

The outstanding listener may be gone.

The completion key remains allocated.

A stale key can satisfy a future lookup immediately

The in-memory store’s callback registration checks whether the requested key already exists.

Conceptually:

if (data_.contains(key)) {
    callback(data_[key]);
    return;
}

callbacks_[key].push_back(callback);

Therefore, if an old completion key remains and the same identifier is later requested:

New AsyncGet(key)
        ↓
old key already exists
        ↓
callback runs immediately

No new GPU transfer needs to finish.

The stale control-plane entry can look like current completion.

This is why key retirement is related to correctness rather than only memory usage.

The false-completion sequence

Consider a completed transfer whose key remains:

Old transfer

Key:
pjrt_gpu_remote_recv_X_0

State:
present in KV store

Now suppose another operation later uses the same key identity.

New receive registers AsyncGet(X_0)

Because the old entry is still present:

AsyncGet callback fires immediately
        ↓
new Promise becomes ready
        ↓
new sequencing event may be recorded

The receiver can conclude that the new destination buffer is ready before the new sender has completed anything.

Old completion marker

interpreted as

new transfer completion

The PR does not report that this exact collision was observed in production.

It states that deletion prevents key collisions, and the key-value-store behavior shows why a stale entry would be semantically dangerous if an identity were reused.

A random transfer ID reduces collision risk but does not define cleanup

PJRT generates a new 64-bit transfer identifier and combines it with the buffer index.

That makes accidental reuse unlikely in ordinary execution.

But uniqueness generation and lifecycle cleanup solve different problems.

Random transfer ID
→ reduce the probability of identity reuse

Delete after consumption
→ ensure old state does not survive under that identity

A random identifier is not a license to keep every key forever.

Long-running coordination stores may contain many transfers.

Processes can restart.

Tests can use controlled identifiers.

Different key-value-store implementations can have different retention behavior.

The PR’s cleanup acts as defense in depth:

choose a fresh identity

and

retire the identity after use

The key-value-store interface makes collision avoidance the caller’s responsibility

The KeyValueStoreInterface documentation says that callers must avoid key collisions between different users of the store.

It also states that ordinary Get, Set, and Delete operations do not provide a general concurrency-ordering guarantee; callers needing broader ordering must impose it through the appropriate coordination mechanism.

The store is therefore not responsible for understanding:

which key belongs to which transfer

when that transfer is complete

when the key can safely be removed

PJRT owns that protocol.

Deletion is part of PJRT fulfilling its namespace-management obligation.

The new cleanup runs inside the `AsyncGet()` completion path

The refactored CrossHostRecvState begins one AsyncGet() for every key.

The callback captures:

AsyncWorkRunner

PromiseOnce

key-value-store reference

rendezvous key

When the lookup completes, the callback schedules another closure on the asynchronous work runner.

That closure performs:

promise.Set(result.status());

if (Status s = kv_store->Delete(key);
    !s.ok()) {
    LOG(DFATAL)
        << "Failed to delete rendezvous key";
}

The ordering is explicit:

1. Publish terminal promise status

2. Delete the completion key

Completion is published before cleanup is attempted

The source does not make buffer readiness depend on successful key deletion.

Promise status
→ published first

Delete
→ attempted afterward

The corresponding Future can therefore become ready before the cleanup result is known.

The buffer’s sequencing-event callback reacts to the future’s status:

Future is OK
→ allocate and record event

Future is error
→ mark event as error

Key deletion is not part of that future status.

This creates two distinct contracts.

Transfer contract

What is the destination buffer’s terminal outcome?

Cleanup contract

Was the consumed rendezvous entry removed?

They are ordered.

They are not transactionally combined.

A cleanup failure does not rewrite the transfer result

Suppose the transfer completed successfully.

AsyncGet status
→ OK

The closure executes:

PromiseOnce.Set(OK)

The future becomes successful.

If Delete(key) then fails:

future
→ remains successful

sequencing event
→ can still be recorded

cleanup failure
→ logged as DFATAL

The code does not change the future from:

OK

to:

cleanup error

afterward.

That is directly visible in the call order.

The PR does not explain whether every key-value-store backend treats cleanup failure as recoverable, fatal, or operationally exceptional beyond the DFATAL log.

The supported statement is narrower:

The buffer’s already-published terminal status is not replaced by a later deletion failure.

Why completion should not necessarily wait for housekeeping

There is a practical reason to separate these boundaries.

The destination buffer’s data may already be complete and valid.

A downstream kernel may be waiting only for the sequencing event.

If buffer readiness required successful control-plane deletion first:

GPU data ready

but

KV cleanup delayed

→ compute remains blocked

The landed code avoids that coupling.

Data completion
→ future can resolve

Key retirement
→ follows as cleanup

The source confirms the ordering.

The broader performance rationale is an inference because the PR does not publish a benchmark of deletion latency.

But calling deletion “mere housekeeping” would also be wrong

Cleanup is after the transfer result, but that does not make it optional.

Failure to delete can create:

retained store entries

stale completion state

future identifier conflicts

immediate callbacks on reused keys

The distinction is:

Deletion is not part of the buffer’s numerical data result.

Deletion is part of the control-plane lifecycle’s correctness.

The current transfer can be valid while the shared coordination namespace is left in a dangerous state for later work.

Why cleanup is scheduled through `AsyncWorkRunner`

The key-value-store AsyncGet() invokes a callback supplied by the caller.

PR #47530 does not perform promise publication and key deletion directly inside that callback.

It schedules them through:

AsyncWorkRunner

The source establishes this ownership transfer:

KV callback
        ↓
AsyncWorkRunner::Execute
        ↓
PromiseOnce.Set
        ↓
Delete key

The PR does not document one definitive reason for the extra scheduling boundary.

A reasonable architectural inference is that OpenXLA avoids performing future propagation, event callbacks, logging, and potentially backend-specific deletion work directly on the key-value-store callback thread.

What is confirmed is:

completion and cleanup are moved onto PJRT’s asynchronous work infrastructure

not the exact performance motive for doing so.

The callback thread no longer owns the full completion chain

Without the worker hop, a key-value-store callback might directly trigger:

promise completion

future callbacks

sequencing-event recording

key deletion

logging

The amount and location of that work would depend on the KVS backend’s callback execution model.

With AsyncWorkRunner:

KVS backend
→ reports lookup result

PJRT worker
→ owns the remaining lifecycle

This gives OpenXLA a more controlled execution boundary.

Again, that ownership structure is visible.

Its measured effect is not published.

Cancellation also reaches the same cleanup path

Part 2 examined the cancellation sequence.

Cancellation first attempts:

promises_[buffer_index].Set(reason)

Then it sets the rendezvous key:

kv_store_->Set(key, "");

The purpose of the key write is to wake the AsyncGet() that was registered when the receive was created.

The complete flow is:

Cancellation publishes error to PromiseOnce
        ↓
Future becomes cancelled
        ↓
cancellation sets rendezvous key
        ↓
AsyncGet callback wakes
        ↓
worker attempts PromiseOnce.Set(OK)
        ↓
later Set is ignored
        ↓
worker deletes key

The cleanup path therefore runs even when successful transfer completion did not win the promise.

The later success callback still has work to do after losing the promise race

If cancellation already resolved the promise, the later lookup callback cannot change the future.

PromiseOnce.Set(OK)
→ returns false or has no effect

But the closure does not stop there.

It still executes:

kv_store->Delete(key)

That is important.

Terminal status arbitration
→ cancellation already won

Completion-key lifecycle
→ still must be closed

Losing the future race does not remove cleanup responsibility.

Success and cancellation share cleanup without sharing the terminal result

The combined state machine is:

                  success lookup
                 /
Unresolved Promise
                 \
                  cancellation

The first accepted status owns the future.

But whichever route causes the rendezvous key to become available ultimately leads the AsyncGet() callback to delete that key.

Future status
→ first result wins

Key lifecycle
→ completion marker is consumed and removed

The two protocols are related but distinct.

Cancellation publication and key wake-up are not atomic

The cancellation implementation performs:

1. PromiseOnce.Set(cancellation reason)

2. kv_store->Set(rendezvous key)

Suppose the first succeeds and the second fails.

The state can become:

Buffer future
→ cancelled

Cancellation acknowledgment
→ reports KV Set failure

Outstanding AsyncGet
→ may remain pending

Key cleanup
→ may not run

The patch does not make these two operations one transaction.

The same observation applies to success publication and deletion:

Future can be complete

while

cleanup can still fail

This is an important evidence boundary.

PromiseOnce provides exactly-once status arbitration.

It does not provide transactional coordination across every external side effect.

The cleanup assumes the transfer’s exactly-once API contract

The source comment states that one transfer must either:

complete successfully

or

be cancelled exactly once

That contract matters to key deletion.

Consider cancellation winning, waking AsyncGet, and causing the key to be deleted.

If an unrelated or late second sender were still permitted to set the same key afterward:

key deleted
        ↓
late Set recreates key

the stale-entry problem could reappear.

The design relies on the higher-level transfer contract preventing multiple legitimate terminal publications for one descriptor.

One transfer identity

→ one terminal outcome

→ one consumed key

→ one retirement

The patch does not implement a general multi-writer transactional key protocol.

The key-value store itself allows overwrite in its default in-memory form

OpenXLA’s default in-memory store is configured to allow overwriting.

Its Set() operation writes:

data_[key] = value

and notifies the callbacks registered for that key.

The constructor also permits a non-overwriting mode when requested.

Allowing overwrite does not eliminate the stale-key problem.

The dangerous point occurs before the new writer arrives.

Old key already present

New AsyncGet registered

Callback fires immediately

New Set has not happened yet

Deleting the old entry is what restores the required initial condition:

New receive begins

→ key absent

→ future remains pending

until

new terminal publication occurs

A key has three identities at once

The rendezvous string carries several meanings.

Namespace identity

Which KVS entry is this?

Transfer identity

Which receive batch and buffer does it belong to?

Temporal identity

Which generation of that operation does the current value represent?

Random naming helps with namespace identity.

The buffer suffix helps with per-batch identity.

Deletion enforces temporal identity.

Without retirement, a name can continue carrying the meaning of an earlier generation.

Stale control-plane state can outlive valid data-plane state

The destination buffer can be reused or destroyed after its sequencing event completes.

The completion key can remain in a remote coordination service independently.

GPU allocation lifecycle
→ finished

PJRT buffer lifecycle
→ finished

rendezvous-key lifecycle
→ still present

This asymmetry is easy to miss because the key contains no tensor data.

But distributed correctness depends on control-plane state being retired along with the operation it names.

Resource growth is also real even without a collision

Suppose every cross-host receive creates:

B keys

and a long-running job performs:

T receive batches

Without deletion, the store can retain approximately:

B × T completion entries

even though every corresponding GPU transfer has finished.

The public PR does not quantify the memory footprint or the rate at which entries accumulated.

It explicitly identifies resource leakage as a reason for deletion.

The exact operational severity depends on the KVS backend and workload.

The lifecycle issue remains independent of the byte count.

The old callback registrations were not the only retained resource

In the in-memory implementation, NotifyCallbacksForKey() removes the callback list after invoking it.

Therefore, the post-completion leak is not accurately described as:

every AsyncGet callback remains registered forever

The more precise statement is:

the stored key-value entry remains after its callback has fired

That retained data is sufficient to:

consume store resources,
make TryGet() continue to succeed,
and cause a later callback registration for the same key to fire immediately.

The new lifecycle separates four moments

The refactored code contains four distinguishable points.

1. Transfer completion

GPU work reaches sender host callback

2. Completion notification

sender sets rendezvous key

3. Buffer-state publication

PromiseOnce resolves
→ Future becomes ready
→ sequencing event can be recorded

4. Coordination cleanup

rendezvous key is deleted

These moments can occur close together.

They should not be collapsed into one vague statement such as:

The transfer completed.

Different bugs belong to different boundaries.

A valid future does not prove cleanup succeeded

A caller can observe:

Buffer future
→ OK

That proves the receive event was completed successfully according to the future path.

It does not prove:

Delete(key)
→ succeeded

The deletion result is not encoded in the future.

The only source-level evidence of cleanup failure is the DFATAL log.

Observability tools treating future completion as complete protocol health could therefore miss a retained-key failure.

A deleted key does not prove the GPU copy was valid by itself

The reverse is also true.

The key deletion means:

the completion marker was retired

It does not independently validate:

the numerical contents of the destination buffer

That is protected by the transfer and sequencing-event contracts.

Cleanup should not be mistaken for data verification.

Transfer correctness

Promise status

Event readiness

Key cleanup

remain separate layers.

Why deletion follows lookup rather than sender `Set()`

The sender cannot safely delete the key immediately after setting it.

If it did:

sender sets key

sender immediately deletes key

receiver has not observed it yet

the receiver’s AsyncGet() could miss the completion signal.

Deletion must be owned by the consumer that knows the signal has been observed.

The new implementation does exactly that.

Sender
→ publishes key

Receiver AsyncGet
→ confirms observation

Receiver-side worker
→ deletes key

This is an acknowledgment-based lifecycle.

The receiver is the natural owner of signal consumption

The sender knows:

I published the completion marker.

The receiver knows:

My completion observer has received the marker.

Only the receiver can determine that the key’s purpose has been fulfilled end to end.

Publication
→ sender responsibility

Consumption and retirement
→ receiver responsibility

That ownership split prevents premature deletion.

The key is deleted even when `AsyncGet()` reports an error

The worker captures:

status = result.status()

It then executes:

promise.Set(status)

Delete(key)

regardless of whether the lookup status is OK.

Therefore:

successful key lookup
→ delete attempted

lookup error
→ delete still attempted

This makes the cleanup path terminal-status agnostic.

The public PR does not enumerate every key-value-store error condition or prove that deletion is always meaningful after each one.

The source confirms that the attempt occurs.

Deleting a missing key is backend-defined

The key-value-store interface returns an absl::Status from Delete().

OpenXLA’s in-memory implementation erases the key and returns OK even if the entry is already absent.

Another backend can implement different failure behavior within the interface contract.

The PJRT code therefore checks the returned status and logs failures.

The patch does not assume every deletion is infallible.

The cleanup failure policy is intentionally visible

Rather than silently ignoring deletion failure, the code emits:

Failed to delete rendezvous key <key>: <status>

through LOG(DFATAL).

That preserves evidence that:

buffer completion succeeded

but

control-plane retirement failed

The source does not retry deletion.

It does not convert the buffer event into an error.

It does not quarantine the key namespace.

Those would be separate policy choices not implemented by this patch.

The patch did not add a dedicated key-lifecycle test

PR #47530 changed one implementation file:

xla/pjrt/gpu/se_gpu_pjrt_client.cc

No test file was changed in the PR.

Existing cross-host tests cover behavior such as:

multi-buffer send and receive

full cancellation

partial cancellation

zero-sized buffers

chained receive-to-local-D2D transfer

Those tests can exercise the general receive path.

The patch did not add a targeted assertion that:

the rendezvous key no longer exists
after terminal completion

Nor did it add a deterministic stale-key-reuse test.

A direct cleanup test could inspect the store after completion

A targeted test could use an instrumented key-value store.

For one successful buffer:

register AsyncGet

complete transfer key

wait for buffer event

verify Delete(key) was called

verify TryGet(key) returns NotFound

For cancellation:

cancel buffer

verify future is cancelled

verify cancellation wakes AsyncGet

verify key is eventually absent

This would protect the key lifecycle directly.

It is a proposed regression design, not a test added by PR #47530.

A stale-key test should reuse the same deterministic identity

The strongest collision test would avoid relying on random chance.

First receive
→ use deterministic Key X
→ complete
→ assert X deleted

Then:

Second receive
→ reuse Key X
→ register AsyncGet
→ do not set X yet
→ assert future remains pending

Finally:

set X for the second transfer
→ assert future becomes ready

Without deletion, the second AsyncGet() could resolve immediately from the old entry.

A test of this shape would convert the PR’s collision rationale into a direct negative control.

The current public patch does not expose such a deterministic-key fixture.

A deletion-failure test would clarify policy

A fake KVS could return an error from:

Delete(key)

The test could then determine the intended behavior.

Current source behavior suggests:

Future status
→ remains the transfer result

Delete error
→ logged separately

A dedicated test could make that separation explicit.

It could also answer whether a future implementation should:

retry,
record a metric,
escalate the event,
or leave the current logging-only policy.

Cancellation wake-up failure deserves a separate boundary test

Because cancellation performs:

PromiseOnce.Set(reason)

then

kv_store->Set(key)

a test KVS could fail the second operation.

The expected state would need to be documented.

Questions include:

Should the future remain cancelled?

Should the cancellation callback return an error?

How is the outstanding AsyncGet retired?

Can the state object be reclaimed safely?

The current code implies some answers.

The PR does not add a test establishing the complete failure contract.

The source directly confirms the new lifecycle

The source and PR confirm that:

a unique rendezvous key is created per buffer within a transfer batch,
the sender sets the key after GPU stream completion reaches its host callback,
the receiver installs one AsyncGet() per key,
lookup completion is moved onto AsyncWorkRunner,
the promise status is set before key deletion,
deletion is attempted for successful and error lookup statuses,
cancellation wakes the lookup by setting the same key,
cleanup still runs if cancellation already won the promise,
deletion failure is logged as DFATAL,
and the PR explicitly cites resource leaks and key collisions as cleanup reasons.

The source does not confirm every lifecycle guarantee

The public evidence does not establish:

a historical production collision caused by a stale key,
the probability of random transfer-ID reuse,
the amount of KVS memory leaked before the fix,
transactional ordering across promise publication, key setting, and deletion,
retry behavior after Delete() failure,
cleanup completion before the sequencing event becomes ready,
physical cancellation of a GPU copy,
a deterministic cleanup or key-reuse regression test,
or identical concurrency semantics across every KVS backend.

The supported conclusion is narrower:

PJRT now retires each rendezvous key after its asynchronous lookup reaches a terminal result, preventing successfully consumed completion entries from remaining indefinitely in the shared coordination store.

Cleanup closes the identity contract

A transfer key is not just a string.

It declares:

This completion belongs to:

one transfer batch

one buffer index

one terminal generation

Creation establishes that identity.

Publication gives it meaning.

Observation consumes it.

Deletion closes it.

Create

→ publish

→ observe

→ retire

Without the final step, the identity remains active after its operation is gone.

The control plane must forget at the right time

Distributed systems need memory.

They also need forgetting.

Deleting too early loses the completion signal.

Delete before receiver observes
→ transfer may wait forever

Deleting too late preserves stale state.

Leave key indefinitely
→ resource growth
→ potential false completion after reuse

The correct boundary is:

receiver has observed the terminal signal
        ↓
future status has been published
        ↓
key can be retired

That is the sequence implemented by PR #47530.

The complete three-part structure

Part 1 — Independent transfers were observed through one batch barrier

Per-buffer GPU completion

→ sequential blocking Get loop

→ all sequencing events delayed until batch completion

The patch initiated all lookups concurrently and resolved each buffer independently.

Part 2 — Success and cancellation competed for one terminal state

AsyncGet status

versus

cancellation reason

→ first PromiseOnce Set wins

Later callbacks cannot overwrite the buffer’s future.

Part 3 — The completion signal had to be retired

Key observed

→ promise status published

→ key deleted

The destination buffer can finish while the shared coordination namespace is restored for later transfers.

The final lesson is that completion has a lifecycle

Asynchronous APIs often focus on:

How does work begin?

How does the caller learn that it finished?

A distributed protocol needs a third question:

How is the completion signal itself retired?

A key that says:

Transfer X is complete

is useful while Transfer X is active.

Afterward, the same key can become:

retained resource

stale generation marker

future collision hazard

A completion signal is not permanent state. It is a one-transfer message that must be consumed exactly once and then removed from the namespace that gave it meaning.

OpenXLA’s refactor made the lookups concurrent.

PromiseOnce made the per-buffer result terminal.

Delete() completed the lifecycle.

OpenXLA #47530 — Why Cross-Host Buffer Completions Were Gated by Sequential KV-Store Lookups
OpenXLA #47530 — Why Transfer Completion and Cancellation Had to Share One PromiseOnce

Related material

Patch status: Merged into OpenXLA main
Key identity: Random transfer ID plus per-buffer index
Key publication: Sender sets the key after GPU stream completion reaches its host callback
Key consumption: One AsyncGet() per receive buffer
Terminal-status order: PromiseOnce.Set(status) before key deletion
Cleanup execution: Scheduled through AsyncWorkRunner
Cancellation cleanup: Cancellation sets the key to wake AsyncGet; the resulting callback deletes it even when cancellation already owns the future
Deletion failure: Logged as DFATAL; not propagated through the already-selected buffer future
Resource rationale: Remove retained KVS entries
Correctness rationale: Prevent stale completion entries from satisfying a later lookup under a reused key
Dedicated cleanup regression: No new key-deletion or deterministic key-reuse test added by PR #47530

This is Part 3 and the final article in the OpenXLA PJRT cross-host transfer series.

Part 1 examined why sequential blocking key lookups turned independent transfer completions into one ordered batch barrier.

Part 2 examined how successful lookup completion, lookup failure, and cancellation compete for one once-only per-buffer future status.

This final article examined why the completion key must be retired after observation, why promise publication and cleanup are separate operations, and how a stale control-plane marker can outlive the GPU transfer it was created to represent.

#OpenXLA #PJRT #GPU #DistributedSystems #CrossHostTransfer #KeyValueStore #AsyncProgramming #PromiseOnce #ResourceLifecycle #Concurrency #CodeAnalysis