Real-time systems / Design study
Recovering a real-time session without losing its meaning
A design study of separate signaling and media lifecycles, bounded recovery, and truthful connection status.
A reconnecting socket does not necessarily mean a recovered conversation. A real-time interface must distinguish transport connectivity, application membership, and media health. Recovery is complete only when the state the product needs has been reconciled, or the interface explains which part remains unavailable.
Technical design study: conceptual guidance with illustrative examples and proposed checks, not a description of deployed client architecture, measured production outcomes, or tests already run.
01 / Lifecycles
Signaling and media fail independently
Imagine a browser conversation that uses a WebSocket for membership and negotiation messages and an RTCPeerConnection for media. Losing the socket can leave an established peer connection carrying audio. Conversely, an open socket says little about whether audio reaches the other participant. Track signaling availability, room membership, and media connection state separately, then derive a user-facing summary from those facts.
The invariant is that the interface never reports a fully recovered session solely because a transport opened. A signaling interruption can display “Reconnecting session controls” while preserving working media. A media failure needs a different explanation. The counterexample is a single connected boolean toggled by both transports: whichever callback runs last hides the other failure and can offer controls that the server is not ready to honor.
02 / Recovery budget
Retry within a deadline and an ownership boundary
Give the session controller a recovery generation, a deadline, and a bounded retry schedule with backoff and jitter. Only one attempt owns the next transition. A new socket must authenticate and reconcile membership before application controls become available. If the server supports resumption, use its cursor or revision contract; otherwise obtain a fresh snapshot. Reconnecting alone does not prove that missed events were delivered.
Treat a failed peer connection separately. An ICE restart can trigger renegotiation through signaling, but cannot guarantee recovery from every network or remote-peer failure. A transient disconnected state can justify observation within the product’s deadline rather than immediately rebuilding the call. If recovery exhausts its budget, stop automatic attempts and offer an explicit rejoin. Rejoining creates a new session generation so old callbacks cannot mutate it.
03 / Reconciliation
Recover membership before replaying commands
A socket may close after the server accepts a command but before its acknowledgment arrives. Blind replay can duplicate a participant action. Where the application protocol supports it, assign an operation ID and have the server return the original outcome for repeated submissions. Where it does not, fetch authoritative state and ask the user to retry ambiguous actions. A transport acknowledgment and an application acknowledgment are different facts.
Leaving the conversation must invalidate recovery immediately, close owned transports, and cancel scheduled attempts. Browser online events can prompt a guarded attempt, but they do not establish reachability to the service. Visibility changes can trigger reconciliation after suspension. Every trigger enters the same controller rather than starting its own reconnect loop. This keeps recovery explainable when the network, route, and user intent change at nearly the same time.
System flow
A bounded recovery sequence
- 01
Classify the failure
Record which transport or membership contract failed without discarding healthy media.
- 02
Attempt recovery
The current generation performs one scheduled attempt within the shared deadline.
- 03
Reconcile authority
Authenticate, restore membership, and obtain a snapshot or supported event continuation.
- 04
Resume or explain
Enable reconciled controls, or stop and offer rejoin with an accurate status.
Decision record
Design decisions and tradeoffs
↔ Scroll horizontally to compare approaches and tradeoffs.
| Decision | Approach | Tradeoff |
|---|---|---|
| Separate state dimensions | Track signaling, membership, and media independently. | More explicit states are needed, but a working audio path can survive a controls outage. |
| Finite automatic recovery | Use a deadline and bounded retries with jitter. | Some sessions require manual rejoin; the interface avoids indefinite activity without progress. |
| Authoritative reconciliation | Resume only through a documented server contract or fresh snapshot. | Recovery may take another round trip, but missing events do not silently become accepted state. |
| Explicit ambiguous actions | Deduplicate by operation ID where supported; otherwise reconcile before retry. | Server cooperation is necessary for safe replay, and some outcomes must remain visibly uncertain. |
Failure cases & validation
Failure scenarios to validate
Proposed validation scenarios and expected behavior. These are design checks, not reported production test results.
Signaling drops while audio still flows.
Audio remains attached and session controls report their restricted availability.
The socket opens but room membership has expired.
The interface waits for membership reconciliation and does not declare recovery.
ICE recovery fails through the configured deadline.
Automatic recovery stops and an explicit rejoin action becomes available.
The user leaves during backoff, then enters another room.
Old timers and callbacks cannot reconnect or update the new session generation.
A command was accepted before its acknowledgment was lost.
A supported operation ID recovers the existing outcome; unsupported replay remains blocked pending reconciliation.
Reflection
Engineering lessons
- 01
Define recovery in terms of application invariants, including membership and command outcomes.
- 02
Give every retry a current owner, a deadline, and a terminal user-visible outcome.
- 03
Preserve healthy media while repairing signaling, and acknowledge uncertainty when delivery is ambiguous.
Evidence & further reading