Realtime without lost events: an append-only log and a cursor

Realtime features usually break in two places: a client reconnects and quietly misses events, or one slow client holds back everyone else. In PromptP I designed the protocol around both from the start. The core idea is small. Every event goes into an append-only log, and every client remembers how far into the log it has read.

Why fire-and-forget fanout isn't enough

Redis pub/sub, like most fanout, delivers a message to whoever is subscribed at that instant. A phone that drops off Wi-Fi for three seconds isn't subscribed for those three seconds, and nothing will ever tell it what it missed. A full reload on every reconnect hides the problem, but then every reconnect is expensive and the client still has to work out what changed.

The log

Redis Streams give you an append-only log per key. XADD appends an entry and returns its ID: a millisecond timestamp and a sequence number, like 1729238400000-0. IDs only ever increase, so the ID is also the order, and nothing else has to agree on ordering.

// Append to the room's log. "*" lets Redis assign the next ID;
// MAXLEN ~ keeps roughly the last 10,000 entries without trimming on every write.
const id = await redis.xadd(
  `room:${roomId}`,
  "MAXLEN", "~", 10000,
  "*",
  "type", event.type,
  "data", JSON.stringify(event.data),
);

Views are derived from the log instead of being mutated in place. A client applies events in ID order, and replaying the same events always produces the same state.

The cursor

Each client keeps the ID of the last event it applied. That one string is its whole position. When it reconnects it sends the cursor, and the server replays everything after it before the client goes live again.

// On (re)connect: join the live room first, then replay what the client missed.
async function resume(socket, roomId, cursor) {
  socket.pending = [];             // live events that arrive while we replay
  rooms.get(roomId).add(socket);   // from here on, nothing new is missed

  // "(" makes the start exclusive (Redis 6.2+): only entries after the cursor.
  const missed = await redis.xrange(`room:${roomId}`, cursor ? `(${cursor}` : "-", "+");
  for (const [id, fields] of missed) deliver(socket, id, fields);

  const queued = socket.pending;
  socket.pending = null;           // live from now on
  for (const [id, fields] of queued) deliver(socket, id, fields);
}

The order of those steps matters. Replay first and join second, and an event written in between is never sent. Join first and send live events straight away, and they can overtake the replay. So the server joins first, holds that socket's live events until the replay is done, then sends them. A few events now arrive twice, once in the replay and once from the queue, and the client drops any ID at or below its cursor. Compare IDs as two numbers, time then sequence, not as strings. Order is kept and nothing is lost.

When the cursor is too old

Trimming keeps the log bounded, so a client that was away long enough can come back with a cursor older than the oldest entry. The server checks by reading the first entry (XRANGE key - + COUNT 1), and if the cursor falls before it, sends a fresh snapshot instead of a replay. It's the rare path, so it's allowed to be the slow one.

Backpressure

Fanout is a loop over sockets. When one of them is on a bad connection its buffers fill up, and a naive server keeps queueing for it until memory runs out or the event loop stalls. Every other client in the room feels it.

The ws library exposes how much is queued for each socket as bufferedAmount, which allows a simple, local rule: past a threshold, stop sending to that client. Its cursor is safe and the log is durable, so the cheapest fix is to close the connection and let it resume once it can keep up.

const HIGH_WATER = 1024 * 1024; // 1 MiB queued for one socket is more than any client should need

// Fanout calls this for every new entry in the room.
function send(socket, id, fields) {
  if (socket.pending) socket.pending.push([id, fields]); // still replaying: hold it
  else deliver(socket, id, fields);
}

function deliver(socket, id, fields) {
  if (socket.bufferedAmount > HIGH_WATER) {
    // Too far behind. Closing costs this client a replay; keeping it costs everyone.
    socket.close(4000, "slow consumer");
    return;
  }
  socket.send(JSON.stringify({ id, ...toObject(fields) }));
}

// ioredis returns an entry's fields as a flat [key, value, key, value] array.
function toObject(fields) {
  const out = {};
  for (let i = 0; i < fields.length; i += 2) out[fields[i]] = fields[i + 1];
  return out;
}

Closing sounds harsh, but it's the honest option. The client reconnects with backoff, replays from its cursor and catches up at its own pace, while the rest of the room never notices.

Optimistic updates

Waiting for the server before showing your own action feels slow, so the client applies it immediately under a temporary ID and sends it. When the event comes back from the log with its real ID, the client swaps the temporary entry for the real one. If the server rejects it, the client removes the temporary entry and says why. The reconcile step is small but unforgiving: the confirmed event and the optimistic one must never both be on screen.

What this buys

• Reconnects are routine: a replay of what was missed, not a full reload.
• A slow client only slows itself.
• Ordering comes from one place, the stream ID, so the client code stays almost boring.