Designing an SFU (e.g., Mediasoup)

Strip away the acronym and an SFU has one job: receive media from each participant, decide who else needs it, and forward it unchanged — no decoding, no re-encoding, no mixing. That single constraint is what makes an SFU cheap to run at scale, and it's also what shapes almost every design decision inside one. Mediasoup is a good reference implementation to reason about, because it exposes that design directly in its API instead of hiding it behind a black box.

Producers and consumers: the fan-out primitive

Mediasoup models the world as two objects. Every media stream a client sends up — a webcam track, a microphone track, a screen share — becomes a Producer on the server. For every other participant who needs to receive that stream, the SFU creates a matching Consumer, which pulls packets from the Producer and forwards them down that participant's connection.

One upload, many downloads: a single Producer can feed dozens of Consumers, and the SFU is just plumbing RTP packets between them. It never touches the payload — no codec work happens on the server at all. That's the whole trick behind an SFU scaling to far more participants per CPU core than an MCU ever could, because an MCU has to decode every incoming stream, composite or mix it, and re-encode outgoing streams per viewer.

// On the sending client: publish local media to the SFU
const sendTransport = await device.createSendTransport(transportParams);

const producer = await sendTransport.produce({
  track: localVideoTrack,
  encodings: [
    { maxBitrate: 150_000,  scaleResolutionDownBy: 4 }, // low
    { maxBitrate: 500_000,  scaleResolutionDownBy: 2 }, // medium
    { maxBitrate: 1_500_000, scaleResolutionDownBy: 1 }, // high
  ],
});

// On the SFU: a new consumer is created per receiver, per producer
router.on("newproducer", async (producer) => {
  for (const receiver of roomPeers(producer.roomId)) {
    if (receiver.id === producer.peerId) continue;

    const consumer = await receiver.recvTransport.consume({
      producerId: producer.id,
      rtpCapabilities: receiver.rtpCapabilities,
    });

    // The SFU never decodes producer's media — it only relays
    // whichever encoded layer suits this consumer's bandwidth.
    consumer.setPreferredLayers({ spatialLayer: receiver.estimatedLayer });
  }
});

In practice a room is just a set of Producers and a many-to-many mesh of Consumers wired between them, created and torn down as people join, leave, mute, or start screen sharing.

Simulcast: relaying, not transcoding

Because the SFU refuses to touch encoded media, it can't magically shrink a stream for a bandwidth-constrained viewer the way an MCU could. Mediasoup's answer is to push that problem back onto the sender: the encoder on the sending client produces simulcast — the same video encoded simultaneously at several independent quality layers, say a low-resolution 150 kbps layer, a medium 500 kbps layer, and a high 1.5 Mbps layer, each as its own RTP stream.

The SFU's job then becomes a routing decision rather than a media-processing one: for each Consumer, look at that receiver's available bandwidth, CPU, and even how large the video tile is rendered on screen, and pick which already-encoded layer to forward. A phone on a shaky 4G connection gets routed the low layer; a pinned speaker on a fast desktop connection gets the high layer — and the SFU can switch a Consumer between layers on the fly as conditions change, without ever decoding a single frame.

Scaling within and across machines

A single SFU process doesn't use one thread for everything. Mediasoup spins up a pool of workers — typically one per CPU core — and each worker independently owns a subset of the rooms (and therefore the Producers, Consumers, and the RTP/RTCP processing that goes with them) running on that machine. Spreading rooms across workers this way lets a single box handle a large number of concurrent calls without any one core becoming a bottleneck, and it keeps the failure domain small: a problem in one worker doesn't take down every room on the host.

Eventually one machine runs out of cores, memory, or network bandwidth, and scaling means adding more SFU nodes and deciding which rooms live where — usually by consistent hashing on room ID, or a control-plane service that assigns rooms to the least-loaded node. The wrinkle is that participants in the same room don't always land on the same node. When a Producer on node A has a Consumer that needs to live on node B, something has to relay that stream between the two SFU instances — a server-to-server (or piped) transport that looks, from each SFU's point of view, like just another Producer/ Consumer pair, except the far end is another server instead of a browser. This is the piece that turns a fleet of independent SFU boxes into something that behaves like one large, elastic SFU.

Bandwidth estimation and graceful degradation

None of the routing decisions above mean anything without a continuous, accurate read on how much bandwidth is actually available on each connection, and that number is always moving — Wi-Fi contention, a neighbor's video call, a phone switching towers. WebRTC leans on two complementary congestion-control signals for this: REMB (Receiver Estimated Maximum Bitrate), where the receiving side periodically reports back the bitrate it believes the link can sustain, and transport-cc, a more precise scheme where the receiver acknowledges packet arrival times so the sender can infer queuing delay and loss trends directly.

The SFU sits in the middle of both feedback loops. It uses the estimate from each downstream connection to decide which simulcast layer to forward to that Consumer, and it relays sender-side feedback back to each Producer so the sending client's encoder can drop its target bitrate — or disable a simulcast layer entirely — before the network actually stalls. Done well, a call under pressure degrades in resolution and frame rate first, in a controlled and mostly imperceptible way, rather than freezing or dropping altogether.