Can RL rediscover a simple streaming rule?

Adaptive bitrate (ABR) streaming balances video quality against stalls. In this controlled simulator, reinforcement learning (RL) learns to choose between two bitrates.

The test: Does the policy learn to favor a higher bitrate as the playback buffer fills?

Adjust the stall penalty and learning rate, then press Train. Click the heatmap to change the starting policy.

Return landscape

Computing the landscape…

Bitrate policy

4.3 Mbps · high bitrate2.85 Mbps · low bitrate

Training step

Step 0 / 10,000

w = 0.00 · b = 0.00

Initial policy: Preparing the experiment.

The controlled ABR experiment

Two failure modes: unstable optimization paths and distorted return landscapes.

Change the optimization path

Change the learning rate or starting point to compare optimization paths on the same landscape.

Change the return landscape

Change the stall penalty to see how the return landscape shifts.

Setup and reward

Each episode has 8 four-second chunks, two bitrates (2.85 and 4.3 Mbps), and a 60-second buffer. Playback stalls when buffered video runs out.

Return is the expected total reward per episode, averaged across 7 constant-bandwidth network traces: 3, 3.5, 4, 4.5, 5, 5.5, and 6 Mbps.

Reward per chunk = bitrate / 4.3 − stall penalty × stall seconds / 4

How RL learns

With parameters $θ = (w, b)$ , buffer state $s = B / 60$ ( $B$ in seconds), and sigmoid $σ$ :

π_{θ} (high ∣ s) = σ (w s + b)

REINFORCE uses $N = 16$ episodes of $T = 8$ chunks to estimate the policy gradient:

\hat{g} (θ) = \frac{1}{N T} \sum_{i = 1}^{N} \sum_{t = 0}^{T - 1} G_{t}^{(i)} \nabla_{θ} \log π_{θ} (a_{t}^{(i)} ∣ s_{t}^{(i)})

\begin{array}{l} \hat{g} (θ) = \frac{1}{N T} \sum_{i = 1}^{N} \sum_{t = 0}^{T - 1} G_{t}^{(i)} \\ \times \nabla_{θ} \log π_{θ} (a_{t}^{(i)} ∣ s_{t}^{(i)}) \end{array}

G_{t}^{(i)} = \sum_{k = t}^{T - 1} r_{k}^{(i)}

$a_{t}$ is the bitrate choice; $r_{t}$ is the chunk reward. Updates use Adam with a small entropy bonus.

Failure modes

F1: Unstable optimization paths. A high-return ridge exists, but optimization misses it.

F2: Distorted return landscapes. Reward or trace changes shift the high-return region toward undesirable policies.