It's 9 pm on a Saturday, the busiest hour of the week, and the main server is crawling. The load average says 38. free -m shows a few hundred megabytes left out of 32 GB. Messages about buffering start coming in, so you do what everyone does and rent another load balancer.
A week later the new one looks exactly the same.
We've had that conversation with operators more times than we'd like. What eats the server is usually the channels themselves, and the fact that most panels keep them running whether anybody is watching or not.
One channel, one ffmpeg process
Most panels you'll come across start a separate program for every channel, usually ffmpeg. That program opens the source, reads it, repackages the video and hands it to viewers. It works, and it's simple to build, which is why it's everywhere.
The catch is that each of those programs carries its own overhead: its own memory, threads, buffers and connection to the provider. One of them is cheap. A thousand of them isn't. A server with 1,500 channels set to always on can end up with 1,500 copies of ffmpeg in memory, each holding tens of megabytes.
It's one reason so many install guides ask for 16 to 32 GB of memory on the main server before you've added a single customer.
The channels nobody is watching
Do the sum for your own list. If you carry 2,000 channels and 300 people are watching, at least 1,700 channels have nobody on them right now. At 4 am it's nearly all of them. They still cost you. The process keeps running, the source connection stays open, and the CPU keeps repackaging video that goes nowhere.
Setting channels to on-demand fixes that in theory. In practice a lot of operators turn it on, get complaints that channels take eight or ten seconds to start, and turn it straight back off. So the idle channels go back to burning CPU all night, and each of them keeps a provider connection too, which is how lines fill up at 8 pm.
Transcoding you never asked for
The fastest way to overload a server is transcoding. Remuxing, which moves the same video into a different container, costs very little. Transcoding means decoding the picture and encoding it again, and one HD channel can keep a whole CPU core busy doing it.
Some streaming software quietly falls back to transcoding when it can't read a source cleanly. Then, when a batch of channels goes dead or silent at the provider, it starts re-encoding all of them at once. We've watched two load balancers go from comfortable to a load average above 600 that way, until the kernel started killing processes to free memory. Nobody had touched a setting.
If your CPU graph jumps at the same time as a batch of channels goes dead, look here first.
Check your own server in five minutes
You don't need a monitoring stack to see where the resources go. On a streaming server, run these:
# How many copies of each program are running
ps -eo comm= | sort | uniq -c | sort -rn | head
# Memory held by all ffmpeg processes together, in MB
ps -C ffmpeg -o rss= | awk '{ sum += $1 } END { print sum / 1024 " MB" }'
# How many of them are re-encoding video (look for an encoder, not "copy")
ps -C ffmpeg -o args= | grep -Ec -- '(-c:v|-vcodec|-codec:v) +(libx26|h264_|hevc_)'The memory figure runs a little high, because libraries shared between processes are counted once for each of them, but it's close enough to show the shape of things.
Now put the first number next to how many people are actually watching. If you see 1,200 ffmpeg processes and 300 viewers, you know where your money is going. If the last command prints anything other than zero and you never set up transcoding on purpose, you've found your CPU.
What a lighter setup looks like
None of this needs special hardware.
A channel should open its source once, and every viewer after the first should be served from that same connection. Ten viewers or a few hundred, the provider sees one connection from that server.
On-demand only works if starting is fast. When a channel can show a picture in about a second, there's no reason to keep it running for nobody, so it can stop when the last viewer leaves.
A source that can't be read should fail loudly, with a reason, instead of being re-encoded in the background.
And the boring one: send less. Playlists and TV guides compress extremely well. A 128 MB playlist goes out as roughly 15 MB with gzip, which adds up when thousands of devices refresh it every evening.
What we built
In Minimus, the panel we build on Rust and GStreamer, channels can start when the first viewer arrives and stop after the last one leaves, and every viewer of a channel on a server shares one source connection. In our lab, a viewer joining a channel that's already running gets a picture in about 0.06 seconds, and a cold start takes about a second on sources with 1-second keyframes. The main server runs on 2 vCPU and 4 GB of memory.
You don't need us for most of this, though. Take the fifty channels nobody watched last week, set them to on-demand, and compare tomorrow evening's load graph with tonight's.