When a streaming server goes down: keeping channels on air

By the Minimus team · · 3 min read

Servers fail. Sometimes it's a full disk, sometimes the host reboots for maintenance it forgot to mention. Your customers can't tell the difference, and they don't care.

On a lot of setups the answer is a black screen on every channel that lived on that server, until somebody wakes up, signs in and moves them by hand. With 1,200 channels over three servers, that's 400 channels to edit at 3 am.

Why channels get stuck on one server

Most panels ask you to choose the servers for each channel. It feels like control. In practice it means the knowledge of where everything runs is spread across hundreds of stream settings, and nothing moves unless you move it.

Add a fourth server and it sits empty until you edit channels to use it. Lose one, and its channels stay down until you edit them again.

What automatic routing needs

For channels to move by themselves, the panel has to know three things about every server, all the time: whether it's up, how busy it is, and whether it has room for one more channel.

Then, when a viewer asks for a channel, the panel picks a healthy server with room and sends the viewer there. If that server stops answering, it's marked as down and the next request for each of its channels goes somewhere else. Players that lose their stream reconnect, and the reconnection lands on a healthy server.

Server 1room leftABDEServer 2offlineDEFServer 3room leftGF Already there Moved there by itself
Server 2 goes offline. Its channels carry on from the servers that had room.

So nothing has to "move the channel" at 3 am. The next request for it just has to go somewhere that works.

Leave room to fail into

Automatic routing only helps if the other servers can take the extra load. If three servers each run at 90% at the busy hour, losing one leaves the other two trying to carry 135% each. You're still down, just a bit more slowly. (If your servers run that hot with few viewers, look at what's using the CPU first.)

A rule of thumb: at your busiest hour, the servers left after losing your biggest one should still sit under about 80%. With three servers of the same size, that means none of them should run above roughly half at peak.

Backup sources are a different problem

Server failover and source failover get mixed up a lot. A backup source helps when the provider's stream dies. It does nothing when your own server dies. You want both, set up separately: a backup source on another line or provider for each channel that matters, and enough servers that any one of them can fail.

And when a channel has moved to its backup source, don't rush it back. A source that failed once tonight can fail again five minutes later. Let the main source prove itself for a while before you switch back.

Test it before you need it

The first time failover runs shouldn't be on your busiest night. Pick a quiet morning, stop the streaming service on one server, and watch. Do channels play from the others? How long does a viewer wait for the picture to come back? Does anything stay black?

Then start it again and check that it starts taking channels again once it's healthy.

How Minimus handles it

Set a channel's servers to Automatic in Minimus and it chooses a healthy server with room for it. A full or offline server is skipped, a new server starts taking channels once it's added, and when a server goes down its channels move and viewers reconnect to a healthy one. If you'd rather stay in control, list the servers in order and the first healthy one carries the channel.

Spare room is still your job. Do the 80% sum anyway.

More from the blog