You test a channel at lunchtime and it plays. At 8 pm a customer says it's black. You try it again and it plays for you, fails for them, and you start to wonder whether they're imagining it.
They aren't. More often than not, the line is full.
How a connection limit works
When you buy a line from a provider you get a username, a password and a number: how many connections that line can hold at the same time. Every time something opens one of the line's streams, that's a connection. The provider counts them on its side and refuses the next one once you hit the limit.
The word that matters is "something". The provider doesn't know or care whether it's a viewer, your panel, one of your load balancers or a script you ran last week. A connection is a connection.
When the line is full, most providers answer with an error instead of video. You'll see codes like 509 or 429 in your logs, or a redirect to a short "limit reached" clip. A 403 can mean the same thing, though it can also mean the provider has blocked your server's address, so check which one you're looking at. Your viewers don't see any of this. They see a channel that won't start.
Where the connections go
When we go through a panel that keeps hitting its limit, the same things turn up again and again.
Channels running for nobody
An always-on channel holds its connection 24 hours a day. If a line allows 100 connections and 80 of them sit under always-on channels, you have 20 left for everything else, and you have them at the busiest hour of the day.
The same channel pulled twice
With several load balancers, each server that carries a channel can open its own connection to the same source. Three servers carrying it means three connections for one channel. It's easy to miss, because each server looks perfectly normal on its own.
Connections that never close
A player drops off badly, a process gets stuck, and the connection stays open on the provider's side. Some providers keep counting it for minutes. A few of these on a busy evening and you're full without anyone watching more than usual.
We once lost most of a weekend to channels that wouldn't start. The cause was a stuck stream that could never let go of its viewers, so the connections behind it never came back, and everything sat at its limit until we found it.
Checks that open a stream
Testing whether a source works by playing it uses a connection. Testing a few hundred sources one after another, in the evening, uses a lot of them, and it's often an automated job nobody remembers setting up.
Copies that need their own connection
A +1 or a delayed version of a channel is often set up as a second stream that pulls the same source again. That's one more connection for something you already have.
Viewers sent straight to the provider
Some panels can send a viewer straight to the provider's link instead of passing the stream through. It saves your bandwidth, but every viewer then takes a seat on the line, so 250 viewers need 250 connections. One busy evening and the line is full.
Is it really the limit? Reading 509 and 429
Before changing anything, confirm it. Look at what the provider answered when the channel failed: a 509 or 429 usually means you're at the limit, not that the channel is down. Then count what each of your servers really holds. This prints a server's open connections to one provider address (put in your provider's):
ss -Htn state established dst 203.0.113.10 | wc -lIf the provider's dashboard counts more than your servers hold, something is leaking. If your servers hold far more than the number of channels being watched, the rest are going to the places above.
And test at the busy hour. Lunchtime tests are how this problem survives for months.
Getting connections back
Share one connection per channel. All the viewers of a channel on a server should come off a single source connection. Then make sure a channel normally runs on one server at a time, so the provider sees one connection per channel, however many people watch it.
Let quiet channels sleep. An on-demand channel opens its source when the first viewer arrives and closes it when the last one leaves. It only works if channels start quickly. If they take eight seconds, customers complain and you're back to always-on. (Idle channels cost you CPU and memory as well.)
Close connections cleanly, and watch for the ones that don't. If your panel can show how many connections each line is using right now, compare that with the provider's figure every so often. A gap means a leak.
Check sources without holding on to them. A check only needs a few seconds of video to confirm that a source plays and what's inside it, then it should let go. And don't check a line that's already full, because you'll take a connection a viewer could have had.
Make copies from what you already have. A delayed channel can play from a recording on your own server instead of pulling the source a second time.
Keep backup sources on a different line. A backup on the same line as the main source is no help when that line is full. Put it on another line, or with another provider. A backup source won't help when your own server fails, though; that needs a different fix.
What we did about it
In Minimus, a channel uses one connection on a server however many people watch it, and it can start with the first viewer and stop after the last. The check button on a stream's sources reads each one once and lets go, and it won't touch a line that's already full. A delayed copy of a channel plays from a recording on your own server, so a +1 made that way never costs a second connection.
Whatever you run, do one thing before you buy another line: count the connections each line uses at 9 pm and put that next to the number of channels people are actually watching on it. If the line holds far more connections than that, a new line won't fix it. It'll just fill up too.