Currently playing with the idea of organizing a #PHP unconference online on @Workadventure_ . Interested? Please give your feedback on this Reddit thread!

RT @mattpocockuk: Here's how to guarantee you've handled every known error in your system. from the very top of a potentially enormous call….

RT @brendt_gd: I'm so proud: two years of working together with a lot of talented programmers, finally we tagged Tempest 1.0: a new framewo….

RT @Workadventure_: No slides, no stress, just sun. Our team retreat = 5 days in the South of France with bikes, beach, paintball. and a….

So it seems that the "Active users" messages from Livekit are generating quite a load (~8MB/s, that's a lot!), but it does not completely explain the very high load on Redis. We need to dig deeper!.

Hypothesis tested: it's the messages related to the active users that generate this load. To test this, we did 2 tests. The first test (on the left), with sound => Redis bandwidth: ~30MB/s.The second test (on the right), without sound => Redis bandwidth: ~22MB/s.

Some more @livekit related Tweets!. Current question: why is Redis under such a high load when using Livekit?. In Yellow: the Livekit output bandwidth during load tests with 70 to 100 users. In Green: the Redis output bandwidth.

We'll test that hypothesis and I'll come back here as we advance in those tests :).

Current hypothesis: the load test scenario we use (based on Playwright) is emitting short "beeps". All 70 users of the test scenario emit the beeps at regular interval and need to dispatch those to the other users (so that's a O(n2) complexity).

It turn out the 20MB/s bandwidth is generated by. the communication between Livekit and Redis 😱😱😱. Redis does not receive any video streams, but it receives messages anytime something happens in a room (like the active user is changing).

Our first hypothesis: the other node was relaying some video streams, somehow. We investigated Kubernetes Service NodePorts and stuff like that to no avail. Livekit is exposing UDP ports directly on the host node, so this part is working correctly.

In yellow: the output bandwidth of the Livekit server (up to ~80MB/s).In green: the output bandwidth of the **other** server (up to ~20MB/s).Why the hell is it taking a part of the bandwidth?.

During our load tests, I see this:

Our test K8S Livekit cluster has 2 nodes. - 1 node that contains ONLY the "Livekit server" we are load testing.- 1 node for all the rest (Redis required by Livekit, Prometheus to gather metrics, etc. ).

I'm keeping doing load-balancing tests with @Livekit for fun and profit. Today's mystery: an unexplained network load on one of our servers. 🧵.

The good news?. Security measures at OVH are in place to keep their system safe. The bad news. I'm not sure how I will circumvent those :). Anyway, it's a fun ride and I'll keep posting my findings as we progress in those load tests!.

BOOM! This is the sound of a livekit server that crashes! Or is it? . After thorough investigation, it turns out that the #OVH anti-hack system was triggered and #OVH simply turned down our Livekit node because they thought it was under attack (it was somehow 😀).

The number of users is piling up to about 70 simultaneous users and at that time. .

This saves a lot of bandwidth and allows.@livekit.to scale greatly. So. in practice, what does my first load test looks like?

They most probably have one stream taking most of the page and maybe 5-10 smaller videos at the top. The beauty of Livekit is that it automatically detects if a video is visible or not on your screen, and will adapt the video stream downloaded automatically.