Investigating Video Player Memory Issues

I know a lot of you have been complaining about tabs crashing in both FireFox and Chrome especially on PC, and we've been listening. The problem is, these issues can be really tricky to figure out, especially when nothing has changed with our video player when these issues crop up so we can't point to one single change and say that was it.

I did a lot of digging and I wanted to summarize my progress so far so you know that we're working on it.

The problem: Chrome/FireFox tabs crashing while watching really long live streams or videos

The devices: Usually a Windows 10 PC running Chrome or FireFox. This doesn't mean the issue only happens on Windows, but it may just be more prevalent there.

Reproducing the problem: So here's a quick test run I ran using my windows. After running a video in our GB player for 30 minutes here's the memory usage statistics for that tab:

Yikes...

So what is causing that gradual increase of memory, let's run the Chrome memory profiler to see if something we're doing in the dom is causing that memory spike, let's take a couple heap snapshots over time and do a timeline profile of that memory to see if there's anything obvious:

Well crap... As you can see there just isn't really anything obvious causing the memory to balloon over time... Hell, the memory timeline peaks at under 30MB, nowhere near the 1GB+ we're seeing in the chrome task manager. This tells us that it's not something obvious we're doing in the video player, like say leaving a bunch of detached dom nodes or event handlers dangling in memory somewhere.

So what to do now?

Well, let's see if we're the only ones having this issue, maybe it's some configuration in how we're playing video or some setting we need to turn off.

Let's run the same stream in another player to make sure it's not just us. Our CDN has some test players that we can use to plug in some stream URLs so let's try that:

http://players.akamai.com/hls/

After running for a few minutes we're already up to 300MB:

So it looks like it may not just be us having the issue. Unfortunately now we have to resort to Googling...

Googling stuff like this is hard... Everybody is having video issues with everything all the time so it's hard to search for individual issues without wading through the deluge on Google of crap from like 2015. One tool that is indispensable is the Google filter results by time period tool so you don't see a bunch of crap from 3 years ago:

Even now though there's just a bunch of links to old github issues that have recent comments or projects that aren't related at all. Lot's of threads about Adobe CS memory issues though of course...

But that first result is promising. HLS.js is a library we use to render video in browsers that don't have native HLS support (like Chrome) has had memory issues in the past.

Let's go to their github page and see if they have any open issues or pull requests that look related:

https://github.com/video-dev/hls.js/issues

Nothing in issues really related to a memory leak (at time of writing) let's pop over to pull requests.

Well that second one there is intriguing.

VTT cues are not cleaned up until a stream is destroyed.

Long Live streams lead to a memory leak, as the TextTrackList continues growing with Cues until the browser runs out of memory and becomes un-responsive.

That looks like that could be related... It has to do with VTT cues which are tags related to video captions, no we don't currently run captions on our videos but we do have that ability built into the player. Maybe it's still saving empty cues that are coming through with each segment and still causing a leak? Maybe it's completely unrelated, not really sure.

Well that's where I'm at now, going to continue investigating and I'll update here as I make more progress. Thank you for your patience as we work through this.

-Will

That was an interesting read, good luck with your continuing efforts

Very interesting. Love the transparency. I definitely noticed this issue on other video players, but not all. So glad I don't have to do the work you're doing.

Thanks for the write-up Will! I don't have much experience debugging front-end issues like this so it was very cool to read your process and where you ended up. As an alternative workaround to refreshing, I've been using the 'video/current-live' API endpoint to get the stream URL and pop it into a desktop video player and popout chat into a separate Firefox window. Working pretty well so far but I do miss using the site player. Best of luck!

Nice to read the thought process!

This got me curious, so I had firefox tell me where the memory usage was going and it indicated it was all in ArrayBuffer objects (which smells like retrieved media data). I spent a little bit of time looking at hls.js, and for a "live stream" it sounds like you'll want to set liveBackBufferLength otherwise it'll use the default of Infinity (which it looks like you are not setting?). However, the memory usage blows up even if it's not a "live" stream, soooo I really don't know what's going on there. I didn't want to spend a lot of time trying to understand HLS.js's non-live buffering code, so that was where I stopped.

I learned some stuff about HLS, so it was still interesting even if I didn't find that much out.

Just wanted to update that this issue seems to be fixed. I had several Windows 10 updates and the latest Firefox update (69) and tabs no longer consume excess memory. Not sure if anything was updated on GB's site, but I thought I'd update this thread with my experience.

This issue is cropping back up for me. Watching the latest Bombcast for example over a span of 15 minutes memory usage of the video player rises from 200mb to 750mb and it keeps going. Running Edge.