Archived Chat Demonstration

Have you ever thought it would be cool if the chat in live streams was archived with the video? I did, so in my spare I time I wrote a little demonstration of the concept (be patience, it may take a moment to load the 4,000 comments). I figured out how to save the chat and link it to a video several weeks ago, but I had to wait for another live stream of a Quick Look so I could demonstrate it with a YouTube video. The popcorn.js API would work with any video, but obviously I can't serve a Giant Bomb video in my own web space. If it bothers you that I'm hosting your comments, let me know and I'll remove them from the demonstration.

To store the live chat feed, I created a local node.js server which stores HTTP get request data to file. Then, I opened the console in Google Chrome, and inserted a callback which is invoked when the chat window updates. This callback removes the new message, parses out its elements, and sends it to the node.js server to be stored. Surprising what you can do to a live website, huh?

Next, I needed to tie the chat to the archived video. This is done using the popcorn.js library. I copied the style of the live stream page so I could duplicate the presentation. Popcorn allows me to add a callback every time the video advances, so every second I repopulate the chat window with comments that would have appeared at that moment during the live stream. I synced the chat to the video by manually determining the time-stamp of the first moment of the video.

Here's a chart of the chat frequency over time for all the comments I recorded:

This is something that could be aligned with the width of the video, giving you an impression where amusing stuff is happening. In this case, the first major peak coincides with Brad driving off of a broken bridge, and the second peak is when Brad escaped certain death with a divekick.

Hope you guys find this interesting! It was fun to do.

Note: I didn't store membership or moderator information, so messages don't show up highlighted or with their appropriate metals. Since this is just a proof-of-concept prototype, I didn't see the need to take the extra effort. No disrespect intended!

Edit: Here are the files for extracting the chat data in your browser:

node.js server for storing the input:

http://www.rabbitfury.com/gbchat/server.js

code inserted into the console during a live show for capture:

http://www.rabbitfury.com/gbchat/gbchat_insertion.txt

Note: the insertion code produces an error after each message is sent to the node.js server because it can't accept the response to the HTTP get request. Its annoying but it doesn't adversely effect operation.

@lokno: Wow, this is really impressive stuff; great work! I can definitely see this becoming a thing, due to the growing popularity of keeping a chat intact with a stream after-the-fact. Great explanation for everything, too, I feel like I'd kind of be able to follow enough to try and replicate it myself as a personal project.

Thanks for sharing!

Wow, that works amazingly well.

Nice work, duder.

That is really cool. Some of the best parts of live shows are the reactions of the chat. Being able to watch an archived feed could be pretty fun.

Wow, this is impressive. I'm way too stupid to understand how you did this but great job!

@equitasinvictus: So there are already sites that archive and play back the chat from live streams? I've never seen this implemented before. Cool to hear its already happening!

@demoskinos: My thoughts exactly! Its a shame not to have the community when you watch the videos later.

Quite impressive duder, and totally awesome.

Actually, not quite! What I meant by what I was saying earlier is that more people are recording chats along with livestreams (VGCW for example) in a way such that videos on sites such as YouTube would include the original chat feed in the actual video of a stream.

Your idea is a much better way than that, though, and could probably catch on quick with those who like to read chat feeds with their stream playbacks.

Dang, nice work! Thanks for sharing how you implemented it.

Rarely has such effort been applied for so ridiculous a purpose. Rarer still has it brought me this much joy. Thanks for the work, duder!

Wow this is great. It doesn't seem to work on my Firefox browser though (works with IE). I have noscript installed but I've whitelisted everything on that site.

Also, it seems like it doesn't autoplay on my Firefox browser (does when I use IE).

Anyways, great work! How long did this take to write?

Edit: The mods should put this up on the front page.

This is pretty cool stuff, nice work.

@equitasinvictus: Yep, as someone who has recorded a show for VGCW, I'm just capturing screen regions in Xsplit. Which falls apart if my Internet lags at all.

Wow. This is really cool. I've honestly had that same thought, but I'm nowhere near skilled at... anything to do what you did. Amazing job, buddy!

Wow that's super cool. Could be cool for the e3 streams.

That is damn impressive. Does this only work with youtube though? If so, i guess this would only work for live quick looks. :(

I guess now my stupid comments are saved for posterity :o It's cool looking over the chat and noticing comments I didn't see before though.

Also, I think the chat is a little late. A lot of comments seem to come a tad later than I remember; for example, i am almost certain that people were making human centipede jokes while the guys were on the ladder.

Holy cow - when the chat paused after I paused the video my mind exploded.

Fantastic work!

I hope they implement something like this for the site. Chat can be as interesting as the show sometimes.

This is goddamn amazing ! The fact that when you pause the chat pauses or it follows you if you jump around in the video is fantastic.

thats pretty impressive, it would be awesome for tnt's and press events like E3 and what not, dont see a point in using it for QL other then novel but all the other things it would be perfect.
most impressive is that you call this a proof-of-concept prototype.

Ugh, that's such a damn great idea, and a fantastic execution. I really love the graph of the chat over time, too, that's a really fucking smart way to measure how 'interesting' parts of the video are.

This is seriously amazing, I actually asked snide if he could do this when he was re designing the site. I always miss the live streams being in Australia and the rare occasion i do get to check it out, i love it. If this could be done for all future live stream Especially the e3 ones, you'll be my favorite person

Perhaps you should have read the original post before you wrote anything. The purpose isn't to archive chat. It's to archive chat with the video so it's part of the experience when you go back to watch archived live-stream footage.

@lokno This is a beautiful implementation. And I'm really into that Nielsen Graph to show when people were interested or something hilarious was happening.

@thunderslash: I didn't test it in Firefox originally, but it autoplays and works correctly for me in Firefox 21, Windows 7 x64. Maybe its an issue with noscript and embedded youtube videos? Glad you think its front page worthy! :D

@beachthunder: Timing is hard to judge, since reactions are naturally delayed from what's happening on screen. There's parts in the video where the responses almost seem too fast. However, its possible that the time-stamps for the messages are a little off because of the round-about way I processed them.

@forteexe21: It would work for any video, and actually my initial test was with an UnProFri archive a couple weeks back. However, I used the YouTube archive so I wouldn't have to host a GB video on my server. This way I'm not stepping on any toes.

Thanks for the feedback everybody! Glad you're all getting a kick out of it. I showed it to Brad on twitter and he really digs it too.

Well done, duder! Out of curiosity, do you have any plans to make this more robust or do you happen to have your source available in a GitHub repo or something. I would love to see this be expanded upon, even if that means doing the work myself.

@chaser324: I don't currently have plans to iterate on the concept. I basically just wanted to show how it could be implemented. A major improvement would be to stream in chat data instead of loading an entire MB upfront. Here, I'll just upload the local server and insertion code:

node.js server for storing the input:

http://www.rabbitfury.com/gbchat/server.js

code inserted into the console during a live show for capture:

http://www.rabbitfury.com/gbchat/gbchat_insertion.txt

Obviously you can just look at the demo source itself, its all on the page linked in the OP. The chat data is stored as JSON in log0605.json

A missing element how I went from the server output to the JSON. That was just done with find/replace regex in Sublime Text. I also strip out the html from the comments, and add it back in later (url links, emote images).

@lokno: Thanks. I'm definitely going to try to build on this a bit.

@chaser324: There's an error in the console every time the insertion code sends a HTTP get request to the local server (on every new message), but it doesn't effect operation because the error is related to the response from the request, which the code doesn't use.

@lokno: I actually have some ideas about alternate methods of capturing and parsing that messages, but I'm going to have to experiment a bit to see if they work. I think you also would ideally want to get that stuff tied into your Node backend, and try to eliminate all of the manual work.