Live Thumbnails for Twitch Generated in the Cloud

In comparison to Figure 1, in the video in Figure 2 you can see that the stream progress has advanced significantly. For example, the match in the ProLeagueCSGO channel has already been started in my generated thumbnails, in comparison to still being at the caster desk in Figure 1. The channel thumbnails are also auto updating every 3 seconds. This allows to stay on the overview until an interesting thing happens on stream. You do not have to keep a stream running in the background to see what is going on.

Architecture Powering the Proof-Of-Concept

The core architecture is build in the Cloud on Amazon Web Services. The goal was to get a good scaling architecture, such that even large amounts of streams can be processed.

From a high-level perspective, this is how the architecture looks like.

The steps performed are:

  1. A GreaseMonkey userscript identifies stream names and their thumbnails that are currently in the viewport of the user’s browser. For each identified stream, it submits a request to Amazon API Gateway to get a more up to date thumbnail.
  2. API Gateway receives the request and passes it over to Amazon Lambda that answers with a JSON object containing the thumbnail in base64 encoded form.
  3. Amazon Lambda reads the request send by the user. It parses the desired channel name and quality that the user has requested. By utilizing the LiveStreamer API, it can extract the Twitch stream URL of the requested stream. Getting the stream URL takes quite some time, thus the stream URL is cached for a few hours inside Dynamo DB, to speed up subsequent requests. The stream URL then allows to perform parts of the HLS protocol. The playlist can be loaded and the latest stream segment containing the video data can be downloaded. FFMPEG is then utilized to extract a thumbnail from the stream segment.

It turns out that once Amazon Lambda is warmed up, this process is quite fast. It takes around 200 milliseconds to get an up-to-date stream thumbnail.

Caveats

Performance: At first, the PoC was struggling to generate thumbnails with a reasonable response time. I identified two main reasons for this. At first, the LiveStreamer API takes quite a while to load. Importing the livestreamer package alone took nearly half a second. Additionally, getting the stream URL from Twitch also took quite some time. Introducing a caching mechanism for the stream URL allowed me to import LiveStreamer only when really needed and also avoid the slow lookup of a stream URL from Twitch.

Passing the thumbnail back to the client: At the time of my experiments, API Gateway did not support responding with binary data, for example an image file. The challenge was to pass back an image in another format. The solution I chose was to base64 encode the image data and pass it as string in a JSON response. On the browser side, the image can then directly be embedded in side the src attribute of an image.

Conclusion

My PoC has successfully demonstrated that it is possible for a 3rd party service to generate more recent and auto refreshing thumbnails for Twitch channels. In comparison to the standard thumbnail delay of about 5 minutes, my PoC keeps up with the live stream and has no noticeable delay.

To bring this idea into a usable service requires some more hardening of the caching architecture to really support a high request rate. The biggest challenge might be to get approval of Twitch to run this service. From my understanding, the Twitch’s Terms of Service do forbid programmatic stream accesses and thus no generation of thumbnails seems possible.