bookmark_borderEncoding Video in Games

A while ago I had a problem in a project I was working on. I needed to record a video of the game and upload it to YouTube, which at first sounds simple but can is in fact quite complex once you factor in everything.

Decisions, Decisions…

One of the most important things to decide upon was how the encoder should function. My criteria was:

  • It should encode in realtime.
  • It needs to record audio from the gameplay.
  • The resulting video needs to be uploaded to YouTube.

Since I could not afford to block the main thread, so I decided upon using a processing thread to encode the frames. For the audio, I ended up writing my own audio mixer as I could not find any suitable audio libraries that could mix to a raw audio buffer. As for YouTube, I decided to use the patent-free Ogg Theora codec.

The specifics

To handle the actual capture, I made a VideoRecorder object which would periodically extract frames and audio samples, and dispatch them to the encoder thread which was managed by a VideoEncoder. This went through several re-designs until I ended up with something like this:

i.e. the VideoRecorder sends the frames to the VideoEncoder which in turn adds them to the Frame Queue which in turn is processed by the actual Encoder, finally inserting them into the encoded video stream (in this case using Ogg Theora).

I ended up using two queues for video and audio frames, as I needed control over when I could insert video/audio into the encoded stream. In fact, this turned out to be one of the major problems.

In Ogg Theora you need to make sure the video and audio pages you generate are inserted in the correct order in the output stream, otherwise you will get odd playback corruption where frames are dropped, since the decoder will get confused with the timestamps.

Beyond that, uploading a video to YouTube is remarkably simple. Though to save time, I ended up using the code from Claus Hofele’s excellent Share Your Experience: YouTube Integration In Games on Gamasutra.