This is the second post in our Proactive Audio series. In the first post we covered the basic concept of what Proactive Audio is. This post is the first of a three-part look into major Proactive Audio development techniques. First up:
MUSIC SYNCED ANIMATIONS
One of the toughest parts of synchronizing action and music in games is dealing with player agency - the player's ability to control the action to some degree. Without player agency, the problem is the same one faced by the film industry and we could simply leverage that industry's tools to perform the synchronization. The film industry uses a host of non-linear editing software to line up various clips of audio and video together in order to produce a final product. Software like this is integrated into most game engines in some form or another to assist in realtime cutscene development.
Setting cutscenes aside, how do we retain strong audio-visual synchronization while maintaining player agency? One obvious approach would involve authoring all animations in the game to line up exactly with musical timing in the game. This, however, imposes large restrictions on how and when animations may be started and blended together and would require exporting different versions of each animation to match each piece of music in the game. Further, handling music transitions or tempo shifts would be a nightmare.
At this point, some of you may be thinking that we could cut down on the animation asset duplication issue by changing the playback speed to match each musical composition's tempo. And you'd be right! But we can take it further than that with Proactive Audio.
TICK WITH TEMPO
At the core of every game engine is Time: it drives physics, animations, audio, gameplay logic, and more. The vast majority of game engines are designed under the assumption that time is measured in seconds. However, it doesn't have to be. The standard unit of time for musical Tempo is the beat, typically expressed in Beats Per Minute (BPM). Replacing time-in-seconds with time-in-beats in the animation system Tick/Update/etc. means that animation speed automatically adjusts to match the content of the music. With properly authored run-cycle animations, your characters can always run on the beat - regardless of music tempo, all with a single animation.
Fortunately, creating "properly authored animations" is also incredibly easy: simply treat your animation timeline as though it were measured in beats, rather than seconds. When animating to seconds, the goal is to find something that feels natural when played back. When animating to beats, the goal is to make impactful moments in the animation occur on beat boundaries. A simple run-cycle, therefore, might be animated to loop at exactly the 2-second mark, with the middle footfall occurring at the 1-second mark. When such an animation is played back in a tempo-driven animation system, the footfall timings will always line up with main beats! (Note that some initial animation offsetting may be required for this effect.)
The details of how to make your animation system tempo-driven are, unfortunately, game engine-specific (and frequently even animation system-specific!). Luckily, the core math for transforming time-in-seconds into time-in-beats is trivial:
time-in-beats = time-in-seconds * ((BPM) / 60)
Using this simple function you can easily convert your frame's delta time from seconds to beats and hand it to your animation system for use. Or, you know, you could simply ask your Proactive Audio system for the latest delta-time-in-beats!
MUSIC-DRIVEN, MOTHER-APPROVED
This Proactive Audio approach to synchronizing music and action is fast, efficient, and, best-of-all, simple. By using music itself as the time source for animation updates, we can lean on established industry standard animation/game development workflows while gaining powerful cinematic synchronization of audio and visuals. And the best part? There's no need to sacrifice on player agency!