(Deprecated) Build a musical game using Oboe

1. Before you begin

This codelab is deprecated and will be removed soon.

In this codelab, you build a simple musical game using the Oboe library, a C++ library that uses the high-performance audio APIs in the Android NDK. The objective of the game is to copy the clapping pattern you hear by tapping the screen.

Prerequisites

• Basic knowledge of C++, including how to use header and implementation files.

What you'll do

• Play sounds using the Oboe library.
• Create low-latency audio streams.
• Mix sounds together.
• Trigger sounds precisely on a timeline.
• Synchronize audio with the on-screen UI.

What you'll need

• Android Studio 4.1 or higher
• Android NDK and build tools installed
• An Android device running Android Lollipop (API level 21) or higher for testing (Pixel devices are best for low-latency audio)

2. How do you play the game?

The game plays a funky four-beat backing track that continually loops. When the game starts, it also plays a clapping sound on the first three beats of the bar.

The user must try to repeat the three claps with the same timing by tapping the screen when the second bar begins.

Each time the user taps, the game plays a clap sound. If the tap happens at the right time, the screen flashes green. If the tap is too early or too late, the screen flashes orange or purple, respectively.

3. Get started

Clone project

Clone the Oboe repository on GitHub and switch to the game-codelab branch.

git clone https://github.com/google/oboe 
cd oboe
git checkout game-codelab

Open the project in Android Studio

Load Android Studio and open the codelab project:

1. File > Open...
2. Select the oboe/samples folder

Note: This project contains all the code samples for the Oboe library. For this codelab, you are only working with the RhythmGame sample.

Run the project

1. Choose the RhythmGame run configuration.

2. Press Control+R to build and run the template app. It should compile and run, but it doesn't do anything except turn the screen yellow. You add functionality to the game during this codelab.

Open the RhythmGame module

The files you work on for this codelab are stored in the RhythmGame module. Expand this module in the Project window, making sure that the Android view is selected.

Now expand the cpp/native-lib folder. During this codelab, you edit Game.h and Game.cpp.

Compare with the final version

During the codelab, it can be useful to refer to the final version of the code, which is stored in the master branch. Android Studio makes it easy to compare changes in files across branches.

1. Right click on a file in the Project view.
2. Go to git > Compare with Branch... > master

This opens a new window with the differences highlighted.

4. Architecture overview

Here's the game architecture:

UI

The left side of the diagram shows objects associated with the UI.

The OpenGL Surface calls tick each time the screen needs to be updated, typically 60 times per second. Game then instructs any UI-rendering objects to render pixels to the OpenGL surface and the screen is updated.

The UI for the game is very simple: the single method SetGLScreenColor updates the color of the screen. The following colours are used to show what's happening in the game:

• Yellow—game is loading.
• Red—game failed to load.
• Grey—game is running.
• Orange—user tapped too early.
• Green—user tapped on time.
• Purple—user tapped too late.

Tap events

Each time the user taps the screen, the tap method is called, passing the time the event occurred.

Audio

The right side of the diagram shows objects associated with audio. Oboe provides the AudioStream class and associated objects to allow Game to send audio data to the audio output (a speaker or headphones).

Each time the AudioStream needs more data it calls AudioStreamDataCallback::onAudioReady. This passes an array named audioData to Game, which must then fill the array with numFrames of audio frames.

5. Play a sound

Start by making a sound! You're going to load an MP3 file into memory and play it whenever the user taps the screen.

Build an AudioStream

An AudioStream allows you to communicate with an audio device, such as speakers or headphones. To create one, you use an AudioStreamBuilder. This allows you to specify the properties that you would like your stream to have once you open it.

Create a new private method called openStream in the Game class. First, add the method declaration to Game.h.

private:
   ...
   bool openStream();

Then add the implementation to Game.cpp.

bool Game::openStream() {
   AudioStreamBuilder builder;
   builder.setFormat(AudioFormat::Float);
   builder.setFormatConversionAllowed(true);
   builder.setPerformanceMode(PerformanceMode::LowLatency);
   builder.setSharingMode(SharingMode::Exclusive);
   builder.setSampleRate(48000);
   builder.setSampleRateConversionQuality(
      SampleRateConversionQuality::Medium);
   builder.setChannelCount(2);
}

There's quite a bit going on here, so break it down.

You created the stream builder and requested the following properties:

• setFormat requests the sample format to be float.
• setFormatConversionAllowed enables Oboe to always provide a stream with float samples, regardless of the underlying stream format. This is important for compatibility with pre-Lollipop devices which only support streams with 16-bit samples.
• setPerformanceMode requests a low-latency stream. You want to minimize the delay between the user tapping the screen and hearing the clap sound.
• setSharingMode requests exclusive access to the audio device.This reduces latency further on audio devices that support exclusive access.
• setSampleRate sets the stream's sample rate to 48000 samples per second. This matches the sample rate of your source MP3 files.
• setSampleRateConversionQuality sets the quality of the resampling algorithm that's used if the underlying audio device does not natively support a sample rate of 48000. In this case, a medium-quality algorithm is used. This provides a good tradeoff between resampling quality and computational load.
• setChannelCount sets the stream's channel count to 2, a stereo stream. Again, this matches the channel count of your MP3 files.

Open the stream

Now that the stream has been set up using the builder, you can go ahead and open it. To do this use the openStream method which takes a std::shared_ptr<AudioStream> as its parameter.

1. Declare a member variable of type std::shared_ptr<AudioStream> inside Game.h.

private:
    ...
    std::shared_ptr<AudioStream> mAudioStream;
}

2. Add the following to the end of openStream in Game.cpp.

bool Game::openStream() {

    [...]
    Result result = builder.openStream(mAudioStream);
    if (result != Result::OK){
        LOGE("Failed to open stream. Error: %s", convertToText(result));
        return false;
    }
    return true;
}

This code attempts to open the stream and returns false if there is an error.

Note: It's a good idea to check the result of stream operations and handle any errors gracefully. Almost all methods in Oboe return a Result object, which can be checked against Result::OK and can be converted to a human-readable string using convertToText.

Load the sound file

The project includes a file in the assets folder named CLAP.mp3, which contains MP3 audio data. You're going to decode that MP3 file and store it as audio data in memory.

Note: If you want to view or play this file on your computer, you can open it using Audacity.

1. Open Game.h, and declare a std::unique_ptr<Player> called mClap and a method called bool setupAudioSources.

private:
    // ...existing code... 
    std::unique_ptr<Player> mClap;
    bool setupAudioSources();

2. Open Game.cpp and add the following code:

bool Game::setupAudioSources() {

   // Set the properties of our audio source(s) to match those of our audio stream.
   AudioProperties targetProperties {
            .channelCount = 2,
            .sampleRate = 48000
   };
   
   // Create a data source and player for the clap sound.
   std::shared_ptr<AAssetDataSource> mClapSource {
           AAssetDataSource::newFromCompressedAsset(mAssetManager, "CLAP.mp3", targetProperties)
   };
   if (mClapSource == nullptr){
       LOGE("Could not load source data for clap sound");
       return false;
   }
   mClap = std::make_unique<Player>(mClapSource);
   return true;
}

This decodes CLAP.mp3 into PCM data with the same channel count and sample rate as our audio stream, then stores it in the Player object.

Set up the callback

So far, so good! You have methods for opening an audio stream and loading your MP3 file into memory. Now, you need to get the audio data from memory into the audio stream.

To do this, you use an AudioStreamDataCallback because this approach provides the best performance. Update your Game class to implement the AudioStreamDataCallback interface.

1. Open Game.h and locate the following line:

class Game {

2. Change it to the following:

class Game : public AudioStreamDataCallback {

3. Override the AudioStreamDataCallback::onAudioReady method:

public:
    // ...existing code... 
     
    // Inherited from oboe::AudioStreamDataCallback
    DataCallbackResult
    onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) override;

The audioData parameter for onAudioReady is an array into which you render the audio data using mClap->renderAudio.

Note: Whenever you receive a container array of type void *, you must remember to cast it to the current stream data format (in your case, float) and supply data only in that format. Failure to do so results in some awful noise!

4. Add the implementation of onAudioReady to Game.cpp.

// ...existing code... 

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
    mClap->renderAudio(static_cast<float *>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

The return value DataCallbackResult::Continue tells the stream that you intend to keep sending audio data so callbacks should continue. If you return DataCallbackResult::Stop, the callbacks would stop and no more audio would be played through the stream.

To complete the data callback setup, you must tell the audio-stream builder where to find the data callback object using setDataCallback in openStream. Do this before the stream is opened.

bool Game::openStream() {
    ...
    builder.setDataCallback(this);
    Result result = builder.openStream(mAudioStream);

Loading

Before the game can be played, there's a couple of things that must happen:

• The audio stream must be opened using openStream.
• Any MP3 files used by the game need to be decoded and loaded into memory using setupAudioSources.

These operations are blocking and, depending on the size of the MP3 files and the speed of the decoder, they might take several seconds to complete. You should avoid performing these operations on the main thread. Otherwise, you might get a dreaded ANR.

Another thing which must happen before the game can be played is starting the audio stream. It makes sense to do this after the other loading operations have completed.

Add the following code to the existing load method. You call this method on a separate thread.

void Game::load() {

   if (!openStream()) {
       mGameState = GameState::FailedToLoad;
       return;
   }

   if (!setupAudioSources()) {
       mGameState = GameState::FailedToLoad;
       return;
   }

   Result result = mAudioStream->requestStart();
   if (result != Result::OK){
       LOGE("Failed to start stream. Error: %s", convertToText(result));
       mGameState = GameState::FailedToLoad;
       return;
   }

   mGameState = GameState::Playing;
}

Here's what's going on. You're using a member variable mGameState to keep track of the game state. This is initially Loading, changing to FailedToLoad or Playing. You update the tick method shortly to check mGameState and update the screen background color accordingly.

You call openStream to open your audio stream, then setupAudioSources to load the MP3 file from memory.

Lastly, you start your audio stream by calling requestStart. This is a non-blocking method, which starts the audio callbacks as soon as possible.

Start asynchronously

All you need to do now is call your load method asynchronously. For this, you can use the C++ async function, which calls a function on a separate thread asynchronously. Update your game's start method:

void Game::start() {
   mLoadingResult = std::async(&Game::load, this);
}

This simply calls your load method asynchronously and stores the result in mLoadingResult.

Update the background color

This step is simple. Depending on the game state, you update the background color.

Update the tick method:

void Game::tick(){
   switch (mGameState){
       case GameState::Playing:
           SetGLScreenColor(kPlayingColor);
           break;
       case GameState::Loading:
           SetGLScreenColor(kLoadingColor);
           break;
       case GameState::FailedToLoad:
           SetGLScreenColor(kLoadingFailedColor);
           break;
   }
}

Handle the tap event

You're almost there, just one more thing to do. The tap method is called each time the user taps the screen. Start the clap sound by calling setPlaying.

1. Add the following to tap:

void Game::tap(int64_t eventTimeAsUptime) {
    if (mClap != nullptr){ 
        mClap->setPlaying(true);
    }
}

2. Build and run the app. You should hear a clap sound when you tap the screen.
3. Give yourself a round of applause!

Note: If you see a red screen, it means something went wrong during game loading. Check the logcat output for more details. If you see the message Input (48000) and output (44100) sample rates do not match, try using a newer version of the Android emulator.

6. Play multiple sounds

Playing a single clap sound is going to get boring pretty quickly. It would be nice to also play a backing track with a beat you can clap along to.

Until now, the game places only clap sounds into the audio stream.

Using a mixer

To play multiple sounds simultaneously, you must mix them together. Conveniently, a Mixer object has been provided, which does this as part of this codelab.

Create the backing track and mixer

1. Open Game.h and declare another std::unique_ptr<Player> for the backing track and a Mixer:

private:
    ..
    std::unique_ptr<Player> mBackingTrack; 
    Mixer mMixer;

2. In Game.cpp, add the following code after the clap sound has been loaded in setupAudioSources.

bool Game::setupAudioSources() {
   ...
   // Create a data source and player for your backing track.
   std::shared_ptr<AAssetDataSource> backingTrackSource {
           AAssetDataSource::newFromCompressedAsset(mAssetManager, "FUNKY_HOUSE.mp3", targetProperties)
   };
   if (backingTrackSource == nullptr){
       LOGE("Could not load source data for backing track");
       return false;
   }
   mBackingTrack = std::make_unique<Player>(backingTrackSource);
   mBackingTrack->setPlaying(true);
   mBackingTrack->setLooping(true);

   // Add both players to a mixer.
   mMixer.addTrack(mClap.get());
   mMixer.addTrack(mBackingTrack.get());
   mMixer.setChannelCount(mAudioStream->getChannelCount());
   return true;
}

This loads the contents of the FUNKY_HOUSE.mp3 asset (which contains MP3 data in the same format as the clap sound asset) into a Player object. Playback starts when the game starts and loops indefinitely.

Both the clap sound and backing track are added to the mixer, and the mixer's channel count is set to match that of your audio stream.

Update the audio callback

You now need to tell the audio callback to use the mixer rather than the clap sound for rendering.

1. Update onAudioReady to the following:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

    mMixer.renderAudio(static_cast<float*>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

2. Build and run the game. You should hear the backing track and the clap sound when you tap the screen. Feel free to jam for a minute!

7. Enqueuing clap sounds

Now things start to get interesting. You're going to start adding the gameplay mechanics. The game plays a series of claps at specific times. This is called the clap pattern.

For this simple game, the clap pattern is just three claps that start on the first beat of the first bar of the backing track. The user must repeat the same pattern starting on the first beat of the second bar.

When should the game play the claps?

The backing track has a tempo of 120 beats per minute or 1 beat every 0.5 seconds. So the game must play a clap sound at the following times in the backing track:

Beat	Time (milliseconds)
1	0
2	500
3	1,000

Synchronizing clap events with the backing track

Each time onAudioReady is called, audio frames from the backing track (through the mixer) are rendered into the audio stream—this is what the user actually hears. By counting the number of frames that have been written, you know the exact playback time and, therefore, when to play a clap.

With this in mind, here's how you can play the clap events at exactly the right time:

1. Convert the current audio frame into a song position in milliseconds.
2. Check whether a clap needs to be played at this song position. If it does, play it.

Note: Why not use a timer?

Using a timer based on the system clock to queue claps would give you suboptimal accuracy for a number of reasons:

• There's no guarantee that a timer triggers exactly on time. It may be preempted by another thread.
• There's no guarantee that the backing track playback position is the same as the system clock. It may have suffered an underrun or there could be clock drift between the audio clock and the system clock.

By counting the number of frames that are written inside onAudioReady, you know the exact playback position and can ensure perfect synchronization with the backing track.

Cross-thread communication

The game has three threads: an OpenGL thread, a UI thread (main thread), and a real-time audio thread.

Clap events are pushed onto the scheduling queue from the UI thread and popped off the queue from the audio thread.

The queue is accessed from multiple threads, so it must be thread-safe. It must also be lock-free so it does not block the audio thread. This requirement is true for any object shared with the audio thread. Why? Because blocking the audio thread can cause audio glitches and noone wants to hear that!

Add the code

The game already includes a LockFreeQueue class template, which is thread-safe when used with a single reader thread (in this case, the audio thread) and a single writer thread (the UI thread).

To declare a LockFreeQueue, you must supply two template parameters:

• Data type of each element: Use int64_t because it allows a maximum duration in milliseconds in excess of any audio track length you would conceivably create.
• Queue capacity (must be a power of 2): There are 3 clap events, so set the capacity to 4.

1. Open Game.h and add the following declarations:

private:
    // ...existing code...  
    Mixer mMixer;
    
    LockFreeQueue<int64_t, 4> mClapEvents;
    std::atomic<int64_t> mCurrentFrame { 0 };
    std::atomic<int64_t> mSongPositionMs { 0 };
    void scheduleSongEvents();

Note: mCurrentFrame and mSongPositionMs are std::atomic because they are accessed from the UI thread.

2. In Game.cpp, create a new method called scheduleSongEvents.
3. Inside this method, enqueue the clap events.

void Game::scheduleSongEvents() {

   // Schedule the claps.
   mClapEvents.push(0);
   mClapEvents.push(500);
   mClapEvents.push(1000);
}

4. Call scheduleSongEvents from your load method before the stream is started so that all events are enqueued before the backing track starts playing.

void Game::load() {
   ...
   scheduleSongEvents();
   Result result = mAudioStream->requestStart();
   ...
}

5. Update onAudioReady to the following:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

   auto *outputBuffer = static_cast<float *>(audioData);
   int64_t nextClapEventMs;

   for (int i = 0; i < numFrames; ++i) {

       mSongPositionMs = convertFramesToMillis(
               mCurrentFrame,
               mAudioStream->getSampleRate());

       if (mClapEvents.peek(nextClapEventMs) && mSongPositionMs >= nextClapEventMs){
           mClap->setPlaying(true);
           mClapEvents.pop(nextClapEventMs);
       }
            mMixer.renderAudio(outputBuffer+(oboeStream->getChannelCount()*i), 1);
       mCurrentFrame++;
   }

   return DataCallbackResult::Continue;
}

The for loop iterates for numFrames. On each iteration, it does the following:

• Converts the current frame into a time in milliseconds using convertFramesToMillis
• Uses that time to check whether a clap event is due. If it is, then the following occurs:
• The event is popped off the queue.
• The clap sound is set to playing.
• Renders a single audio frame from mMixer ( Pointer arithmetic is used to tell mMixer where in audioData to render the frame.)

6. Build and run the game. Three claps should be played exactly on the beat when the game starts. Tapping the screen still plays the clap sounds.

Feel free to experiment with different clap patterns by changing the frame values for the clap events. You can also add more clap events. Remember to increase the capacity of the mClapEvents queue.

8. Add scoring and visual feedback

The game plays a clap pattern and expects the user to imitate the pattern. Finally, complete the game by scoring the user's taps.

Did the user tap at the right time?

After the game claps three times in the first bar, the user should tap three times starting on the first beat of the second bar.

You shouldn't expect the user to tap at exactly the right time—that would be virtually impossible! Instead, allow some tolerance before and after the expected time. This defines a time range, which you call the tap window.

If the user taps during the tap window, make the screen flash green, too early—orange, and too late—purple.

Storing the tap windows is easy.

1. Store the song position at the center of each window in a queue, the same way you did for clap events. You can then pop each window off the queue when the user taps the screen.

The song positions at the center of the tap window are as follows:

Beat	Time (milliseconds)
5	2,000
6	2,500
7	3,000

2. Declare a new member variable to store these clap windows.

private: 
    LockFreeQueue<int64_t, kMaxQueueItems> mClapWindows;

3. Add the song positions for the clap windows.

void Game::scheduleSongEvents() {

    ...
    // Schedule the clap windows.
    mClapWindows.push(2000);
    mClapWindows.push(2500);
    mClapWindows.push(3000);
}

Compare tap events with the tap window

When the user taps the screen, you need to know whether the tap fell within the current tap window. The tap event is delivered as system uptime (milliseconds since boot), so you need to convert this to a position within the song.

Luckily, this is simple. Store the uptime at the current song position each time onAudioReady is called.

1. Declare a member variable in the header to store the song position:

private: 
    int64_t mLastUpdateTime { 0 };

2. Add the following code to the end of onAudioReady:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
   ...
   mLastUpdateTime = nowUptimeMillis();

   return DataCallbackResult::Continue;
}

3. Update the tap method to the following:

void Game::tap(int64_t eventTimeAsUptime) {

   if (mGameState != GameState::Playing){
       LOGW("Game not in playing state, ignoring tap event");
   } else {
       mClap->setPlaying(true);

       int64_t nextClapWindowTimeMs;
       if (mClapWindows.pop(nextClapWindowTimeMs)){

           // Convert the tap time to a song position.
           int64_t tapTimeInSongMs = mSongPositionMs + (eventTimeAsUptime - mLastUpdateTime);
           TapResult result = getTapResult(tapTimeInSongMs, nextClapWindowTimeMs);
           mUiEvents.push(result);
       }
   }
}

You use the provided getTapResult method to determine the result of the user's tap. You then push the result onto a queue of UI events. This is explained in the next section.

Update the screen

Once you know the accuracy of a user's tap (early, late, perfect), you need to update the screen to provide visual feedback.

To do this, you use another instance of LockFreeQueue class with TapResult objects to create a queue of UI events.

1. Declare a new member variable to store these.

private: 
    LockFreeQueue<TapResult, kMaxQueueItems> mUiEvents;

2. In the tick method, pop any pending UI events and update the screen color accordingly.
3. Update the code for the Playing state in tick.

case GameState::Playing:
   TapResult r;
   if (mUiEvents.pop(r)) {
       renderEvent(r);
   } else {
       SetGLScreenColor(kPlayingColor);
   }
   break;

4. Build and run the game.

You should hear three claps when the game starts. If you tap exactly on the beat three times in the second bar, you should see the screen flash green on each tap. If you're early, it flashes orange. If you're late, it flashes purple. Good luck!

9. Stop and restart the music

Handle pause and close events

If the user closes or pauses the app the audio must be stopped. Add the following code to the stop method in Game.cpp

void Game::stop(){
   if (mAudioStream){
       mAudioStream->stop();
       mAudioStream->close();
       mAudioStream.reset();
   }
}

This stops the audio stream, meaning that the data callback will no longer be called and the game will no longer produce any audio. close frees up any resources associated with the audio stream, and reset releases ownership so the stream can be safely deleted.

The Game::stop method will be called whenever the game's onPause method is called.

Connect headphones

When a user connects an external audio device, such as headphones, the audio is not automatically routed to that audio device. Instead, the existing audio stream is disconnected. This is intentional, allowing you to tailor your content for the properties of the new audio device.

For example, you might resample your audio sources to the native sample rate of the audio device to avoid CPU overhead introduced by resampling.

To listen for audio stream disconnect events you override AudioStreamErrorCallback::onErrorAfterClose. This will be called after an audio stream has been closed.

1. Open Game.h and locate the following line:

class Game : public AudioStreamDataCallback {

2. Change it to the following:

class Game : public AudioStreamDataCallback, AudioStreamErrorCallback {

3. Override the AudioStreamErrorCallback::onErrorAfterClose method:

public:
    // ...existing code... 
     
    // Inherited from oboe::AudioStreamErrorCallback.
    void onErrorAfterClose(AudioStream *oboeStream, Result error) override;

4. Add the implementation of onErrorAfterClose to Game.cpp.

// ...existing code... 

void Game::onErrorAfterClose(AudioStream *audioStream, Result error) {
   if (result == Result::ErrorDisconnected){
       mGameState = GameState::Loading;
       mAudioStream.reset();
       mMixer.removeAllTracks();
       mCurrentFrame = 0;
       mSongPositionMs = 0;
       mLastUpdateTime = 0;
       start();
   } else {
       LOGE("Stream error: %s", convertToText(result));
   }
}

This resets the game to its initial state and restarts it if the audio stream is disconnected.

Build and run the game. If you put it into the background the audio should stop. Bringing it back to the foreground should restart the game.

Now try connecting and disconnecting headphones. The game should restart each time.

In a real game you may want to save and restore the game state when the audio device changes.

10. Congratulations

Congratulations! You built a musical game with the Oboe library.

Learn more

• Oboe full guide
• Oboe samples
• Low latency audio - Because your ears are worth it (ADC ‘18)