XNA for Silverlight developers: Part 8 - Music and sound effects

This article is part 8 of the series "XNA for Silverlight developers".

This article is compatible with Windows Phone "Mango".
Latest update: Aug 1, 2011.

What would a game be without sound effects and music? Naturally Windows Phone 7 offers enough possibilities to add a rich acoustic experience to your apps and games. However, due to the nature of the device, there are some things to keep in mind. Even with the lack of real multi-tasking, you don't have the whole platform at your disposal exclusively. The user may already listen to music from the media library when they start your game, for example. In addition, a mobile device may also be used in an environment where noise is not welcome. These are things you have to consider when you add audio to your game. Some of these aspects are even regulated in the certification requirements for Windows Phone 7 applications and are a potential source for certification failure if implemented incorrectly. In this article, we'll take a look at how audio can be integrated in your games and what the implications are for playing sound and music in Silverlight and XNA. As always, the source code is available at the end of the article.

Silverlight

In Silverlight, there is a dedicated element for audio and video playback: the MediaElement. This component easily enables playback of audio in your Silverlight application. Simply add an instance of it to your page, point it to the source file (needs to be embedded as "Resource") or an Uri, and set some basic values:

 1: <MediaElement x:Name="Media"

 2:               Source="Song.mp3"

 3:               Volume="0.9"                          

 4:               AutoPlay="True" />

Of course you can control the playback through various methods like Play, Pause and Stop, and properties like Position allow to implement seeking and similar features. In addition, the media element has events to notify you about e.g. when the media has finished playing. All in all it's a quick way to add media to your application or Silverlight game.

However, one huge limitation of the media element is that only one is supported at a time. It is not possible to play multiple media files simultaneously, and even having multiple elements that don't play at the same time on the same page is not supported. To overcome this limitation, you can use the XNA libraries for audio playback. XNA's audio features are one of those parts which are allowed and safe to use from Silverlight. In addition this approach opens the door to a whole set of features and parameters that aren't accessible through the media element and allow to create even more interesting sound effects. If you are interested in doing that, I strongly recommend downloading the simple "Silverlight Sound Sample" from the Windows Phone 7 code samples page on MSDN. It demonstrates how to use the XNA features that we will learn about below in a Silverlight project. All of the following applies to both XNA and Silverlight if you're doing that.

Differentiation and certification requirements

Regarding audio on Windows Phone 7, especially the certification requirements differentiate between music and sound effects. Sound effects are played through the XNA class named alike (see below). Those sounds are mixed with music that is played by the media player automatically and it is allowed to play them at any time, even simultaneously to already running music. For background music on the other hand you are supposed to use a different method (the requirements explicitly state that the sound effects class should not be used for that), and your app needs to be aware of already running music. In particular (section 6.5.1, Initial Launch Functionality):

When the user is already playing music on the phone when the application is launched, the application must not pause, resume, or stop the active music [...]

And:

If the application plays its own background music or adjusts background music volume, it must ask the user for consent to stop playing/adjust the background music (e.g. message dialog or settings menu).

Additionally, the requirements state that if your application plays background music, you must provide a user interface that allows configuration of the settings for that.

To many developers the wording of this section is confusing, and working with these requirements can be quite tricky and cause certification failure easily. So what does all this mean?

Sample startup procedure

When your game starts, you can use the media player's static property "GameHasControl". If it returns true it means that there's no currently playing music and you can start your own background music right away. If it returns false, you are not allowed to manipulate the currently playing music (like stop or pause it) without asking the user first. So what you need to do is present the user a message where they confirm that it is ok to stop the currently running music. Only if that request is confirmed by the user you are allowed to replace it with your own. You can use the message box functionality of the Guide class for the dialogs.

There is a full example with additional details available in the education section of the App Hub, named Music Management for Windows Phone 7. If you think about adding background music to your game, I recommend downloading and working with that sample. The included helper class is a good starting point and can easily be extended to fit your needs.

Background music

Now that we have clarified the dos and don'ts of background music, let's see how to implement it. The central element for this is XNA's Song class. Like with any other media, the standard procedure for music and sound effects in XNA is to import it through your content project. When you add an MP3 file (or any other of the supported audio formats) you can choose "Song – XNA Framework" as content processor to turn it into a song:

Playing the song then is a matter of only a few lines of code:

 1: // load the song

 2: _backgroundMusic = Content.Load<Song>("Song");

 3:  

 4: // try to start the music if we are allowed to do that

 5: if (MediaPlayer.GameHasControl)

 6: {

 7:     try

 8:     {

 9:         // start playing the music

 10:         MediaPlayer.Play(_backgroundMusic);

 11:  

 12:         // set some parameters

 13:         MediaPlayer.Volume = 0.9f;

 14:         MediaPlayer.IsRepeating = true;

 15:  

 16:         _message = "You should hear the background music now.";

 17:     }

 18:     catch (InvalidOperationException)

 19:     {

 20:         // you cannot play songs if the device is connected to Zune 

 21:         _message = "Playing the background music failed.";

 22:     }

 23: }

 24: else

 25: {

 26:     _message = "Already playing music detected.";

 27: }

Important requirement

The above sample code already incorporates some changes that will be introduced by the first minor update to Windows Phone 7 ("NoDo") which is coming soon to existing devices and already used by some of the currently sold new phones. The problem: the media player on the phone does not have a separate volume control on a per-song basis or mute functionality. That means that if you used parameters like the above "Volume" property while music from the media library was playing, you were able to make changes that the user could not undo later through the media player controls anymore.

To overcome this problem, changing any of these parameters is ignored when the currently playing music source is located in the media library. Here is an example:

 1: MediaPlayer.Play(new MediaLibrary().Songs);

 2:  

 3: // setting this has no effect;

 4: // Volume is 1.0f after this line!

 5: MediaPlayer.Volume = 0.5f;

So it is important to remember and/or get used to the following recommended pattern of actions:

• First start playing your own music.
• Then adjust the necessary parameters like volume etc.

If you follow this recommendation, playing your music will work as expected in both the RTM and updated version of Windows Phone 7. The topic is described in more detail in this recent blog post by Nick Gravelyn.

Sound effects

Sound effects, just like music, are imported into your game through the content project. This time however, select "Sound Effect – XNA Framework" as content processor:

Loading the sound effect also is nothing new to us:

 1: // load sound effect

 2: _soundEffect = Content.Load<SoundEffect>("ShotEffect");

When you look at the properties and methods the sound effect class offers for a second, you can see that it in fact has a "Play" method that can play the sound in a "fire and forget" fashion. This is somewhat similar to what we have seen for the Song type above. An overload of that method allows you to pass in values for various parameters, like volume, pitch and pan.

 1: // play sound 

 2: var pan = (float)((_rnd.NextDouble() - 0.5) * 2.0);

 3: var pitch = (float)((_rnd.NextDouble() - 0.5) * 2.0);

 4: var volume = (float)((_rnd.NextDouble() / 2.0) + 0.5);

 5:  

 6: _soundEffect.Play(volume, pitch, pan);

It's noteworthy that this method also allows you to play the same sound multiple times simultaneously.

Modifying sounds as they play

A more interesting approach to play sound effects however is to create a sound effect instance. The great thing is that by doing this you have a reference to the sound that currently plays and can change its properties on the fly. For example, let's create a sound that pans from the far left to the far right or vice versa as it goes. First of all we create the sound instance by using the respective method of the sound effect, set some initial values and start playing:

 1: // set the pan speed so a full pan (-1.0 to 1.0 = 2.0)

 2: // is achieved over the duration of the sound

 3: _panSpeed = 2.0f / (float)_soundEffect.Duration.TotalSeconds;

 4:  

 5: // randomly choose panning direction

 6: _panSpeed = _rnd.NextDouble() > 0.5 ? _panSpeed : -_panSpeed;

 7:  

 8: // create the actual effect and play it

 9: _panningSoundEffect = _soundEffect.CreateInstance();

 10: _panningSoundEffect.Pan = _panSpeed > 0.0f ? -1.0f : 1.0f;

 11: _panningSoundEffect.Play();

Then, in the "Update" method of the game, as long as the sound effect instance plays, we update its pan property:

 1: // update panning sound if present

 2: if (_panningSoundEffect != null)

 3: {

 4:     // remove sound if it has finished

 5:     if (_panningSoundEffect.State == SoundState.Stopped)

 6:     {

 7:         _panningSoundEffect.Dispose();

 8:         _panningSoundEffect = null;

 9:     }

 10:     else

 11:     {

 12:         // caluclate new pan value

 13:         var newPan = _panningSoundEffect.Pan +

 14:             (float)(gameTime.ElapsedGameTime.TotalSeconds * _panSpeed);

 15:  

 16:         // make sure the new value doesn't exceed

 17:         // the allowed bounds

 18:         newPan = Math.Max(-1.0f, Math.Min(newPan, 1.0f));

 19:  

 20:         // update pan value

 21:         _panningSoundEffect.Pan = newPan;

 22:     }

 23: }

An important detail to note here is the use of the "Dispose" method. When you're done using a sound effect instance, let it clean up its unmanaged resources that way and remove all references in your code so the garbage collector can do the rest.

3D Sounds

The above example tries to mimic positioning a sound in the coordinate space around the player by manipulating the pan value in real-time. The sound effect instance class actually offers a more sophisticated feature to accomplish this for more complex scenarios: 3D sounds. You can turn any sound effect into a 3D sound by calling the "Apply3D" method at least once before you call Play. To make use of the 3D features, you have to repeat this frequently, for example call the method in the Update method of your game.

 1: // create a new sound effect

 2: _3dSoundEffect = _soundEffect.CreateInstance();

 3:  

 4: // set looping and make it a 3D sound

 5: _3dSoundEffect.IsLooped = true;

 6: _3dSoundEffect.Apply3D(_audioListener, _audioEmitter);

 7: _3dSoundEffect.Play();

As you can see, the method takes an audio listener and an audio emitter as arguments. Both of these are entities that are located in 3D space and share the following properties of interest:

• Position: A Vector3 that describes the current position.
• Velocity: A Vector3 that describes the current movement (= speed + direction) of the entity.
• Up: A Vector3 that points upward in the entity's space.
• Forward: A Vector3 that is the direction the entity is facing at.

This allows to set up pretty complex scenarios, where entities are tilted and move sideways, with relative speed to each other. The sound mixing reacts accordingly; the sound is ought to come from the emitter and will be distorted to match what the listener would hear in its current position (using things like attenuation and the doppler effect).

A lot of games conceptionally work in a way that the player (the listener) stays positioned in the origin, with the forward and up vectors axis-aligned to the y- and z-axes, and never actually moves around. Instead, the emitter (and the rest of the world) is moved around the player. I have added a sample for this kind of a 3D positioned sound in the source code for this article. It moves an emitter around the player in an ellipse to demonstrate how the sound mixing applies not only panning, but also adjusts the volume as the emitter is moving away from and back towards the player.

 1: // calculate the new position

 2: var emitterPosition = new Vector3(

 3:     (float)Math.Cos(gameTime.TotalGameTime.TotalSeconds),

 4:     0,

 5:     (float)Math.Sin(gameTime.TotalGameTime.TotalSeconds) * 3);

 6:  

 7: // update the audio emitter position

 8: _audioEmitter.Position = emitterPosition;

 9:  

 10: // update the 3D sound

 11: _3dSoundEffect.Apply3D(_audioListener, _audioEmitter);

The question of course is whether that level of sophistication is needed on a mobile device (and in 2D games). However, it adds interesting new possibilities to the use of audio, something a lot of games tend to ignore a bit. And when the player wears headphones it can actually create quite a nice acoustic environment.

Summary

Adding sounds and music to your game is pretty straight forward, and as long as you follow a few rules certification for the market place should not become a problem. Audio is one of the areas where XNA has clear advantages regarding the feature set, and luckily Silverlight developers can make use of that easily too. Even on the limited mobile platform the sound libraries have pretty sophisticated support for advanced audio scenarios, including the use of 3D sounds. I hope I was able to demonstrate some of those possibilities in this article. Feel free to leave any feedback or comments.

Download source code

About the author

Peter Kuhn aka "Mister Goodcat" has been working in the IT industry for more than ten years. After being a project lead and technical director for several years, developing database and device controlling applications in the field of medical software and bioinformatics as well as for knowledge management solutions, he founded his own business in 2010. Starting in 2001 he jumped onto the .NET wagon very early, and has also used Silverlight for business application development since version 2. Today he uses Silverlight, .NET and ASP.NET as his preferred development platforms and offers training and consulting around these technologies and general software design and development process questions.

He created his first game at the age of 11 on the Commodore 64 of his father and has finished several hobby and also commercial game projects since then. His last commercial project was the development of Painkiller:Resurrection, a game published in late 2009, which he supervised and contributed to as technical director in his free time. You can find his tech blog here: http://www.pitorque.de/MisterGoodcat/