The ABC of streaming in Silverlight

Advanced media integration is one of Silverlight’s greatest strengths. In this tutorial we delve into this topic by carrying out some simple exercises. We will learn how to play a movie, how to interact with webcam and microphone and how to create a live streaming solution using Expression Encoder 4 and VLC and capture it in a Silverlight application.

Play a movie

In Silverlight there is a native control called MediaElement which allows you to do a lot of things. The first thing obviously is the possibility to play a video: using its Source property you can define the location of the video you want to see and you can use either a relative or absolute URL. Suppose you have a collection of videos in a subdirectory called “Videos” of your web application and you want to play one of them. The xaml code snippet below is all you will need:

<Grid x:Name="LayoutRoot" >

<MediaElement  x:Name="mediaEl" Source="videos/video1.wmv" />

</Grid>

If you launch the application the movie will start automatically. This happens because the Autostart property of the MediaElement is true by default. To control the video, the MediaElement exposes the canonical Play(), Pause() and Stop() methods. At this point adding a commandbar to control the video is fairly easy ; just add a StackPanel in the xaml as below:

<StackPanel Name="CommandBar" Orientation="Horizontal" Background="Beige" Height="39" Width="783" Margin="5" HorizontalAlignment="Center" Canvas.Left="3">

<Button Name="PlayMovie" Background="AntiqueWhite" Content="Play" FontWeight="Bold" Click="PlayMovie_Click" Margin="50,5,0,5" Width="100"/>

<Button Name="PauseMovie" Background="AntiqueWhite" Content="Pause" FontWeight="Bold" Click="PauseMovie_Click" Margin="50,5,0,5" Width="100"/>

<Button Name="VideoStop" Background="AntiqueWhite" Content="Stop" FontWeight="Bold" Click="VideoStop_Click" Margin="50,5,0,5" Width="100" />

</StackPanel>

And, in the code behind, the following event handlers:

private void PlayMovie_Click(object sender, RoutedEventArgs e)

        {

            if(mediaEl.CurrentState != MediaElementState.Opening ||

                mediaEl.CurrentState == MediaElementState.Stopped ||

                mediaEl.CurrentState == MediaElementState.Paused)

                mediaEl.Play();

        }

        private void PauseMovie_Click(object sender, RoutedEventArgs e)

        {

            if (mediaEl.CurrentState == MediaElementState.Playing)

                mediaEl.Pause();

        }

        private void VideoStop_Click(object sender, RoutedEventArgs e)

        {

            if(mediaEl.CurrentState == MediaElementState.Playing ||

                mediaEl.CurrentState == MediaElementState.Paused)

                mediaEl.Stop();

        }

Let’s focus on the video formats supported. The list below recaps all the formats recognized by Silverlight 4:

Drop & Know…

How do we know if a video is supported? Well, instead of digging into the properties of the file and compare them with the list above, we can be way more pragmatic, i.e. by simply dragging the video from the file system to the Silverlight app and dropping it over the MediaElement. If the MediaElment does not recognize the video, it fires an exception that we can catch the internal error and redirect to the user as a comprehensible message. Let’s see how we can achieve that.

After seeing the list of the properties of the MediaElement you will find that it exposes an AllowDrop property since it derives from UIElement . Yet hooking a drop event here won’t work because the Mediaelement is not a container. No panic! You can embed the MediaElement in a Canvas container and handle the drop event of this one. In the xaml below the Canvas object incorporates the StackPanel above mentioned and the MediaElement:

<Canvas Drop="VideoCanvas_Drop" x:Name="VideoCanvas" Width="800" Height="600" AllowDrop="True" Background="RosyBrown" HorizontalAlignment="Center">

    <StackPanel Name="CommandBar" Orientation="Horizontal" Background="Beige" Height="39" Width="783" Margin="5" HorizontalAlignment="Center" Canvas.Left="3">

        <Button Name="PlayMovie" Background="AntiqueWhite" Content="Play" FontWeight="Bold" Click="PlayMovie_Click"

            Margin="50,5,0,5" Width="100"/>

        <Button Name="PauseMovie" Background="AntiqueWhite" Content="Pause" FontWeight="Bold" Click="PauseMovie_Click"

                Margin="50,5,0,5" Width="100"/>

        <Button Name="VideoStop" Background="AntiqueWhite" Content="Stop" FontWeight="Bold" Click="VideoStop_Click"

                Margin="50,5,0,5" Width="100" />

    </StackPanel>

    <MediaElement x:Name="mediaEl" Canvas.Top="50"  Width="800" Height="450" AutoPlay="False" />

</Canvas>

Let’s take a look at the VideoCanvas_Drop event handler:

void VideoCanvas_Drop(object sender, DragEventArgs e)

{

    DragEventArgs dr = e as DragEventArgs;

    if (dr.Data == null)

        return;

    IDataObject dataObject = dr.Data as IDataObject;

    FileInfo[] files = dataObject.GetData(DataFormats.FileDrop) as FileInfo[];

    if(files.Count() > 0)

        mediaEl.SetSource(files[0].OpenRead());

}

The trick here is to use the SetSource property of the MediaElement instead of Source. Since it accepts a Stream as an argument, we can pass it the FileStream related to the dropped file. If the FileStream does not contain a natively supported media source then the MediaElement fires a MediaFailed event that we can catch as below:

public MainPage()

{

    InitializeComponent();

    mediaEl.MediaFailed += new EventHandler<ExceptionRoutedEventArgs>(mediaEl_MediaFailed);

    …

}

void mediaEl_MediaFailed(object sender, ExceptionRoutedEventArgs e)

{

    WarningTxtBlock.Text = "Cannot open this media:\r\n" + e.ErrorException.Message;

}

Using a webcam

Starting From Silverlight 4 it is possible to access to webcam and microphone of the machine. Fortunately the steps required to get a working application are nicely easy. We need substantially two objects: a VideoCaptureDevice and a CaptureSource.

private VideoCaptureDevice myWebCam;
private CaptureSource myCaptureSource;

The first represents the physical webcam and the second one offers some methods to manage the webcam. In this case we won’t use a MediaElement but a VideoBrush object to paint a Rectangle with the video stream coming from the webcam. Let’s create a new Silverlight project and add the following xaml code to the MainPage.xaml:

<Grid x:Name="LayoutRoot" Background="Black" Width="1024" Height="768" >

    <Grid.ColumnDefinitions>

        <ColumnDefinition />

        <ColumnDefinition />

    </Grid.ColumnDefinitions>

    <Grid.RowDefinitions>

        <RowDefinition Height="*"/>

        <RowDefinition Height="70"/>

    </Grid.RowDefinitions>

    <StackPanel Orientation="Horizontal" 

              HorizontalAlignment="Center" 

              Grid.ColumnSpan="2" 

              Grid.Row="1">

        <Button Width="70" Height="30" Margin="8"

             x:Name="StartButton" Click="StartButton_Click"

             Content="Start" />

        <Button Width="70" Height="30" Margin="8"

             x:Name="StopButton" Click="StopButton_Click" 

             Content="Stop" />

        <Button Width="70" Height="30" Margin="8"

             x:Name="SnapshotButton" Click="SnapshotButton_Click" 

             Content="Snapshot" />

    </StackPanel>

    <Border Background="DarkRed" BorderBrush="Beige" BorderThickness="5" Height="380" Margin="10,25,10,10"> 

        <Rectangle x:Name="webcamRectangle" />

    </Border>

    <Border Background="Gray" BorderBrush="Beige" Grid.Row="0" Grid.Column="1" BorderThickness="5" Height="380" Margin="10,25,10,10">

        <Rectangle x:Name="snapshotRectangle" />

    </Border>

</Grid>

As you can see there are two rectangles; the first will be painted with the VideoBrush and the second will be used to show a snapshot from the video. Now in the .cs file we can initialize the two objects mentioned above (myWebCam and myCaptureSource) and prepare everything for the video capture:

private void InitializeWebCam()

{

    myCaptureSource = new CaptureSource();

    //1° step: get the default capture device

    myWebCam = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

    // 2° step: indicate the video capture device to be used by the CaptureSource 

    myCaptureSource.VideoCaptureDevice = myWebCam;

    // 3° step: initialize a VideoBrush with the CaptureSource just initialized 

    myVideoBrush = new VideoBrush();

    myVideoBrush.SetSource(myCaptureSource);

    // 4° step: fill the Rectangle with the stream provided by the VideoBrush

    webcamRectangle.Fill = myVideoBrush;

As you see, getting the VideoCaptureDevice that represents the webcam available is just a matter of calling the GetDefaultVideoCaptureDevice() of the static CaptureDeviceConfiguration class. Then we have to indicate the myCaptureSource object which is the video capture device from which we want to actually capture. The following step is to initialize a VideoBrush object and set its source to the myCaptureSource. In this way the VideoBrush is ready to paint a target with the stream coming from the capture device. This is what has been done in the fourth step. We then have to handle the click events of the play and stop buttons to control the video:

private void StartButton_Click(object sender, RoutedEventArgs e)

{

    if (CaptureDeviceConfiguration.AllowedDeviceAccess ||

        CaptureDeviceConfiguration.RequestDeviceAccess())

        myCaptureSource.Start();

}

private void StopButton_Click(object sender, RoutedEventArgs e)

{

    if (CaptureDeviceConfiguration.AllowedDeviceAccess ||

        CaptureDeviceConfiguration.RequestDeviceAccess())

        myCaptureSource.Stop();

}

As you may notice before starting or stopping the capture we need to make a formal request in order to use the webcam and/or the microphone. The first time involves opening a dialog box asking for an explicit consent for access to the webcam and microphone as in the image below:

And what about a snapshot taken from the video? Nothing complicated! just modify the previous InitializeWebcam() method by adding this code snippet at the end:

mySnapshotBrush = new ImageBrush();

snapshotRectangle.Fill = mySnapshotBrush;

myCaptureSource.CaptureImageCompleted +=new EventHandler<CaptureImageCompletedEventArgs>(myCaptureSource_CaptureImageCompleted);

In the code above the snapshotRectangle is the Rectangle where we want to place the snapshot and its Fill property is set to an ImageBrush. So when the ImageBrush contains an image it is painted on the rectangle. How do we acquire an image from the video? You can use the CaptureImageAsync() method of the CaptureSource object. Since it is an asynchronous method you need to add an event handler to the CaptureImageCompleted event of the CaptureSource as shown below:

void myCaptureSource_CaptureImageCompleted(object sender, CaptureImageCompletedEventArgs e)

{

    mySnapshotBrush.ImageSource = e.Result;

}

You are almost done;

the only thing to add is calling the async method when the snapshot button is clicked:

private void SnapshotButton_Click(object sender, RoutedEventArgs e)

{

    if (myCaptureSource.VideoCaptureDevice != null &&

        myCaptureSource.State == CaptureState.Started)

    {

        myCaptureSource.CaptureImageAsync();

    }

}

Capturing a live stream

To capture a live stream we will use the MediaElement object again. But first we need a media streaming source. Fortunately there is more than one way to create your own little home streaming service to experiment a little. In the next paragraph we will learn how to broadcast the webcam on our pc live over the network using two different programs, Microsoft Expression Encoder 4 and VLC media player.

Live broadcasting with Expression Encoder

Microsoft Expression Encoder 4 is freely downloadable from here . Unlike the professional version, i.e. Encoder PRO 4, it does not support live Smooth Streaming and does not have support for H.264 but it is more than enough for our purposes. When you start the program you will find three options as in the image below:

Then Expression Encoder offers you a rich UI to create your own streaming project. The first step to do is to add a Live Source. After doing so a yellow framed box appears in the “Live Sources” area on the left:

Focusing on the “Live Source 1” just created you can attach a video device to it by choosing “Video Device” from the drop down list. If you have at least one webcam on board it should be listed in the drop down as below:

Similarly you can add an audio device choosing from the drop down list below. By the way, there is no simple way here to set an IP webcam as a video source since Encoder 4 has no support for RTSP streams. In that case you need a directshow source filter.
Once you have your working live source you have to activate it by clicking on the “Cue” button in the box:

Now let’s focus on the “Presets” section on the right. Starting from the top there is a System tab which collects several predefined encoding profiles. In my experience the best compromise between video quality and latency (I will go back to this later on) is the “Standard Encoding / VC-1 High Speed Broadband” preset.

You can try to change the video parameters, for instance reducing the Buffer Window but probably you will lose quality in the video.

The final step is set your output type. In order to be able to capture the live stream from a Silverlight application we will choose the broadcast - streaming option on port 8080 as below:

Now you are ready to press the start button in the centre:

Live broadcasting with VLC

VLC is known as an efficient and free multimedia player but perhaps not everybody knows that it can be used also as a streaming server. The nice thing is that it is able to stream all that it can read, for instance it can perform a live broadcasting stream even from an IP camera. But let’s stay on track: here as in the Expression Encoder tutorial above, we want to capture the video from the built-in webcam of our PC and stream it over the network. So let’s download the software from here and read the following steps.

The first thing to do is to open the “Media” menu and choose “Open Capture Device…”:

In the dialog box make sure to select your built-in webcam in the “Video device name” section as in the image below:

As for the audio device, you can do the same. Then click on the pick button at the right of the play button to get the other button options as below:

A new stream configuration wizard will appear. On the first page just click on the “next” button to jump directly to the “Destinations” configuration:

In the “Destinations” page there are two things to do: one is configuring the streaming method and the other is choosing the appropriate transcoding. Let’s start with the first one. Open the “New destination” dropdown list and choose the MS-WMSP (MMSH) option as below:

Don’t forget to set the “Display locally” check box in order to be able to capture the live stream from the same PC. Then click to “Add” button on the right of the drop down list. You should see something like that:

The streaming method configuration is completed. Lets’ focus on the transcoding options below now. Expand the Profile dropdown list and choose the “video – VMW + WMA (ASF) option. Then click on the edit button:

In the Edit Trascoding dialog box, click on the “Video codec tab and select the WMV2 codec as below (it could be already automatically set):

Click on “Save” to return to the “Destinations” page. Click on “Next” to go to the final page where there is a useful output string which allows you to launch vlc.exe with the same configuration by adding it to the command line. Click on the “Stream” button and you are done.

The output string is (in case you want to skip all the boring steps above and launch VLC from cmd):

Vlc.exe :sout=#transcode{vcodec=WMV2,vb=800,scale=1,acodec=wma2,ab=64,channels=2,samplerate=22050} :no-sout-rtp-sap :no-sout-standard-sap :ttl=1 :sout-keep

Using the live broadcasting with Silverlight

In the two previous paragraphs we have seen how to create a live broadcasting using two different softwares and you may expect that in Silverlight you have to configure the source of the stream differently in the two cases. Fortunately it is not true: if you use the configurations shown above you will be able to use one or the other option to your liking. The only requirement you need is to set the Source property of the MediaElement as below using the mms moniker:

<MediaElement x:Name="MediaElement" Width="320" Height="240" Source="mms://127.0.0.1:8080"/>

Let’s start a new Silverlight project to create a very basic live TV. Include this xaml in the MainPage.xaml:

<Grid x:Name="LayoutRoot">

  <StackPanel Orientation="Vertical">

      <MediaElement x:Name="MediaElement" Width="320" Height="240" Source="mms://127.0.0.1:8080"/>

      <StackPanel Orientation="Horizontal" HorizontalAlignment="Center">

          <TextBlock x:Name="Status" Text="status" Margin="0,5"/>

          <TextBlock x:Name="Buffer" Text="buffer" Margin="10,5"/>

      </StackPanel>

      <StackPanel Background="Beige" Orientation="Horizontal" HorizontalAlignment="Center">

          <Button x:Name="Play" Content="Play" Click="Play_Click"/>

          <Button x:Name="Pause" Content="Pause" Click="Pause_Click"/>

          <Button x:Name="Stop" Content="Stop" Click="Stop_Click"/>

      </StackPanel>

  </StackPanel>

Grid>

And in the code behind:

public MainPage()

{

    InitializeComponent();

    MediaElement.BufferingTime = new TimeSpan(0,0,0);

    MediaElement.CurrentStateChanged += (sender, e) => {

        Status.Text = MediaElement.CurrentState.ToString(); 

        Buffer.Visibility = MediaElement.CurrentState == MediaElementState.Buffering ? Visibility.Visible : Visibility.Collapsed;

    };

    this.MediaElement.BufferingProgressChanged += (sender, e) => {

        this.Buffer.Text = string.Format("{0:0.0} %", this.MediaElement.BufferingProgress * 100);

    }; 

}

private void Play_Click(object sender, RoutedEventArgs e) 

{

    this.MediaElement.Play();

}

private void Pause_Click(object sender, RoutedEventArgs e) 

{

    this.MediaElement.Pause();

}

private void Stop_Click(object sender, RoutedEventArgs e) 

{

    this.MediaElement.Stop();

} 

As you see, there is a rude panel which contains the basic functionalities (Play, Pause, Stop) to interact with the video. It is interesting that I set to zero the BufferingTime of the MediaElement in order to reduce as much as possible the delay between the real events and what we see through the MediaElement. In fact, in my attempts I measured a latency of around 7 seconds using EE4 and 1,5 seconds using VLC on a private LAN. This latter value is not too bad to me.

Conclusion

Streaming media with Silverlight is easy and funny. The MediaElement object is like a Swiss army knife which allows you to create a complete media player capable of playing a movie, capturing the output of a webcam and even reproducing a live stream. Microsoft Expression encoder 4 and VLC are two great free programs which allow you to create multimedia platforms to play with Silverlight. For more advanced scenarios it could be useful to take a look at the following topics:

• MediaStreamSource: a part of the Silverlight runtime which allows direct access to APIs to manipulate audio and video streams. (An example application here)
• IIS Smooth Streaming: an advanced platform which enables you adaptive streaming of media to Silverlight over HTTP.
• Microsoft Media Platform-Player Framework v.2.5: an open source platform which allows developers to create sophisticated video players for IIS Smooth Streaming delivery. It is available on Codeplex.

About the Author

Walter Ferrari is an environmental engineer and cultivates his passion for software development for a long time. He is currently a consultant working for his company, Abertech. He develops applications based on Microsoft technologies since 1995 and works primarily with .NET since 2003. He is currently focused on Silverlight and Sharepoint and acts as representative of CompletIT/SilverlightShow.
Walter is used to wearing a helmet while writing code..just to avoid serious damages when slamming his head against the monitor :) He blogs about his passion at http://www.snello.it/eng