Windows Phone 7.5 - Manipulating camera stream

In the previous article from this series I've introduced how to access the cameras of the phone and emulate a photo camera using the Windows Phone API. These API are really straightforward and almost everyone can take pictures easily for the purposes of the application. But for the most demanding, taking pictures does not suffice.

These two samples, but also lot of other I can imagine, require the access to the raw stream coming from the camera. Only in this way the developer can examine the stream, detect shapes, perform calculations, and so on. And only in this way he can develop the most compelling features.

Download the source code

Accessing the raw stream

For the purpose of demonstrating how to access the camera stream, I've created a simple histogram calculator app. A color histogram is a simple chart showing the distribution of colours along the pixels of the image. Lot of professional and semi-professional cameras on the market offer this feature that is useful to people to improve the quality of the pictures. The approach to use in this case, is to capture a snapshot of the raw stream and then iterate the pixels counting the single color occurrences.

Just to be clear, it cannot be a simple image capture performed using the CaptureImage() method I've described in the last article. This method is good to take still pictures to save to the media library, but it cannot be so fast as I need, to calculate the histogram on the fly on the continuously changing image.

To be fast enough I have to access the raw stream and for this purpose they exist two methods called GetPreviewBufferArgb32 and GetPreviewBufferYCbCr. They acts making a copy of the current frame, incoming from the stream, to a buffer. The two methods differ for the format of the buffer they fill. The GetPreviewBufferYCbCr is able to encode the information using the YCbCr format that is an expression of Luminance and Chrominance. In this way Cb and Cr describe the chrominance component made of a composition of red and blue and on the other side the Y describes the luminance component. The Y taken by itself is an index of light intensity.

The YCbCr format can be useful in some applications but it is for sure hard to manipulate because there is not a direct relationship between each value and a single pixel of the image. Also, for the purpose of calculating an histogram, it would be better to have the picture represented as a simple RGB buffer to iterate, to easily count color occurrences. The GetPreviewBufferArgb32 method is the best choice because it fills the buffer representing for each pixel a Int32 value that express Red, Green and Blue plus the Alpha component, determining the level of transparency of the pixel. To extract values from the Int32 you can use the following formula

• Blue = pixel & 0xf
• Green = (pixel & 0xf0) >> 8
• Red = (pixel & 0xf00) >> 16
• Alpha = (pixel & 0xf000) >> 24

Unfortunately it does not exists a "buffer changed" event so, to write the histogram example, I have to start a timer that trigger a method to recalculate le histogram every single timeout. The calculation of the histogram is not so heavy so the lenght of the timeout may be sufficently short to give to the user the appearance of a chart update almost in realtime, also if this is not exacly the truth. Here is the code to start the timer:

   1: public Histogram()

   2: {

   3:     InitializeComponent();

   4:  

   5:     // add here further initializations

   6:  

   7:     if (PhotoCamera.IsCameraTypeSupported(CameraType.Primary))

   8:     {

   9:         this.Camera = new PhotoCamera(CameraType.Primary);

  10:         this.Camera.Initialized += new EventHandler<CameraOperationCompletedEventArgs>(Camera_Initialized);

  11:         this.cameraViewBrush.SetSource(this.Camera);

  12:         this.CorrectViewFinderOrientation(this.Orientation);

  13:     }

  14:     else

  15:         throw new NotSupportedException("Camera is not supported");

  16: }

  17:  

  18: private void Camera_Initialized(object sender, CameraOperationCompletedEventArgs e)

  19: {

  20:     Timer timer = new Timer(UpdateHistogram, null, 0, 1000);

  21: }

In the constructor of the page I make the needed checks to verify that a camer is available and I also setup the brush to project the viewfinder to the screen. This part is exacly the same I've explained in the last article. Then, when the camera has been initialized I create the timer to call the UpdateHistogram method once a second. This is not really a fast update but you can try shorter values by yourself.

In the UpdateHistogram method I access the buffer using the GetPreviewBufferArgb32 method and then I calculate the chart on the returned buffer. The calculation is made iterating each pixel of the image and calculating the average value between the RGB components. Then the resulting valaue is used as an index on the array representing the counts for the 256 possible values of the average. At the end of the loop the array contains a count value for each color present in the picture:

   1: private void UpdateHistogram(object state)

   2: {

   3:     int max = (int)this.Camera.PreviewResolution.Width * (int)this.Camera.PreviewResolution.Height;

   4:     int[] buffer = new int[max];

   5:     this.Camera.GetPreviewBufferArgb32(buffer);

   6:  

   7:     int [] histogram = new int[256];

   8:  

   9:     foreach (int pixel in buffer)

  10:     {

  11:         byte r = (byte)((pixel >> 16) & 0xff);

  12:         byte g = (byte)((pixel >> 8) & 0xff);

  13:         byte b = (byte)(pixel & 0xff);

  14:         int value = (r + g + b) / 3;

  15:         histogram[value]++;

  16:     }

  17:  

  18:     int h = histogram.Max();

  19:  

  20:     Deployment.Current.Dispatcher.BeginInvoke(

  21:         () =>

  22:         {

  23:             this.Bitmap.Clear();

  24:  

  25:             for(int i = 0; i<256; i++)

  26:             {

  27:                 int v = histogram[i] * 100 / h;

  28:                 this.Bitmap.DrawLine(i, 100, i, 100 - v, Colors.Red);

  29:             }

  30:  

  31:             this.Bitmap.Invalidate();

  32:         });

  33: }

Interesting to say, the asynchronous nature of the Timer is here an opportunity to perform the histogram calculation on a separated thread without impacting the update of the user interface. Only when the buffer has been scanned and the histogram is calculated the Dispatcher marshal the thread to the user interface and the calculated values are plotted to a WriteableBitmap that shows an horizontal bar chart of 256 values. This chart is then presented to the ui with the call to Invalidate().

Saving the stream to a file

The capability of directly access the raw stream does not means we are able to save the stream to a file. Given that we are not able to record the every single change in the buffer, taking a snaphot of the stream with GetPreviewBufferArgb32 does not suffice to say we can be so fast as it is required to record a video without hiccups.

It is for this reason it has been included a FileSink class that is able to automatically read the stream from a camera and save it to the IsolatedStorage. This class is really easy to use, and it can automatically encode the raw stream to a MP4 video. This format can be direclty handled by the MediaElement, so you can easily play again a video you have recorded.

The FileSink class acts as a junction between the camera and a file in the isolated storage. To initialize the FileSink class you have to provide an instance of VideoCaptureDevice that represent the camera you have to record, and the filename of the output file. To get access to the VideoCaptureDevice you have to use the Capturesource class:

   1: this.Source = new CaptureSource();

   2: this.Source.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

The GetDefaultVideoCaptureDevice method give you a reference to the default camera but you are able to enumerate the available cameras using the GetAvailableVideoCaptureDevices method. Using the returned VideoCaptureDevice class you can detect supported formats, and other properties like the FriendlyName. Once you get the reference to the device you can connect it to the FileSink:

   1: this.Sink = new FileSink();

   2: this.Sink.CaptureSource = this.Source;

   3: this.Sink.IsolatedStorageFileName = "video.mp4";

   4:  

   5: this.Source.Start();

After the device has beend start, using the Start method, the FileSink class begins to pump data from the camera directly to the "video.mp4" file, located in the IsolatedStorage. During this process you can also redirect the output of the CaptureSource to a VideoBrush for the purpose of showing a preview of the recorded video.

   1: this.Brush = new VideoBrush();

   2: this.cameraView.Background = this.Brush;

   3: this.Brush.SetSource(this.Source);

To end the recording you have to call the Stop method on the CaptureSource and then you have to disconnect the FileSink from the source setting its properties to null. This automatically free the output stream and becomes the recorded video available for the play. If you have a preview on a VideoBrush you have also ti disconnect the Source from the brush to be sure you can restart the capture in a later time.

   1: this.Source.Stop();

   2: this.Sink.CaptureSource = null;

   3: this.Sink.IsolatedStorageFileName = null;

   4: this.cameraView.Background = null;

   5: this.Brush = null;

To play the recorded video you have to connect it to a MediaElement. This means opening a stream to the Isolated Storage file and pass it as a source for the MediaElement:

   1: this.Stream = new IsolatedStorageFileStream("video.mp4", FileMode.Open, FileAccess.Read, IsolatedStorageFile.GetUserStoreForApplication());

   2: this.media.SetSource(this.Stream);

   3: this.media.Play();

As a side note please be aware that the CaptureSource class is also able to take still pictures using the CaptureImageAsync method. It may be an alternative way to the use of the PhotoCamera class. Of course, once you have recorded a video it is in charge to you the code to upload it to the network if you want to persist it to a file on a pc. Isolated Storage in windows Phone is limited only by the size of the available memory, but it is not a good idea to waste it with huge video files.

Updated example

In the solution I provided with the last article I've added another project that shows the things I've explained in this rows. The first example shows how to calculate the histgram and if you load it on your development device you can appreciate the resulting chart. Also another example shows how to record to a file. The application alternatively record to a file and reply the recorded video. The sequence show how to correcly attach and detach the stream to avoid the lock of something.