When you create an app like the SilverlightShow app, that accesses remote data frequently, thinking about a suitable strategy for local storage and caching is vital to provide a great user experience. In the previous parts of this series, we have already seen how several optimizations are used to improve the networking performance of the app. Overall, the costs to pull data from the portal could be reduced to far less than 1/10th of the original traffic by applying some relatively simple changes. However, this is only half the story. What we also wanted to achieve is that content that has been retrieved once should not be fetched again – we needed a solution for storing those items locally.

Local Storage Options

When you think about persisting and restoring data locally on the phone, you have a variety of features at your disposal. With the first versions of Windows Phone, you could either use the ApplicationSettings for trivial situations, or you had to pretty much do all the work manually, as in writing to and reading from Isolated Storage files.

With the platform maturing more and more, additional options are established. In particular, Microsoft added local database support in Windows Phone 7.1, built on SQL Server of course. This gives you the comfort of working with relational data and the well known features of LinqToSql for queries. If you want to learn more about local database support, take a look at Andrea Boschin's article about it.

The option we decided to use is a community project originally designed and developed by Jeremy Likness: The Sterling Database. The project has been around for two years, is now maintained and contributed to by various community members, and has proven its power in several apps on the Marketplace that are using it today.

What Sterling is

Let me quote from the Sterling project page on CodePlex to give you a basic understanding of its nature:

"Sterling is a lightweight NoSQL object-oriented database for .Net 4.0, Silverlight 4 and 5, and Windows Phone 7 that works with your existing class structures. Sterling supports full LINQ to Object queries over keys and indexes for fast retrieval of information from large data sets."

What exactly does this mean? For starters, instead of mapping your classes to a relational database model, you can simply use your existing data structures right away. In situations like ours, where a data structure has already been established or is determined by some external factors, this can result in less work. It may also give you a more natural and easier way of working with your data, as you don't have to deal with that extra layer and conversion logic between your application objects and data storage.

Although Sterling is a very lightweight project (the core assembly is only ~80 kilobytes big), it still is very flexible. For example, in addition to well-known features like triggers you can also register so-called interceptors that let you change the way Sterling works on a low level, which enables interesting options for example regarding transparent data encryption or optimization (more on that later).

Behind the scenes, Sterling of course also uses Isolated Storage for persistence, but it uses binary serialization, which results in very compact file sizes compared to the built-in serialization options:

(Image taken from the Sterling project page)

The way Sterling performs queries on its keys and indexes (explained below) gives a massive performance benefit, as shown by the following comparison:

(Image taken from the Sterling project page)

So in certain situations, for example when disk memory consumption or query speeds are crucial and your data is suitable to be stored and handled by Sterling, using that database can be a huge improvement over alternative methods, and/or save you a considerable amount of time compared to custom implementations.

Setup

Setting up a Sterling database is easy to do. After pulling in the required references (you can use the available NuGet package for that) simply derive from the built-in database instance base class. You can then provide so-called table definitions that determine what types you want to store in your database, and what indexes should be created for them. For example, the following piece of code creates a table definition for "Article" items, uses the "Guid" property as key, and adds an index for the "PublishDate" property:

public class SterlingDatabase : BaseDatabaseInstance

{

  protected override List<ITableDefinition> RegisterTables()

  {

    return new List<ITableDefinition>

      {

        CreateTableDefinition<Article, string>(article => article.Guid)

          .WithIndex<Article, DateTime, string>("ArticleDateIndex", article => article.PublishDate)

      };

  }

}

All that's left to do is initialize the database. A good pattern is to do that when your application starts or gets activated, and to execute the corresponding clean-up when the user leaves your application. The following shows the required code that can then be called in the respective lifetime events of the PhoneApplicationService:

private SterlingEngine _engine;

private ISterlingDatabaseInstance _sterlingDatabaseInstance;

private void ActivateSterling()

{

  _engine = new SterlingEngine();

  _engine.Activate();

  _sterlingDatabaseInstance = _engine.SterlingDatabase.RegisterDatabase<SterlingDatabase>(new IsolatedStorageDriver());

}

private void DeactivateSterling()

{

  _sterlingDatabaseInstance.Flush();

  _engine.Dispose();

  _sterlingDatabaseInstance = null;

  _engine = null;

}

Do not forget to clean-up in the "Deactivated" event too, as there is no guarantee that the user will return to your app, resulting in tombstoning and silent removal of your app, with the potential to leave your database corrupted.

Basic Operations

From the moment of activation on, you can use the database instance to save and load items very easily:

// save an article

_sterlingDatabaseInstance.Save(article);

_sterlingDatabaseInstance.Flush();

// load back

var loadedArticle = Load<Article>(article.Guid);

As you can see, the snippet uses both "Save" as well as an additional "Flush" method to save the object. Since Sterling is optimized for performance, it keeps its current keys and indexes in-memory only, until you call "Flush" explicitly. This allows you to do multiple successive "Save" operations without the performance hit of persisting keys and indexes between each of these operations. The drawback is that for single operations, you have to call "Flush" explicitly to keep the database consistent on disk.

One important thing that often causes confusion with inexperienced users, is that the "Load" operation does not cache items in memory. This means that every time you use the "Load" operation, a new instance is returned, i.e. two successive invokes of "Load" with the same "article.Guid" value will return two different objects (for the same article though). To benefit from caching, you have to use queries (see below).

Loading and saving an object will treat the whole object graph connected to that object through properties. Depending on how you set up your database the behavior will be slightly different. To learn more about the details of loading and saving, I recommend reading the excellent documentation of Sterling: Saving Instances and Loading Instances

Deletion of objects is performed using either the object itself, or by passing in the key for that object (which eliminates the need to explicitly load an object just for deletion):

// delete by reference

var article = _sterlingDatabaseInstance.Load<Article>(guid);

_sterlingDatabaseInstance.Delete(article);

// or delete by key

_sterlingDatabaseInstance.Delete(typeof(Article), key);

Sterling of course also supports deleting all objects of a certain type (truncating a table) and even purging the whole database. You can learn more about this here.

Queries

The power of Sterling comes with queries. In the above sample, we had added an index for the publish date of articles. This allows us to access stored article instances by their publish dates extremely fast because Sterling holds that data in memory. It uses a lazy load mechanism that only needs to access the disk once you actually need to retrieve the object content. Let me give you an example:

public DateTime DetermineNewestArticle()

{

  var result = _sterlingDatabaseInstance.Query<Article, DateTime, string>("ArticleDateIndex")

                 .OrderByDescending(o => o.Index)

                 .Select(o => o.Index)

                 .FirstOrDefault();

  return result;

}

This retrieves the date of the newest available article using the respective index, without loading the actual article from disk. Because the index is available in memory, this is a Linq to Objects operation that is really fast. You can have multiple indexes for each type, to enable different of these query scenarios at the same time.

If you need to actually access the content of the newest article, then the code could look like this:

public Article GetNewestArticle()

{

  var result = _sterlingDatabaseInstance.Query<Article, DateTime, string>("ArticleDateIndex")

                 .OrderByDescending(o => o.Index)

                 .Select(o => o.LazyValue.Value)

                 .FirstOrDefault();

  return result;

}

Only when you access the "LazyValue.Value" property does Sterling load the data from Isolated Storage. Queries of course are much more powerful than simply loading single objects. You can also combine them and make use of joins, projections and all the other available features. To learn more about complex queries, the documentation is a good starting point.

Caching

As mentioned above, as soon as you use queries the retrieved values are cached. This means that only the first access of a certain (lazy) value in the key or an index collection results in an actual load operation. Successive queries return the previously loaded object and hence will execute much faster. Don't worry: as soon as you change the object and save it back to Sterling, the cache is cleared for that object, and the next query operation will reload your changed values from disk again (with the result being cached again). Sterling keeps track of save operations internally.

While caching can improve performance of your app tremendously, it also can have some unexpected side effects when you use the same data in different parts of your app at the same time. To learn more about these edge cases, take a look at the docs here.

Interceptors

The data of the SilverlightShow app is easily compressible, because it's mostly text. One of the nice hooks Sterling allows you to use therefore came in handy, to apply a general compression to all the data that is written and read – without the need to explicitly take care of this manually during each and every save and load operation. The way this works is to add a custom interceptor implementation during the initialization phase of the database:

public void ActivateSterling()

{

  _engine = new SterlingEngine();

  _engine.Activate();

  _sterlingDatabaseInstance = _engine.SterlingDatabase.RegisterDatabase<SterlingDatabase>(new IsolatedStorageDriver());

  // register a custom interceptor

  _sterlingDatabaseInstance.RegisterInterceptor<CompressionInterceptor>();

}

To create such an interceptor, you can conveniently derive from the existing "BaseSterlingByteInterceptor" base class and override the required methods:

public class CompressionInterceptor : BaseSterlingByteInterceptor

{

  override public byte[] Save(byte[] sourceStream)

  {

    // do anything you want with the incoming bytes

  }

  override public byte[] Load(byte[] sourceStream)

  {

    // reverse whatever you did to the bytes in the Save method here

  }

}

Your interceptor is used behind the scenes in all the operations, you don't have to change anything in the way you work with Sterling. So this is the perfect place to apply things like on-the-fly compression or encryption.

Conclusion

Sterling is an interesting alternative for local storage of data in Windows Phone apps. We were able to reuse our existing data structures without the need to introduce any storage-related changes to them, and the way Sterling handles key and indexes is perfect for our requirements of searching and sorting available items without the need to deserialize their (heavy) content from disk. The possibility to implement a generic data compression easily was the icing on the cake that helped create a great user experience.