Max Paulousky has not only a VS2010 add-on to minimize your XAP size, but he also demonstrates how to build it.
I was working at the Modularity in Silverlight applications post and bumped into the Jeff Prosise's blog post Cool Silverlight Trick #3. He mentioned, that an assembly can be added to a project but it may not be embedded into XAP file. Developers should just change the CopyLocal option in assembly properties in Visual Studio from True to False. So, a project had a reference to an assembly but that assembly was not in the XAP file. That was what I needed.