I have been asked by a few people how sites like WP7AppList get their data.  The Windows Phone Marketplace, which you access on your PC via Zune software, uses XML to get data over the wire.  I wanted to share a couple of code snippets which might help an erstwhile data junkie on their way.  This code works.  It may not be the most elegant solution, but it works, and I wanted to share it with others in case they wanted to see how to parse the XML, or how to write LINQ queries against it.

Caveat – this is a geek enthusiast post.  I used Fiddler to figure out how to parse the XML.  This was something I did over Christmas break to give me a project about which I could be excited, and learn some more about parsing XML with LINQ.  I also wanted to do some large database stuff, and this crawler throws off a ton of data.  I did not use an proprietary knowledge about how our backend systems are working.  This is all done against the public XML feeds.

First up, we are going to need to create some data structures to catch all of the inbound data.  You can use anonymous types with LINQ, but I liked having a measure of control, and having the ability to handle null values and potential errors in the feed.

public class ZestAppData
{
    public string Title { get; set; }
    public string Id { get; set; }
    public DateTime ReleaseDate { get; set; }
    public DateTime Updated { get; set; }
    public string Version { get; set; }
    public string ShortDescription { get; set; }
    public decimal AverageUserRating { get; set; }
    public int UserRatingCount { get; set; }
    public string ImageId { get; set; }

    public IList<ZestCategory> Categories = new List<ZestCategory>();
    public IList<ZestPublisher> Publisher = new List<ZestPublisher>();
    public IList<ZestOffer> Offers = new List<ZestOffer>();
}

public class ZestCategory
{
    public string Id { get; set; }
    public string IsRoot { get; set; }
    public string Title { get; set; }
}

public class ZestOffer
{
    public string OfferId { get; set; }
    public string MediaInstanceId { get; set; }
    public decimal Price { get; set; }
    public string PriceCurrencyCode { get; set; }
    public string LicenseRight { get; set; }
    public List<string> PaymentType = new List<string>();
}

public class ZestPublisher
{
    public string Id { get; set; }
    public string Name { get; set; }
}

You are also going to want to have a bunch of variables defined for the URLs where the XML is coming from, the XML namespaces, etc:

const string BaseAppsUrl = "http://catalog.zune.net";
const string BaseImageUrl = "http://image.catalog.zune.net";

const string ZestVersion = "/v3.2/";
const string ZestImageVersion = "/v3.0/";

const string BaseApps = "apps/";
const string BaseImage = "image/";

const string BaseAppsResource = "?clientType=WinMobile%207.0&store=Zest&orderby=downloadRank";
const string BaseCommentsResource = "/reviews/?store=Zest&chunkSize=10";
const string BaseImageResource = "?width=240&height=240";

ZestCrawlEntities ZestCrawlContext;

XNamespace ns = "http://www.w3.org/2005/Atom";
XNamespace zestns = "http://schemas.zune.net/catalog/apps/2008/02";

public string LangCode = "en-us"; //setting the default value

public List<string> ValidLangCodes = new List<string>(
    new string[] {  "en-us", "en-gb", "de-de",
                    "fr-fr", "es-es", "it-it",
                    "en-au", "de-at", "fr-be",
                    "fr-ca", "en-ca", "en-hk",
                    "en-in", "en-ie", "es-mx",
                    "en-nz", "en-sg", "de-ch",
                    "fr-ch" });

public string AppAfterMarkerUrl { get; set; }
public bool HasMoreApps = true;
public string AppsResponseString { get; set; }
public XElement ReturnedAppsXml;

Have a look at the ValidLangCodes list.  that’s the coding we have on the URLs for country specific data.  So if you want to get the data from Mexico, us “es-mx.”  The first two letters are the language code, and the second two are the country code.  If an app is listed in the feed, it is active.  The list returned is ordered, meaning the first app is ranked #1.  I am pulling the ALL APPs list, which is the orderby clause on the BaseAppsResource.

The ZextCrawlContext is the ADO.NET DB model.  Create your own and stuff the data however you want.

Now that we have the code segments, you are going to need a way to get the XML from MSFT servers.

public void GetAppsResponse()
{
    string FullUrl;
    bool done = false;

    if (!String.IsNullOrEmpty(AppAfterMarkerUrl))
    {
        FullUrl = AppAfterMarkerUrl;
    }
    else
    {
        FullUrl = BaseAppsUrl + ZestVersion + LangCode + "/"
            + BaseApps + BaseAppsResource;
    }

    while (!done)
    {
        try
        {
            var request = WebRequest.Create(FullUrl) as HttpWebRequest;
            request.KeepAlive = false;

            var response = request.GetResponse() as HttpWebResponse;

            if (request.HaveResponse == true && response != null)
            {
                var reader = new StreamReader(response.GetResponseStream());
                ReturnedAppsXml = XElement.Parse(reader.ReadToEnd());
                done = true;
            }
        }
        catch
        {
            Console.WriteLine("yeah, your connection was likely aborted");
            done = false;
        }
    }
}

Now comes the fun part.  Remember, the XML is coming over the wire, and it comes 100 elements at a time.  So you have to parse the stream, stuff them somewhere and get the next stream.  Included in the XML returned is the token for how you request the next bit of XML. (note, yes I know I am using RegEx where I could be using String.Replace; also sorry about the wonky formatting, but my blog has width issues)

public IEnumerable<ZestAppData> GetAppEntries()
{
    //first we have to parse the feed which came back
    IEnumerable<ZestAppData> entries =
        from e in ReturnedAppsXml.Elements(ns + "entry")
        select new ZestAppData
        {

            Title = e.Element(ns + "title").Value,

            Id = Regex.Replace(e.Element(ns + "id").Value, "(urn:uuid:)(.)", "$2"),

            ReleaseDate = DateTime.Parse(e.Element(zestns + "releaseDate").Value),

            Updated = DateTime.Parse(e.Element(ns + "updated").Value),

            ShortDescription = e.Element(zestns + "shortDescription") == null
                ? "" : e.Element(zestns + "shortDescription").Value,

            AverageUserRating = decimal.Parse(e.Element(zestns + "averageUserRating").Value),

            UserRatingCount = int.Parse(e.Element(zestns + "userRatingCount").Value),

            Version = e.Element(zestns + "version").Value,

            ImageId = Regex.Replace(e.Element(zestns + "image").Element(zestns + "id").Value, "(urn:uuid:)(.)", "$2"),

            Categories = (
                from category in e.Elements(zestns + "categories").Elements(zestns + "category")
                select new ZestCategory
                {
                    Id = category.Element(zestns + "id").Value,
                    Title = category.Element(zestns + "title").Value,
                    IsRoot = category.Element(zestns + "isRoot").Value
                }).ToList(),

            Publisher = (
                from publisher in e.Elements(zestns + "publisher")
                select new ZestPublisher
                {
                    Id = publisher.Element(zestns + "id").Value,
                    Name = publisher.Element(zestns + "name").Value
                }).ToList(),

            Offers = (
                from offer in e.Elements(zestns + "offers").Elements(zestns + "offer")
                select new ZestOffer
                {
                    OfferId = offer.Element(zestns + "offerId").Value,
                    MediaInstanceId = offer.Element(zestns + "mediaInstanceId").Value,
                    Price = decimal.Parse(offer.Element(zestns + "price").Value),
                    PriceCurrencyCode = offer.Element(zestns + "priceCurrencyCode").Value,
                    LicenseRight = offer.Element(zestns + "licenseRight").Value,
                    PaymentType = (
                        from paymenttype in offer.Elements(zestns + "paymentTypes").Elements()
                        select paymenttype.Value).ToList()
                }).ToList()
        };

    //now I need to get the AfterMarkerUrl from the XML feed
    var afterMarker =
        from e in ReturnedAppsXml.Elements(ns + "link")
        where e.Attribute("rel").Value == "next"
        select (string)e.Attribute("href").Value;

    if (afterMarker.Count() > 0)
    {
        AppAfterMarkerUrl = BaseAppsUrl + afterMarker.Single();
    }
    else
    {
        HasMoreApps = false;
    }

    return entries;
}

Now you have all the data you need to crawl the marketplace whenever you want.  The LINQ stuff is really, really fast.  Crawling the marketplaces can be a bit slow.  I crawl each one individually when my code runs, and I store app lists for each of the markets.

One of the mistakes I made was having ZestAppData.Udpated be a DateTime and not a Date.  I only crawl once per day, so I don’t need all the extra data.  The Zest feeds update daily, I think every couple of hours.

There was a story which surfaced yesterday about vulnerabilities exposed in a wide swath of Android apps.  SMobile Systems conducted research in the Android app space and found that some 20% of the apps allow third-party apps to gain access to sensitive or private information.

It would be easy to dismiss this article by pointing out that by simply downloading an app, a customer is making a explicit allowance to an app to access data on the phone.  What I found troubling about the findings wasn’t necessarily the point about access to the data, but rather that, 5% of the apps surveyed could make calls and 2% could send premium SMS messages.  Talk about a surprise cell phone bill.

The key word in the above statements, however, is “could.’”  Yes, customers make the decision to download those apps, but they have no way of knowing with certainty what those apps are doing behind the scenes.  UPDATE: Ben points out below that the customer us warned of all the APIs used, which is true, but they aren’t told *how* they are used.

Further, because of the multi-tasking architecture of Android, the apps have the potential to be doing a bunch of bad things in the background when the phone is not in use.

Google has been quick to point out that the architecture of Android would limit what actual damage one of these apps could do, but that’s really not the point.  What is being lost in this discussion is that there is no curation of the Android marketplace.  For all the grumbling and grousing about the Apple AppStore, their review process would likely catch these abuses.  There is no such level of certification for the Android marketplace.  Customers don’t want to think about needing anti-spyware software for their phone, as the article implies is one solution for Android.

The Windows Phone Marketplace certainly believes in the curation model, and we have placed user security as a top priority.  This is one of the main reasons that we have our app certification process, and why (UDPATE: “at least in version 1”) apps are run in sandboxes, with no access to any data other than its own isolated storage, or the ability to communicate with other apps.  UPDATE: The goal is to ensure that absolute best customer experience when using their phone.

UDPATE 6/25/10

I’m not one who believes in conspiracy theories or anything, but I do find the timing of this announcement from Google that they can remotely wipe apps from phones a bit curious.  I have to go do some digging, but I’d be interested to know if the Android developer agreement has specificity about what would constitute grounds for a remote wipe.