Windows Phone 7 301 Redirect Bug

I checked with the Windows Phone 7 team before putting this post out, and got the OK to write about it.  I hit this bug when building out my FriendLinks Windows Phone 7 application, and was surprised to find it was a bona fide bug and not a problem with my coding skills.  If there was a default value in the constructor for any new project I start, it would look something like:

codingErrorResponsibilty == this.programmerName;

In any event, here’s the cruxt of the bug.  If you are using HttpWebRequest developing on Windows Phone 7, and you get a 301 redirect returned from the web server, and that web server sets a cookie, the current build (as of 03/16/10 at Mix10) of Windows Phone 7 will throw an exception.

In the case of my applications, I was trying to determine the title of the page as defined by the “<title></title>” tags for a given URL.  The function I wrote for this was:

try
{
    WebResponse response = ((HttpWebRequest)result.AsyncState)

                            .EndGetResponse(result);
    StreamReader reader = new StreamReader(response.GetResponseStream());
    string responseString = reader.ReadToEnd();
    string regex = @"(?<=<title.*>)([\s\S]*)(?=</title>)";

    if (new List<string>(response.Headers.AllKeys).Contains("Content-Type"))
    {
        if (response.Headers["Content-Type"].StartsWith("text/html"))
        {
            Regex ex = new Regex(regex, RegexOptions.IgnoreCase);

            selectedUrlTitle = ex.Match(responseString).Value.Trim();

            if (String.IsNullOrEmpty(selectedUrlTitle))
            {
                selectedUrlTitle = "No title for page";
            }
        }

    }
    

    UrlTitleRequestCompleted(this, EventArgs.Empty);
}

For some URLs this was throwing the most elusive of all exceptions: CookieException.  Behold, I present to you a StackOverflowWhack! [note: a nod to GoogleWhacks]  There should really be a better name for that.

There is only one response for this search query at StackOverflow.  Interestingly enough, when I first hit this bug, there were no questions – this one was asked in the last week.

The debug information from Visual Studio was even less useful.  It was telling me that the cookie was too big.  A little searching revealed that this exception is thrown when you try to set a cookie larger than MaxCookieSize for a CookieContainer.  That value cannot be changed in Silverlight, and since I don’t control the remote web servers, I can’t control the cookie coming back.  Worse, whether or not the cookie was being set on the 301 redirects varied from domain to domain.  Some set it, some didn’t.

In using Fiddler2 to debug this problem, here is what a redirect looks like that won’t throw the CookieException:

HTTP/1.1 301 Moved Permanently
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Transfer-Encoding: chunked
Via: 1.1 RED-PRXY-21
Date: Wed, 03 Mar 2010 02:55:15 GMT
Location: http://blogs.bnet.com/career-advice/?p=101
Content-Type: text/html; charset=UTF-8
Server: Apache
Set-Cookie: supr=supr301; path=/; domain=.su.pr
Vary: Accept-Encoding

In this case, su.pr didn’t try to set the cookie.  And here is a 301 redirect that tries to set a cookie:

HTTP/1.1 301 Moved
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 381
Via: 1.1 RED-PRXY-21
Date: Wed, 03 Mar 2010 02:57:15 GMT
Location: http://www.allfacebook.com/2010/03/facebook-developers-see-

               dramatic-drop-in-traffic-following-removal-of-notifications/
Content-Type: text/html; charset=utf-8
Server: nginx/0.7.42
Set-Cookie: _bit=4b8dd00b-0013c-01992-b5a08fa8;domain=.bit.ly;

                expires=Sun Aug 29 22:57:15 2010;path=/; HttpOnly
MIME-Version: 1.0

In this case, bit.ly is in fact trying to set a cookie.  What helped lead me down the path of figuring this out was a short.to link that forwarded to a bit.ly link.  Here’s the chain of raw headers:

HTTP/1.1 301 Moved Permanently
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 0
Via: 1.1 RED-PRXY-21
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Wed, 03 Mar 2010 02:58:43 GMT
Location: http://bit.ly/8ZyVoz
Content-Type: text/html; charset=utf-8
Server: Google Frontend
Cache-Control: no-cache
X-XSS-Protection: 0

HTTP/1.1 301 Moved
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 312
Via: 1.1 RED-PRXY-21
Date: Wed, 03 Mar 2010 02:58:44 GMT
Location: http://news.cnet.com/8301-13578_3-10462563-38.html
Content-Type: text/html; charset=utf-8
Server: nginx/0.7.42
Set-Cookie: _bit=4b8dd064-000f6-0198d-b5a08fa8;domain=.bit.ly;

                expires=Sun Aug 29 22:58:44 2010;path=/; HttpOnly
MIME-Version: 1.0

As you can see here, the first header revealed a 301 to a bit.ly URL, which then tried to redirect to a news.com article.  Some other offenders of this cookie setting on 301 redirects are FourSquare and TechCrunch.  Mind you, this isn’t a bug in their 301 redirect, it’s a bug in how Windows Phone 7 handles the redirect.

I ultimately had to abandon this path to get the page title because once that CookieException is thrown, there is no data in the exception that allows you to get access to the header.  The exception is thrown when the WebResponse object is being set to the result of reading to the end of the asynch response.  The CookieException itself doesn’t reveal any of the 301 redirect information, so recovering from this exception is next to impossible, except to abandon trying to get the URL title.

For my specific app, I needed to get that title information, and since the vast majority of links coming through Twitter were bit.ly, I needed another way.  Mercifully, TechMeme reveals this information through their API.

So there you go – if you are getting a 301 redirect in a HttpWebRequest, and your code is throwing a CookieException, now you know why.  This is a filed bug and should be fixed, but given the amount of excitement around Windows Phone 7 development right now, I wanted to try and save people pain and suffering.  It’s better that only one of us lose hair.

Today at Mix10, Microsoft released the developer tools for Windows Phone 7 Series.  I’m excited about seeing what apps will get built now that there is general availability of the Windows Phone dev tools.  That’s general availability, as in for everyone!

Charlie Kindel (if you aren’t, you really should follow him) was nice enough to get me access to the tools about 3 weeks ago, despite the fact that he and his team have had plenty on their plate since going public with WP7.  I have spent some time in the last few winks building and tinkering.

A little background on my dev skills.  I can write basic applications, and have been known to favor Python when trying out new ideas.  I have dabbled a bit with our ASP.NET MVC (MVC v2 just released – way to go guys!!) and taught myself enough C# to be dangerous.  What coding I do, I do for fun and in my free time.  I call it my nocturnal nerdiness, and have been logging some of my projects using the n00bnotes tag.  Prior to 3 weeks ago, I had never written one line of Silverlight (or WPF for that matter) code, nor any XAML.  I was really excited to have the opportunity to build apps for this mobile platform, as I once tried to get along with iPhone development, and while it’s clear that Apple has created tools that developers seem to love, I couldn’t get along with ObjectiveC.  That’s a me issue, and not a statement about ObjectiveC.  I get along famously with Python, but me and Ruby are not friends.  That’s just the way my brain works.

With that as a preamble, I wanted to share what I have created in just the last 3 weeks, working largely in what spare time I could find when not doing my day job or dealing with an recalcitrant 8 month old girl who refuses to sleep.  The main thing I want people to take away from this is that it in incredibly easy to built apps for Windows Phone 7.  If I can figure it out, anyone can.  The team has delivered a great development experience built on top of Visual Studio Express.  When you fire up the development environment, everything you need is there and you are ready to go.  It was a pretty painless experience to get the environment up and running, and it includes templates for Silverlight apps as well as XNA games.  While I have only been able to deploy to an actual phone once, the emulator felt like a software version of the phone.

Over the next few days of Mix10, I am going to put up a few posts about my experiences with the development tools, highlighting some of the blockers I hit, how I solved them, and for some of them, how I should have solved them, which I eventually fixed during code refactoring.

In the meantime, I wanted to share a link to the current version of the code.  This is my FriendLinks application, built for Windows Phone 7 [UPDATE: 3/24/10 – oh the joy of forgetting to remove commented code with your twitter pass.  Ooops…code updated.]  You will need the development tools in order to open, edit view.  The only disclaimer I make is that the code works.  Not all of it is pretty, and in some places I haven’t gone back to fix things I fixed elsewhere (i.e. walking XML for Bit.ly versus for Twitter).

This specific post is about some of the things that gave me the biggest problems in getting started.  The app that I built is pretty simple.  It’s meant to allow you to connect to Twitter, pull down your friend timeline, and parse the timeline looking for URLs sent by people you follow.  I use Twitter for content discovery, and this is my ultimate time waster app.  When you click on a link the listbox, some additional calls are made via the Bit.ly API, and the TweetMeme API, to get additional information like the number of retweets that article has, the title of the page referenced and the number of clicks as tracked by Bit.ly.

Making Async Calls to Web Services

Wow, what a huge pain this was for me to figure out.  When you do some web searching about how to connect to web services in C#, you will invariably find yourself staring at content about WebClient class.  I couldn’t make this work for me, and now that I am 3 weeks into it, I can’t remember what the specific issue was.  Something about Twitter not doing Basic Auth correctly, and needing to set the username and password in the header, necessitating the use of NetwrokCredentials.

In any event, I had to use the HttpWebRequest.  This is where things got challenging for me, since I had never done any async programming in C# or Silverlight.  [Apologies for the wonky formatting in the code samples, but I have a narrow blog and the style isn’t doing auto wrap]

public void GetStatuses()
{
    NetworkCredential nc = new NetworkCredential { UserName =

                          _username, Password = _password };

    string url = http://twitter.com/statuses/friends_timeline.xml?

                               + "count=" + _count + (_sinceID > 0 ?

                               "&since_id=" + _sinceID : "");
    System.Net.HttpWebRequest request =

                              (HttpWebRequest)HttpWebRequest.Create(url);
    request.Method = "GET";

    if (_username != null && _password != null)
    {
        request.Credentials = nc;
    }

    IAsyncResult token = request.BeginGetResponse(

                                  new AsyncCallback(GetStatusesCallBack),

                                  request);

}

Sorting out how to correctly get the async call done, and then set up the callback, was where I completely threw a rod.  Basically, you need to set up a web request to happen on its own thread, and then you need to assign a delegate function to process the response.  Figuring out how to do this took me 2 days, mostly because I didn’t know what questions to ask, or what terms to use when searching online, and in this specific case, only code that I did discover was useless for me because I was trying to learn something new and didn’t understand the samples I found.

Looking at the call, you may notice that I am using the REST API from Twitter, and setting the since_id and count variables to ensure that I am getting the data that I want.  This is so I can reuse this function to make subsequent calls while the app is running and only get the new tweets.  Twitter makes things pretty easy to get the data you want.

Here’s how you handle the async callback function to actually issue the HTTP request and process the data which comes back:

public void GetStatusesCallBack(IAsyncResult result)
{
    bool foundNewSinceID = false;

    try
    {

        WebResponse response = ((HttpWebRequest)result.AsyncState)

                                       .EndGetResponse(result);
        StreamReader reader = new StreamReader(response

                                       .GetResponseStream());
        string responseString = reader.ReadToEnd();