Recently I was working on a project that requires to programmatically access and download files from SharePoint Online (part of Office 365) document library. Currently remote authentication (SP Managed Client OM / SP Web Services) is bit challenging to deal with as discussed here but a decent workaround (for now?) is available on MSDN. The first thing you will notice is that authentication is more powerful and more complex because SharePoint Online (so as SharePoint Foundation 2010 | SharePoint Server 2010) embrace Claims Based Authentication. The MSDN sample code seems to work fine so I went ahead and use it as a starting point.
**I used Word 2010 to create and publish this post and somehow images quality gets deteriorated during publishing process. If you want to read this post with high quality images its available as pdf here
Issue:
Everything went smooth (authentication + accessing list items) till I actually try to download a file from a document library. I was using File.OpenBinraryDirect method (see code snippet below) and got The remote server returned an error: (403) Forbidden exception.
As I investigate this issue by looking into the exception details I found out that basically it’s complaining about the fact that before opening files in this location I must select the option to login automatically.
This gives me the clue that there is something missing at a basic level. I went ahead and open up a browser and browse to the SharePoint online site from which I am trying to download the file. I use a tool to monitor the request from the browser and eureka I see User-Agent header with a valid value (as expected) which was never set in the code.
The code from the MSDN does not set the UserAgent property of the WebRequest so it remains null. Normally when you are using browser it’s automatically set up for you but in this case you have to do it yourself though code. So you fix this part and you are good to go.
Fix:
Locate the method GetAuthenticatedContext (inside class ClaimsClientContext within ClaimsAuth project). And add following line of code before you initialize cookie container. You are setting up UserAgent property for the WebRequest to a valid user agent. I have used following value for user agent but you can try different but valid value and it may work too (but I didn’t tried it yet).
e.WebRequestExecutor.WebRequest.UserAgent = “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)”;
That’s it! You should now able to download the files without any issues.
**On a side note although above solutions works I have a feeling that there is more to it because X-MSDAVEST_Error: 917656 is an Authentication level error and related to WebDav protocol more info here. I may investigate it further when time permits and share my finding in this blog post.
Razi bin Rais - Blog 
Thanks champ, nice investigation, had a quick fix done for a customer in a glimpse thanks to you
Mate, I love you. You just solved a huge issue that prevented me from finishing my application. Many thanks!
Thanks man, this saved my application.
I started having this problem all the sudden in the middle of my development cycle, possibly after installing some other software. Setting the user agent gives a 405 error. I’m wondering why I all the sudden have this problem. It must be some configuration in IE.
Great work, saved me 🙂
Awesome !!
You have done a really good job of detecting that the UserAgent is not set.
Thanks a lote ! It saved me a lot of trouble.
Hi…I am having same issue…I am trying to upload document in online Sharepoint document library using REST api. I tried with above trick but it is not solving my problem. Please guide.
Great fix, worked great for me in consuming Sharepoint hosted REST data in an SSIS script task. It’s amazing that this is the only place I found this. Thanks.