Tuesday, March 12, 2013

Azure Management, PowerShell deployment, and a Forefront monkey wrench

How “Man-in-the-Middle” (MITM) interferes with Azure Management API authentication

A client needed an automated deployment of a PaaS web role into Azure using PowerShell.  I followed all appropriate steps up to the point of calling Publish-AzureServiceProject.  As the cmdlet progresses, it blows up at the point of verifying a storage account with an error “CData elements not valid at top level of an XML document. line 1 position 3”. Odd... 

I started researching: there seemed to be no relationship to PowerShell, Azure, or using Azure Management API.  Because some posts on troubleshooting this error indicated that it could be caused by a corrupt installation or configuration, we uninstalled and reinstalled anything Azure, and then even the .NET frameworks.  No difference whatsoever.  Some post mentioned that the content of the response that they could examine contained an authentication error that WCF/REST failed to properly parse.  I would have liked to pursue this lead: the only problem is PowerShell cmdlet is communicating over HTTPS, and it uses client side certificate to authenticate itself to the Azure Management API services.

Next, I made a copy of the project, put it on my (as opposed to the client’s) machine, and deployed it: typical “it works on my machine”.  While this sanity check was personally comforting, it did not get me any closer to the root cause of the error, and that little thing of client being able to deploy from THEIRS and NOT from my machine.

Providentially, a couple weeks back I was researching “Man-in-the-Middle” vulnerabilities of TLS and SSL and stumbled across MSDN articles about Forefront Endpoint Protection (like this).  This is similar to how Fiddler allows decryption of HTTPS traffic.

Since I wanted to examine the HTTPS requests and responses that caused the error message, I naturally tried to configure Fiddler.  When I had finished using Fiddler last time, I removed DO_NOT_TRUST Fiddler certificates from Trusted CAs.  That broke Fiddler’s configuration for HTTPS decryption.  I had to clean up certificates and the RSA work folder in C:\Users\\[username]\AppData\Roaming\Microsoft\Crypto\RSA\\[folder-with-big-name]\ to bring HTTPS decryption in Fiddler back from the dead.  After all this excitement on my machine was over, I was ready to see the request/response HTTPS traffic, on my machine first.  Then I got an HTTP 403 error at the same step of Publish-AzureServiceProject cmdlet.  image

I asked the client: “Do you happen to use Forefront HTTPS inspection in your organization?”  And what do you know!

We had their IT guys turn off Forefront HTTPS inspection on one machine.  We put the project there, ran Publish-AzureServiceProject, and we were in business!  It appeared that the error caused by Fiddler certificate based HTTPS “Man-in-the-Middle” was related to the original error caused by Forefront trying to inject itself between the PowerShell process with client certificate and the Azure Management services.

Thank God for drawing those lines in my head, one dot to the other, from “CData elements not valid at top level of an XML document. line 1 position 3” all the way to “Forefront is trying to read my encrypted traffic.”  I am thinking about contacting guys at Microsoft about defining Forefront configuration guidelines for these scenarios.