Wednesday, August 24, 2011
I spent the last few hours debugging an issue where a query in Windows Azure table storage was not returning any results, even though I knew that data was there. It didn't start that way of course. Rather, stuff that should have been working and previously was working, just stopped working. Tracing through the code and debugging showed me it was a case of a method not returning data when it should have.
Now, I have known for quite some time that you must handle continuation tokens and you can never assume that a query will return data always (Steve talks about it waaaay back when here). However, what I did not know was that different methods of enumeration will give you different results. Let me explain by showing the code.
var q = this.CreateQuery()
.Where(filter)
.Where(f => f.PartitionKey.CompareTo(start.GetTicks()) > 0)
.Take(1)
.AsTableServiceQuery();
var first = q.FirstOrDefault();
if (first != null)
{
return new DateTime(long.Parse(first.PartitionKey));
}
In this scenario, you would assume that you have continuation tokens nailed because you have the magical AsTableServiceQuery extension method in use. It will magically chase the tokens until conclusion for you. However, this code does not work! It will actually return null in cases where you do not hit the partition server that holds your query results on the first try.
I could easily reproduce the query in LINQPad:
var q = ctx.CreateQuery<Foo>("WADPerformanceCountersTable")
.Where(f => f.RowKey.CompareTo("9232a4ca79344adf9b1a942d37deb44a") > 0 && f.RowKey.CompareTo("9232a4ca79344adf9b1a942d37deb44a__|") < 0)
.Where(f => f.PartitionKey.CompareTo(DateTime.Now.AddDays(-30).GetTicks()) > 0)
.Take(1)
.AsTableServiceQuery()
.Dump();
Yet, this query worked perfectly. I got exactly 1 result as I expected. I was pretty stumped for a bit, then I realized what was happening. You see FirstOrDefault will not trigger the enumeration required to generate the necessary two round-trips to table storage (first one gets continuation token, second gets results). It just will not force the continuation token to be chased. Pretty simple fix it turns out:
var first = q.AsEnumerable().SingleOrDefault();
Hours wasted for that one simple line fix. Hope this saves someone the pain I just went through.
Thursday, July 14, 2011
I was working on a Windows Azure website solution the other day and suddenly started getting this error when I tried to run the site with a debugger:
This error is one of the hardest to diagnose. Typically, it means that there is something crashing in your website before the debugger can attach. A good candidate to check is your global.asax to see if you have changed anything there. I knew that the global.asax had not been changed, so it was puzzling. Naturally, I took the normal course of action:
- Run the website without debug inside the emulator.
- Run the website with and without debugging outside the emulator.
- Tried it on another machine
None of these methods gave me any clue what the issue was as they all worked perfectly fine. It was killing me that it only happened on debugging inside the emulator and only on 1 machine (the one I really wanted to work). I was desperately looking for a solution that did not involve rebuilding the machine. I turned on SysInternal's DebugView to see if there were some debug messages telling me what the message was. I saw an interesting number of things, but nothing that really stood out as the source of the error. However, I did notice the process ID of what appeared to be reporting errors:
Looking at Process Explorer, I found this was for DFAgent.exe (the Dev Fabric Agent). I could see that it was starting with an environment variable, so I took a look at where that was happening:
That gave me a direction to start looking. I opened the %UserProfile%\AppData\Local\Temp directory and found a conveniently named file there called Visual Studio Web Debugger.log.
A quick look at it showed it to be HTML, so one rename later and viola!
One of our developers had overridden the <httpErrors> setting in web.config that was disallowed on my 1 machine. I opened my applicationHost.config using a Administatrive Notepad and sure enough:
So, the moral of the story is next time, just take a look at this log file and you might find the issue. I suspect the reason that this only happened on debug and not when running without the debugger was that for some reason the debugger is looking for a file called debugattach.aspx. Since this file does not exist on my machine, it throws a 404, which in turn tries to access the <httpErrors> setting, which culminates in the 500.19 server error. I hope this saves someone the many hours I spent finding it and I hope it prevents you from rebuilding your machine as I almost did.
Tuesday, May 17, 2011
One of the common patterns I have noticed across many customers is the desire to download a resource from the web and execute it as part of the bootup process for a Windows Azure web or worker role. The resource could be any number of things (e.g. exe, zip, msi, cmd, etc.), but typically is something that the customer does not want to package directly in the deployment package (cspkg) for size or update/maintainability reasons.
While the pattern is pretty common, the ways to approach the problem can certainly vary. A more complete solution will need to deal with the following issues:
- Downloading from arbitrary http(s) sources or Windows Azure blob storage
- Logging
- Parsing configuration from the RoleEnvironment or app.config
- Interacting with the RoleEnvironment to get ports, DIP addresses, and Local Resource paths
- Unzipping resources
- Launching processes
- Ensuring that resources are only installed once (or downloaded and unzipped once)
With these goals in mind, we built the Windows Azure Bootstrapper. It is a pretty simple tool to use and requires only packaging of the .exe and the .config file itself in your role. With these two items in place, you can then script out fairly complicated installations. For example, you could prepare your roles with MVC3 using a command like this:
bootstrapper.exe -get http://download.microsoft.com/download/F/3/1/F31EF055-3C46-4E35-AB7B-3261A303A3B6/AspNetMVC3ToolsUpdateSetup.exe -lr $lr(temp) -run $lr(temp)\AspNetMVC3ToolsUpdateSetup.exe -args /q
Check out the project page for more examples, but the possibilities are pretty endless here. One customer uses the Bootstrapper to download agents and drivers from their blob storage account to install at startup for their web and worker roles. Other folks use it to simply copy files out of blob storage and lay them out correctly on disk.
Of course, none of this would be available in the community if not for the great guys working at National Instruments. They allowed us to take this code written for them and turn it over to the community.
Enjoy and let us know your feedback or any bugs you find.
Tuesday, February 22, 2011
Just before I left Microsoft, I got a new Windows Phone 7 device. I duly synced it to the company Exchange server and forgot about it. Once I left Microsoft, I simply deleted the connection in WP7. That turned out to be a big mistake. Not only did I lose the email and calendar (which I expected), but it also blew away all my contacts on the phone as well. That was tragic. You see, I had been building these contacts in my Outlook and phones for years. These were mostly personal contacts and lasted across many jobs. These contacts existed nowhere else in the same form. Heck, I didn't even have my brother's phone number as I only use my cell phone as my number memory.
Well, it turns out that my old Windows Mobile phone was my saving grace. Since I had recently upgraded phones to my WP7 device, I had simply turned it off. However, it had a complete copy of my contacts. Here is how I fixed it (and will never have this issue again).
First, you need to find and install the excellent little app called PIM Backup. This app look pretty sophisticated as it will backup pretty much everything on the device, including the contacts which is what I cared about. Once you backup the device, you should go ahead and delete the Exchange partnership on the phone. This will wipe the device of all your contacts. You should simply go and restore the backup at this point and your contacts will be back and in good shape on the Windows Mobile phone.
Next, you should create a new Exchange Sync partnership on Windows Mobile and this time use Google. It is super simple to setup and there are numerous guides (I followed this one). You just need to remember to use your full email address for the username and no domain. I chose to only sync contacts and within a minute, I had all of my phone's contacts in Google Contacts.
What followed for the next 30 mins or so was a simple cleanup of years of contact data. I made extensive use of the 'Find & Merge Duplicates' option in Google Contacts.
Now, WP7 phones will automatically sync with Google Contacts, so I thought everything would be great. However, there is one complication with that. By default, WP7 treats Windows Live as special. It appears that you cannot choose to ignore your Windows Live contacts and only use the Google Contacts. This leads to an ugly mess on the phone as you will potentially have duplicate accounts or partial data in each.
The solution is to make sure that your Windows Live account is up to date with the right contacts as well. For me, this was a simple as opening Windows Live Mail, deleting every single contact in it and importing the contacts from Google Contact. You can export from Google Contacts pretty easily, so this was a very quick operation. In about 5 mins time, my WP7 phone was synced and ready with my years of contacts. Furthermore, this shouldn't be an issue anymore as I know have the contacts stored in both Google and Windows Live.
Next time, I will remember to delete partnerships with great care.
Tuesday, January 11, 2011
As some of you may know, I recently left Microsoft after almost 4 years. It was a wonderful experience and I met and worked with some truly great people there. I was lucky enough to join at the time when the cloud was just getting started at Microsoft. Code words like CloudDB, Strata, Sitka, Red Dog, and others I won't mention here flew around like crazy.
I started as the SQL Server Data Services (SSDS) evangelist and later became the Windows Azure Evangelist (after SSDS eventually became SQL Azure). For the last 2 years, Windows Azure has been my life and it was great seeing a technology very rapidly grow from CTP to a full featured platform in such a short time.
Well, I am not moving far from Windows Azure - I still strongly believe in what Microsoft has done here. I recently joined Cumulux as Director of Cloud Services. I will be driving product strategy, some key customer engagements, and take part in the leadership team there. Cumulux is a close Microsoft partner with a Windows Azure focus, so it is a great fit for me for both personal as well as professional reasons.
While I will miss the great folks I got to work with at Microsoft and being in the thick of everything, I am very excited to begin this new career path with Cumulux.
Sunday, December 19, 2010
Update: Looks like Wade also blogged about this, showing a similar way with some early scripts I wrote. However, there are some important differences around versioning of WebDeploy and creating users that you should pay attention to here. Also, I am using plugins, which are much easier to consume.
One of the features announced at PDC was the coming ability to use the standard IIS7 WebDeploy capability with Windows Azure. This is exciting news, but it comes with an important caveat. This feature is strictly for development purposes only. That is, you should not expect anything you deploy using WebDeploy to persist longterm on your running Windows Azure instances. If you are familiar with Windows Azure, you know the reason is that any changes to the OS post-startup are not durable.
So, the use case for WebDeploy in Windows Azure is in the case of a single instance of a role during development. The idea is that you would:
- Deploy your hosted service (using the .cspkg upload and deploy) with a single instance with WebDeploy enabled.
- Use WebDeploy on subsequent deploys for instant feedback
- Deploy the final version again using .cspkg (without WebDeploy enabled) so the results are durable with at least 2 instances.
The Visual Studio team will shortly be supplying some tooling to make this scenario simple. However, in the interim, it is relatively straightforward to implement this yourself and do what the tooling will eventually do for you.
If you look at the Windows Azure Training Kit, you will find the Advanced Web and Worker Lab Exercise 3 and it will show you the main things to get this done. You simply need to extrapolate from this to get the whole thing working.
We will be using Startup Tasks to perform a little bit of bootstrapping when the role instance (note the singular) starts in order to use Web Deploy. This will be implemented using the Role Plugin feature to make this a snap to consume. Right now, the plugins are undocumented, but if you peruse your SDK bin folder, it won't be too hard to figure out how they work.
The nice thing about using a plugin is that you don't need to litter your service definition with anything in order to use the feature. You simply need to include a single "Import" element and you are done!
Install the Plugin
In order to use this feature, simply extract the contents of the zip file into "%programfiles%\Windows Azure SDK\v1.3\bin\plugins\WebDeploy". You might have to extract the files locally in your profile first and copy them into this location if you run UAC. At the end of the day, you need a folder called "WebDeploy" in this location with all the files in this zip in it.
Use the Plugin
To use the plugin you simply need to add one line to your Service Definition:
<Imports>
<Import moduleName="RemoteAccess" />
<Import moduleName="RemoteForwarder" />
<Import moduleName="WebDeploy"/>
</Imports>
Notice, in my example, I am also including the Remote Access and Remote Forwarder plugins as well. You must have RemoteAccess enabled to use this plugin as we will rely on the user created here. The Remote Forwarder is required on one role in your solution.
Next, you should hit publish on the Windows Azure project (not Web Site) and setup an RDP user. We will be using this same user later in order to deploy because by default the RDP user is also an Admin with permission to use Web Deploy. Alternatively, you could create an admin user with a Startup Task, but this method is easier (and you get RDP). If you have not setup the certificates for RDP access before, it is a one time process outlined here.
Now, you publish this to Windows Azure using the normal Windows Azure publishing process:
That's it. Your running instance has WebDeploy and the IIS Management Service running now. All you had to do was import the WebDeploy plugin and make sure you also used RDP (which most developers will enable during development anyway).
At this point, it is a pretty simple matter to publish using WebDeploy. Just right click the Web Site (not the Cloud Project) and hit publish:
You will need to type in the name of your .cloudapp.net project in the Service URL. By default, it will assume port 8172; if you have chosen a different public VIP port (by editing the plugin -see *hint below), here is where you need to update it (using the full https:// syntax).
Next, you need to update the Site/Application. The format for this is ROLENAME_IN_0_SITENAME, so you need to look in your Service Definition to find this:
Notice, in my example, my ROLENAME is "WebUX" and the Site name is "Web". This will differ for each Web Role potentially, so make sure you check this carefully.
Finally, check the "Allow untrusted certificate" option and use the same RDP username and password you created on deploy. That's all. It should be possible for you to use Web Deploy now with your running instance: Hit Publish and Done!
How it works
If you check the plugin, you will see that we use 2 scripts and WebPI to do this. Actually, this is the command line version of WebPI, so it will run without prompts. You can also download it, but it is packaged already in the plugin.
The first script, called EnableWebAdmin.cmd simply enables the IIS Management service and gets it running. By default, this sets up the service to run and listen on port 8172. If you check the .csplugin file, you will notice we have opened that port on the Load Balancer as well.
start /w ocsetup IIS-ManagementService
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WebManagement\Server /v EnableRemoteManagement /t REG_DWORD /d 1 /f
net start wmsvc
sc config WMSVC start= auto
exit /b 0
*Hint: if you work at a place like Microsoft that prevents ports other than 443 to host SSL traffic, here would be a good place to map 443 to 8172 in order to deploy. For most normal environments, this is not necessary.
Next, we have the proper WebDeploy installation command using WebPI command line.
@echo off
ECHO "Starting WebDeploy Installation" >> log.txt
"%~dp0webpicmd\WebPICmdLine.exe" /Products: WDeploy /xml:https://www.microsoft.com/web/webpi/2.0/RTM/WebProductList.xml /log:webdeploy.txt
ECHO "Completed WebDeploy Installation" >> log.txt
net stop wmsvc
net start wmsvc
You will notice one oddity here - namely, we are using the WebPI 2.0 feed to install an older version of WebDeploy (v1.1). If you leave this off, it will default to the 3.0 version of WebPI that uses the Web Deploy 2.0 RC. In my testing, Web Deploy 2.0 RC only works sporadically and usually not at all. The 1.1 version is well integrated in VS tooling, so I would suggest this one instead until 2.0 works better.
Also, you will notice that I am stopping and restarting the IIS Management Service here as well. In my testing, WebDeploy was was unreliable on the Windows Server 2008 R2 family (osFamily=2). Starting and stopping the service seems to fix it.
Final Warning
Please bear in mind that this method is only for proto-typing and rapid development. Since the changes are not persisted, they will disappear the next time you are healed by the fabric controller. Eventually, the Visual Studio team will release their own plugin that does essentially the same thing and you can stop using this. In the meantime, have fun!
Tuesday, November 30, 2010
If you haven't already read on the official blog, Windows Azure SDK 1.3 was released (along with Visual Studio tooling). Rather than rehash what it contains, go read that blog post if you want to know what is included (and there are lots of great features). Even better, goto microsoftpdc.com and watch the videos on the specific features.
If you are a user of the Windows Azure MMC or the Windows Azure Service Management Cmdlets, you might notice that stuff gets broken on new installs. The underlying issue is that the MMC snapin was written against the 1.0 version of the Storage Client. With the 1.3 SDK, the Storage Client is now at 1.1. So, this means you can fix this two ways:
- Copy an older 1.2 SDK version of the Storage Client into the MMC's "release" directory. Of course, you need to either have the .dll handy or go find it. If you have the MMC installed prior to SDK 1.3, you probably already have it in the release directory and you are good to go.
- Use Assembly Redirection to allow the MMC or Powershell to call into the 1.1 client instead of the 1.0 client.
To use Assembly Redirection, just create a "mmc.exe.config" file for MMC (or "Powershell.exe.config" for cmdlets) and place it in the %windir%\system32 directory. Inside the .config file, just use the following xml:
<configuration>
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="Microsoft.WindowsAzure.StorageClient"
publicKeyToken="31bf3856ad364e35"
culture="neutral" />
<bindingRedirect oldVersion="1.0.0.0"
newVersion="1.1.0.0"/>
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="Microsoft.WindowsAzure.StorageClient"
publicKeyToken="31bf3856ad364e35"
culture="neutral" />
<publisherPolicy apply="no" />
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>
Tuesday, November 16, 2010
I am back from TechEd EU (and PDC)! It was a long and exciting last few weeks getting first ready for PDC and then flying to Germany to deliver a couple sessions. It was a great, if not hectic experience for the last few weeks. For PDC, I had a part to play in the keynote demos as well as the PDC Workshop on the following Saturday. Those two things together took more time than I would care to admit. Once PDC was a wrap, I got to start building my talks for TechEd EU.
You can find both of my talks on the TechEd Online Site and embedded below. The embedding doesn't work great on the blog because of layout issues I need to fix, but RSS readers should have a better time.
Thanks to everyone who attended these sessions in person and all the nice comments. I have almost completely recovered my voice at this point - my cold last just the duration of the trip of course!
Friday, October 15, 2010
For the 30th Cloud Cover show, Steve and I talked about coordinating work. It is not an entirely new topic as there have been a couple variations on the topic before. However, the solution I coded is actually different than both of those. I chose to simply iterate through all the available instances and fire a WCF message to each one. I could have done this is parallel or used multi-casting (probably more elegant), but for simplicity sake: it just works.
The reason I needed to coordinate work was in context of load testing a site. Someone had asked me about load testing in Windows Azure. The cloud is a perfect tool to use for load testing. Where else can you get a lot of capacity for a short period of time? With that thought in mind, I searched around to find a way to generate a lot of traffic quickly.
I built a simple 2 role solution where a controller could send commands to a worker and the worker in turn would spawn a load testing tool. For this sample, I chose a tool called ApacheBench. You should watch the show to understand why I chose ApacheBench instead of the more powerful WCAT.
Get the source - It's Hammer Time!
PS: Big props go to Steve for AJAX'ing the source up side the head.
Thursday, October 14, 2010
I have been meaning to update my blog for some time with the routing solution I presented in Cloud Cover 24. One failed laptop harddrive later and well. I lost my original post and didn't get around until now to try to replace it. Anyhow, enough of my sad story.
One of the many great benefits of using Windows Azure is that networking is taken care of for you. You get connected to a load balancer for all your input endpoints and we automatically round-robin the requests to all your instances that are declared to be part of the role. Visually, it is pretty easy to see:
However, this introduces a slight issue common to all webfarm scenarios: state doesn't work very well. If your service likes to rely on the fact that it will communicate with exactly the same machine as last time, you are in trouble. In fact, this is why you will always hear an overarching theme of designing to be stateless in the cloud (or any scale-out architecture).
There are inevitably scenarios where sticky sessions (routing when possible to the same instance) will benefit a given architecture. We talked a lot about this on the show. In order to produce sticky sessions in Windows Azure, you need a mechanism to route to a given instance. For the show, I talked about how to do this with a simple socket listener.
We introduce a router between the Load Balancer and the Web Roles in this scenario. The router will peek (1) the incoming headers to determine via cookies or other logic where to route the request. It then forwards the traffic over sockets to the appropriate server (2). For all return responses, the router will optionally peek that stream as well to determine whether to inject the cookie (3). Finally, if necessary, the router will inject a cookie that states where the user should be routed to in subsequent requests (4).
I should note that during the show we talked about how this cookie injection could work, but the demo did not use it. For this post, I went ahead and updated the code to actually perform the cookie injection. This allows me to have a stateless router as well. In the demo on the show, I had a router that simply used a dictionary lookup of SessionIDs to Endpoints. However, we pointed out that this was problematic without externalizing that state somewhere. A simple solution is to have the client deliver to the router via a cookie where it was last connected. This way the state is carried on each request and is no longer the router's responsibility.
You can download the code for this from Code Gallery. Please bear in mind that this was only lightly tested and I make no guarantees about production worthiness.