On 1/18 we quietly released a version of our scalable task scheduler (creatively named 'Scheduler' for right now) to the Windows Azure Store.  If you missed it, you can see it in this post by Scott Guthrie.  The service allows you to schedule re-occurring tasks using the well-known cron syntax.  Today, we support a simple GET webhook that will notify you each time your cron fires.  However, you can be sure that we are expanding support to more choices, including (authenticated) POST hooks, Windows Azure Queues, and Service Bus Queues to name a few.

In this post, I want to share a bit about how we designed the service to support many tenants and potentially millions of tasks.  Let's start with a simplified, but accurate overall picture:

We have several main subsystems in our service (REST API façade, CRON Engine, and Task Engine) and additionally several shared subsystems across additional services (not pictured) such as Monitoring/Auditing and Billing/Usage.  Each one can be scaled independently depending on our load and overall system demand.  We knew that we needed to decouple our subsystems such that they did not depend on each other and could scale independently.  We also wanted to be able to develop each subsystem potentially in isolation without affecting the other subsystems in use.  As such, our systems do not communicate with each other directly, but only share a common messaging schema.  All communication is done over queues and asynchronously.

REST API

This is the layer that end users communicate with and the only way to interact with the system (even our portal acts as a client).  We use a shared secret key authentication mechanism where you sign your requests and we validate them as they enter our pipeline.  We implemented this REST API using Web API.  When you interact with the REST API, you are viewing fast, lightweight views of your scheduled task setup that reflects what is stored in our Job Repository.  However, we never query the Job Repository directly to keep it responsive to its real job - providing the source data for the CRON Engine.

CRON Engine

This subsystem was designed to do as little as possible and farm out the work to the Task Engine.  When you have an engine that evaluates cron expressions and fire times, it cannot get bogged down trying to actually do the work.  This is a potentially IO-intensive role in the subsystem that is constantly evaluating when to fire a particular cron job.  In order to support many tenants, it must be able run continuously without bogging down in execution.  As such, this role only evaluates when a particular cron job must run and then fires a command to the Task Engine to actually execute the potentially long running job.

Task Engine

The Task Engine is the grunt of the service and it performs the actual work.  It is the layer that will be scaled most dramatically depending on system load.  Commands from the CRON Engine for work are accepted and performed at this layer.  Subsequently, when the work is done it emits an event that other interested subsystems (like Audit and Billing) can subscribe to downstream.  The emitted event contains details about the outcome of the task performed and is subsequently denormalized into views that the REST API can query to provide back to a tenant.  This is how we can tell you your job history and report back any errors in execution.  The beauty of the Task Engine emitting events (instead of directly acting) is that we can subscribe many different listeners for a particular event at any time in the future.  In fact, we can orchestrate very complex workflows throughout the system as we communicate to unrelated, but vital subsystems.  This keeps our system decoupled and allows us to develop those other subsystems in isolation.

Future Enhancements

Today we are in a beta mode, intended to give us feedback about the type of jobs, frequency of execution, and what our system baseline performance should look like.  In the future, we know we will support additional types of scheduled tasks, more views into your tasks, and more complex orchestrations.  Additionally, we have setup our infrastructure such that we can deploy to to multiple datacenters for resiliency (and even multiple clouds).  Give us a try today and let us know about your experience.

This is a quick post today that might save folks the same trouble I had to go through when upgrading my Windows Identity Foundation (WIF) enabled MVC website to the latest version of .NET.  The scenario is that you might want to enrich the claims coming from your STS with additional claims of your choosing.  To do this, there is a common technique of creating a class the derives from ClaimsAuthenticationManager and overrides the Authenticate method.  Consider this sample ClaimsAuthenticationManager:

The issue we have is that we need to provide an implementation of ITenantRepository here in order to lookup the data for the additional claims we are adding.  If you are lucky enough to find the article on MSDN, it will show you how to wire in a custom ClaimsAuthenticationManager using the web.config.  I don't want to hardcode references to an implementation of my TenantRepository, so using config is not a great option for me.

In the older WIF model (Microsoft.IdentityModel) for .NET <= 4.0, you hooked the ServiceConfigurationCreated event:

But, in .NET 4.5, all of the namespaces and a lot of the classes are updated (System.IdentityModel).  It took me a long time in Reflector to figure out how to hook the configuration being created again.  Turns out you need to reference System.IdentityModel.Services and find the FederatedAuthentication class.  Here you go:

Happy WIF-ing.

I am excited to announce that I have officially joined Aditi Technologies as Director of Product Services.  Taking what I have learned building large scale solutions in Windows Azure, I will be responsible for building Aditi's own portfolio of SaaS services and IP/frameworks.  We have a number of exciting projects underway and I hope to blog more about what we are building soon.

Along with this move, I get to rejoin Wade (now my boss!) and Steve as well as some of my former Cumulux colleagues.  I took this role because I see a great opportunity to build software and services in the 'cloud' and I am convinced that Aditi has been making the right investments.  It doesn't hurt at all that I get to work with top-notch technologists either.

Along the way, I plan to build a team to deliver on these cloud services.  If you think you have what it takes to build great software, send me a note and your resume.  Thanks!

At this point in our diagnostics saga, we have our instances busily pumping out the data we need to manage and monitor our services.  However, it is simply putting the raw data in our storage account(s).  What we really want to do is query and analyze that data to figure out what is happening.

The Basics

Here I am going to show you the basic code for querying your data.  For this, I am going to be using LINQPad.  It is a tool that is invaluable for ad hoc querying and prototyping.  You can cut & paste the following script (hit F4 and add references and namespaces for Microsoft.WindowsAzure.StorageClient.dll and System.Data.Service.Client.dll as well).

          void Main() { var connectionString
= "DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=yourkey";
var account = CloudStorageAccount.Parse(connectionString); var client = account.CreateCloudTableClient();
var ctx = client.GetDataServiceContext(); var deploymentId = new Guid("25d676fb-f031-42b4-aae1-039191156d1a").ToString("N").Dump();
var q = ctx.CreateQuery<PerfCounter>("WADPerformanceCountersTable")
.Where(f => f.RowKey.CompareTo(deploymentId) > 0 && f.RowKey.CompareTo(deploymentId
+ "__|") < 0) .Where(f => f.PartitionKey.CompareTo(DateTime.Now.AddHours(-2).GetTicks())
> 0) //.Take(1) .AsTableServiceQuery() .Dump(); //(q
as DataServiceQuery<Foo>).RequestUri.AbsoluteUri.Dump(); //(q
as CloudTableQuery<Foo>).Expression.Dump(); } staticclass Funcs
{ publicstaticstring GetTicks(this DateTime
dt) { return dt.Ticks.ToString("d19");
} } [System.Data.Services.Common.DataServiceKey("PartitionKey", "RowKey")] class PerfCounter
{ publicstring PartitionKey {
get; set; } publicstring RowKey
{ get; set; } public DateTime Timestamp { get; set; } publiclong EventTickCount
{ get; set; } publicstring Role
{ get; set; } publicstring DeploymentId
{ get; set; } publicstring RoleInstance
{ get; set; } publicstring CounterName
{ get; set; } publicstring CounterValue
{ get; set; } publicint Level
{ get; set; } publicint EventId
{ get; set; } publicstring Message
{ get; set; } }

What I have done here is setup a simple script that allows me to query the table storage location for performance counters.  There are two big (and 1 little) things to note here:

1. Notice how I am filtering down to the deployment ID (also called Private ID) of the deployment I am interested in seeing.  If you use same storage account for multiple deployments, this is critical.
2. Also, see how I have properly formatted the DateTime such that I can select a time range from the Partition Key appropriated.  In this example, I am retrieving the last 2 hours of data for all roles in the selected deployment.
3. I have also commented out some useful checks you can use to test your filters.  If you uncomment the DataServiceQuery<T> line, you also should comment out the .AsTableServiceQuery() line.

Using the Data

If you haven't set absurd sample rates, you might actually get this data back in a reasonable time.  If you have lots of performance counters to monitor and/or you have high sample rates, be prepared to sit and wait for awhile.  Each tick is a single row in table storage.  You can return 1000 rows in a single IO operation.  It can take a very long time if you ask for large time ranges or have lots of data.

Once you have the query returned, you can actually export it into Excel using LINQPad and go about setting up graphs and pivot tables, etc.  This is all very doable, but also tedious.  I would not recommend this for long term management, but rather some simple point in time reporting perhaps.

For AzureOps.com, we went a bit further.  We collect the raw data, compress, and index it for highly efficient searches by time.  We also scale the data for the time range, otherwise you can have a very hard time graphing 20,000 data points.  This makes it very easy to view both recent data (e.g. last few hours) as well as data over months.  The value of the longer term data cannot be overstated.

Anyone that really wants to know what their service has been doing will likely need to invest in monitoring tools or services (e.g. AzureOps.com).  It is simply impractical to pull more than a few hours of data by querying the WADPeformanceCountersTable directly.  It is way too slow and way too much data for longer term analysis.

The Importance of Long Running Data

For lots of operations, you can just look at the last 2 hours of your data and see how your service has been doing.  We put that view as the default view you see when charting your performance counters in AzureOps.com.  However, you really should back out the data from time to time and observe larger trends.  Here is an example:

This is actual data we had last year during our early development phase of the backend engine that processes all the data.  This is the Average CPU over 8 hours and it doesn't look too bad.  We really can't infer anything from this graph other than we are using about 15-35% of our CPU most of the time.

However, if we back that data out a bit.:

This picture tells a whole different story.  We realized that we were slowly doing more and more work with our CPU that did not correlate with the load.  This was not a sudden shift that happened in a few hours.  This was manifesting itself over weeks.  Very slow, for the same amount of operations, we were using more CPU.  A quick check on memory told us that we were also chewing up more memory:

We eventually figured out the issue and fixed it (serialization issue, btw) - can you tell where?

Eventually, we determined what our threshold CPU usage should be under certain loads by observing long term trends.  Now, we know that if our CPU spikes above 45% for more than 10 mins, it means something is amiss.  We now alert ourselves when we detect high CPU usage:

Similarly, we do this for many other counters as well.  There is no magic threshold to choose, but if you have enough data you will be able to easily pick out the threshold values for counters in your own application.

In the next post, I will talk about how we pull this data together analyzers, notifications, and automatically scale to meet demand.

Shameless plug:  Interesting in getting your own data from Windows Azure and monitoring, alerting, and scaling?  Try AzureOps.com for free!

Assuming you know what to monitor and you have configured your deployments to start monitoring, now you need to actually get the data and do something with it.

First, let's briefly recap how the Diagnostics Manager (DM) stores data.  Once it has been configured, the DM will start to buffer data to disk locally on the VM using the temporal scratch disk*.  It will buffer it using the quota policy found in configuration.  By default, this allocates 4GB of local disk space to hold diagnostics data.  You can change the quota with a little more work if you need to hold more, but most folks should be served just fine with the default.  Data is buffered as FIFO (first in, first out) in order to age out the oldest data first.

Scheduled versus OnDemand

Once the data is buffering locally on the VM, you need to somehow transfer the data from the VM to your cloud storage account.  You can do this by either setting a Scheduled or OnDemand transfer.  In practice, I tend to recommend always using Scheduled transfers and ignoring the OnDemand option (it ends up being a lot easier). 

But, for completeness, here is an example of setting an OnDemand transfer:

          void Main() { var account = new CloudStorageAccount( new StorageCredentialsAccountAndKey("dunnry", "yourkey"), true );
var mgr = new DeploymentDiagnosticManager(account, "6468a8b749a54c3..."); foreach (string role in mgr.GetRoleNames())
{ var ridm = mgr.GetRoleInstanceDiagnosticManagersForRole(role); var options = new OnDemandTransferOptions()
{ From = DateTime.UtcNow - TimeSpan.FromMinutes(10), To = DateTime.UtcNow, NotificationQueueName
= "pollme" }; var qc = account.CreateCloudQueueClient();
var q = qc.GetQueueReference("pollme"); q.CreateIfNotExist(); foreach (var
i in ridm) { //cancel all pending transfersforeach (var
pt in i.GetActiveTransfers()) { i.CancelOnDemandTransfers(pt.Key);
} var key = i.BeginOnDemandTransfer(DataBufferName.Logs, options); //poll
here... why bother... } } }

It's not exactly straightforward, but essentially, you need to specify the time range to transfer and optionally a queue to notify when completed.  You must ensure that all outstanding OnDemand transfers are canceled and then you can begin the transfer and ideally you should also cancel the transfer when it is completed.  In theory, this gives you some flexibility on what you want transferred.

As with most things in life, there are some gotchas to using this code.  Most of the time, folks forget to cancel the transfer after it completes.  When that happens, it prevents any updates to the affected data source.  This can impact you when you try to set new performance counters and see an error about an OnDemand transfer for instance.  As such, you end up writing a lot of code to detect and cancel pending transfers first before doing anything else in the API.

Using Scheduled transfers ends up being easier in the long run because you end up getting the same amount of data, but without having the pain of remembering to cancel pending transfers and all that.  Here is similar code (you should adapt for each data source you need to transfer):

          void Main() { var account = new CloudStorageAccount( new StorageCredentialsAccountAndKey("dunnry", "yourkey"), true );
var mgr = new DeploymentDiagnosticManager(account, "6468a8b749a54c3..."); foreach (string role in mgr.GetRoleNames())
{ var ridm = mgr.GetRoleInstanceDiagnosticManagersForRole(role); foreach (var
idm in ridm) { var config = idm.GetCurrentConfiguration()

                ?? DiagnosticMonitor.GetDefaultInitialConfiguration();
            config.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(5);
            
            //set
other scheduled intervals here... idm.SetCurrentConfiguration(config); } }
}

This ends up being the technique we use for AzureOps.com.  When you setup your subscription with us, we detect the diagnostics connection string and allow you to change your data source settings.  For Performance Counters, we force the transfer to 5 minutes to today (a good compromise) and allow you to choose the interval for other sources (i.e. Traces, Windows Event Logs).  When you use a provider like AzureOps, it is usually best to stream the data in in relatively small chunks as opposed to say transferring once an hour.  Firstly, we won't be able to do anything with your data until we see it and you probably want to be notified sooner than 1 time an hour.  Secondly, when you set long transfer period times, there is a risk that you exceed the buffer quota and start to lose data that was never transferred.  In practice, we have not observed any noticeable overhead by transferring more often.  When in doubt, pick 5 mins.

Whew!  If you have made it this far, you now have a reasonable set of performance counters and trace information that is both being collected on your VMs in Windows Azure as well as being persisted to your storage account.  So, essentially, you need to now figure out what to do with that data.  That will be the subject of the next post in this series.

*if you are interested, RDP into an instance and check the resource drive (usually C:) under /Resources/Directory/<roleuniquename>/Monitor to see buffered data.

In order to actually monitor anything in Windows Azure, you need to use the Diagnostics Manager (DM) that ships out of box.  SaaS providers like AzureOps rely on this data in order to tell you how your system is behaving.  The DM actually supports a few data sources that it can collect and transfer:

• Performance Counters
• Trace logs
• IIS Logs
• Event Logs
• Infrastructure Logs
• Arbitrary logs

One of the most common issues I hear from customers is that they don't know how to get started using the DM or they think they are using it and just cannot find the data where they think they should.  Hopefully, this post will clear up a bit about how the DM actually works and how to configure it.  The next post will talk about how to get the data once you are setup.

Setting UP The Diagnostics Manager

Everything starts by checking a box.  When you check the little box in Visual Studio that says "Enable Diagnostics", it actually modifies your Service Definition to include a role plugin.  Role plugins are little things that can add to your definition and configuration similar to a macro.  If you have ever used Diagnostics or the RDP capability in Windows Azure, you have used a role plugin.  For most of their history, these plugins have been exclusively built by Microsoft, but there is really nothing stopping you from using it yourself (that is another topic).

If we check our SDK folder in the plugins directory in the 'diagnostics' folder, you will actually find the magic that is used to launch the DM.

          <?
          xml
          version
          ="1.0" ?><RoleModulexmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition"namespace="Microsoft.WindowsAzure.Plugins.Diagnostics"><Startuppriority="-2"><TaskcommandLine="DiagnosticsAgent.exe"executionContext="limited"taskType="background"/><TaskcommandLine="DiagnosticsAgent.exe
/blockStartup"executionContext="limited"taskType="simple"/></Startup><ConfigurationSettings><Settingname="ConnectionString"/></ConfigurationSettings></RoleModule>

Here, we can see that the DM is implemented as a pair of startup tasks.  Notice, it is using a task type of background (the other is blocking until it gets going).  This means that the DM exists outside the code you write and should be impervious to your code crashing and taking it down as well.  You can also see that the startup tasks listed here will be run with a default priority of -2.  This just tries to ensure that they run before any other startup tasks.  The idea is that you want the DM to start before other stuff so it can collect data for you.

You can also see in the definition the declaration of a new ConfigurationSettings with a single Setting called 'ConnectionString'.  If you are using Visual Studio when you import the Diagnostics plugin, you will see that the tooling automatically combines the namespace with the settings name and creates a new Setting called Microsoft.Windows.Plugins.Diagnostics.ConnectionString.  This setting will not exist if you are building your csdef or cscfg files by hand.  You must remember to include it.

Once you have the plugin actually enabled in your solution, you will need to specify a valid connection string in order for the DM to operate.  Here you have two choices:

1. Running in emulation, it is valid to use "UseDevelopmentStorage=true" as ConnectionString.
2. Before deploying to cloud, you must remember to update that to a valid storage account (i.e. "DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=nrIXB.")

Common Pitfalls

It seems simple enough, but here come the first set of common pitfalls I see:

1. Forgetting to set the ConnectionString to a valid storage account and deploying with 'UseDevelopmentStorage=true'.  This has become less of a factor in 1.6+ SDK tooling because you will notice the checkbox that says, "Use publish storage account as connection string when you publish to Windows Azure".  However, tooling will not help you here for automated deploys or when you forget to check that box.
2. Using "DefaultEndpointsProtocol=http" in the connection string (note the missing 's' from 'https').  While it is technically possible to use the DM with an http connection, it is not worth the hassle.  Just use https and save yourself the hassle of troubleshooting this later.
3. Setting an invalid connection string.  Hard to believe, but I see it all the time now on AzureOps.  This usually falls into two categories: deleting a storage account, and regenerating a storage key.  If you delete a storage account, but forget to remove that as the ConnectionString, things won't work (shocking, I know).  Further, if you decide to regenerate the primary or secondary storage keys and you were using them, stuff won't work here either.  Seems obvious, but you won't actually get any warning on this.  Stuff won't work and you will have to figure that out yourself.  A good 3rd party provider (like AzureOps) will let you know however.
4. Forgetting to co-locate the diagnostics storage account with the hosted service.  This one might not show itself until you see the bill.  The diagnostics agent can be pretty chatty.  I have seen GBs of data logged in a single minute.  Forgetting to co-locate that would run you a pretty hefty bandwidth bill in addition to slowing you down.

Best Practices

Setting up the Diagnostics Manager is not terribly hard, but easy to get wrong if you are not familiar with it.  There are some other subtle things you can do here that will shoot yourself in the foot however.  Here are some things you can do that will make your life easier:

1. Always separate your diagnostics storage account from other storage accounts.  This is especially important for production systems.  You do not want diagnostics competing with your primary storage account for resources.  There is an account wide, 5000 transactions per second limit across tables, queues, and blobs.  When you use a single storage account for both, you could unintentionally throttle your production account.
2. If possible, use a different diagnostics storage account per hosted service.  If that is not practical, at least try to separate storage accounts for production versus non-production systems.  It turns out that querying diagnostics data can be difficult if there are many different systems logging to the same diagnostics tables.  What I have seen many times is someone use the same diagnostics account for load testing against non-production systems as their production system.  What happens is that the amount of data for the non-production system can greatly exceed the production systems.  The query mechanism for then finding production data is akin to finding a needle in the haystack.  It can take a very long time in some cases to query even for simple things.
3. Don't' use the Anywhere location.  This applies to all storage accounts and all hosted services.  This might seem obvious, but I see it all the time.  It is possible to use Anywhere location with affinity groups and avoid pitfall #4, but it is not worth the hassle.  Additionally, if you have a 3rd party (like AzureOps) that is monitoring your data, we cannot geo-locate a worker next to you to pull your data.  We won't know where you are located and it could mean big bandwidth bills for you.

At this point, if you have enabled the DM, and remembered to set a valid connection string, you are almost home.  The last thing to do is actually get the data and avoid common pitfalls there.  That is the topic for the next post.

One of the first questions I often get when onboarding a customer is "What should I be monitoring?".  There is no definitive list, but there are certainly some things that tend to be more useful.  I recommend the following Performance Counters in Windows Azure for all role types at bare minimum:

• \Processor(_Total)\% Processor Time
• \Memory\Available Bytes
• \Memory\Committed Bytes
• \.NET CLR Memory(_Global_)\% Time in GC

You would be surprised how much these 4 counters tell someone without any other input.  You can see trends over time very clearly when monitoring over weeks that will tell you what is a 'normal' range that your application should be in.  If you start to see any of these counters spike (or spike down in the case of Available Memory), this should be an indicator to you that something is going on that you should care about.

Web Roles

For ASP.NET applications, there are some additional counters that tend to be pretty useful:

• \ASP.NET Applications(__Total__)\Requests Total
• \ASP.NET Applications(__Total__)\Requests/Sec
• \ASP.NET Applications(__Total__)\Requests Not Authorized
• \ASP.NET Applications(__Total__)\Requests Timed Out
• \ASP.NET Applications(__Total__)\Requests Not Found
• \ASP.NET Applications(__Total__)\Request Error Events Raised
• \Network Interface(*)\Bytes Sent/sec

If you are using something other than the latest version of .NET, you might need to choose the version specific instances of these counters.  By default, these are going to only work for .NET 4 ASP.NET apps.  If you are using .NET 2 CLR apps (including .NET 3.5), you will want to choose the version specific counters.

The last counter you see in this list is somewhat special as it includes a wildcard instance (*).  This is important to choose in Windows Azure as the names of the actual instance adapter can (and tends to) change over time and deployments.  Sometimes it is "Local Area Connection* 12", sometimes it is "Microsoft Virtual Machine Bus Network Adapter".  The latter one tends to be the one that you see most often with data, but just to be sure, I would include them all.  Note, this is not an exhaustive list - if you have custom counters or additional system counters that are meaningful, by all means, include them.  In AzureOps, we can set these remotely on your instances using the property page for your deployment.

Choosing a Sample Rate

You should not need to sample any counter faster than 30 seconds.  Period.  In fact, in 99% of all cases, I would actually recommend 120 seconds (that is our default we recommend in AzureOps).  This might seem like you are losing too much data or that you are going to miss something.  However, experience has shown that this sample rate is more than sufficient to monitor the system over days, weeks, and months with enough resolution to know what is happening in your application.  The difference between 30 seconds and 120 seconds is 4 times as much data.  When you sample at 1 and 5 second sample rates, you are talking about 120x and 24x the amount of data.  That is per instance, by the way.  If you are have more than 1 instance, now multiply that by number of instances.  It will quickly approach absurd quantities of data that costs you money in transactions and storage to store, and that has no additional value to parse, but a lot more pain to keep.  Resist the urge to put 1, 5, or even 10 seconds - try 120 seconds to start and tune down if you really need to.

Tracing

The other thing I recommend for our customers is to use tracing in their application.  If you only use the built-in Trace.TraceInformation (and similar), you are ahead of the game.  There is an excellent article in MSDN about how to setup more advanced tracing with TraceSources that I recommend as well.

I recommend using tracing for a variety of reasons.  First, it will definitely help you when your app is running in the cloud and you want to gain insight into issues you see.  If you had logged exceptions to Trace or critical code paths to Trace, then you now have potential insight into your system.  Additionally, you can use this as a type of metric in the system to be mined later.  For instance, you can log length of time a particular request or operation is taking.  Later, you can pull those logs and analyze what was the bottleneck in your running application.  Within AzureOps, we can parse trace messages in variety of ways (including semantically).  We use this functionality to alert ourselves when something strange is happening (more on this in a later post).

The biggest obstacle I see with new customers is remembering to turn on transfer for their trace messages.  Luckily, within AzureOps, this is again easy to do.  Simply set a Filter Level and a Transfer Interval (I recommend 5 mins).

The Filter Level will depend a bit on how you use the filtering in your own traces.  I have seen folks that trace rarely, so lower filters are fine.  However, I have also seen customers trace upwards of 500 traces/sec.  As a point of reference, at that level of tracing, you are talking about 2GB of data on the wire each minute if you transfer at that verbosity.  Heavy tracers, beware!  I usually recommend verbose for light tracers and Warning for tracers that are instrumenting each method for instance.  You can always change this setting later, so don't worry too much right now.

Coming Up

In the next post, I will walk you through how to setup your diagnostics in Windows Azure and point out some common pitfalls that I see.

For the last year since leaving Microsoft, I have been deeply involved in building a world class SaaS monitoring service called AzureOps.  During this time, it was inevitable that I see not only how to best monitor services running in Windows Azure, but also see the common pitfalls amongst our beta users.  It is one thing to be a Technical Evangelist like I was and occasionally use a service for a demo or two, and quite another to attempt to build a business on it.

Monitoring a running service in Windows Azure can actually be daunting if you have not worked with it before.  In this coming series, I will attempt to share the knowledge we have gained building AzureOps and from our customers.  The series will be grounded in these 5 areas:

1. Choosing what to monitor in Windows Azure.
2. Setting up the Diagnostics Monitor in Windows Azure
3. Getting the diagnostics data from Windows Azure
4. Interpreting diagnostics data and making adjustments
5. Maintaining your service in Windows Azure.

Each one of these areas will be a post in the series and I will update this post to keep a link to the latest.  I will use AzureOps as an example in some cases to highlight both what we learned as well as the approach we take now due to this experience.

If you are interested in monitoring your own services in Windows Azure, grab an invite and get started today!.

I spent the last few hours debugging an issue where a query in Windows Azure table storage was not returning any results, even though I knew that data was there.  It didn't start that way of course.  Rather, stuff that should have been working and previously was working, just stopped working.  Tracing through the code and debugging showed me it was a case of a method not returning data when it should have.

Now, I have known for quite some time that you must handle continuation tokens and you can never assume that a query will return data always (Steve talks about it waaaay back when here).  However, what I did not know was that different methods of enumeration will give you different results.  Let me explain by showing the code.

          
var q = this.CreateQuery() 
          
    .Where(filter)

          
    .Where(f => f.PartitionKey.CompareTo(start.GetTicks()) > 0)

          
    .Take(1)

          
    .AsTableServiceQuery();

          
          
          
var first = q.FirstOrDefault();

          
            if (first
!= null) 
          
{

          
            return
            new DateTime(long.Parse(first.PartitionKey)); 
          
}
        

In this scenario, you would assume that you have continuation tokens nailed because you have the magical AsTableServiceQuery extension method in use.  It will magically chase the tokens until conclusion for you.  However, this code does not work!  It will actually return null in cases where you do not hit the partition server that holds your query results on the first try.

I could easily reproduce the query in LINQPad:

          
var q = ctx.CreateQuery<Foo>("WADPerformanceCountersTable") 
          
    .Where(f => f.RowKey.CompareTo("9232a4ca79344adf9b1a942d37deb44a")
> 0 && f.RowKey.CompareTo("9232a4ca79344adf9b1a942d37deb44a__|")
< 0) 
          
    .Where(f => f.PartitionKey.CompareTo(DateTime.Now.AddDays(-30).GetTicks()) > 0)

          
    .Take(1)

          
    .AsTableServiceQuery()

          
    .Dump();    

          
          
        

Yet, this query worked perfectly.  I got exactly 1 result as I expected.  I was pretty stumped for a bit, then I realized what was happening.  You see FirstOrDefault will not trigger the enumeration required to generate the necessary two round-trips to table storage (first one gets continuation token, second gets results).  It just will not force the continuation token to be chased.  Pretty simple fix it turns out:

          
var first = q.AsEnumerable().SingleOrDefault();
        

Hours wasted for that one simple line fix.  Hope this saves someone the pain I just went through.

I was working on a Windows Azure website solution the other day and suddenly started getting this error when I tried to run the site with a debugger:

This error is one of the hardest to diagnose.  Typically, it means that there is something crashing in your website before the debugger can attach.  A good candidate to check is your global.asax to see if you have changed anything there.  I knew that the global.asax had not been changed, so it was puzzling.  Naturally, I took the normal course of action:

1. Run the website without debug inside the emulator.
2. Run the website with and without debugging outside the emulator.
3. Tried it on another machine

None of these methods gave me any clue what the issue was as they all worked perfectly fine.  It was killing me that it only happened on debugging inside the emulator and only on 1 machine (the one I really wanted to work).  I was desperately looking for a solution that did not involve rebuilding the machine.   I turned on SysInternal's DebugView to see if there were some debug messages telling me what the message was.  I saw an interesting number of things, but nothing that really stood out as the source of the error.  However, I did notice the process ID of what appeared to be reporting errors:

Looking at Process Explorer, I found this was for DFAgent.exe (the Dev Fabric Agent).  I could see that it was starting with an environment variable, so I took a look at where that was happening:

That gave me a direction to start looking.  I opened the %UserProfile%\AppData\Local\Temp directory and found a conveniently named file there called Visual Studio Web Debugger.log. 

A quick look at it showed it to be HTML, so one rename later and viola!

One of our developers had overridden the <httpErrors> setting in web.config that was disallowed on my 1 machine.  I opened my applicationHost.config using a Administatrive Notepad and sure enough:

So, the moral of the story is next time, just take a look at this log file and you might find the issue.  I suspect the reason that this only happened on debug and not when running without the debugger was that for some reason the debugger is looking for a file called debugattach.aspx.  Since this file does not exist on my machine, it throws a 404, which in turn tries to access the <httpErrors> setting, which culminates in the 500.19 server error.  I hope this saves someone the many hours I spent finding it and I hope it prevents you from rebuilding your machine as I almost did.

One of the common patterns I have noticed across many customers is the desire to download a resource from the web and execute it as part of the bootup process for a Windows Azure web or worker role.  The resource could be any number of things (e.g. exe, zip, msi, cmd, etc.), but typically is something that the customer does not want to package directly in the deployment package (cspkg) for size or update/maintainability reasons.

While the pattern is pretty common, the ways to approach the problem can certainly vary.  A more complete solution will need to deal with the following issues:

• Downloading from arbitrary http(s) sources or Windows Azure blob storage
• Logging
• Parsing configuration from the RoleEnvironment or app.config
• Interacting with the RoleEnvironment to get ports, DIP addresses, and Local Resource paths
• Unzipping resources
• Launching processes
• Ensuring that resources are only installed once (or downloaded and unzipped once)

With these goals in mind, we built the Windows Azure Bootstrapper.  It is a pretty simple tool to use and requires only packaging of the .exe and the .config file itself in your role.  With these two items in place, you can then script out fairly complicated installations.  For example, you could prepare your roles with MVC3 using a command like this:

          
bootstrapper.exe -get http://download.microsoft.com/download/F/3/1/F31EF055-3C46-4E35-AB7B-3261A303A3B6/AspNetMVC3ToolsUpdateSetup.exe -lr $lr(temp) -run $lr(temp)\AspNetMVC3ToolsUpdateSetup.exe -args /q

          
          
        

Check out the project page for more examples, but the possibilities are pretty endless here.  One customer uses the Bootstrapper to download agents and drivers from their blob storage account to install at startup for their web and worker roles.  Other folks use it to simply copy files out of blob storage and lay them out correctly on disk.

Of course, none of this would be available in the community if not for the great guys working at National Instruments.  They allowed us to take this code written for them and turn it over to the community.

Enjoy and let us know your feedback or any bugs you find.

Just before I left Microsoft, I got a new Windows Phone 7 device.  I duly synced it to the company Exchange server and forgot about it.  Once I left Microsoft, I simply deleted the connection in WP7.  That turned out to be a big mistake.  Not only did I lose the email and calendar (which I expected), but it also blew away all my contacts on the phone as well.  That was tragic.  You see, I had been building these contacts in my Outlook and phones for years.  These were mostly personal contacts and lasted across many jobs.  These contacts existed nowhere else in the same form.  Heck, I didn't even have my brother's phone number as I only use my cell phone as my number memory.

Well, it turns out that my old Windows Mobile phone was my saving grace.  Since I had recently upgraded phones to my WP7 device, I had simply turned it off.  However, it had a complete copy of my contacts.  Here is how I fixed it (and will never have this issue again).

First, you need to find and install the excellent little app called PIM Backup.  This app look pretty sophisticated as it will backup pretty much everything on the device, including the contacts which is what I cared about.  Once you backup the device, you should go ahead and delete the Exchange partnership on the phone.  This will wipe the device of all your contacts.  You should simply go and restore the backup at this point and your contacts will be back and in good shape on the Windows Mobile phone.

Next, you should create a new Exchange Sync partnership on Windows Mobile and this time use Google.  It is super simple to setup and there are numerous guides (I followed this one).  You just need to remember to use your full email address for the username and no domain.  I chose to only sync contacts and within a minute, I had all of my phone's contacts in Google Contacts.

What followed for the next 30 mins or so was a simple cleanup of years of contact data.  I made extensive use of the 'Find & Merge Duplicates' option in Google Contacts.

Now, WP7 phones will automatically sync with Google Contacts, so I thought everything would be great.  However, there is one complication with that.  By default, WP7 treats Windows Live as special.  It appears that you cannot choose to ignore your Windows Live contacts and only use the Google Contacts.  This leads to an ugly mess on the phone as you will potentially have duplicate accounts or partial data in each.

The solution is to make sure that your Windows Live account is up to date with the right contacts as well.  For me, this was a simple as opening Windows Live Mail, deleting every single contact in it and importing the contacts from Google Contact.  You can export from Google Contacts pretty easily, so this was a very quick operation.  In about 5 mins time, my WP7 phone was synced and ready with my years of contacts.  Furthermore, this shouldn't be an issue anymore as I know have the contacts stored in both Google and Windows Live.

Next time, I will remember to delete partnerships with great care.

As some of you may know, I recently left Microsoft after almost 4 years.  It was a wonderful experience and I met and worked with some truly great people there.  I was lucky enough to join at the time when the cloud was just getting started at Microsoft.  Code words like CloudDB, Strata, Sitka, Red Dog, and others I won't mention here flew around like crazy.

I started as the SQL Server Data Services (SSDS) evangelist and later became the Windows Azure Evangelist (after SSDS eventually became SQL Azure).  For the last 2 years, Windows Azure has been my life and it was great seeing a technology very rapidly grow from CTP to a full featured platform in such a short time.

Well, I am not moving far from Windows Azure - I still strongly believe in what Microsoft has done here.  I recently joined Cumulux as Director of Cloud Services.  I will be driving product strategy, some key customer engagements, and take part in the leadership team there.  Cumulux is a close Microsoft partner with a Windows Azure focus, so it is a great fit for me for both personal as well as professional reasons.

While I will miss the great folks I got to work with at Microsoft and being in the thick of everything, I am very excited to begin this new career path with Cumulux.

Update:  Looks like Wade also blogged about this , showing a similar way with some early scripts I wrote.  However, there are some important differences around versioning of WebDeploy and creating users that you should pay attention to here.  Also, I am using plugins, which are much easier to consume.

One of the features announced at PDC was the coming ability to use the standard IIS7 WebDeploy capability with Windows Azure.  This is exciting news, but it comes with an important caveat.  This feature is strictly for development purposes only.  That is, you should not expect anything you deploy using WebDeploy to persist longterm on your running Windows Azure instances.  If you are familiar with Windows Azure, you know the reason is that any changes to the OS post-startup are not durable.

So, the use case for WebDeploy in Windows Azure is in the case of a single instance of a role during development.  The idea is that you would:

1. Deploy your hosted service (using the .cspkg upload and deploy) with a single instance with WebDeploy enabled.
2. Use WebDeploy on subsequent deploys for instant feedback
3. Deploy the final version again using .cspkg (without WebDeploy enabled) so the results are durable with at least 2 instances.

The Visual Studio team will shortly be supplying some tooling to make this scenario simple.  However, in the interim, it is relatively straightforward to implement this yourself and do what the tooling will eventually do for you.

If you look at the Windows Azure Training Kit, you will find the Advanced Web and Worker Lab Exercise 3 and it will show you the main things to get this done.  You simply need to extrapolate from this to get the whole thing working.

We will be using Startup Tasks to perform a little bit of bootstrapping when the role instance (note the singular) starts in order to use Web Deploy.  This will be implemented using the Role Plugin feature to make this a snap to consume.  Right now, the plugins are undocumented, but if you peruse your SDK bin folder, it won't be too hard to figure out how they work.

The nice thing about using a plugin is that you don't need to litter your service definition with anything in order to use the feature.  You simply need to include a single "Import" element and you are done!

Install the Plugin

In order to use this feature, simply extract the contents of the zip file into "%programfiles%\Windows Azure SDK\v1.3\bin\plugins\WebDeploy".  You might have to extract the files locally in your profile first and copy them into this location if you run UAC.  At the end of the day, you need a folder called "WebDeploy" in this location with all the files in this zip in it.

Use the Plugin

To use the plugin you simply need to add one line to your Service Definition:

          
            <
            Imports
            >
          
          
            <
            Import
            moduleName="RemoteAccess"/>
          
            <
            Import
            moduleName="RemoteForwarder"/>
          
            <
            Import
            moduleName="WebDeploy"/>
          
            </
            Imports
            >
          
        

Notice, in my example, I am also including the Remote Access and Remote Forwarder plugins as well.  You must have RemoteAccess enabled to use this plugin as we will rely on the user created here.  The Remote Forwarder is required on one role in your solution.

Next, you should hit publish on the Windows Azure project (not Web Site) and setup an RDP user.  We will be using this same user later in order to deploy because by default the RDP user is also an Admin with permission to use Web Deploy.  Alternatively, you could create an admin user with a Startup Task, but this method is easier (and you get RDP).  If you have not setup the certificates for RDP access before, it is a one time process outlined here.

Now, you publish this to Windows Azure using the normal Windows Azure publishing process:

That's it.  Your running instance has WebDeploy and the IIS Management Service running now.  All you had to do was import the WebDeploy plugin and make sure you also used RDP (which most developers will enable during development anyway).

At this point, it is a pretty simple matter to publish using WebDeploy.  Just right click the Web Site (not the Cloud Project) and hit publish:

You will need to type in the name of your .cloudapp.net project in the Service URL.  By default, it will assume port 8172; if you have chosen a different public VIP port (by editing the plugin -see *hint below), here is where you need to update it (using the full https:// syntax).

Next, you need to update the Site/Application.  The format for this is ROLENAME_IN_0_SITENAME, so you need to look in your Service Definition to find this:

Notice, in my example, my ROLENAME is "WebUX" and the Site name is "Web".  This will differ for each Web Role potentially, so make sure you check this carefully.

Finally, check the "Allow untrusted certificate" option and use the same RDP username and password you created on deploy.  That's all.  It should be possible for you to use Web Deploy now with your running instance:  Hit Publish and Done!

How it works

If you check the plugin, you will see that we use 2 scripts and WebPI to do this.  Actually, this is the command line version of WebPI, so it will run without prompts.  You can also download it, but it is packaged already in the plugin.

The first script, called EnableWebAdmin.cmd simply enables the IIS Management service and gets it running.  By default, this sets up the service to run and listen on port 8172.  If you check the .csplugin file, you will notice we have opened that port on the Load Balancer as well.

          
start /w ocsetup IIS-ManagementService

          
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WebManagement\Server /v EnableRemoteManagement /t REG_DWORD /d 1 /f

          
net start wmsvc

          
sc config WMSVC start= auto

          
          
          
exit /b 0
        

*Hint: if you work at a place like Microsoft that prevents ports other than 443 to host SSL traffic, here would be a good place to map 443 to 8172 in order to deploy.  For most normal environments, this is not necessary.

Next, we have the proper WebDeploy installation command using WebPI command line.

          
@echo off

          
          
          
ECHO "Starting WebDeploy Installation" >> log.txt

          
"%~dp0webpicmd\WebPICmdLine.exe" /Products: WDeploy /xml:https://www.microsoft.com/web/webpi/2.0/RTM/WebProductList.xml /log:webdeploy.txt

          
            

ECHO "Completed WebDeploy Installation" >> log.txt

            

 

            

net stop wmsvc

            

net start wmsvc

          
        

You will notice one oddity here - namely, we are using the WebPI 2.0 feed to install an older version of WebDeploy (v1.1).  If you leave this off, it will default to the 3.0 version of WebPI that uses the Web Deploy 2.0 RC.  In my testing, Web Deploy 2.0 RC only works sporadically and usually not at all.  The 1.1 version is well integrated in VS tooling, so I would suggest this one instead until 2.0 works better.

Also, you will notice that I am stopping and restarting the IIS Management Service here as well.  In my testing, WebDeploy was was unreliable on the Windows Server 2008 R2 family (osFamily=2).  Starting and stopping the service seems to fix it.

Final Warning

Please bear in mind that this method is only for proto-typing and rapid development.  Since the changes are not persisted, they will disappear the next time you are healed by the fabric controller.  Eventually, the Visual Studio team will release their own plugin that does essentially the same thing and you can stop using this.  In the meantime, have fun!

If you haven't already read on the official blog, Windows Azure SDK 1.3 was released (along with Visual Studio tooling).  Rather than rehash what it contains, go read that blog post if you want to know what is included (and there are lots of great features).  Even better, goto microsoftpdc.com and watch the videos on the specific features.

If you are a user of the Windows Azure MMC or the Windows Azure Service Management Cmdlets, you might notice that stuff gets broken on new installs.  The underlying issue is that the MMC snapin was written against the 1.0 version of the Storage Client.  With the 1.3 SDK, the Storage Client is now at 1.1.  So, this means you can fix this two ways:

1. Copy an older 1.2 SDK version of the Storage Client into the MMC's "release" directory.  Of course, you need to either have the .dll handy or go find it.  If you have the MMC installed prior to SDK 1.3, you probably already have it in the release directory and you are good to go.
2. Use Assembly Redirection to allow the MMC or Powershell to call into the 1.1 client instead of the 1.0 client.

To use Assembly Redirection, just create a "mmc.exe.config" file for MMC (or "Powershell.exe.config" for cmdlets) and place it in the %windir%\system32 directory.  Inside the .config file, just use the following xml:

              <
              configuration
              >
            

              <
              runtime
              >
            

              <
              assemblyBinding
              xmlns
              ="urn:schemas-microsoft-com:asm.v1"
              >
            

              <
              dependentAssembly
              >
            

              <
              assemblyIdentity
              name
              ="Microsoft.WindowsAzure.StorageClient"
            

              publicKeyToken
              ="31bf3856ad364e35"
            

              culture
              ="neutral"
              />
            

              <
              bindingRedirect
              oldVersion
              ="1.0.0.0"
            

              newVersion
              ="1.1.0.0"
              />
            

              </
              dependentAssembly
              >
            

              <
              dependentAssembly
              >
            

              <
              assemblyIdentity
              name
              ="Microsoft.WindowsAzure.StorageClient"
            

              publicKeyToken
              ="31bf3856ad364e35"
            

              culture
              ="neutral"
              />
            

              <
              publisherPolicy
              apply
              ="no"
/
              >
            

              </
              dependentAssembly
              >
            

              </
              assemblyBinding
              >
            

              </
              runtime
              >
            

              </
              configuration
              >
            

I am back from TechEd EU (and PDC)!  It was a long and exciting last few weeks getting first ready for PDC and then flying to Germany to deliver a couple sessions.  It was a great, if not hectic experience for the last few weeks.  For PDC, I had a part to play in the keynote demos as well as the PDC Workshop on the following Saturday.  Those two things together took more time than I would care to admit.  Once PDC was a wrap, I got to start building my talks for TechEd EU.

You can find both of my talks on the TechEd Online Site and embedded below.  The embedding doesn't work great on the blog because of layout issues I need to fix, but RSS readers should have a better time.

Thanks to everyone who attended these sessions in person and all the nice comments.  I have almost completely recovered my voice at this point - my cold last just the duration of the trip of course!

For the 30th Cloud Cover show, Steve and I talked about coordinating work.  It is not an entirely new topic as there have been a couple variations on the topic before.  However, the solution I coded is actually different than both of those.  I chose to simply iterate through all the available instances and fire a WCF message to each one.  I could have done this is parallel or used multi-casting (probably more elegant), but for simplicity sake: it just works.

The reason I needed to coordinate work was in context of load testing a site.  Someone had asked me about load testing in Windows Azure.  The cloud is a perfect tool to use for load testing.   Where else can you get a lot of capacity for a short period of time?  With that thought in mind, I searched around to find a way to generate a lot of traffic quickly.

I built a simple 2 role solution where a controller could send commands to a worker and the worker in turn would spawn a load testing tool.  For this sample, I chose a tool called ApacheBench.  You should watch the show to understand why I chose ApacheBench instead of the more powerful WCAT.

Get the source - It's Hammer Time!

PS: Big props go to Steve for AJAX'ing the source up side the head.

I have been meaning to update my blog for some time with the routing solution I presented in Cloud Cover 24.  One failed laptop harddrive later and well. I lost my original post and didn't get around until now to try to replace it.  Anyhow, enough of my sad story.

One of the many great benefits of using Windows Azure is that networking is taken care of for you.  You get connected to a load balancer for all your input endpoints and we automatically round-robin the requests to all your instances that are declared to be part of the role.  Visually, it is pretty easy to see:

However, this introduces a slight issue common to all webfarm scenarios:  state doesn't work very well.  If your service likes to rely on the fact that it will communicate with exactly the same machine as last time, you are in trouble.  In fact, this is why you will always hear an overarching theme of designing to be stateless in the cloud (or any scale-out architecture).

There are inevitably scenarios where sticky sessions (routing when possible to the same instance) will benefit a given architecture.  We talked a lot about this on the show.  In order to produce sticky sessions in Windows Azure, you need a mechanism to route to a given instance.  For the show, I talked about how to do this with a simple socket listener.

We introduce a router between the Load Balancer and the Web Roles in this scenario.  The router will peek (1) the incoming headers to determine via cookies or other logic where to route the request.  It then forwards the traffic over sockets to the appropriate server (2).  For all return responses, the router will optionally peek that stream as well to determine whether to inject the cookie (3).  Finally, if necessary, the router will inject a cookie that states where the user should be routed to in subsequent requests (4).

I should note that during the show we talked about how this cookie injection could work, but the demo did not use it.  For this post, I went ahead and updated the code to actually perform the cookie injection.  This allows me to have a stateless router as well.  In the demo on the show, I had a router that simply used a dictionary lookup of SessionIDs to Endpoints.  However, we pointed out that this was problematic without externalizing that state somewhere.  A simple solution is to have the client deliver to the router via a cookie where it was last connected.  This way the state is carried on each request and is no longer the router's responsibility.

You can download the code for this from Code Gallery.  Please bear in mind that this was only lightly tested and I make no guarantees about production worthiness.

For Episode 19 of the Cloud Cover show, Steve and I discussed the importance of setting the Content-Type on your blobs in Windows Azure blob storage.  This was was especially important for Silverlight clients.  I mentioned that there was a way to look up a Content Type from your registry as opposed to hardcoding a list.  The code is actually pretty simple.  I pulled this from some code I had lying around that does uploads. 

Here it is:

          
            private
            static
            string GetContentType(string file) 
          
{

          
            string contentType
= "application/octet-stream"; 
          
          
          
            string fileExt
= System.IO.Path.GetExtension(file).ToLowerInvariant(); 
          
	RegistryKey fileExtKey = Registry.ClassesRoot.OpenSubKey(fileExt);

          
          
          
            if (fileExtKey
!= null && fileExtKey.GetValue("Content
Type") != null) 
          
	{

          
		contentType = fileExtKey.GetValue("Content
Type").ToString(); 
          
	}

          
          
          
            return contentType; 
          
}

          
          
        

I remember when I was in consulting some years ago now.  At the time, I was a fairly senior technical resource and I had a pretty wide variety of clients and scenarios under my belt.  I was... experienced.

Anyhow, as a consultant it is pretty common to be interviewed by your potential client before arriving.  Clients don't like to have total n00bs on the job if they are paying consulting rates.  I used to have a number of these sort of interviews before being accepted to work on a project.

Even though I forget exactly now which client it was that interviewed me, I still remember the interview.  At the time, .NET 1.1 was the norm and 2.0 was in beta or so.  I was interviewing for a technical position on corporate account - architect or senior dev, if I recall correctly.  The customer got me on a conference call and I breezed through the background portion of the interview.  Next came the 'technical' interview.  I apparently got some young buck, fresh from his MSCD.NET exam that decided to test me on .NET trivia.  After answering a number of general .NET questions correctly, he asked me two questions that stick in my mind:

• What does 'private internal' mean?
• If you were to throw a custom exception in .NET, how would you do it?

Now, I said at the time that I wasn't sure on the first one and that I thought it might be both private and internal visibility.  The interviewer delighted in telling me it was private OR internal (the UNION of the two).  For the next question, I said I would simply derive from Exception and do what I needed.  The interviewer again (with 'Aha!' in his voice) told me that (of course) I should derive from ApplicationException.

I was passed over for the role for not being technical enough.

Now, in retrospect, it is laughable.  First, you only need to be told once what something means in order for it to stick.  Next, it turns out I was right on the second one.  The interviewer was so caught up on catching me on what I didn't know that he never considered if it mattered.  To this day, I have never seen a single instance of using "private internal" on anything - ever.  Furthermore, he ignored the fact that I had a vast amount of experience in his industry and had demonstrated enough .NET knowledge to determine I wasn't joking when I said I had coded in it.

It would have been much better to ask meaningful questions that actually test what matters (in this case systems thinking):

• When and why would you use a private and internal visible declaration - a.k.a. 'private internal'?
• Why would you throw a custom exception in .NET?  When do you throw exceptions?

Those kinds of questions get much deeper into the psyche of the developer and let you understand what kind of developer you are dealing with.

I contrast this with my Microsoft interview.  I had one of the most intimidating interviews ever with Vittorio.  He asked me (paraphrasing), "How would you design Twitter?  Take some time and think about it - trivial answers without consideration will not bode well for you".  It was beautiful in hindsight.  I got to tell him about the architectural considerations that I thought mattered and in broad strokes what I thought about.  It must have worked - we are now on the same team and he is a good colleague and friend.  Had he asked me about advanced threading or WCF bindings, I might not have 'passed' his exam.  Turns out that those topics don't matter for the job.  He asked exactly what did matter - could I think about systems as a whole and I did I have enough understanding of the pieces?

Next time you interview someone, make sure you measure what matters.

Last week, I had the opportunity to talk with Hal and Jonathan on the PowerScripting podcast about Windows Azure.  It was a fun chat - lots on Windows Azure, a bit on the WASM cmdlets and MMC, and it revealed my favorite comic book character.

Listen to it now.

This post is a bit overdue:  Steve threatened to blog it himself, so I figured I should get moving.  In one of our Cloud Cover episodes, we covered how to host WCF services in Windows Azure.  I showed how to host both publically accessible ones as well as how to host internal WCF services that are only visible within a hosted service.

In order to host an internal WCF Service, you need to setup an internal endpoint and use inter-role communication.  The difference between doing this and hosting an external WCF service on an input endpoint is mainly in the fact that internal endpoints are not load-balanced, while input endpoints are hooked to the load-balancer.

Hosting an Internal WCF Service

Here you can see how simple it is to actually get the internal WCF service up and listening.  Notice that the only thing that is different is that the base address I pass to my ServiceHost contains the internal endpoint I created.  Since the port and IP address I am running on is not known until runtime, you have to create the host and pass this information in dynamically.

          
            public
            override
            bool OnStart() 
          
{

          
            //
Set the maximum number of concurrent connections 
          
          
    ServicePointManager.DefaultConnectionLimit = 12;

          
          
          
    DiagnosticMonitor.Start("DiagnosticsConnectionString"); 
          
          
          
            //
For information on handling configuration changes
          
          
            //
see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
          
          
    RoleEnvironment.Changing += RoleEnvironmentChanging;

          
          
          
    StartWCFService();

          
          
          
            return
            base.OnStart(); 
          
}

          
          
          
            private
            void StartWCFService() 
          
{

          
    var baseAddress = String.Format(

          
        "net.tcp://{0}", 
          
        RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["EchoService"].IPEndpoint 
          
        );

          
          
          
    var host = new ServiceHost(typeof(EchoService), new Uri(baseAddress)); 
          
          
          
    host.AddServiceEndpoint(typeof(IEchoService), new NetTcpBinding(SecurityMode.None),
"echo"); 
          
          
          
    host.Open();

          
}
        

Consuming the Internal WCF Service

From another role in my hosted service, I want to actually consume this service.  From my code-behind, this was all the code I needed to actually call the service.

          
            protected
            void Button1_Click(object sender,
EventArgs e) 
          
{

          
    var factory = new ChannelFactory<WorkerHost.IEchoService>(new NetTcpBinding(SecurityMode.None)); 
          
          
          
    var channel = factory.CreateChannel(GetRandomEndpoint());

          
          
          
    Label1.Text = channel.Echo(TextBox1.Text);

          
}

          
          
          
            private EndpointAddress
GetRandomEndpoint() 
          
{

          
    var endpoints = RoleEnvironment.Roles["WorkerHost"].Instances 
          
        .Select(i => i.InstanceEndpoints["EchoService"]) 
          
        .ToArray();

          
          
          
    var r = new Random(DateTime.Now.Millisecond); 
          
          
          
            return
            new EndpointAddress( 
          
        String.Format(

          
            "net.tcp://{0}/echo", 
          
            endpoints[r.Next(endpoints.Count() - 1)].IPEndpoint)

          
            );

          
}
        

The only bit of magic here was querying the fabric to determine all the endpoints in the WorkerHost role that implemented the EchoService endpoint and routing a request to one of them randomly.  You don't have to route requests randomly per se, but I did this because internal endpoints are not load-balanced.  I wanted to distribute the load evenly over each of my WorkerHost instances.

One tip that I found out is that there is no need to cache the IPEndpoint information you find.  It is already cached in the API call.  However, you may want to cache your ChannelFactory according to best practices (unlike me).

Hosting Public WCF Services

This is all pretty easy as well.  The only trick to this is that you need to apply a new behavior that knows how to deal with the load balancer for proper MEX endpoint generation.  Additionally, you need to include a class attribute on your service to deal with an address filter mismatch issue.  This is pretty well documented along with links to download the QFE that contains the behavior patch out on the WCF Azure Samples project on Code Gallery in Known Issues. Jim Nakashima actually posted about this the other day as well in detail on his blog as well, so I won't dig into this again here.

Lastly, if you just want the code from the show, have at it!

I am happy to announce the public release of the Windows Azure MMC - May Release.  It is a very significant upgrade to the previous version on Code Gallery.  So much, in fact, I tend to unofficially call it v2 (it has been called the May Release on Code Gallery).  In addition to all-new and faster storage browsing capabilities, we have added service management as well as diagnostics support.  We have also rebuilt the tool from the ground up to support extensibility.  You can replace or supplement our table viewers, log viewers, and diagnostics tooling with your own creation.

This update has been in the pipeline for a very long time.  It was actually finished and ready to go in late January.  Given the amount of code however that we had to invest to produce this tool, we had to go through a lengthy legal review and produce a new EULA.  As such, you may notice that we are no longer offering the source code in this release to the MMC snap-in itself.  Included in this release is the source for the WASM cmdlets, but not for the MMC or the default plugins.  In the future, we hope to be able to release the source code in its entirety.

Features At A Glance:

	Hosted Services	Upload / configure / control / upgrade / swap / remove Windows Azure application deployments
	Diagnostics	Configure instrumentation for Windows Azure applications (diagnostics) per source (perf counters, file based, app logs, infrastructure logs, event logs).   Transfer the diagnostic data on-demand or scheduled. View / Analyze / Export to Excel and Clear instrumentation results.
	Certificates	Upload / manage certificates for Windows Azure applications
	Storage Services	Configure Storage Services for Windows Azure applications
	BLOBs and Containers	Add / Upload / Download / Remove BLOBs and Containers and connect to multiple storage accounts
	Queues	Add / Purge / Delete Windows Azure Queues
	Tables	Query and delete Windows Azure Tables
	Extensibility	Create plugins for rich diagnostics data visualization (e.g. add your own visualizer for performance counters). Create plugins for table viewers and editors or add completely new modules!  Plugin Engine uses MEF (extensibility framework) to easily add functionality.
	PowerShell-based backend	The backend is based on PowerShell cmdlets. If you don't like our UI, you can still use the underlying cmdlets and script out anything we do

How To Get Started:

There are so many features and updates in this release that I have prepared a very quick 15-min screencast on the features and how to get started managing your services and diagnostics in Windows Azure today!

I have been largely offline for the last 3 weeks because I have been on parental leave (again).  I was lucky enough to have another baby girl last December right around Christmas.  Working for Microsoft, this means that within the first year, you can take 4 weeks off to be at home with your new baby.  It is a terrific benefit.  Hence, for the last 3 weeks I have been largely caring for my youngest alone while her older sister is at daycare.  What a unique bonding experience - it is something that has been wonderful.

Anyhow, I will be back to work next week and I know that I will miss my time here.  It has gone by far too fast.

In Episode 3 of Cloud Cover, I mentioned the tip of the week was how to measure your database size in SQL Azure.  Here is the exact queries you can run to do it:

The first one will give you the size of your database in MB and the second one will do the same, but break it out for each object in your database.

Hat tip to David Robinson and Tony Petrossian on the SQL Azure team for the query.

I am happy to announce the updated release of the Windows Azure Service Management (WASM) Cmdlets for PowerShell today. With these cmdlets you can effectively automate and manage all your services in Windows Azure. Specifically,

• Deploy new Hosted Services
  • Automatically upload your packages from the file system to blob storage.
• Upgrade your services
  • Choose between automatic or manual rolling upgrades
  • Swap between staging and production environments
• Remove your Hosted Services
  • Automatically pull down your services at the end of the day to stop billing. This is a critical need for test and development environments.
• Manage your Storage accounts
  • Retrieve or regenerate your storage keys
• Manage your Certificates
  • Deploy certificates from your Windows store or the local filesystem
• Configure your Diagnostics
  • Remotely configure the event sources you wish to monitor (Event Logs, Tracing, IIS Logs, Performance Counters and more)
• Transfer your Diagnostics Information
  • Schedule your transfers or Transfer on Demand.

Why did we build this?

The WASM cmdlets were built to unblock adoption for many of our customers as well as serve as a common underpinning to our labs and internal tooling. There was an immediate demand for an automation API that would fit into the standard toolset for IT Pros. Given the adoption and penetration of PowerShell, we determined that cmdlets focused on this core audience would be the most effective way forward. Furthermore, since PowerShell is a full scripting language with complete access to .NET, this allows these cmdlets to be used as the basis for very complicated deployment and automation scripts as part of the application lifecycle.

How can you use them?

Every call to the Service Management API requires an X509 certificate and the subscription ID for the account. To get started, you need to upload a valid certificate to the portal and have it installed locally to your workstation. If you are unfamiliar with how to do this, you can follow the procedure outlined on the Windows Azure Channel9 Learning Center here.

Here are a few examples of how to use the cmdlets for a variety of common tasks:

Common Setup

Each script referenced below will refer to the following variables:

Listing My Hosted Services

View Production Service Status

Creating a new deployment

Removing a deployment

Upgrading a single Role

Configuring Diagnostics - Adding a Performance Counter to All Running Instances

More Examples

You can find more examples and documentation on these cmdlets by typing 'Get-Help <cmdlet> -full' from the PowerShell cmd prompt.

If you have any questions for feedback, please send it directly to me through the blog (look at right hand navigation pane for Contact Ryan link).

Occasionally, I will check online to see the reviews for our book.  It is now almost 4 years old and a bit outdated with respect to the changes that came with .NET 3.5.  The website and forums however has been trucking along (thanks Joe!) these years.  We never sold a ton of books about this particular niche topic, but for the audience size, it wasn't too shabby.  Joe and I are not about to retire on our royalties, however.

I was a bit surprised to see this online however at Amazon:

Now, it is a bit silly for two reasons, a.) our book as a collectible, really?  Who would actually collect it?  and b.) It says it was signed by both authors.  Now, there is a slight chance it is authentic as I think Joe and I signed maybe 2 books together ever.  However, the odds of it being authentic are exceedingly slim.  Who knows. it's funny regardless.

One of my most popular posts to this day is the hack to install ADAM (now called AD LDS) on Vista.  It is the source of numerous emails as well.  So today, I am happy to say that AD LDS (aka ADAM) is available for Windows 7 officially - no hacks needed.

Download it here

Now, I fully realize this doesn't help those that are using Vista today, but there are quite a few folks that are using Windows 7 or upgrading from Vista or XP to 7.  LDAP directories are fabulous tools and AD LDS is a great one.  Enjoy.

Windows Azure storage makes use of a symmetric key authentication system.  Essentially, we take a 256-bit key and sign each HTTP request to the storage subsystem.  In order to access storage, you have to prove you know the key.  What this means of course is that you need to protect that key well.  It is an all-or-nothing scenario, if you have the key, you can do anything.  If you don't possess the key, you can do nothing*.

The natural question for most folks when they understand this model is, how can I grant access to someone without compromising my master key?  The solution turns out to be something called Shared Access Signatures (SAS) for Windows Azure.  SAS works by specifying a few query string parameters, canonicalizing those parameters, hashing them and signing the hash in the query string.  This creates a unique URL that embeds not only the required access, but the proof that it was created by someone that knew the master key.  The parameters on the query string are:

• st - this is the start time of when the signature becomes valid.  It is optional.  If not supplied, then now is implied.
• se - this the the expiration date and time.  All signatures are timebound and this parameter is required.
• sr - this is the resource that the signature applies to and will be either (b)lob or (c)ontainer.  This is required.
• sp -this is the permission set that you are granting - (r)ead, (w)rite, (d)elete, and (l)ist.  This is required.
• si - this is a signed identifier or a named policy that can incorporate any of the previous elements. Optional.
• sig - this is the signed hash of the querystring and URI that proves it was created with the master key.  It is required.

There is one other caveat that is important to mention here.  Unless you use a signed identifier - what I refer to as a policy - there is no way to create a signature that has a lifetime longer than an hour.  This is for good reason.  A SAS URL that was mistakenly created with an extremely long lifetime and without using the signed idenifier (policy) could not be revoked without changing the master key on the storage account.  If that URL was to leak, your signed resource would be open to abuse for a potentially long time.  By making the longest lifetime of a signature only an hour, we have limited the window in which you are exposed.

If you want to create a longer-lived SAS, you must create a policy.  The policy is very interesting.  Because this policy contains any of those parameters mentioned above and is stored at the service, it means that we can revoke those permission or completely change them instantly.

Let's walk through an example using MyAzureStorage.com, where it is trivial to create a SAS.  Once I login to the site, I am going to select the BLOBs option from the navigation tabs on top.  Here I will see a list of all my containers.  I can select a container's Actions menu and click Manage Policies.

Next, I am going to create two policies (signed identifiers), called Read and Write.  These will have different expirations dates and permission sets.  Notice I am not specifying a Start Date, so they are immediately valid.

Next, I am going to select the Actions menu for one of the blobs under this container and click Share.  I am going to apply one of the policies (shared access signature) that I just created by selecting it from the dropdown.

You will notice that the value that I created in the policy fill in the values in the Share BLOB UI and disable you from changing them.  The reason is that a policy is like a template.  If you don't set a value, then you can set (or must set it if it is required).  However, if the policy states one of the parameters, you cannot supply that parameter.  In this case, the 'read' policy that was created specified the expiration (se) and permissions (sp).  It is implied from the dialog selection that the resource (sr) is a (b)lob.  The only value that could be supplied here outside of the policy is the start time (st), which I am not supplying as it is optional.

When I click the Get URL button, I get back a URL that looks like this:

http://<account>.blob.core.windows.net/shared/blobname?sr=b&si=read&sig=hIbD%.%3D

Now, if that URL was to leak and I no longer wanted to provide read access to the blob, I could simply delete the 'read' policy or change the expiration date.  It would instantly be invalidated.  Compare this to the the same signature created without a policy:

http://<account>.blob.core.windows.net/shared/blobname?se=2010-02-13T02%3A17%3A46Z&sr=b&sp=r&sig=bYfBBb1yf.%3D

This signature could not be revoked until it either expired or I regenerated the master key.

If you want to see how the SAS feature works or easily share blobs or containers in your Windows Azure storage account, give it a try at MyAzureStorage.com and see how easy it is to do.

* assuming the key holder has not marked the container as blob- or container-level public access already, in which case it is public read-only.

It wasn't too long ago when Karsten Januszewski came to my office looking for a Windows Azure token (back when we were in CTP).  I wondered what cool shenanigans the MIX team had going.  Turns out it was for the Incarnate project (explained here).  In short, this service finds all the different avatars you might be using across popular sites* and allows you to select an existing one instead of having to upload one again and again.

You will note that there is another 'dunnry' somewhere on the interwebs, stealing my exclusive trademark.  I have conveniently crossed them out for reference. ;)

Since the entire Incarnate service is running in Windows Azure, I was interested in Karsten's experience:

We chose Windows Azure to host Incarnate because there was a lot of uncertainty in traffic.  We didn't know how popular the service would be and knowing that we could scale to any load was a big factor in choosing it.

I asked him how the experience was, developing for Windows Azure

There is a ton of great documentation and samples.  I relied heavily on the Windows Azure Platform Kit as well as the samples in the SDK to get started.  Once I understood how the development environment worked and how the deployment model functioned, I was off and running. I'd definitely recommend those two resources as well as the videos from the PDC for people who are getting started.

I love it when I hear that.  Karsten was able to get the Incarnate service up and running on Windows Azure easily and now he is scale-proof in addition to the management goodness that is baked into Windows Azure.

Checkout more about Incarnate and Karsten's Windows Azure learning on the MIX blog.

*turns out you can extend this to add a provider for any site (not just the ones that ship in source).

This might come as a surprise to some folks, but in Windows Azure you are billed when you deploy, not when you run.  That means we don't care about CPU hours - we care about deployed hours.  Your meter starts the second you deploy, irrespective of the state of the application.  This means that even if you 'Suspend' your service so it is not reachable (and consumes no CPU), the meter is still running.

Visually, here is the meter still running:

Here is when the meter is stopped:

Right now, there is a 'free' offering of Windows Azure that offers a limited # of hours per month.  If you are using MSDN benefits for Windows Azure, there is another offering that offers some bucket of 'free' hours.  Any overage and you start to pay.

Now, if you are like me and have a fair number of hosted services squirreled around, you might forget to go to the portal and delete the deployments when you are done.  Or, you might simply wish to automate the removal of your deployments at the end of day.  There are lots of reasons to remove your deployments, but the primary one is to turn the meter off.  Given that re-deploying your services is very simple (and can also be automated), this is not a huge issue to remove the deployment when you are done.

Automatic Service Removal

For those folks that wish an automated solution, it turns out that this is amazingly simple when using the Service Management API and the Azure cmdlets.  Here is the complete, deployment-nuking script:

That's it - just 6 lines of Powershell. BE CAREFUL.  This script will iterate through all the services in your subscription ID, stop any deployed service, and then remove it.  After this runs, every hosted service will be gone and the billing meter has stopped (for hosted services anyway).

A few customers have asked how they can use tools like wazt, Windows Azure MMC, the Azure Cmdlets, etc. when they are behind proxies at work that require basic authentication.  The tools themselves don't directly support this type of proxy.  What we are doing is simply relying on the fact that the underlying HttpRequest object will pick up your IE's default proxy configuration.  Most of the time, this just works.

However, if you are in an environment where you are prompted for your username and password, you might be on a basic auth proxy and the tools might not work.  To work around this, you can actually implement a very simple proxy handler yourself and inject it into the application.

Here is one that I wrote to support wazt.  To use this, add the following to your app.config and drop the output assembly from this project into your execution directory.  Note, this would work with any tool in .NET that uses HttpWebRequest under the covers (like csmanage for instance).

<!-- basic auth proxy section declaration area-->
<!-- proxyHostAddress="Auto" : use Internet explorer configuration for name of the proxy -->
<configSections>
  <sectionGroup name="proxyGroup">
    <section name="basicProxy"
             type="Proxy.Configuration.CustomProxySection, Proxy" />
  </sectionGroup>
</configSections>

<system.net>
  <defaultProxy enabled="true" useDefaultCredentials="false">
    <module type="Proxy.CustomProxy, Proxy"/>
  </defaultProxy>
</system.net>

<proxyGroup>   
  <basicProxy proxyHostAddress="Auto" proxyUserName="MyName" proxyUserPassword="MyPassword" />
</proxyGroup>

Download the source here.

I love elegant software.  I knew about CloudXplorer from Clumsy Leaf for some time, but I hadn't used it for awhile because the Windows Azure MMC and MyAzureStorage.com have been all I need for storage for awhile.  Also, I have a private tool that I wrote awhile back to generate Shared Access signatures for files I want to share.

I decided to check out the progress on this tool and noticed in the change log that support for Shared Access signatures is now included.  Nice!  So far, this is the only tool* that I have seen handle Shared Access signatures in such an elegant and complete manner.  Nicely done!

Definitely a recommended tool to keep on your shortlist.

*My tool is complete, but not nearly as elegant.

Some time back, I put in a request to LINQPad's feature request page to support SQL Azure.  I love using LINQPad for basically all my quick demo programs and prototypes.  Since all I work with these days is the Windows Azure platform, it was killing me to have to go to SSMS to do anything with SQL Azure.

Well, my request was granted!  Today, you can use the beta version of LINQPad against SQL Azure and get the full LINQ experience.  Behold:

In this case, I am querying the firewall rules on my database using LINQ.  Hot damn.  Nice work Joe!  If you pay a few bucks, you get the intellisense version of the tool too, which is well worth it.  This tool has completely replaced SnippetCompiler for me and continues to get better and better.  Now, if Joe would add F# support.

LINQPad Beta