Monday, April 27, 2009

Overlooking the Obvious

I was trying to troubleshoot a bug in my worker role in Windows Azure the other day.  To do this, I have a very cool tool (soon to be released) that lets me peek messages.  The problem was that I couldn't get ahold of any messages, it was like they were disappearing right from under my nose.  I would see them in the count, but couldn't get a single one to open.  I was thinking that I must have a bug in the tool.

Suddenly, the flash of insight came: something was actually popping the messages.  While I remembered to shut down my local development fabric, I forgot all about the version I had running in the cloud in the staging environment.  Since I have been developing against cloud storage, it is actually a shared environment now.  My staging workers were popping the messages, trying to process them and failing (it was an older version).  More frustrating, the messages were eventually showing up again, but getting picked up before I could see them in the tool.

So, what is the lesson here:  when staging, use a different account than your development accounts.  In fact, this is one of the primary reasons we have the cscfg file:  don't forget about it.