Friday, July 9, 2010

Measure What Matters

I remember when I was in consulting some years ago now.  At the time, I was a fairly senior technical resource and I had a pretty wide variety of clients and scenarios under my belt.  I was... experienced.

Anyhow, as a consultant it is pretty common to be interviewed by your potential client before arriving.  Clients don't like to have total n00bs on the job if they are paying consulting rates.  I used to have a number of these sort of interviews before being accepted to work on a project.

Even though I forget exactly now which client it was that interviewed me, I still remember the interview.  At the time, .NET 1.1 was the norm and 2.0 was in beta or so.  I was interviewing for a technical position on corporate account - architect or senior dev, if I recall correctly.  The customer got me on a conference call and I breezed through the background portion of the interview.  Next came the 'technical' interview.  I apparently got some young buck, fresh from his MSCD.NET exam that decided to test me on .NET trivia.  After answering a number of general .NET questions correctly, he asked me two questions that stick in my mind:

  • What does 'private internal' mean?
  • If you were to throw a custom exception in .NET, how would you do it?

Now, I said at the time that I wasn't sure on the first one and that I thought it might be both private and internal visibility.  The interviewer delighted in telling me it was private OR internal (the UNION of the two).  For the next question, I said I would simply derive from Exception and do what I needed.  The interviewer again (with 'Aha!' in his voice) told me that (of course) I should derive from ApplicationException.

I was passed over for the role for not being technical enough.

Now, in retrospect, it is laughable.  First, you only need to be told once what something means in order for it to stick.  Next, it turns out I was right on the second one.  The interviewer was so caught up on catching me on what I didn't know that he never considered if it mattered.  To this day, I have never seen a single instance of using "private internal" on anything - ever.  Furthermore, he ignored the fact that I had a vast amount of experience in his industry and had demonstrated enough .NET knowledge to determine I wasn't joking when I said I had coded in it.

It would have been much better to ask meaningful questions that actually test what matters (in this case systems thinking):

  • When and why would you use a private and internal visible declaration - a.k.a. 'private internal'?
  • Why would you throw a custom exception in .NET?  When do you throw exceptions?

Those kinds of questions get much deeper into the psyche of the developer and let you understand what kind of developer you are dealing with.

I contrast this with my Microsoft interview.  I had one of the most intimidating interviews ever with Vittorio.  He asked me (paraphrasing), "How would you design Twitter?  Take some time and think about it - trivial answers without consideration will not bode well for you".  It was beautiful in hindsight.  I got to tell him about the architectural considerations that I thought mattered and in broad strokes what I thought about.  It must have worked - we are now on the same team and he is a good colleague and friend.  Had he asked me about advanced threading or WCF bindings, I might not have 'passed' his exam.  Turns out that those topics don't matter for the job.  He asked exactly what did matter - could I think about systems as a whole and I did I have enough understanding of the pieces?

Next time you interview someone, make sure you measure what matters.