Wednesday, 28 May 2014

How Does Brewmaster Work?

In this post, I will cover at a high level what Brewmaster does behind the scenes to enable your templates to deploy in Microsoft Azure.  There are around 8 major steps that we have to orchestrate the template deployment.

First, let's start with an architectural overview:


We built Brewmaster as an API first service.  The portal where you register has no abilities other than Registration that cannot be done via API.  That is, once you are registered with us, you have a subscription and a secret key that can be used to deploy.  In some subsequent posts, I will detail how to use the API.  The key components to Brewmaster are the API layer that everyone interacts with, our Template Validation logic that helps ensure you don't have common deployment errors, as well as our Workflow Engine that handles failure, retries, and allows us to efficiently scale our resources.

All of the actions that we take on your behalf in Azure are recorded and presented back to the user in our View repositories.  You could drive your own dashboard with this data or simply use ours.

Template Registration


Brewmaster templates are stored in Git repositories.  Today, we support both Bitbucket and Github public repositories.  There is a known layout and convention for creating templates.  When you first come to Brewmaster, you will register your Git repository for your template, or choose one of our pre-existing templates (hosted at Github).  We chose Git because we wanted to be able to version the templates and always have a repeatable deployment.  This has the effect that you can have a single template with different branches for each deployment type (Production, Development, etc.)

Once you register your Git repository with us, it becomes available as a deployment option:


When you click the Deploy button, we will pull that template from Git, parse it for any parameters you have defined and generate a UI to accept those params.  Once you submit the parameters, we will combine the template and params together, expand the template into a full deployment schema and then run our Validation logic.  As a template author, you have control over validation logic as well for any parameters coming into the template.  Once the template is validated, it is passed to our Workflow engine to start the execution.

Azure Provisioning

Brewmaster templates are scoped to a single Azure Affinity Groups (AG).  We are requiring that any Azure assets you want to either use or create must reside in the same AG within a single template.  While that sounds restrictive, it really isn't.  There is not much you can do that is interesting in the IaaS world without your resources in a single AG.  As such, this also means that if you want to deploy to multiple AGs, you must create multiple templates.

Today, we support the provisioning of IaaS VMs, Cloud Services, Storage Accounts, and Virtual Networks through Brewmaster.  We may expand that to other Azure resources (e.g. Hadoop, Service Bus, CDN, etc.) as demand dictates.



In the simple (80%) cases, Brewmaster will deploy your entire set Azure resources in a single go.  However, for more complicated scenarios where there are dependencies between Azure resources, you can actually stage them using the concept of Deployment Groups.  This allows for some sets of Azure resources to be provisioned and configured before other sets.  That is a more advanced topic that I will delve into more later.

Bootstrap the VM

In order to communicate with the Windows machines and allow it to be configured, we must enable a few things on the node.  For instance, Powershell Remoting is required along with Powershell 4.  The node might be rebooted during this time if required.

Brewmaster is dependent on Desired State Configuration (DSC).  In a nutshell, DSC is a declarative Powershell syntax where you define 'what to do' and leave the 'how to do' up to DSC.  This fits perfectly in our own declarative template model.  As such, we support DSC completely.  That means you can not only use any of the built-in DSC resource, but we also support any DSC resources - community released, or subsequently released "Wave" resources.  Don't see a DSC resource you need?  No worries, we also give you access to the 'Script' DSC resource, which will allow you to call into arbitrary Powershell code (or your existing scripts).

Once we have ensured that DSC is available, Brewmaster dynamically creates a DSC configuration script from the template and pushes it onto each node.

During this process, we also push down to each a node a number of scripts and modules that the node will later execute locally.  Once your nodes have been configured with DSC and PS Remoting, we are ready to download your template package.

Pulling the Template Package

We refer to the Brewmaster template as a JSON file that describes your Azure and VM topology.  However, we refer the template package as both the template JSON file as well as any other supporting assets required for configuring the VM.  Inside the package you can include things like custom scripts or DSC resources, installers, certificates, or other file assets.  The package has a known layout that will be used by Brewmaster.

Brewmaster instructs the node to pull the template package from the Git repository.  The node must pull the package from Git as opposed to Brewmaster pushing the repository.  The main reason for this is for quality of service and unintentional (or intentional) denial of service.  Theoretically, the template package could be very big and we would not want to lock our Brewmaster workers while trying to push a large package to each node.

Running the DSC Configuration

At this point, your node has all the assets from the template package and the DSC configuration that describes how to use them all.  Brewmaster now instructs the local node to compile the DSC into the appropriate MOF file and start a staged execution.

Brewmaster periodically connects to the node to discover information about the running DSC configuration (error, status, etc.).  It pulls what it finds back into Brewmaster and exposes that data to the user at the website (or via API).  This is how you can track status of a running configuration and know when something is done.

Wrap up

Once a deployment runs to its conclusion (either failure or success), Brewmaster records the details and presents it back to the user.  In this way the user has a complete picture of what was deployed and any logs or errors that occurred.  Should something go wrong during a deployment, the user can simply fix the template definition (or an asset in the template package) and hit the 'Deploy' button again.  Brewmaster will re-execute the deployment, but this time, it will skip the things it has already done (like provisioning in Azure or full Bootstrapping) and will re-execute the DSC configurations.  Given the efficient nature of DSC, it will also only re-execute resources that are not in the 'desired state'.  As such, subsequent re-deploys can be magnitudes faster that the initial.

Next up

In the coming installments, I will talk more about the key Brewmaster concepts and what each part of the schema does and about how to troubleshoot your templates and debug them.