Bernie Cook's Blog

Azure, C#, .NET, Architecture & Related Tech News

Distributed Caching in Azure – Cache Worker Role

4 Comments

In June 2012 version 1.7 of the Windows Azure Platform release was introduced and with it came the new cache worker role. This provided another distributed cache management option for Azure developers alongside the likes of AppFabric Caching, or Memcached, to name a few. There are a number of ways to utilise and configure cache worker roles and this post covers one of them, providing a step by step guide to creating a new cloud solution where a web and worker role (cache clients) share the same cache worker role (cache cluster).

As to whether or not this approach is suitable for you depends on your requirements. Hopefully you’ll find this of some benefit, and excuse me for being a bit gung-ho on screenshots.

The source code for this solution is available on GitHub and has been tested locally and in Azure.

Assumptions

I’m assuming you have the following tools and knowledge:

Step 1 of 5. Create a New Project

Start off by creating a new “Windows Azure Cloud Service” project in Visual Studio:

Next configure the project so that it has the following 3 roles; a web role (I chose a blank MVC 4 web role), a worker role and a cache worker role.

Once Visual Studio has done it’s work you’ll have your Solution Explorer looking something like this:

Step 1 complete.

Step 2 of 5. Configure the Cache Worker Role

Because Visual Studio has done a lot of work for us already there is very little to change. Although the “Worker.cs” file may look empty don’t remove it, all roles need an entry point.

Start by opening up the properties for the cache worker role – not the project’s properties but the cloud project’s role properties. Under the “Caching” tab you’ll see the following:

There’s a bit going on here so lets walk through it.

It’s clear from the first checkbox labelled “Enable Caching” that caching is (you guessed it) enabled – very important 🙂

We want a dedicated cache cluster in this solution, and by this I mean one that runs on it’s own Windows Azure virtual machines (VM). You have the option to set up a co-located cache cluster, one that shares a VM with an existing web or worker role (so it shares bandwidth, CPU and memory), but that’s not the particular topology we’re after in this guide. We want our web and worker role to share the same dedicated cache cluster so “Dedicated Role” is the option we’re interested in. MSDN has a great article titled: “Windows Azure Caching on Existing Roles” which covers co-location if you’re interested.

Next we need to configure the Azure solution so that the configuration settings pertaining to the cache cluster are stored somewhere central; this is where Azure Storage comes in handy. So by default Visual Studio has entered “UseDevelopmentStorage=true“. You can change this here or edit it within the Azure Service Configuration Schema file (ServiceConfiguration.Local.cscfg) if you like, just look for the setting with the name: “Microsoft.WindowsAzure.Plugins.Caching.ConfigStoreConnectionString“. Whatever you do ensure that when you deploy to a staging or production environment you’ve set it to a valid Azure storage location using the appropriate end points and account name and key.

Next on the list is the Named Cache Settings section. You’ll always need at least one named cache, and in the initial setup Visual Studio has created a default one called (you guessed it again) “default“. You can run with this or add more named caches depending on your requirements. What they provide is specific cache settings per named cache, i.e. different eviction policies, expiry types, TTL, etc. If you add more you’ll have to ensure that when you create your DataCache instance that you’re using the appropriate one – the setting is available in the DataCache constructor. For now though we’re going to use the default.

So with little or no changes Visual Studio has done all the work necessary to get your cache cluster up and running. If you run your solution at this point, regardless of how the web role will respond in your browser, you’ll find that the cache cluster is running. See Task Manager > Details and look for processes with the “Cache” prefix 🙂

Excellent, step 2 complete.

Step 3 of 5. Wiring Up the Web Role

So all systems are go regarding the cache cluster, now we need to get our web role cache client talking to it.

First things first though, all cache clients need the right Windows Azure Caching assemblies. So get the NuGet Packages window up for your web role (right-click the web role’s References > Manage NuGet Packages… option). Search for “windows azure caching” and select the result which matches the one in the screenshot below, or a more recent iteration. Then click “Install” and let NuGet do what it does best. Or using Package Manager Console simply enter “Install-Package WindowsAzure.Caching

Once NuGet is done you’ll notice a few new assemblies in your solution:

  • Microsoft.ApplicationServer.Caching.AzureClientHelper
  • Microsoft.ApplicationServer.Caching.AzureCommon
  • Microsoft.ApplicationServer.Caching.Client
  • Microsoft.ApplicationServer.Caching.Core

A new folder with some cache diagnostics files:

  • Microsoft.WindowsAzure.Caching\ClientPerfCountersInstaller.exe
  • Microsoft.WindowsAzure.Caching\ClientPerfCountersInstaller.exe.config
  • Microsoft.WindowsAzure.Caching\PerformanceCounters.xml

And a few additions to your web.config:

  • <dataCacheClients>
  • <cacheDiagnostics>
  • <caching>
  • <sessionState>
  • Several <section> entries

I’ll walk you through the web.config changes as you need to be aware of what they mean and what you need to change.

<dataCacheClients>
This is already wired up, it’s aimed at the “default” named cache that was mentioned above and it’s been enabled. The only thing to change is the “autoDiscover” identifier value. It needs to change from this:

<autoDiscover isEnabled ="true" identifier="[cache cluster role name] " />

to this (or a variation depending on the name of your cache worker role):

<autoDiscover isEnabled ="true" identifier="AzureCache.CacheWorkerRole " />

There is a <localCache> node commented out – this is for utilising local VM cache, and I won’t go into this here other than to say it’s as you’d expect – you can store objects locally on the VM. If you want to read more about local cache then please do so, actually I encourage it.

<cacheDiagnostics>
Cache diagnostics is turned off, and again I won’t go into it here other than to say if you want to turn it on do some reading so that you’re aware of the what you’re getting into.

<cache>
If you’re going to be using output cache then uncomment this. You don’t need to make any changes, it’s ready to go if you haven’t changed any default settings thus far.

<sessionState>
If you’re going to be using the your cache cluster to support session state then uncomment this block, and comment out the current <sesstionState/> entry which is most likely set to use local memory – not recommend for a cloud application 🙂

Now lets add some code to test our setup. I created a blank MVC 4 web role so I’ve added the following in:

Model

using System.ComponentModel;

namespace AzureCache.WebRole.Models.HomeModels
{
    public class IndexModel
    {
        [DisplayName("Name")]
        public string Name { get; set; }

        [DisplayName("Value")]
        public string Value { get; set; }

        public IndexModel()
        {
            Name = "Tim";
        }
    }
}

Controller

using AzureCache.WebRole.Models.HomeModels;
using Microsoft.ApplicationServer.Caching;
using System.Web.Mvc;
using Microsoft.WindowsAzure.ServiceRuntime;

namespace AzureCache.WebRole.Controllers
{
    public class HomeController : Controller
    {
        public HomeController()
        {
            ViewBag.RoleInstanceId = RoleEnvironment.CurrentRoleInstance.Id;
        }

        [HttpGet]
        public ActionResult Index()
        {
            return View(new IndexModel());
        }

        [HttpPost]
        public ActionResult Index(
            IndexModel indexModel,
            string button)
        {
            var dataCache = new DataCache("default");

            if (button.Equals("set"))
            {
                dataCache.Put(indexModel.Name, indexModel.Value);
            }
            else
            {
                var value = dataCache.Get(indexModel.Name);
                if (value != null)
                {
                    ModelState.Remove("Value");
                    indexModel.Value = (string)value;
                }
            }

            return View(indexModel);
        }
    }
}

View

@model AzureCache.WebRole.Models.HomeModels.IndexModel
@{
    ViewBag.Title = "Azure Cache";
}
<hgroup>
    <h1>Azure Cache</h1>
    <h2>Role Instance Id: @ViewBag.RoleInstanceId</h2>
</hgroup>

@using (Html.BeginForm())
{
    <p>Set or get cache:</p>

    @Html.LabelFor(m => m.Name)
    @Html.TextBoxFor(m => m.Name)

    <br/>

    @Html.LabelFor(m => m.Value)
    @Html.TextBoxFor(m => m.Value)

    <br/>

    <button name="button" type="submit" value="set">Set</button>
    <button name="button" type="submit" value="get">Get</button>
}

Now do be warned, this view is going to look so ugly it may cause blindness. I desperately wanted to Bootstrap it but I didn’t want to introduce another element into the equation, so please excuse me on this.

Now a few points before you start debugging (F5):

  • The View and Model have been kept basic so there is nothing out of the ordinary to highlight
  • The Controller contains the logic to connect up to the cache cluster, which you can see in the HttpPost method.
  • I’ve purposefully made a call to output the current role instance identifier – this will come in handy when we want to confirm that our separate web role instances are all talking to the same cache cluster
  • Tim Tams are good for you – this will make more sense in a minute

How we’ve created the DataCache client is via one of two possible methods. The other utilises the DataCacheFactory which I’d encourage you to read up on if you’re unfamiliar with it.

Before we proceed with a test increase the number of web role instances in the “ServiceConfiguration.Local.cscfg” file for both the AzureCache.CacheWorkerRole and AzureCache.WebRole. A tip on working with Azure – avoid developing/testing against a single role instance.

So if this all makes sense F5 the solution, and prepare for the ugliest UI you’ve ever seen (except for this … maybe).

Now fire up a second browser and check that it’s Role Instance Id value is different between the two browsers. Set the value to in one browser to “Tam” or something else (chocolatey), then in the other browser click to get the value. You’ll see that the same value sent to the cache cluster in one web role is being read from the cache cluster in another.

This confirms everything is working as planned.

Step 3 of 5. Wiring Up the Worker Role

First off get the NuGet packages for Windows Azure Caching – same as before. The worker role, for our given purpose, is considered another cache client so we need the appropriate assemblies.

Open the app.config once the assemblies have been added and change this line:

<autoDiscover isEnabled ="true" identifier="[cache cluster role name]" />

to this (or a variation depending on the name of your cache worker role):

<autoDiscover isEnabled ="true" identifier="AzureCache.CacheWorkerRole " />

Next lets add some code to the “WorkerRole.cs“. We want to connect to the cache cluster and pull back a value for “Tim” if it exists.

using Microsoft.ApplicationServer.Caching;
using Microsoft.WindowsAzure.ServiceRuntime;
using System.Diagnostics;
using System.Net;
using System.Threading;

namespace AzureCache.WorkerRole
{
    public class WorkerRole : RoleEntryPoint
    {
        public override void Run()
        {
            DataCache dataCache = null;
            var roleInstanceId = RoleEnvironment.CurrentRoleInstance.Id;

            while (true)
            {
                if (dataCache == null)
                {
                    try
                    {
                        dataCache = new DataCache("default");
                    }
                    catch (DataCacheException)
                    {
                        Thread.Sleep(1000);
                        continue;
                    }
                }

                var value = dataCache.Get("Tim");
                if (value != null)
                {
                    EventLog.WriteEntry(
                        string.Concat("AzureCache ", roleInstanceId),
                        string.Concat("Tim = ", (string)value));
                }

                Thread.Sleep(2000);
            }
        }

        public override bool OnStart()
        {
            ServicePointManager.DefaultConnectionLimit = 12;

            return base.OnStart();
        }
    }
}

I’ve stripped out a lot of the normal comments, unused assembly references, added some test code and have added in a try-catch block. The reason for the latter is that our worker role VM might start before our cache cluster has a chance to initialise itself. That being said we need to handle this scenario and keep retrying until it’s available. We’re doing a broad spectrum exception catch for any DataCacheExceptions so use something slightly more refined in production.

The rest of the code simply outputs the cache object value for “Tim“, which we’re going to set in a web role.

So what we can expect to happen when this runs is that event log entries will be written containing “Tim = Tam“, and each event log entry will have a source that looks like this “AzureCache deployment18(467).AzureCache.AzureCache.WorkerRole_IN_0“, with the last digital being equal to 0, 1 or 2 depending on the instance that wrote it out.

Step 4 of 5. Testing All 3 Roles, Locally

Okay so now the plumbing is done I’ve increased the default instance counts across all roles to 3.

If we run the solution we’ll get our “fancy” page.

Further to this if we open the Computer Emulator UI we should see 3 roles with 3 instances each, all with green lights. Excellent.

Now if we go to the Event Viewer’s Windows Logs > Application we won’t see any new entries with the AzureCache prefix, yet. This is because your cache cluster has no object by the name “Tim“, yet. So if we switch to the browser and enter “Tam” into the value field and hit the Set button, then switch back to the Event Viewer and refresh the Windows Logs we’ll see something that looks like this:

This tells us that each worker role instance is talking to the cache cluster and can retrieve a value for my cache object named “Tim“. Now if we take it one step further and open up a new browser (Firefox just to mix it up) then go to the same address, hit the “Get” button (ensuring the Role Instance Id is different) we’ll find that “Tam” is returned.

So quite proof that our cache clients are both communicating to the same cache cluster in our local environment. Excellent.

Step 5 of 5. Testing All 3 Roles, In the Cloud

Time for a few more tests, this time we want to see how it runs on some real Azure VMs.

First we need to change the cache Cloud ConfigStoreConnectionString setting so it doesn’t try to use the “UseDevelopmentStorage=true” value. Grab one from your Azure account and enter it here.

Next off we should add a bit more logic to the worker role so if the worker role finds the “Tim” cache object it can append the web role instance id to it. This means that if I do a get on “Tim” in my browser I’ll get back a value updated by the worker role, which saves me having to remote desktop into the VMs to check their event logs like I did locally.

So add this just below the EventLog.WriteEntry() call in the “Worker.cs” file:

dataCache.Put("Tim", string.Concat((string)value, RoleEnvironment.CurrentRoleInstance.Id));

Finally lets update the instance count for all roles to 2, you’ll see why in a moment.

Now we’re ready to do some testing. Deploy the solution to Azure.

Once it’s running open up the DNS name assigned (i.e. http://berniecook.cloudapp.net/) in the browser and set a value for the “Tim” cache object. Then clear the value from it’s field and perform a get on “Tim” and you should get back “Tam“, great. If you see the Role Instance Id change you’ll know you’re hitting different VMs so both instances are talking to the same cache cluster successfully.

If you keep hitting Get button you’ll also see the value change from “Tam” to “TamAzureCache deployment18(476).AzureCache.AzureCache.WorkerRole_IN_0AzureCache deployment18(476).AzureCache.AzureCache.WorkerRole_IN_1“. Although this looks messy it’s proof that our worker roles are picking up the “Tim” cache object and successfully setting a new value against it.

Now if we run another test, this time lets reduce the role instance count for the cache cluster from 2 to 1 and see what that does (I’m using Cloud Storage Studio in the following screenshots). So this turns it from this…

into this…

Then we return to the browser and put a new cache object into the cache cluster, Name: “Car“, Value: “Lamborghini” and perform a get on a different web role instance I get back the value for “Car” we end up with “Lamborghini“. If we do a get on “Tim” we’ll find we still get back the “TamAzureCache deployment18(476).Azur…” value, albeit a bit longer this time as the worker roles are constantly increasing it’s size.

So what this says is that even with only one VM running in our cache cluster it can still operate properly.

Now one final test, what if we simulate the last cache worker role being disabled by rebooting it (you can’t set the instance count to 0) so that for several seconds the cache cluster will in fact be down. So lets do that.

If we try to set or get the “Tim” cache object in a browser we’ll get this error page. This is as expected because our web role will be throwing an exception when it tries to create it’s DataCache instance – no cache cluster means no DataCache instance.

Then if we wait for the cache cluster to finish rebooting and we do a get for “Tim” we’ll find that we get nothing back in return.

This is as expected because when we disabled our cache cluster all of our cache objects were removed. Leaving at least one cache worker role running is enough, but without that we loose our cache objects entirely.

So the main point is having at least one VM supporting the cache cluster is fine (although the Windows Azure SLA recommends more than one) so do keep that in mind.

Final Thoughts

Options, options, options. As with all things Azure there are usually an array of options. Get to know what the pros, cons and costs are relating to caching in Azure so you know that you’re making the right choice with a degree of confidence. With distributed caching, co-location caching, session state, output caching, VM sizes and their associated costs, AppFabric Caching, Memcached, named caches, expiration type, TTLs, etc. there is a fair bit to know but it’s important to do the reading and testing nonetheless.

Good luck.

Source Code

The entire sourcecode for this solution is available on GitHub at https://github.com/BernieCook/AzureCache

Author: Bernie

I currently live and work in the UK, just outside of London. I've been working in IT for 15+ years and in that time have solved many technical problems via blogs, forums, tutorials etc. and feel like I should be giving something back. This blog post is my way of contributing and I hope it proves just as useful to others as their contributions have to me.

4 thoughts on “Distributed Caching in Azure – Cache Worker Role

  1. Hi Bernie,

    This article is great and covers all the required topics in detail.
    Is there a way we can use the cache worker role in a web site that is hosted on IIS? I am not creating it as a web worker project because of some reasons. We have to deploy this app on IIS only. Same sample you created works fine when web application is created as a web role, but when I create a web site and host it on IIS it gives me below error “system.runtime.interopservices.sehexception external component has thrown an exception. azure”. Is there a way we can access Azure app fabric caching capabilities in a web site hosted on IIS?

    Thanks,
    Krishna

    • Hi Krishna

      There are a few questions here so I’ll try to answer each of them individually.

      • Given we’re using a worker role (a cache worker role to be exact) that runs on a VM which manages our cache entities then I doubt you’ll be able to get it working with a straight up IIS deployment that isn’t a web role. Having said that I’m sure you could work out how to run your IIS website separately from a cache worker role and have them communicate. Although then you’re introducing new performance issues, which may defeat the purpose of using a cache worker role altogether
      • Having said that you could use AppFabric Caching as it sits separate to your IIS solution, but again be aware of the performance issues – this article on MSDN might be useful: . Do ensure though that your website is running in the same affinity group (or region) as your AppFabric cache if you head down this route.

      I hope this is of some help.

      Bernie.

  2. Hi Bernie, I must appreciate the depth of this article. I’m a little late to the party but I have a question for you. I’m going to evaluate a few distributed caching solutions for Microsoft Azure. NCache is on top of my list. So have you tested it with Azure or any experiences related to it?
    Link here: http://www.alachisoft.com/ncache/azure-index.html

    Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s