Durable Task Framework (Preview) with Azure Service Bus
In todays landscape we have connected clients and continuous services that are powering rich and connected experiences for users. Developers face challenges in solving various challenges when writing code for both clients and services. I recently presented a session at the TechEd conference covering some of the challenges that connected clients face and how Service Bus can help with these (Connected Clients and Continuous Services with Windows Azure Service Bus)
From the continuous services perspective consider some of the scenarios were you have to perform long running operations spanning several services in a reliable manner.Consider some examples:
Compositions: Upload video -> Encode -> Send Push Notification
Provisioning: Bring up a Database followed by a Virtual Machine
For each of these you will generally need a state management infrastructure that will then need to be maintained and incorporated in your code. With Windows Azure Service Bus providing durable messaging features including advanced features like sessions and transactions we have released a preview of Durable Task Framework that allows you to write these long running orchestrations easily using C# Task code. Following is a developer guide and additional information for this:
Service Bus Durable Task Framework (Preview) Developer guide
Code Samples for Durable Task Framework
NuGet library for Durable Task Framework
Please give this a try and let us know what you think!
Rock - Paper - Scissors - Cloud! 
Wow, very nice! We have been looking for library like this.
I do not yet fully understand message flow/ relationship between taskhubclient and taskhubworker? I started 4 instances of signal sample (removed hub creation and hub.stop). In one process you can have one client and several workers? but every process should have one client for orchestration initialization?
I also noticed that instances can handle other instances activities.
Our orchestration goes like
socketserver puts messages to queue – parse message – add data from external webservice if needed – add data from another external webservice if needed – create alarm if needed – save data to db as batched write.
We will have a lot of messages ( 500 – 1000 msg/s). Should we have several hubs?
-Petri
Thanks Petri. Glad to know this helps.
The taskhubclient is only for starting and querying task orchestrations.
The taskhubworker is the thing that actually executes the task orchestrations. In addition the taskhubworker also executes task activities.
In a typical deployment, you would run multiple instances of the taskhubworker which would process all the orchestrations and activities. You would run the taskhubclient in a different tier of your application where it is simply being used to start the required task orchestrations.
So for the 500-1000 msg/sec, would you actually create a new orchestration for every message? If yes then you should probably shard to multiple hubs.
Thank you for your answer.
So you can start from another tier (different worker role instance) orchestration that can span to several worker roles? If true then this suits just perfectly for us if performance is good.
My hubclient reads messages from queue and sets message payload as parameter to orchestration. I thought that i need to create new instance for every message, how can i reuse completed orchestrations?
So the way it works is that the taskhubclient creates an orchestration instance (most probably from a different tier). Execution of this orchestration can span multiple worker roles but at one point only one of the workers will be executing it. Basically a node will execute a part of it and then checkpoint, after which another worker might pick up this instance and run upto the next checkpoint and so on.
Re performance, in your scenario, it does seem like you need to create an orchestration for every message in your input queue? You can start off by experimenting with a single hub and see if you are be able to achieve the required throughput by scaling out the number of taskhubworkers reading from this hub.
This is a great framework and incredibly useful! Not to mention very easy to get up and running. One major problem I have is with the extensibility of the framework. I’d like to be able to use unity with the framework but don’t see how I can right now.
For example, when creating a new client and worker, I’d like to be able to pass in an instance of my unity container that was created by the worker role. The container could then be passed down in to each orchestration and then on to the tasks. As it stands now, I would need to create a new instance of my container for each task. This adds overhead and makes testing very difficult.
Having looked through the code for the framework it doesn’t look like there’s any way I can subclass things and add support for this myself (some key classes are sealed or internal etc). Do you have any thoughts on this?
This is a great framework and incredibly useful! Not to mention very easy to get up and running. One major problem I have is with the extensibility of the framework. I’d like to be able to use unity with the framework but don’t see how I can right now.
For example, when creating a new client and worker, I’d like to be able to pass in an instance of my unity container that was created by the worker role. The container could then be passed down in to each orchestration and then on to the tasks. As it stands now, I would need to create a new instance of my container for each task. This adds overhead and makes testing very difficult.
Having looked through the code for the framework it doesn’t look like there’s any way I can subclass things and add support for this myself (some key classes are sealed or internal etc). Do you have any thoughts on this?
Hi Sam, there are a number of overloads of the TaskHubWorker.AddTaskOrchestration & AddTaskActivities methods that might help. These overloads allow you to add taskorchestration ‘factories’ (ObjectCreator). When the framework needs access to your taskorchestration or taskactivity, it will invoke the provided factory for that object.
In this factory you can instantiate your object however you want. E.g. you can return a singleton or instantiate a new object and pass it the instance of the unit container to the ctor etc.
Hope that helps.
Hi Sam, there are a number of overloads of the TaskHubWorker.AddTaskOrchestration & AddTaskActivities methods that might help. These overloads allow you to add taskorchestration ‘factories’ (ObjectCreator). When the framework needs access to your taskorchestration or taskactivity, it will invoke the provided factory for that object.
In this factory you can instantiate your object however you want. E.g. you can return a singleton or instantiate a new object and pass it the instance of the unit container to the ctor etc.
Hope that helps.
Is the source of the Framework available? I would like to recompile it against the latest versions of the dependencies. Currently update the WindowsAzure.Storage to v3 causes assembly binding errors.
Hy, I have another question because I found no information for that on the internet. Is it possible to get the total number of messages of a queue or topic which have been processed since the beginning. So I want to see all messages which already sent to a subscriber and waiting to send to a subscriber.
Best regards
Michi
I use the Windows Service Bus 1.0 with the Service Bus 1.0 SDK.
Or can you tell me how to get the following performance coutners in my c# application
http://msdn.microsoft.com/en-us/library/windowsazure/jj192996.aspx
I can check the counters in the Perfomance pane in the Windows Performance Monitor, that works fine. I need two of the performance counter for my application. How can I access the performance counters in my c# applicationtion? As I wrote above I only use the service bus for windows and not the whole azure service.
Best regards