Skip to content

Durable Task Framework (Preview) with Azure Service Bus

In todays landscape we have connected clients and continuous services that are powering rich and connected experiences for users. Developers face challenges in solving various challenges when writing code for both clients and services. I recently presented a session at the TechEd conference covering some of the challenges that connected clients face and how Service Bus can help with these (Connected Clients and Continuous Services with Windows Azure Service Bus)

From the continuous services perspective consider some of the scenarios were you have to perform long running operations spanning several services in a reliable manner.Consider some examples:

Compositions: Upload video -> Encode -> Send Push Notification

Provisioning: Bring up a Database followed by a Virtual Machine

For each of these you will generally need a state management infrastructure that will then need to be maintained and incorporated in your code. With Windows Azure Service Bus providing durable messaging features including advanced features like sessions and transactions we have released a preview of Durable Task Framework that allows you to write these long running orchestrations easily using C# Task code. Following is a developer guide and additional information for this:

Service Bus Durable Task Framework (Preview) Developer guide

Code Samples for Durable Task Framework

NuGet library for Durable Task Framework

Please give this a try and let us know what you think!

Receiving messages using NetMessagingBinding with custom formats

As a follow up to my earlier post on Formatting the content for Service Bus messages where we walked through the BrokeredMessage APIs that allow you to have custom formats for message content, I wanted to cover how you can use NetMessagingBinding (WCF) to receive such messages. This scenario would be relevant when your clients/senders are using REST API or non-.NET SDKs like Java, Python, PHP and Node.js and you server/receivers want to use WCF to process the messages.

NetMessagingBinding always builds a channel stack with BinaryMessageEncodingBindingElement+NetMessagingTransportBindingElement. If the BrokeredMessages in your ServiceBus Queue/Subscription are plain text, xml, then BinaryMessageEncoding won’t work, using WCF you’d have use a CustomBinding with TextMessageEncoder and NetMessagingTransportBindingElement instead.

You need to use a CustomBinding with TextMessageEncodingBindingElement (with MessageVersion = None) and a NetMessagingTransportBindingElement, make sure Action=”*”, and set AddressFilterMode=Any on your ServiceBehavior.

Here are two ways to read a plain XML message using NetMessagingTransportBindingElement:

Solution #1 Use System.ServiceModel.Channels.Message in the ServiceContract and call Message.GetBody()

namespace MessagingConsole
{
    static class Constants {
        public const string ContractNamespace = "http://contoso";
    }

    [DataContract(Namespace = Constants.ContractNamespace)]
    class Record
    {
        [DataMember]
        public string Id { get; set; }
    }

    [ServiceContract]
    interface ITestContract
    {
        [OperationContract(IsOneWay = true, Action="*")]
        void UpdateRecord(Message message);
    }

    [ServiceBehavior(
        AddressFilterMode = AddressFilterMode.Any)] // This is another way to avoid “The message with To ” cannot be processed at the receiver…”
    class TestService : ITestContract
    {
        [OperationBehavior]
        public void UpdateRecord(Message message)
        {
            Record r = message.GetBody<Record>();
            Console.WriteLine("UpdateRecord called! " + r.Id);
        }
    }

    class ServiceProgram
    {
        static void Main(string[] args)
        {
            string connectionString = "Endpoint=sb://<Service Bus connection string from Azure portal>";
            string topicPath = "Topic2";
            string subscriptionName = "Sub0";

            MessagingFactory factory = MessagingFactory.CreateFromConnectionString(connectionString);
            TopicClient sender = factory.CreateTopicClient(topicPath);
            SubscriptionClient receiver = factory.CreateSubscriptionClient(topicPath, subscriptionName, ReceiveMode.ReceiveAndDelete);

            string interopPayload = "<Record xmlns='" + Constants.ContractNamespace + "'><Id>4</Id></Record>";
            BrokeredMessage interopMessage = new BrokeredMessage(new MemoryStream(Encoding.UTF8.GetBytes(interopPayload)), true);
            sender.Send(interopMessage);

            CustomBinding binding = new CustomBinding(
                new TextMessageEncodingBindingElement { MessageVersion = MessageVersion.None },
                new NetMessagingTransportBindingElement());
            ServiceHost serviceHost = new ServiceHost(typeof(TestService), new Uri(solution));
            ServiceEndpoint endpoint = serviceHost.AddServiceEndpoint(typeof(ITestContract), binding, topicPath + "/Subscriptions/" + subscriptionName);
            endpoint.Behaviors.Add(new TransportClientEndpointBehavior(tokenProvider));
            serviceHost.Open();
            Console.WriteLine("Service is running");
            Console.ReadLine();            
        }
    }
}

Solution #2 Define a MessageContract data type to make the expected Soap contract match what the interop client is sending:

namespace MessagingConsole
{
    static class Constants
    {
        public const string ContractNamespace = "http://contoso";
    }

    [DataContract(Namespace = Constants.ContractNamespace)]
    class Record
    {
        [DataMember]
        public string Id { get; set; }
    }

    [MessageContract(IsWrapped=false)]
    class RecordMessageContract
    {
        [MessageBodyMember(Namespace = Constants.ContractNamespace)]
        public Record Record { get; set; }
    }

    [ServiceContract]
    interface ITestContract
    {
        [OperationContract(IsOneWay = true, Action="*")]
        void UpdateRecord(RecordMessageContract recordMessageContract);
    }

    class ServiceProgram
    {
        static void Main(string[] args)
        {
            string connectionString = "Endpoint=sb://<Service Bus connection string from Azure portal>"
            MessagingFactory factory = MessagingFactory.CreateFromConnectionString(connectionString);
            TopicClient sender = factory.CreateTopicClient(topicPath);
            SubscriptionClient receiver = factory.CreateSubscriptionClient(topicPath, subscriptionName, ReceiveMode.ReceiveAndDelete);

            string interopPayload = "<Record xmlns='" + Constants.ContractNamespace + "'><Id>5</Id></Record>";
            BrokeredMessage interopMessage = new BrokeredMessage(new MemoryStream(Encoding.UTF8.GetBytes(interopPayload)), true);
            sender.Send(interopMessage);

            CustomBinding binding = new CustomBinding(
                new TextMessageEncodingBindingElement { MessageVersion = MessageVersion.None },
                new NetMessagingTransportBindingElement());
            ServiceHost serviceHost = new ServiceHost(typeof(TestService), new Uri(solution));
            ServiceEndpoint endpoint = serviceHost.AddServiceEndpoint(typeof(ITestContract), binding, topicPath + "/Subscriptions/" + subscriptionName);
            endpoint.Behaviors.Add(new TransportClientEndpointBehavior(tokenProvider));
            serviceHost.Open();
            Console.WriteLine("Service is running");
            Console.ReadLine();
        }
    }

    [ServiceBehavior(
        AddressFilterMode = AddressFilterMode.Any
        )]
    class TestService : ITestContract
    {
        [OperationBehavior]
        public void UpdateRecord(RecordMessageContract recordMessageContract)
        {
            Record r = recordMessageContract.Record;
            Console.WriteLine("UpdateRecord called! " + r.Id);
        }
    }
}

Enterprise Integration Patterns with Service Bus (Part 2)

Priority-Queues

The scenario here is that a receiver is interested in receiving messages in order of priority for a single or multiple senders. A common use cases for this is event notification, where critical alerts need to be processed first. Today Service Bus Queues do not have the capability to internally sort messages by priority but we can achieve this pattern using Topics and Subscriptions.

We achieve this scenario by routing messages with different priorities to different subscriptions. Routing is done based on the Priority property that the sender adds to the message. The recipient processes messages from specific subscriptions and thus achieves the desired priority order of processing.

The Service Bus implementation for this is thru use of SQL Rules on Subscriptions. These Rules contain Filters that are applied on the properties of the message and determine if a particular message is relevant to that Subscription.

Service Bus Features used:

1. SQL Filters can specify Rules in SQL 92 syntax

2. Typically Subscriptions have one Rule but multiple can be applied

3. Rules can contain Actions that may modify the message (in that case a copy
of the message is created by each Rule that modifies it)

4. Actions can be specified in SQL 92 syntax too.

 

The code sample for this is available here.

Service Bus symmetry

Whether your application runs in the cloud or on premises, it often needs to integrate with other applications or other instances of the application. Windows Azure Service Bus provides messaging technologies including Relay and Brokered messaging to achieve this. You also have the flexibility of using the Azure Service Bus (multi-tenant PAAS) and/or Service Bus 1.0 (for Windows Server). This post takes a look at both these hosting options from the application developer perspective.

The key principle in providing these offerings is to enable applications to be developed, hosted and managed consistently between cloud service and on-premise hosted environments. Most features in Service Bus are available in both environments and only those that are clearly not applicable to a certain hosting environment are not symmetric. Applications can be written against the common set of features and then can be run between these environments with configuration only changes.

Overview

The choice of using Azure Service Bus and Service Bus on-premise can be driven by several factors. Understanding the differences between these offering will help guide the right choice and produce the best results. Azure Service Bus is a multi-tenant cloud service, which means that the service is shared by multiple users. Consuming this service requires no administration of the hosting environment, just provisioning through your subscriptions. Service Bus on-premise is a when you install the same service bits on machines and thus manage tenancy and the hosting environment yourself.

image

Figure 1: Windows Azure Service Bus (PAAS) and Service Bus On-premise

Development

To use any of the Service Bus features, Windows applications can use Windows Communication Foundation (WCF). For queues and topics, Windows applications can also use a Service Bus-defined Messaging API. Queues and topics can be accessed via HTTP as well, and to make them easier to use from non-Windows applications, Microsoft provides SDKs for Java, Node.js, and other languages.

All of these options will be symmetric between Azure Service Bus and Service Bus 1.0, but given the Beta nature of the release, this symmetry is not yet available. The key considerations are called out below:

Similarities

  • The same APIs and SDKs can be used to target Azure Service Bus and Service Bus on-premise
  • Configuration only changes can target application to the different environments
  • The same application can target both environments

Differences

  • Identity and authentication methods will vary thus having application configuration impact
  • Latency, throughput and other environmental differences can affect application performance since these are directly tied to the physical hardware that the service is hosted in
  • Quotas vary between environments (details here)

It’s important to understand that there is only one instance of Azure Service Bus that is available as a PAAS service but several on-premise environments may exists either thru third-party hosters or self-managed IT departments. Since the service is continually evolving with new features and improvements it is a significant factor in deciding with features to consume based on the environment targeted. Below is a conceptual timeline of how the features will be released (note this does NOT track to any calendar dates):

image

Figure 2: Client SDK release and compatibility timeline

Application Considerations

The key considerations from an application perspective can be driven by business or engineering needs. The key similarities and differences from this perspective are listed below:

Similarities

  • Namespaces are the unit for identity and management of your artifacts
  • Queues/Topics are contained within a Namespace
  • Claims based permissions can be managed on a per-entity basis
  • Size constraints are applied on Queues/Topics

Differences

  • Relay messaging is currently unavailable on Service Bus on-premise
  • Service Registry is currently available on Service Bus on-premise
  • Token based Identity providers for Service are ACS and for on-premise is AD and Self-signed tokens
  • SQL Server is the storage mechanism that is provisioned and managed by you for on-premises
  • Latency and throughput of messages vary between the environments
  • The maximum allowable values for message size and entity size vary

Do give the Service Bus 1.0 Beta a try, following are some additional resources:

Formatting the content for Service Bus messages

A message broker has to deal with scenarios where clients are using different protocol and client library versions to send and receive messages. In addition different platforms and protocols have varying support for serialization. For Windows Azure Service Bus, following are some examples of using different methods to create messages to send to Topics and Queues, and the corresponding approaches that can be used to receive the messages from Queues and Subscriptions.

If you are using REST protocol to send messages and receive them thru .NET API then you can directly manipulate the message body as:

Stream bodyStream = receiveMessage.GetBody<Stream>();

The following examples apply when using the .NET API to send and receive messages.

Example 1: Using string

When creating a BrokeredMessage with a string and the default (DataContract + Binary) serializer:

BrokeredMessage stringDefaultMessage = new BrokeredMessage("default string");

You can receive this message as:

string s = receiveMessage.GetBody<string>();

Example 2: Using Text instead of Binary XML in DataContractSerializer

When creating a BrokeredMessage with a string and a DataContractSerializer and use Text instead of Binary Xml:

BrokeredMessage stringDataContractMessage = new BrokeredMessage("DataContract string", new DataContractSerializer(typeof(string)));

You can receive this message as:

string s = receiveMessage.GetBody<string>(new DataContractSerializer(typeof(string)))

Example 3: Using raw UTF8 string

When creating a BrokeredMessage with just a raw UTF 8 string:

string utf8String = "raw UTF8 string";

BrokeredMessage utf8StringMessage = new BrokeredMessage(new MemoryStream(Encoding.UTF8.GetBytes(utf8String)), true);

You can receive this message as:

string s = new StreamReader(receiveMessage.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();

Example 4: Using raw UTF16

When creating a BrokeredMessage with just a raw UTF16 (“Unicode”) string:

string utf16String = "raw UTF16 string";

BrokeredMessage utf16StringMessage = new BrokeredMessage(new MemoryStream(Encoding.Unicode.GetBytes(utf16String)), true);

You can receive this message as:

string s = new StreamReader(receiveMessage.GetBody<Stream>(), Encoding.Unicode).ReadToEnd();

Example 5: Using a custom DataContract

When creating a BrokeredMessage using custom DataContract type and using default serializer (DataContract + Binary Xml):

[DataContract(Namespace = "")]

class Record {

[DataMember]

public string Id { get; set; }

}

[…]

Record recordDefault = new Record { Id = "default Record" };

BrokeredMessage recordDefaultMessage = new BrokeredMessage(recordDefault);

You can receive this message as:

Record r = receiveMessage.GetBody<Record>();

Example 6: Using custom DataContract and Text

When creating a BrokeredMessage using custom DataContract type and using DataContractSerializer (will use Text Xml instead of Binary):

Record recordDataContract = new Record { Id = "DataContract Record" };

BrokeredMessage recordDataContractMessage = new BrokeredMessage(recordDataContract, new DataContractSerializer(typeof(Record)));

You can receive this message as:

Record r = receiveMessage.GetBody<Record>(new DataContractSerializer(typeof(Record)));

Presenting at TechDays Hong Kong

One week to go for TechDays in Hong Kong! I am looking forward to meeting developers at this premier technology training event and will be presenting several sessions on Windows Azure which is part of the following key focus areas:

  • VIEW Soon-to-be-released products such as Microsoft System Center 2012, SQL Server 2012 and Windows Server® 8
  • LEARN about Windows® Phone 7.5; Kinect™ for Windows, Azure™, Visual Studio®
  • EXPLORE the latest technologies in hands-on labs, with certification for early bird participants
  • SHARE field experiences and expert tips and tricks

Below is a list of the specific Azure related sessions:

5th May @ 10:00am PBC258 Abhishek Lal

Windows Azure: An Overview

6th May @ 9:00am PBC353 Scott Golightly

Controlling Application Access with Windows Azure

6th May @ 11:00am PBC305 Abhishek Lal

Using Microsoft Visual Studio® to Build Applications that Run on Windows Azure

6th May @ 1:30pm PBC216 Sam Leung

Understanding the Application Portfolio Assessment and Migration Strategy to Windows Azure

6th May @ 4:45pm PBC384 Abhishek Lal

Windows Azure Service Bus Introduction: Why, What, How

7th May @ 9:30am PBC276 Ben Ng

A Lap Around Microsoft Dynamics CRM and Microsoft Dynamics CRM Online

7th May @ 11:00am PBC389 Scott Golightly

Windows Azure and Windows Phone – Creating Great Apps

7th May @ 1:30pm PBC283 Matt Valentine

Coding 4Fun – Kinect, Microcontrollers and Windows Phone

7th May @ 3:15pm PBC379 Abhishek Lal

Windows Azure Service Bus: Advanced Messaging Features

Azure Datacenter IP Ranges

When using Window Azure Service Bus Relay Bindings, you need to open outbound TCP connections from on-premise servers. If your firewall configuration restricts outbound traffic, you will need to perform the addition step of opening your outbound TCP port range 9350-9353 to the IP range associated with your selected regional data center. This information is now available at here

Enterprise Integration Patterns with Service Bus (Part 1)

Last week I presented at TechReady a session titled: Achieving Enterprise Integration Patterns with Service Bus. Folks who are familiar with the work of Gregor Hohpe, and the book / site http://www.eaipatterns.com/ already know the power of these building blocks in solving integration problems. I will not cover all the material in detail here but wanted to share the key concepts that enable the use of Windows Azure Service Bus in implementing these patterns. As the title of this post indicates, this is a first step of many and here I will cover the following:

  1. Messaging Channel Pattern
    • Publish-Subscribe
    • Content based routing
    • Recipient List

 

Publish-Subscribe

The scenario here is that a sender broadcasts an event to all interested receivers. A common use cases for this is event notification.

The way to implement this with Service Bus is to have a Topic with several Subscriptions. Each Subscription is for a recipient of the event notification. A single message sent to the Topic will be delivered to each Subscription (each will receive a copy of the message)

image

Some key features of Service Bus relevant to this pattern:

  1. A single Topic can have up-to 2000 Subscriptions
  2. Subscription can be individually managed with permissions
  3. Once a Subscription is created, it gets all subsequent messages sent to the topic
  4. Reliable delivery can be achieve thru PeekLock and Transactions

The code sample for this is available here.

Content based Routing

This is when we want to route a message to different recipients based on data contained in the message. The sender is not aware of the specific recipients but just sets appropriate metadata in the message. The recipients decide if a message is relevant to them based on its content.

Any scenario where you need to perform a single logical operation across different systems would need this, say an order processing system that routes orders to different departments.

The Service Bus implementation for this is thru use of SQL Rules on Subscriptions. These Rules contain Filters that are applied on the properties of the message and determine if a particular message is relevant to that Subscription.

image

Service Bus Features used:

  1. SQL Filters can specify Rules in SQL 92 syntax
  2. Typically Subscriptions have one Rule but multiple can be applied
  3. Rules can contain Actions that may modify the message (in that case a copy of the message is created by each Rule that modifies it)
  4. Actions can be specified in SQL 92 syntax too.

The code for this sample is available here.

Recipient List

There are scenarios where a sender wants to address a message to specific recipients. Here the sender sets metadata on the message that routes the message to specific subscribers.

Common use cases for this are order processing systems where the order needs to be fulfilled by a specific vendor or department that is selected by the user submitting the order. Email is another such messaging pattern.

The Service Bus implementation for this is thru use of SQL Rules on Subscriptions. We use the LIKE clause in SQL filters to achieve the desired results. The sender can just add the addresses of recipients in a comma-separated manner.

image

Service Bus Features used:

  1. SQL Filters with Rules in SQL 92 syntax
  2. LIKE clause for finding items addressed to Recipient
  3. Comma separated values to address multiple recipients

The code for this sample is available here.

Happy messaging!

Pointers to Service Bus information

Following is some collateral to get you started and I am happy to answer any questions/concerns.

  1. Release announcement: http://blogs.msdn.com/b/windowsazure/archive/2011/09/16/the-service-bus-september-2011-release.aspx
  2. MSDN Docs (API Reference/Tutorial etc.): http://msdn.microsoft.com/sb
    1. Service Bus Authentication and Authorization with the Access Control Service:  http://msdn.microsoft.com/en-us/library/windowsazure/hh403962.aspx
    2. Service Bus performance – best practices :http://msdn.microsoft.com/en-us/library/hh528527.aspx
  3. SDK & Samples: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=27421
  4. Blog post describing CTP to PROD API changes: http://rickgaribay.net/archive/2011/09/14/azure-appfabric-service-bus-brokered-messaging-ga-amp-rude-ctp.aspx
  5. Videos from //build conference:
    1. Building loosely-coupled apps with Windows Azure Service Bus Topics and Queues: http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-862T
    2. Service Bus Queues and Topics Advanced: http://channel9.msdn.com/posts/ServiceBusTopicsAndQueues
    3. Securing Service Bus with ACS: http://channel9.msdn.com/posts/Securing-Service-Bus-with-ACS

Happy messaging!

Windows Azure Service Bus Queues, Topics / Subscriptions

We just released to production the Queues, Publish-Subscribe with Topics / Subscriptions features that were earlier showcased in Community Technology Preview for Service Bus. For an introductory period these Brokered Messaging features are free, but you continue to pay Azure data transfer. Also the Relay features continue to be charged as before. Following are steps to quickly get started, and the links to additional resources:

The steps below are to download and run a simple application that showcases the publish-subscribe capabilities of Service Bus. You will need to have Visual Studio and NuGet installed. We will start by creating a new console application in VS:

1blog

 

Open the Project –> Properties and change the Target framework to .NET Framework 4

2blog

 

From the References node in Solution Explorer click on the context menu item for Add Library Package Reference. This will show only if you have NuGet Extension installed. (To learn more about NuGet see this TechEd video).

3blog

 

Search for “AppFabric” and select the Windows Azure AppFabric Service Bus Samples – PublishSubscribe item. Then complete the Install and close this dialog.

4blog

 

Note that the required client assemblies are now referenced and some new code files are added.

5blog

 

Add the following line to the main method in Program.cs and hit F5

 Microsoft.Samples.PublishSubscribe.Program.Start(null);

At this point you will be prompted to provide a ServiceNamespace and Issuer Name and Key. You can create your own namespace from https://windows.azure.com/

6blog

 

Once a new namespace has been created you can retrieve the key from the Properties section by clicking View under Default Key:

7blog

These values can now be used in the Console application:

8blog

At this point you can run thru the different scenarios showcased by the sample. Following are some additional resources:

Download page for SDK and Samples

Documentation for Service Bus September Release

Forum for feedback

Looking forward to your feedback / questions / concerns / suggestions.

%d bloggers like this: