Microservices with Dapr mini-series (Pub/Sub + Tye)

Overview

Swoop provides instant loan offers (aka ILO as we call) to the customers powered by API to API automated journey. Since we will be connecting to a number of APIs, we needed a way to manage and abstract the common configuration in a distributed way. As our platform employs the sidecar pattern with envoy proxy and kong proxy (for API gateway), we were looking for options to separate the application core logic and common configuration as same as the sidecar pattern that can be secured and scaled at speed.

What is Dapr?

Dapr is a Distributed Application Runtime, which may come across as Japanese… but let’s break it down.

• Distributed Application refers to software components that are networked and which work together to achieve a common goal, examples of which are things like blockchains and microservices.
• Runtime is referring to software that’s designed to support the execution of an application, which in this specific case, means supporting the execution of distributed applications.

So, in English, the same way in which the .NET Framework uses the Common Language Runtime (CLR) that makes .NET developers life easier by providing services like thread management, garbage collection, type-safety and exception handling, Dapr provides services that simplify the development and execution of microservices.

What does it do exactly?

Dapr enables you to create microservices without all the complexities that are usually attached to microservices. It does this by abstracting away the details of using and managing state stores, secrets, pub/sub infrastructures and enables the configuration of those dependencies outside of the microservice you’re developing, in effect, reducing the complexity of the software we’re developing. 

So, for instance, let’s say you want to use Redis for messaging… no problem…
Changed your mind and now want to use RabbitMQ instead? Again, no problem!

You don’t change a single line of code in your microservice to accommodate that change! All you do is update the relevant manifests belonging to your application and Dapr takes care of the rest!

How does it work?

Dapr uses the Sidecar Architecture, which is a programming model where an application has another application that runs side by side with it, it resembles a sidecar attached to a motorbike and it’s a good analogy because the sidecar application shares the same lifecycle as its parent.

When using Dapr, your application will be communicating with the Dapr sidecar that spins up at the same time your application does. The two will then communicate over either HTTP or gRCP.

For each microservice you have, you’ll have a Dapr sidecar attached to it where your app will be communicating with the Dapr sidecar and the Dapr sidecar will be communicating with other Dapr sidecars, which in turn will communicate with the microservice the sidecar is attached to.

A practical example would be a scenario where we are tasked with creating a system that monitors spikes of activity on a web application and sends notifications when a spike occurs. 

The monitoring system, using Dapr would look something along the lines of this

The system is composed of two microservices:

• The monitoring service (MonitoringService)
• The notification service (NotificationService)

Each of which has a Dapr sidecar attached

• The monitoring service (MonitoringService)
  • The monitoring service Dapr sidecar (Dapr_MonitoringService)
• The notification service (NotificationService)
  • The notification service Dapr sidecar (Dapr_NotificationService)

When MonitoringService finds a spike of activity, it will communicate with its own Dapr sidecar Dapr_MonitoringService, telling it that it needs to fire off a notification.

The Dapr_MonitoringService will then locate and inform the Dapr_NotificationService that a notification has to be fired passing along the details of the notification in question.

The Dapr_NotificationService will then tell the NotificationService to fire off that notification.

The communication between each service and sidecar occurs over either HTTP or gRCP, though this is abstracted away from the developer within the Dapr SDKs, you could still manually post an HTTP request directly to the Dapr sidecar if you wanted to.

Building Blocks

Dapr has a number of different building blocks that it exposes which allow us to do a number of different things…

Getting started

Here at Swoop, we use the Publish and Subscribe building block the most and it’s the underlying tech that helps our applications work asynchronously. 

During the first article in this Dapr mini-series, we will be covering the Publish and Subscribe building block first.

Prerequisites

You will need to have the following installed.

Performing the steps above will have prepared our environment to begin developing our microservice and demonstrating Daprs capabilities.

Getting started with the Pub/Sub building block

To get started, let’s create a new ASP.NET Core WebApi solution.

Next, let’s name our solution DaprBuildingBlocks and our project DaprBuildingBlocks.PubSubBlock

After we click Next, on the Additional information page, make sure that the project is configured as below.

Now that our solution has been created, we need to install the Dapr NuGet packages before we do anything else.

To install the Dapr NuGet package, either search for Dapr.AspNetCore NuGet package in the NuGet Package Manager or install it using the Package Manager Console with the command below

Install-Package Dapr.AspNetCore -Version 1.5.0

Configuration

Once the NuGet package has been installed, we need to configure our Startup.cs class to make sure that the Daprdependency is configured and injected into our application.

Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    // Adds the Dapr client to the IoC container
    services.AddDaprClient(); 

    // `.AddDapr()` Adds the Dapr integration for MVC
    services.AddControllers()
            .AddDapr(); 

    services.AddSwaggerGen(c =>
    {
        c.SwaggerDoc("v1", new OpenApiInfo { Title = "DaprBuildingBlocks.PubSubBlock", Version = "v1" });
    });
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseSwagger();
        app.UseSwaggerUI(c => c.SwaggerEndpoint("https://marketing-website-prod-uk-south-cdn-bmffeydtbxdthpa2.a02.azurefd.net/swagger/v1/swagger.json", "DaprBuildingBlocks.PubSubBlock v1"));
    }

    app.UseRouting();

    // Ensure that you are including thise line
    // Without which, the contents of your published 
    // object will contain nothing but default values.
    app.UseCloudEvents();

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
        
        // We also require this line to subscribe to the 
        // messages that we publish.
        endpoints.MapSubscribeHandler();
    });
}

Publishing

To publish a message, let’s navigate to the WeatherForecastController.cs file.

Create a new controller action named PublishAsync like so:

[HttpPost("publish")]
public async Task<IActionResult> PublishAsync([FromBody]WeatherForecast forecast, [FromServices] DaprClient daprClient)
{
    await daprClient.PublishEventAsync("pubsub", "forecast", forecast);
    return Ok();
}

What we’re doing above is, starting from the method signature is injecting the DaprClient into the controller action once the controller action is invoked ([FromServices] DaprClient daprClient).

We then use the DaprClient to publish an event using PublishEventAsync with:

• The name of the PubSub
• The name of the Topic
• The object itself that we want to broadcast

As method arguments, and with that, we will have published a message to our Pub/Sub topic, it’s that easy!

Subscribing

To subscribe to a topic, let’s create a new controller action called Subscribe with the following contents:

[HttpPost("subscribe")]
[Topic("pubsub", "forecast")]
public IActionResult Subscribe(WeatherForecast forecast)
{
    forecast.GetType()
            .GetProperties()
            .Select(x => $"{x.Name}:{x.GetValue(forecast)}")
            .ToList()
            .ForEach(Console.WriteLine);

    return Ok();
}

As you’ll notice, there’s a special Attribute we’re using here called Topic. Using this Attribute is what enables us to subscribe to messages broadcasted on the topic specified within its parameters.

Tying it all together

To tye things up, let’s navigate to the root directory of our project using the file explorer.

In this directory, create a new file called tye.yaml, we are going to use this manifest to make sure all of our dependencies are available and configured for us to use when we start testing our application. Let’s open tye.yaml and use the following configurations.

name: dapr
extensions:
- name: dapr
  components-path: ".tye_components"
  
services:
- name: redis
  image: redis
  bindings:
    - port: 6380
      containerPort: 6379
   
- name: pubsub-api
  project: DaprBuildingBlocks.PubSubBlock/DaprBuildingBlocks.PubSubBlock.csproj
  bindings:
    - name: http
      port: 35002
      protocol: http
  env:
    - ASPNETCORE_URLS=http://+:35002

As you’ll notice, we’ve specified a components-path field to .tye_components, under the extension name dapr. 

What is happening here is we’re letting dapr know that the manifests relevant to it, is stored under that directory which is what we will be creating next.

Under services, you’ll also notice that we’re using Redis and we also specify the relative path to the very project we just created under the field project. tye will use the services we define here and if it doesn’t exist it will try to fetch it, it will know what to look for if the image field is set.

So, moving on, let’s create our .tye_components directory.

Within that directory let’s create a new manifest called pubsub.yaml

\.tye_components\pubsub.yaml

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: localhost:6380

This manifest is used by dapr to use the pubsub building block, in particular, the redis pubsub building block within our application.

Once you’ve saved the manifest, let’s go back to our root directory.

Open a PowerShell window here at the root and execute the command tye run –watch –dashboard.

To add some context on what that command does is, first we’re telling tye to run our application.

Then we’re telling it to –watch our project for any changes, if you make any changes within the code of the project, tyewill recompile the application and restart our service to reflect our changes which makes it excellent for debugging.

Now let’s navigate to our Swagger documentation for the Web API we’ve just developed and fire off a sample request.

{
  "date": "2022-01-09T22:55:32.093Z",
  "temperatureC": 100,
  "summary": "432fasfas"
}

The response we get back should be a status code 200. If you run into errors where you’re posting this data immediately after you’ve started the application, there may be the possibility that Dapr hasn’t had the opportunity to initialise properly yet, try again every few seconds or so, if that is indeed the case, you should start seeing 200 responses within a few seconds.

Once you do receive a status code 200 response, let’s go back to the tye dashboard to view the logs of our application.

And there we have it… the message that we published to the forecast topic was received by the controller action that was subscribed to that very topic, the controller action was invoked and the message was printed to the console, reflected in the logs we see here.

How Swoop uses Dapr

Here at Swoop, we use the pub/sub building block to make asynchronous workflows, so for instance, most recently we’ve been working on integrating an open banking provider for our Australian customers. The open banking integration provider posts back the customer’s bank statements to us through a web-hook hosted by us once our customers complete the integration journey. 

Once that web-hook is invoked, an asynchronous process is started on our end, a process that is required because we have multiple consumers of the transaction data we receive. 

We receive the data from our integration partner in a number of formats including JSON and PDF, consumers of this data are independent of one another, some deal with persistence to a MongoDb data store, whilst PDFs are stored in our Azure blob storage containers, some of this data is also exposed through SalesForce. 

If so… how can we expose this data so that in future, we never have to modify our existing code for this integration, but have other microservices that can subscribe to this data in almost real-time?

This is where Dapr comes in and is heavily used, we’re able to keep the door open to future expansion of how we make use of this data by publishing to topics, as there are multiple consumers of these bank transactions, we’re able to create event-driven processes where the publisher never needs to know about any of the subscribers, it just publishes the data and consumers never have to poll for updates, the relevant data is pushed to them, and data gets processed the moment it is received. 

A side benefit of Dapr is also resiliency, published messages can be configured to require acknowledgement by the subscriber, or else to retry delivery. 

When we aggregate all of the benefits what we’re left with is first and foremost: reduction in complexity within our codebase. All the resiliency, pub/sub dependencies, retry rules etc are managed within the manifests and outside of our application. 

We don’t need to write any additional code within our application for those benefits, Dapr handles it all.

References

• https://docs.dapr.io/
• https://github.com/dotnet/tye
• https://docs.docker.com/