Welcome to Day 21 of #30DaysOfServerless!
We've so far walked through what Azure Event Grid is and how it generally works. Today, let's discuss how Azure Event Grid deals with CloudEvents.
What We'll Cover
- What We'll Cover
- What is CloudEvents?
- How Azure Event Grid brokers CloudEvents
- How Azure Logic Apps consumes CloudEvents
- Exercise: Try this yourself!
- Resources: For self-study!
OK. Let's get started!
What is CloudEvents?
Needless to say, events are everywhere. Events come not only from event-driven systems but also from many different systems and devices, including IoT ones like Raspberry PI.
But the problem is that every event publisher (system/device that creates events) describes their events differently, meaning there is no standard way of describing events. It has caused many issues between systems, mainly from the interoperability perspective.
- Consistency: No standard way of describing events resulted in developers having to write their own event handling logic for each event source.
- Accessibility: There were no common libraries, tooling and infrastructure to deliver events across systems.
- Productivity: The overall productivity decreases because of the lack of the standard format of events.
Therefore, CNCF (Cloud-Native Computing Foundation) has brought up the concept, called CloudEvents. CloudEvents is a specification that commonly describes event data. Conforming any event data to this spec will simplify the event declaration and delivery across systems and platforms and more, resulting in a huge productivity increase.
How Azure Event Grid brokers CloudEvents
Before CloudEvents, Azure Event Grid described events in their own way. Therefore, if you want to use Azure Event Grid, you should follow the event format/schema that Azure Event Grid declares. However, not every system/service/application follows the Azure Event Grid schema. Therefore, Azure Event Grid now supports CloudEvents spec as input and output formats.
Azure Event Grid for Azure
Take a look at the simple diagram below, which describes how Azure Event Grid captures events raised from various Azure services. In this diagram, Azure Key Vault takes the role of the event source or event publisher, and Azure Logic Apps takes the role of the event handler (I'll discuss Azure Logic Apps as the event handler later in this post). We use Azure Event Grid System Topic for Azure.
Therefore, let's create an Azure Event Grid System Topic that captures events raised from Azure Key Vault when a new version of a secret is added.
As Azure Event Grid makes use of the pub/sub pattern, you need to create the Azure Event Grid Subscription to consume the events. Here's the subscription that uses the Event Grid data format:
![Azure Event Grid System Subscription for Key Vault in Event Grid Format][./img/21-cloudevents-via-event-grid-03.png]
Once you create the subscription, create a new version of the secret on Azure Key Vault. Then, Azure Key Vault raises an event, which is captured in the Event Grid format:
[
{
"id": "6f44b9c0-d37e-40e7-89be-f70a6da291cc",
"topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-aegce-krc/providers/Microsoft.KeyVault/vaults/kv-xxxxxxxx",
"subject": "hello",
"eventType": "Microsoft.KeyVault.SecretNewVersionCreated",
"data": {
"Id": "https://kv-xxxxxxxx.vault.azure.net/secrets/hello/064dfc082fec463f8d4610ed6118811d",
"VaultName": "kv-xxxxxxxx",
"ObjectType": "Secret",
"ObjectName": "hello",
"Version": "064dfc082fec463f8d4610ed6118811d",
"NBF": null,
"EXP": null
},
"dataVersion": "1",
"metadataVersion": "1",
"eventTime": "2022-09-21T07:08:09.1234567Z"
}
]
So, how is it different from the CloudEvents format? Let's take a look. According to the spec, the JSON data in CloudEvents might look like this:
{
"id" : "C234-1234-1234",
"source" : "/mycontext",
"specversion" : "1.0",
"type" : "com.example.someevent",
"comexampleextension1" : "value",
"time" : "2018-04-05T17:31:00Z",
"datacontenttype" : "application/cloudevents+json",
"data" : {
"appinfoA" : "abc",
"appinfoB" : 123,
"appinfoC" : true
}
}
This time, let's create another subscription using the CloudEvents schema. Here's how to create the subscription against the system topic:
Therefore, Azure Key Vault emits the event data in the CloudEvents format:
{
"id": "6f44b9c0-d37e-40e7-89be-f70a6da291cc",
"source": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-aegce-krc/providers/Microsoft.KeyVault/vaults/kv-xxxxxxxx",
"specversion": "1.0",
"type": "Microsoft.KeyVault.SecretNewVersionCreated",
"subject": "hello",
"time": "2022-09-21T07:08:09.1234567Z",
"data": {
"Id": "https://kv-xxxxxxxx.vault.azure.net/secrets/hello/064dfc082fec463f8d4610ed6118811d",
"VaultName": "kv-xxxxxxxx",
"ObjectType": "Secret",
"ObjectName": "hello",
"Version": "064dfc082fec463f8d4610ed6118811d",
"NBF": null,
"EXP": null
}
}
Can you identify some differences between the Event Grid format and the CloudEvents format? Fortunately, both Event Grid schema and CloudEvents schema look similar to each other. But they might be significantly different if you use a different event source outside Azure.
Azure Event Grid for Systems outside Azure
As mentioned above, the event data described outside Azure or your own applications within Azure might not be understandable by Azure Event Grid. In this case, we need to use Azure Event Grid Custom Topic. Here's the diagram for it:
Let's create the Azure Event Grid Custom Topic. When you create the topic, make sure that you use the CloudEvent schema during the provisioning process:
If your application needs to publish events to Azure Event Grid Custom Topic, your application should build the event data in the CloudEvents format. If you use a .NET application, add the NuGet package first.
dotnet add package Azure.Messaging.EventGrid
Then, create the publisher instance. You've already got the topic endpoint URL and the access key.
var topicEndpoint = new Uri("<Azure Event Grid Custom Topic Endpoint URL>");
var credential = new AzureKeyCredential("<Azure Event Grid Custom Topic Access Key>");
var publisher = new EventGridPublisherClient(topicEndpoint, credential);
Now, build the event data like below. Make sure that you follow the CloudEvents schema that requires additional metadata like event source, event type and content type.
var source = "/your/event/source";
var type = "com.source.event.your/OnEventOccurs";
var data = new MyEventData() { Hello = "World" };
var @event = new CloudEvent(source, type, data);
And finally, send the event to Azure Event Grid Custom Topic.
await publisher.SendEventAsync(@event);
The captured event data looks like the following:
{
"id": "cc2b2775-52b8-43b8-a7cc-c1c33c2b2e59",
"source": "/your/event/source",
"type": "com.source.event.my/OnEventOccurs",
"data": {
"Hello": "World"
},
"time": "2022-09-21T07:08:09.1234567+00:00",
"specversion": "1.0"
}
However, due to limitations, someone might insist that their existing application doesn't or can't emit the event data in the CloudEvents format. In this case, what should we do? There's no standard way of sending the event data in the CloudEvents format to Azure Event Grid Custom Topic. One of the approaches we may be able to apply is to put a converter between the existing application and Azure Event Grid Custom Topic like below:
Once the Function app (or any converter app) receives legacy event data, it internally converts the CloudEvents format and publishes it to Azure Event Grid.
var data = default(MyRequestData);
using (var reader = new StreamReader(req.Body))
{
var serialised = await reader.ReadToEndAsync();
data = JsonConvert.DeserializeObject<MyRequestData>(serialised);
}
var converted = new MyEventData() { Hello = data.Lorem };
var @event = new CloudEvent(source, type, converted);
The converted event data is captured like this:
{
"id": "df296da3-77cd-4da2-8122-91f631941610",
"source": "/your/event/source",
"type": "com.source.event.my/OnEventOccurs",
"data": {
"Hello": "ipsum"
},
"time": "2022-09-21T07:08:09.1234567+00:00",
"specversion": "1.0"
}
This approach is beneficial in many integration scenarios to make all the event data canonicalised.
How Azure Logic Apps consumes CloudEvents
I put Azure Logic Apps as the event handler in the previous diagrams. According to the CloudEvents spec, each event handler must implement request validation to avoid abuse. One good thing about using Azure Logic Apps is that it has already implemented this request validation feature. It implies that we just subscribe to the topic and consume the event data.
Create a new Logic Apps instance and add the HTTP Request trigger. Once it saves, you will get the endpoint URL.
Then, create the Azure Event Grid Subscription with:
- Endpoint type: Webhook
- Endpoint URL: The Logic Apps URL from above.
Once the subscription is ready, this Logic Apps works well as the event handler. Here's how it receives the CloudEvents data from the subscription.
Now you've got the CloudEvents data. It's entirely up to you to handle that event data however you want!
Exercise: Try this yourself!
You can fork this GitHub repository to your account and play around with it to see how Azure Event Grid with CloudEvents works. Alternatively, the "Deploy to Azure" button below will provision all necessary Azure resources and deploy an Azure Functions app to mimic the event publisher.
Resources: For self-study!
Want to know more about CloudEvents in real-life examples? Here are several resources you can take a look at: