Hi Microsoft 365 devs !
Here we go again with the part III of my blog post series about Microsoft Graph Subscriptions (aka Webhooks). In this post, we will see how we can implement a real world solution that handles the lifetime of our Microsoft Graph subscriptions !
What's the matter with subscription lifetime ?
Microsoft Graph subscriptions are great to hook our custom logic triggered by the events that occur in our corporate information (groups, users, events, and so on...). However, as we have seen in part I & II of this series, the subscriptions we create expire after quite a small amount of time. This expiration allows Microsoft Graph to spare execution cycles that may, in fact, be totally useless since the remote system may no longer use them! So, as long as we want our custom solution to handle properly the notifications and keep receiving them, we need to implement some logic that will allow us to do so... It requires a bit of a significant effort to achieve something that might seem trivial, but it is probably also a way to ensure that, we, third parties, are not overwhelming Microsoft Graph without caring what we do without a very good reason to do it.
Let's architect a viable solution !
In the previous part, we saw how to implement our Webhook as a pro with Azure Functions using TypeScript and VS Code. We will rely on these foundations to make sure that the lifetime of our subscription are properly extended anytime it is needed. Concretely, it means that, whenever a notification is sent to our system, the Webhook will not only trigger our custom business logic to be executed, but will also update the subscription to postpone the date of its expiration. In order to respond as fast as possible to Graph (as expected from Webhook endpoints), we will delegate the tasks to other functions that gets executed when a message is pushed into a queue. Let's see how the code of our Webhook will look like
import { AzureFunction, Context, HttpRequest } from "@azure/functions"
import { INotificationsPayload } from "../entities/INotification";
import { SUBSCRIPTIONS_QUEUE_NAME, BUSINESS_TASKS_QUEUE_NAME } from "../helpers/configure";
import { addMessageToQueue } from "../helpers/azure-queue-helper";
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
context.log('HTTP trigger function processed a request.');
// Validate the subscription creation
if (req.query.validationToken) {
context.log('Validating new subscription...');
context.log('Validation token:');
context.log(req.query.validationToken);
context.res = {
headers: {
'Content-Type': 'text/plain'
},
body: req.query.validationToken
};
}
else {
context.log('Received new notification from Microsoft Graph...');
context.log('Notifications: ');
const payload = req.body as INotificationsPayload;
payload.value.forEach(async (n, i) => {
context.log(` Notification #${i} `);
context.log(`----------------------------------------------------------`);
context.log(`Subscription Id : ${n.subscriptionId}`);
context.log(`Expiration : ${n.subscriptionExpirationDateTime}`);
context.log(`Change Type : ${n.changeType}`);
context.log(`Client State : ${n.clientState}`);
context.log(`Resource : ${n.resource}`);
const resourceData = n.resourceData && JSON.stringify(n.resourceData);
context.log(`Resource Data : ${resourceData}`);
context.log(`----------------------------------------------------------`);
// ====================================================================================
// HERE we delegate the business logic work to a dedicated queue triggered function
// We pass any needed parameter that might come from the notification
// ====================================================================================
try {
const taskArgs = {
eventId: n.resource
};
await addMessageToQueue(BUSINESS_TASKS_QUEUE_NAME, JSON.stringify(taskArgs));
}
catch (err) {
context.log.error(err);
}
// ====================================================================================
// ====================================================================================
// HERE Ensure the subscription lifetime will be extended
// ====================================================================================
// we push the subscription id to the subscriptions queue
// The EnsureSubscription queue-triggered function
// will in turn update the expiration of the subscription if needed
try {
await addMessageToQueue(SUBSCRIPTIONS_QUEUE_NAME, n.subscriptionId);
}
catch (err) {
context.log.error(err);
}
});
context.res = { body: "" };
}
};
export default httpTrigger;
As we can see at line 60, we push the subscription id to a queue, an other Azure Function we call EnsureSubscription will be triggered and will be responsible for updating the expiration date of the subscription. Another queue-triggered Function we call BusinessTask will achieve what we really want to do with the notification.
Setup some prerequisites
- It will use Microsoft Graph to update the subscription expiration. Thus, it needs to be allowed to. It will do it in an unattended manner. If you follow me, it means we need to register an AAD application and grant it the needed application permissions
- To implement the calls to Microsoft Graph in the easiest possible way, I will use the awesome PnP JS library (however, that is not strictly required)!
Register the AAD application
From the Azure AD portal, go to the App registrations section and click Add new registration
Select a name and register the new application. We'll need to grant our application the required permissions, From the API Permissions section, add the following permissions to Microsoft Graph, don't forget to grant admin consent once you set the API permissions
We will put the needed AAD app information in our function app settings. At development time, it will be in the local.settings.json file that should probably already exist in your solution. If it does not, you can create it with the keys we'll set the values later on
{
"IsEncrypted": false,
"Values": {
"TENANT": "",
"AAD_APP_ID": "",
"AAD_APP_SECRET": "",
"OVERRIDDEN_WEBSITE_HOSTNAME": "",
"SUBSCRIBED_RESOURCE_URI" : "",
"FUNCTIONS_WORKER_RUNTIME": "node",
"AZURE_STORAGE_CONNECTION_STRING": "",
"AzureWebJobsStorage": ""
}
}
From the
- Copy the Directory (tenant) Id and paste it as the value of the TENANT app setting
- Copy the Application (client) Id and paste it as the value of the AAD_APP_ID app setting
Since our Azure Function will use Microsoft Graph in an unattended manner (without any user interaction), We will use client credentials authentication flow, we then need to generate a client secret.
From the Certificates & secret section, click the New client secret button to generate a new one, copy the generated secret as the value of the key AAD_APP_SECRET in your app settings. Make sure to copy it immediately, because you will never be able to see it again. Note: I'll use a client secret it to avoid bring more complexity to the steps here. It is actually recommended that you rather use client certificates instead which are safer.
Install the dependencies
As mentioned above, we will use PnP JS, we will also need to use the Azure Storage API. In a console in your solution folder, type the following command
npm install @pnp/graph-commonjs @pnp/logging @pnp/nodejs-commonjs azure-storage --save
Tip for development time
Since we are now doing much more development, it will probably be easier to run our azure function locally, it will be much easier to debug ! But As you probably understood by now, Microsoft Graph subscriptions need to access a public URI. That's most likely not the case of our development web server. We will use ngrok to redirect a public facing URL to our local development machine. We'll need to download the small tool. Run the tool as such
ngrok http 7071
Note that 7071 is the http port used by the development Azure Function runtime, you'll probably have to make sure it is the same port used in your local runtime, but it is most likely the case ! We will then need to copy the ngrok public URL to our app settings
Copy the https URL as the value of OVERRIDDEN_WEBSITE_HOSTNAME app setting.
Configure the Microsoft Graph Resource to subscribe to
In the app settings, we'll also need to specify the resource we want to subscribe to, concretely, it means set the Microsoft Graph relative URL of the resource. For instance, here I will subscribe to the events of my user, I will need to specify it using the user id like "users/efda010b-19a4-4303-bf7b-70bd6e62e7d2/events", since we will use Microsoft Graph using an app-only token, we cannot use "me/events"
Let's create a few helpers
In a helpers folder: create a configure.ts with the following code
import { graph } from "@pnp/graph-commonjs";
import { AdalFetchClient } from "@pnp/nodejs-commonjs";
import { Context } from "@azure/functions";
export const SUBSCRIPTIONS_QUEUE_NAME = "subscriptions-to-ensure";
export const BUSINESS_TASKS_QUEUE_NAME = "business-tasks";
const TENANT = process.env.TENANT;
const AAD_APP_ID = process.env.AAD_APP_ID;
// TODO That should rather be fetched from Azure Keyvault instead
const AAD_APP_SECRET = process.env.AAD_APP_SECRET;
export const WEBSITE_HOSTNAME = process.env.OVERRIDDEN_WEBSITE_HOSTNAME || process.env.WEBSITE_HOSTNAME;
// Probably a good idea to store these in app settings or in any easily updatable location
export const SETUP_CHANGE_TYPE = "updated";
export const SETUP_NOTIFICATION_URL = `${WEBSITE_HOSTNAME}/api/Webhook`;
export const SETUP_RESOURCE = process.env.SUBSCRIBED_RESOURCE_URI;
// NOTE Cannot subscribe with app permissions to groups events
// export const SETUP_RESOURCE = "groups/a619f4ce-xxxxxx-xxxx-f5df845c5e96/calendar/events";
export const SETUP_EXPIRATION_DELAY_IN_DAYS = 3;
export function configure(functionContext: Context) {
functionContext.log(`Using function app base url: '${WEBSITE_HOSTNAME}'`);
functionContext.log(`Using tenant : '${TENANT}'`);
functionContext.log(`Using ADD App Id : '${AAD_APP_ID}'`);
graph.setup({
graph: {
fetchClientFactory: () => {
return new AdalFetchClient(TENANT, AAD_APP_ID, AAD_APP_SECRET);
},
},
});
}
create a file date-helper.ts
export const DEFAULT_SAFETY_WINDOW_IN_MSECS = 4860000; // 1h31 (Graph common delay is 4230 rounded minutes (70.5 h))
export const getExpirationDateTimeISOString = (fromNowInDays: number, safetyWindowInMsecs?: number) => {
const now = Date.now();
// day 24h 60m 60s ms
// const expiration = now + (fronNowInDays * 24 * 60 * 60 * 1000);
const expiration = now + (fromNowInDays * 86400000) - (safetyWindowInMsecs || DEFAULT_SAFETY_WINDOW_IN_MSECS);
return new Date(expiration).toISOString();
};
export const isNearlyExpired = (dateIsoString: string, safetyWindowInMsecs?: number) => {
const date = Date.parse(dateIsoString);
return (date - (safetyWindowInMsecs || DEFAULT_SAFETY_WINDOW_IN_MSECS)) <= Date.now();
}
create a file azure-queue-helper.ts
import * as azure from "azure-storage";
let _queueService = null;
const getQueueService = () => {
if (!_queueService) {
_queueService = azure.createQueueService();
}
return _queueService;
}
export const addMessageToQueue = async (queue: string, message: string): Promise<void> => {
return new Promise((resolve, reject) => {
const queueService = getQueueService();
queueService.createQueueIfNotExists(queue, (error, results, response) => {
if (!error) {
const base64EncodedMessage = new Buffer(message).toString("base64");
queueService.createMessage(queue, base64EncodedMessage, (err, msgResult, msgResp) => {
if (err) {
reject("Could not send message to queue");
}
resolve();
})
} else {
reject("Queue could not be created.");
}
});
});
}
create a file subscription-helpers.ts
import { Logger } from "@azure/functions"
import { graph } from "@pnp/graph-commonjs";
import { Subscription } from "@microsoft/microsoft-graph-types";
const isValidForCreate = (subscription: Subscription) => {
return !!(subscription.changeType && subscription.notificationUrl && subscription.resource && subscription.expirationDateTime);
}
const isValidForUpdate = (subscription: Subscription) => {
return !!(subscription.id && subscription.expirationDateTime);
}
export const getSubscription = async function (subscriptionId: string, log: Logger): Promise<Subscription> {
try {
const subscription = await graph.subscriptions.getById(subscriptionId).get();
return subscription;
} catch (error) {
log.warn(`An error occured while trying to get subscription with Id ${subscriptionId}`, error);
return null;
}
}
export const createSubscription = async function (subscription: Subscription, log: Logger): Promise<Subscription> {
if (isValidForCreate(subscription)) {
log("Creating new subscription...");
const { changeType, notificationUrl, resource, expirationDateTime } = subscription;
const result = await graph.subscriptions.add(changeType, notificationUrl, resource, expirationDateTime, subscription);
log(`New subscription to ${changeType} of ${resource} has been created.`);
return result.data;
}
else {
log.error("Received request is not valid to create a subscription...");
}
}
export const updateSubscription = async function (subscription: Subscription, log: Logger): Promise<void> {
if (isValidForUpdate(subscription)) {
log(`Updating subscription ${subscription.id}...`);
await graph.subscriptions.getById(subscription.id).update(subscription);
log(`Subscription ${subscription.id} has been updated.`);
} else {
log.error("Received request is not valid to update the subscription...");
}
};
and a last one subscription-setup.ts
import { Logger } from "@azure/functions";
import { createSubscription } from "./subscriptions-helper";
import { getExpirationDateTimeISOString } from "./date-helper";
import { SETUP_RESOURCE, SETUP_CHANGE_TYPE, SETUP_NOTIFICATION_URL } from "./configure";
export const setupNewSubscription = async function(log: Logger) {
log("Let's create a new subscription...");
const created = await createSubscription({
resource: SETUP_RESOURCE,
changeType: SETUP_CHANGE_TYPE,
notificationUrl: SETUP_NOTIFICATION_URL,
expirationDateTime: getExpirationDateTimeISOString(3)
}, log);
log(`A new subscription has been created with id ${created.id}`);
log("Please update the configuration accordingly...");
}
Let's code the EnsureSubscription function
First of all, let's create a new queue-triggered function, from VS Code Azure extension, click the "Create function" button. Select "Azure Queue Storage Triggered" and name it "EnsureSubscription". Select the app setting you want to use to store the Azure Storage connection string. You will have to put the connection string to your storage account. The VS Code extensions also allows to create a new storage account if you don't already have one. Name the azure queue as "subscriptions-to-ensure" Then let's write the following code as the body of our EnsureSubscription function
import { AzureFunction, Context } from "@azure/functions"
import { isNearlyExpired, getExpirationDateTimeISOString } from "../helpers/date-helper";
import { updateSubscription, getSubscription } from "../helpers/subscriptions-helper";
import { setupNewSubscription } from "../helpers/subscription-setup";
import { configure, SETUP_EXPIRATION_DELAY_IN_DAYS } from "../helpers/configure";
const queueTriggeredEnsureSubscription: AzureFunction = async function (context: Context, subscriptionId: string): Promise<void> {
context.log(`Ensuring subscription ${subscriptionId}...`);
configure(context);
const foundSubscription = await getSubscription(subscriptionId, context.log);
if (foundSubscription) {
context.log("A subscription has been found !");
context.log(`Resource : ${foundSubscription.resource}`);
context.log(`Change type : ${foundSubscription.changeType}`);
context.log(`Expires on : ${foundSubscription.expirationDateTime}`);
// At this point, if we realize the subscription is expiring soon, we update it with a new maximum delay starting right now !
if (isNearlyExpired(foundSubscription.expirationDateTime)) {
const expirationDateTime = getExpirationDateTimeISOString(SETUP_EXPIRATION_DELAY_IN_DAYS);
context.log(`Will update the expiration to ${expirationDateTime}`);
await updateSubscription({
id: foundSubscription.id,
expirationDateTime
},
context.log);
context.log("Subscription updated.");
} else {
context.log("The subscription will not expire soon...");
}
} else {
context.log(`No subscription has been found with id ${subscriptionId}...`);
// If subscription is not found, it might have been deleted in the meantime for some reason
// In order to keep the whole system stable, we recreate a new subscription here
await setupNewSubscription(context.log);
}
};
export default queueTriggeredEnsureSubscription;
Let's code the Business Task function
We will create another queue-triggered Azure Function that will execute the actual task we want to achieve whenever a notification is received. Here my business task will be to simply log the Id of the updated event. It is here that you will have to change to code to achieve the task according to your requirements. Repeat the steps above to create a new queue-triggered Function and name the queue "business-tasks". Just for information, the code I have in this function is
import { AzureFunction, Context } from "@azure/functions";
import { IBusinessTaskArgs } from "../entities/IBusinessTaskArgs";
import { GraphHttpClient } from "@pnp/graph-commonjs/graphhttpclient";
import { configure } from "../helpers/configure";
const queueTrigger: AzureFunction = async function (context: Context, taskArgs: IBusinessTaskArgs): Promise<void> {
// The calendar event is post process by our custom logic
context.log(`Received arguments: ${JSON.stringify(taskArgs)}`);
configure(context);
try {
// TODO Here implement custom logic
// e.g. Update the updated event's title
const resourceAbsoluteUrl = `https://graph.microsoft.com/v1.0/${taskArgs.eventResourceUri}`;
const client = new GraphHttpClient();
const eventResponse = await client.get(resourceAbsoluteUrl);
const event: { subject: string; } = await eventResponse.json();
// If event is not already processed
// Since we are updating the notified updated event, we have to check the custom task is not already done
// Otherwise, a notification will be resent and cause an "infinite" loop
if (event.subject.indexOf("[UPDATED]") != 0) {
const eventUpdate = {
subject: `[UPDATED] ${event.subject}`
}
const patchResponse = await client.patch(resourceAbsoluteUrl, {
body: JSON.stringify(eventUpdate)
});
if (patchResponse.ok) {
console.log("Event has been updated.");
} else {
console.log("Event could not be updated");
}
}
}
catch (error) {
context.log.error("An error occured while processing custom business task", error);
}
};
export default queueTrigger;
This custom task will grab the updated event and update its subject if it is not already properly updated (This will avoid resulting in an infinite loop!).
Let's test it
We should be good now, hit F5 in VS Code to run your Azure Function locally ! We will then need to trigger the subscription to be created ! To make sure all is going well, set a breakpoint in the EnsureSubscription index.ts file at line 32 From the Azure Portal, access your storage account, select the subscriptions-to-ensure queue and add a new message with any dummy content
Your breakpoint should be hit almost at once. Continue the execution and your subscription should be added. To test your webhook solution, set a breakpoint in your BusinessTask function and then add an event to the calendar of the user you specified (or do any action that should triggered the specific notification you requested). In my case, I create or update an event in my calendar (It is the resource monitored by the Microsoft Graph Subscription), and I see it gets updated after a few seconds
Now time to deploy !
If everything in debug works as expected you can now deploy the Azure Function to Azure, in the VS Code extension tab, click the "Deploy to Function App.." button. Then you will have to make sure you copy the following app settings to the Function App
- TENANT
- AAD_APP_ID
- AAD_APP_SECRET
- SUBSCRIBED_RESOURCE_URI
Conclusion
We now have a solution that will keep our subscriptions alive as long as it is used. Take care however, with this solution as is, if there is no activity during the maximum lifetime of your particular subscription, it will get deleted ! Fortunately, with the solution described in this post, the only thing you will have to do is to setup a time triggered function that will trigger the EnsureSubscription, But that will be part of a next post :) You can get the whole solution on the GitHub repo here
Hope you liked it and please let me know any feedback you might have !
Cheers !
Yannick