Azure Files Streaming Configuration

This document provides information on how to configure Azure Files connection with real-time events monitoring and data streaming.

Prerequisites

1. Existing Azure Files connection: An Azure Files scan configuration must already exist.

   • If an Azure Files scan does not already exist, follow this guide to create a new Azure Files scan and ensure the necessary credentials are set up.

Steps to Enable Data Streaming

1. Select an Existing Scan Configuration

1. Go to the Scan configurations page in the product UI.

2. Locate your existing Azure Files scan configuration and select Edit Configuration from the options menu. Note the configured path (folder) and save it, as it will be used in step 9 to replace {FolderPath}.

2. Enable Data Streaming

1. Within the Edit Azure Files Scan Configuration page, toggle Data Streaming to ON.

2. Copy the Webhook URL provided, as you will use it later in the Azure Portal. Save this Webhook URL, as it will be used in step 9 to replace {WebhookUrl}.

1. Click Save & Close button to save configuration.

3. Create Azure Event Hub

1. Navigate to Azure Portal Event hubs and click Create

1. In Create Namespace Window fill in the details

   1. Give it a Name

   2. Select your subscription and resource group

   3. Select location

   4. Pricing tier - standard

   5. Throughput Units - 1

1. Click on Review + Create and then Create after validation

1. After namespace is created, click on + Event Hub button

   
2. In Create Event Hub Window fill in name and click Create + Review and Create after validation. Save the name of the Event Hub you created in this step, as it will be used later in step 9 to replace {eventHubName}.

1. Configure access policy

   1. In the event hubs namespace window click on Settings/Shared access policies and then +Add button

      

   2. Fill in the details in the new tab, set LogicAppsListenerPolicy as name, select Listen policy, and click Save.

   3. Click on the newly created policy, then copy and save the Connection string–primary key. This will be needed later in step 8b.

4. Configure Azure Storage Diagnostic settings

1. Navigate to Azure Portal and open your Storage Account.

   
2. Select needed account from the Storage Accounts

   
3. In the left-hand menu, select Monitoring/Diagnostic settings and click file

   
4. In Diagnostic settings Window click on "+ Add diagnostic setting" button

   
5. In Create Diagnostic setting Window fill in the details:

   1. Give it a Name

   2. Select Category groups allLogs

   3. Select Destination details Stream to an event hub and select newly created Event Hub Namespace and Event Hub

      
   4. Click Save.

5. Configure Azure Logic Apps

1. Go to Azure logic apps and click "Add" button

   
2. In Create Logic App Window select Workflow Service Plan

3. In Create Logic App (Workflow Service Plan) Window fill in the details and click "Create + Review":

   1. Select your subscription and resource group

   2. Give logic app name

   3. Select region

   4. Pricing plan should be WS1

   5. In the monitoring tab select No for the application insights

      

   6. Click Review + create button

4. Click Create after validation

5. In newly created logic app click on Workflows/Workflows and then +Add button

6. In new workflow tab fill in name, select State type: Stateful and click Create

   
7. In created workflow go to Developer/Designer and click on Add a trigger, then in search type "Event hub" and select "When events are available in Event Hub"

   
8. Configure API connection

   1. Click on the trigger, set "Temp" for Event Hub Name and then click on Change connection.

      
   2. Then click Add New and fill in the details. Enter any name for the connection name and use the connection string {Connection string–primary key} from step 3.6.c.

   3. On the Change Connection tab, click Details and copy the Name from the connection details. Save this Name, as it will be used later in step 9 to replace {connectionName}.

      

   4. Click save on workflow designer window

      
9. In workflow navigation tab go to Developer/Code and set the provided code, then click save:

   1. Replace {FolderPath} with a path to the streaming folder. For ex., you want to get events from the folder "StreamingFolder" which is located in file share with the name "DocumentsShare" and in the folder with the name "Personal". In this case, the path should be "DocumentsShare/Personal/StreamingFolder"

   2. Replace {WebhookUrl} with webhook url provided in the application in the scan configuration window

   3. Replace {eventHubName} with azure event hub name that was created previously

   4. Replace {connectionName} with connection name from previouse step

      {
          "definition": {
              "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
              "actions": {
                  "Filter_Records": {
                      "type": "Query",
                      "inputs": {
                          "from": "@triggerBody()?['ContentData']?['records']",
                          "where": "@and(not(empty(item()?['uri'])),or(contains(item()?['uri'], '{FolderPath}/'),contains(item()?['uri'], '{FolderPath}?')))"
                      },
                      "runAfter": {}
                  },
                  "Condition": {
                      "type": "If",
                      "expression": "@greater(length(body('Filter_Records')), 0)",
                      "actions": {
                          "HTTP-copy": {
                              "type": "Http",
                              "inputs": {
                                  "uri": "{WebhookUrl}",
                                  "method": "POST",
                                  "headers": {
                                      "Content-Type": "application/json"
                                  },
                                  "body": {
                                      "event": "@setProperty(triggerBody(),'ContentData',setProperty(triggerBody()?['ContentData'],'records',body('Filter_Records')))"
                                  }
                              },
                              "runAfter": {}
                          }
                      },
                      "else": {},
                      "runAfter": {
                          "Filter_Records": [
                              "Succeeded"
                          ]
                      }
                  }
              },
              "contentVersion": "1.0.0.0",
              "outputs": {},
              "triggers": {
                  "When_events_are_available_in_Event_Hub": {
                      "type": "ApiConnection",
                      "inputs": {
                          "host": {
                              "connection": {
                                  "referenceName": "{connectionName}"
                              }
                          },
                          "method": "get",
                          "path": "/@{encodeURIComponent('{eventHubName}')}/events/batch/head",
                          "queries": {
                              "contentType": "application/json",
                              "consumerGroupName": "$Default",
                              "maximumEventsCount": 50
                          }
                      },
                      "recurrence": {
                          "interval": 30,
                          "frequency": "Second"
                      },
                      "splitOn": "@triggerBody()"
                  }
              }
          },
          "kind": "Stateful"
      }
      

Next Steps

After configuring the event subscription:

• You may upload documents to the configured path.

• The events triggered by these uploads will be processed by the Data Streaming setup, and the results will appear in your Getvisibility dashboard.

Troubleshooting

If you experience any issues with the configuration, ensure that:

1. The Webhook URL is correct and matches the configuration in Azure.

2. Steps 5.8 and 5.9 properly executed and all the variables are replaced with real values.

3. You can also check if the trigger was unsuccessful by navigating to your configured in previos steps Logic App, then Workflow and Trigger History. If you see any failed triggers, you can inspect the error details to identify the issue.