This is the third and final post in my series around using your voice to query/search Microsoft Identity Manager or as I’m now calling it, the Voice Assistant for Microsoft Identity Manager.
The two previous posts in this series detail some of my steps and processes in developing and fleshing out this Voice Assistant for Microsoft Identity Manager concept. The first post detailed the majority of the base functionality whilst the second post detailed the auditing and reporting aspects into Table Storage and Power BI.
My final architecture is depicted below.
Voice Assistant for Microsoft Identity Manager Architecture
I’ve put together more of an overview in a presentation format embedded here.
Last month I wrote this post that detailed using your voice to search/query Microsoft Identity Manager. That post demonstrated a working solution (GitHub repository coming next month) but was still incomplete if it was to be used in production within an Enterprise. I hinted then that there were additional enhancements I was looking to make. One is an Auditing/Reporting aspect and that is what I cover in this post.
Overview
The one element of the solution that has visibility of each search scenario is the IoT Device. As a potential future enhancement this could also be a Bot. For each request I wanted to log/audit;
Device the query was initiated from (it is possible to have many IoT devices; physical or bot leveraging this function)
The query
The response
Date and Time of the event
User the query targeted
To achieve this my solution is to;
On my IoT Device the query, target user and date/time is held during the query event
At the completion of the query the response along with the earlier information is sent to the IoT Hub using the IoT Hub REST API
The event is consumed from the IoT Hub by an Azure Event Hub
The message containing the information is processed by Stream Analytics and put into Azure Table Storage and Power BI.
Azure Table Storage provides the logging/auditing trail of what requests have been made and the responses. Power BI provides the reporting aspect. These two services provide visibility into what requests have been made, against who, when etc. The graphic below shows this in the bottom portion of the image.
Voice Search for Microsoft Identity Manager Auditing and Reporting
Sending IoT Device Events to IoT Hub
I covered this piece in a previous post here in PowerShell. I converted it from PowerShell to Python to run on my device. In PowerShell though for initial end-to-end testing when developing the solution the body of the message being sent and sending it looks like this;
[string]$datetime = get-date
$datetime = $datetime.Replace("/","-")
$body = @{
deviceId = $deviceID
messageId = $datetime
messageString = "$($deviceID)-to-Cloud-$($datetime)"
MIMQuery = "Does the user Jerry Seinfeld have an Active Directory Account"
MIMResponse = "Yes. Their LoginID is jerry.seinfeld"
User = "Jerry Seinfeld"
}
$body = $body | ConvertTo-Json
Invoke-RestMethod -Uri $iotHubRestURI -Headers $Headers -Method Post -Body $body
Event Hub and IoT Hub Configuration
First I created an Event Hub. Then on my IoT Hub I added an Event Subscription and pointed it to my Event Hub.
Azure IoT Hub Events
Streaming Analytics
I then created a Stream Analytics Job. I configured two Inputs. One each from my IoT Hub and from my Event Hub.
Azure Stream Analytics Inputs
I then created two Outputs. One for Table Storage for which I used an existing Storage Group for my solution, and the other for Power BI using an existing Workspace but creating a new Dataset. For the Table storage I specified deviceId for Partition key and messageId for Row key.
Azure Stream Analytics Outputs
Finally as I’m keeping all the data simple in what I’m sending, my query is basically copying from the Inputs to the Outputs. One is to get the events to Table Storage and the other to get it to Power BI. Therefore the query looks like this.
Azure Stream Analytics Query
Events in Table Storage
After sending through some events I could see rows being added to Table Storage. When I added an additional column to the data the schema-less Table Storage obliged and dynamically added another column to the table.
Table Storage Events
A full record looks like this.
Voice Search Table Storage Audit Record
Events in Power BI
Just like in Table Storage, in Power BI I could see the dataset and the table with the event data. I could create a report with some nice visuals just as you would with any other dataset. When I added an additional field to the event being sent from the IoT Device it magically showed up in the Power BI Dataset Table.
PowerBI Voice Search Analytics
Summary
Using the Azure IoT Hub REST API I can easily send information from my IoT Device and then have it processed through Stream Analytics into Table Storage and Power BI. Instant auditing and reporting functionality.
Yes, you’ve read the title correctly. Speaking to Microsoft Identity Manager. The concept behind this was born off the back of some other work I was doing with Microsoft Cognitive Services. I figured it shouldn’t be that difficult if I just break down the concept into individual elements of functionality and put together a proof of concept to validate the idea. That’s what I did and this is the first post of the solution as an overview.
Here’s a quick demo.
Overview
The diagram below details the basis of the solution. There are a few extra elements I’m still working on that I’ll cover in a future post if there is any interest in this.
The solution works like this;
You speak to a microphone connected to a single board computer with the query for Microsoft Identity Manager
The spoken phrase is converted to text using Cognitive Speech to Text (Bing Speech API)
The text phrase is;
sent to Cognitive Services Language Understanding Intelligent Service (LUIS) to identify the target of the query (firstname lastname) and the query entity (e.g. Mailbox)
Microsoft Identity Manager is queried via API Management and the Lithnet REST API for the MIM Service
The result is returned to the single board computer as a text result phase which it then uses Cognitive Services Text to Speech to convert the response to audio
Leveraging a lot of Serverless (PaaS) Services, a bunch of scripting (Python on the ReSpeaker and PowerShell in the Azure Function) and the Lithnet REST API it was pretty simple to integrate the ReSpeaker with Microsoft Identity Manager. An alternative to MIM could be any other service you have an API interface into. MIM is obviously a great choice as it can aggregate from many other applications/services.
Why a female voice? From a small response it was the popular majority.
Different IoT Devices have different capabilities. Whether it is a Micro-controller or Single Board Computer your options will vary. In this post I detailed using MQTT to send messages from an IoT Device to an Azure IoT Hub as well as using the AzureIoT PowerShell Module.
For a current project I needed to send the events from an IoT Device that runs Linux and had Python support. The Azure IoT Hub includes an HTTPS REST endpoint. For this particular application using the HTTPS REST endpoint is going to be much easier than compiling the Azure SDK for the particular flavour of Linux running on my device.
Python isn’t my language of choice so first I got it working in PowerShell then converted it to Python. I detail both scripts here as a guide for anyone else trying to do something similar but also for myself as I know I’m going to need these snippets in the future.
PowerShell Device to Cloud Events using HTTPS and REST Script
Here is the PowerShell version of the script. Update Line 3 for your DeviceID, Line 5 for your IoT Hub Name and LIne 11 for your SAS Token.
Using Device Explorer to Monitor the Device on the associated IoT Hub I can see that the message is received.
Python Device to Cloud Events using HTTPS and REST Script
Here is my Python version of the same script. Again update Line 5 for your IoT DeviceID, Line 7 for your IoT Hub and Line 12 for the SAS Token.
And in Device Explorer we can see the message is received.
Summary
When you have a device that has the ability to run Python you can use the IoT Hub HTTPS REST API to send messages from the Client to Cloud negating the need to build and compile the Azure IoT SDK to generate client libraries.
