Using Azure Cognitive Services to Empower the IT Service Desk

Tonight (29 August 2018) I presented the following presentation on using Azure Cognitive Services to Empower the IT Service/build Business Applications to the Sydney Azure User Group.

I walked through how to leverage the following Azure Cognitive API Services;

  • Speech to Text
  • Text to Speech
  • Language Understanding Intelligent Service (LUIS)
  • Text Language Translation

For each I show a working example and demo.

I then walked through my Voice Assistant for Microsoft Identity Manager and how I integrated three of those services along with Azure IoT, Azure Serverless and Azure PaaS Services.

GitPitch Presents: github/darrenjrobinson/SydAzureUG-CognitiveServices

The Markdown Presentation Service on Git.

If you weren’t present I hope this presentation can still give you a start to integrating and leveraging Azure Services.

Find me on Twitter @darrenjrobinson

Using Azure Cognitive Services Language Text Translation with PowerShell

Introduction

Over the last few months whilst developing my Voice Assistant for Microsoft Identity Manager I’ve been leveraging a number of the Azure Cognitive Services. Each one has its own nuance as they all appear to be in differing iterations of maturity. My first hurdle when looking to leverage one, is the examples provided. Often the samples are in languages I’m not fluent in and pretty much always there is no examples of using PowerShell and the awesome Invoke-RestMethod call to interact with them. Of course there are the PowerShell Modules, but I normally like to go direct and not have dependancies on a module.

Once I’ve worked it out how to leverage each service with PowerShell I’ve posted how to set up an API call for future reference. Here are the previous ones;

The final service I was looking to leverage (with respect to Audio and Text) is the Language Translator. This is yet another API with its own quirks and it took me longer than it should have. So as I know I’ll need it again in the future and I’m sure it will help others, I’m detailing it here.

Getting Started with the Microsoft Translate Text Cognitive Service

Like the other Cognitive Services I’ve detailed in the past, the Translator has its own API which is currently up to version 3. Obtain a Translator Text API Key free trial from here.

Here is an example PowerShell script that you will be able to leverage as a getting started guide to interfacing with the Translator Text API with PowerShell. Once I got it working it is quite simple. Here it is;

  • update Line 2 for your API key
  • update Lines 6 and 9 for your From and To languages
  • update Line 17 for the text string you want to convert

Summary

Updating a few lines and stepping through the script we can see that it is possible to quickly leverage the Text Translator service to convert (in this example) from English to German.

Translating Language Text with PowerShell.PNG

Interestingly Klingon is an option to convert to too. Change Line 9 from ‘de‘ to ‘tlh‘ if you want to try it for yourself. It looks like quite a concise language ūüėČ

'Translating between languages is easy with Azure' converted to 'mugh SabtaHbogh Hol ngeD Azure'

A Voice Assistant for Microsoft Identity Manager

This is the third and final post in my series around using your voice to query/search Microsoft Identity Manager or as I’m now calling it, the Voice Assistant for Microsoft Identity Manager.

The two previous posts in this series detail some of my steps and processes in developing and fleshing out this concept. The first post detailed the majority of the base functionality whilst the second post detailed the auditing and reporting aspects into Table Storage and Power BI.

My final architecture is depicted below.

Identity Manager integration with Cognitive Services and IoT Hub 4x3
Voice Assistant for Microsoft Identity Manager Architecture

I’ve put together more of an overview in a presentation format embedded here.

GitPitch Presents: github/darrenjrobinson/MIM-VoiceAssistant/presentation

The Markdown Presentation Service on Git.

If you’re interested in building the solution checkout the Github Repo here which includes the Respeaker Python Script, Azure Function etc.

Let me know how you go @darrenjrobinson

Using your Voice to Search Microsoft Identity Manager ‚Äď Part 2

Introduction

Last month I wrote this post that detailed using your voice to search/query Microsoft Identity Manager. That post demonstrated a working solution (GitHub repository coming next month) but was still incomplete if it was to be used in production within an Enterprise. I hinted then that there were additional enhancements I was looking to make. One is an Auditing/Reporting aspect and that is what I cover in this post.

Overview

The one element of the solution that has visibility of each search scenario is the IoT Device. As a potential future enhancement this could also be a Bot. For each request I wanted to log/audit;

  • Device the query was initiated from (it is possible to have many IoT devices; physical or bot leveraging this function)
  • The query
  • The response
  • Date and Time of the event
  • User the query targeted

To achieve this my solution is to;

  • On my IoT Device the query, target user and date/time is held during the query event
  • At the completion of the query the response along with the earlier information is sent to the IoT Hub using the IoT Hub REST API
  • The event is consumed from the IoT Hub by an Azure Event Hub
  • The message containing the information is processed by Stream Analytics and put into Azure Table Storage and Power BI.

Azure Table Storage provides the logging/auditing trail of what requests have been made and the responses.  Power BI provides the reporting aspect. These two services provide visibility into what requests have been made, against who, when etc. The graphic below shows this in the bottom portion of the image.

Auditing Reporting Searching MIM with Speech.png
Voice Search for Microsoft Identity Manager Auditing and Reporting

Sending IoT Device Events to IoT Hub

I covered this piece in a previous post here in PowerShell. I converted it from PowerShell to Python to run on my device. In PowerShell though for initial end-to-end testing when developing the solution the body of the message being sent and sending it looks like this;

[string]$datetime = get-date
$datetime = $datetime.Replace("/","-")
$body = @{
 deviceId = $deviceID
 messageId = $datetime
 messageString = "$($deviceID)-to-Cloud-$($datetime)"
 MIMQuery = "Does the user Jerry Seinfeld have an Active Directory Account"
 MIMResponse = "Yes. Their LoginID is jerry.seinfeld"
 User = "Jerry Seinfeld"
}

$body = $body | ConvertTo-Json
Invoke-RestMethod -Uri $iotHubRestURI -Headers $Headers -Method Post -Body $body

Event Hub and IoT Hub Configuration

First I created an Event Hub. Then on my IoT Hub I added an Event Subscription and pointed it to my Event Hub.

IoTHub Event Hub.PNG
Azure IoT Hub Events

Streaming Analytics

I then created a Stream Analytics Job. I configured two Inputs. One each from my IoT Hub and from my Event Hub.

Stream Analytics Inputs.PNG
Azure Stream Analytics Inputs

I then created two Outputs. One for Table Storage for which I used an existing Storage Group for my solution, and the other for Power BI using an existing Workspace but creating a new Dataset. For the Table storage I specified deviceId for Partition key and messageId for Row key.

Stream Analytics Outputs.PNG
Azure Stream Analytics Outputs

Finally as I’m keeping all the data simple in what I’m sending, my query is basically copying from the Inputs to the Outputs. One is to get the events to Table Storage and the other to get it to Power BI. Therefore the query looks like this.

Stream Analytics Query.PNG
Azure Stream Analytics Query

Events in Table Storage

After sending through some events I could see rows being added to Table Storage. When I added an additional column to the data the schema-less Table Storage obliged and dynamically added another column to the table.

Table Storage.PNG
Table Storage Events

A full record looks like this.

Full Record.PNG
Voice Search Table Storage Audit Record

Events in Power BI

Just like in Table Storage, in Power BI I could see the dataset and the table with the event data. I could create a report with some nice visuals just as you would with any other dataset. When I added an additional field to the event being sent from the IoT Device it magically showed up in the Power BI Dataset Table.

PowerBI.PNG
PowerBI Voice Search Analytics

Summary

Using the Azure IoT Hub REST API I can easily send information from my IoT Device and then have it processed through Stream Analytics into Table Storage and Power BI. Instant auditing and reporting functionality.

Let me know what you think on twitter @darrenjrobinson

Using your Voice to Search Microsoft Identity Manager – Part 1

Introduction

Yes, you’ve read the title correctly. Speaking to Microsoft Identity Manager. The concept behind this was born off the back of some other work I was doing with Microsoft Cognitive Services. I figured it shouldn’t be that difficult if I just break down the concept into individual elements of functionality and put together a proof of concept to validate the idea. That’s what I did and this is the first post of the solution as an overview.

Here’s a quick demo.

 

Overview

The diagram below details the basis of the solution. There are a few extra elements I’m still working on that I’ll cover in a future post if there is any interest in this.

Searching MIM with Speech Overview

The solution works like this;

  1. You speak to a microphone connected to a single board computer with the query for Microsoft Identity Manager
  2. The spoken phrase is converted to text using Cognitive Speech to Text (Bing Speech API)
  3. The text phrase is;
    1. sent to Cognitive Services Language Understanding Intelligent Service (LUIS) to identify the target of the query (firstname lastname) and the query entity (e.g. Mailbox)
    2. Microsoft Identity Manager is queried via API Management and the Lithnet REST API for the MIM Service
  4. The result is returned to the single board computer as a text result phase which it then uses Cognitive Services Text to Speech to convert the response to audio
  5. The result is spoken back

Key Functional Elements

  • The microphone array I’m using is a ReSpeaker Core v1 with a ReSpeaker Mic Array
  • All credentials are stored in an Azure Key Vault
  • An Azure Function App (PowerShell) interfaces with the majority of the Cognitive Services being used
  • Azure API Management is used to front end the Lithnet MIM Webservice
  • The Lithnet REST API for the MIM Service¬†provides easy integration with the MIM Service

 

Summary

Leveraging a lot of Serverless (PaaS) Services, a bunch of scripting (Python on the ReSpeaker and PowerShell in the Azure Function) and the Lithnet REST API it was pretty simple to integrate the ReSpeaker with Microsoft Identity Manager. An alternative to MIM could be any other service you have an API interface into. MIM is obviously a great choice as it can aggregate from many other applications/services.

Why a female voice? From a small response it was the popular majority.

Let me know what you think on twitter @darrenjrobinson

Utilising Azure Speech to Text Cognitive Services with PowerShell

Introduction

Yesterday I posted about using Azure Cognitive Services to convert text to speech. I also eluded that I’ve been leveraging Cognitive Services to do the conversion from Speech to Text. I detail that in this post.

Just as with the Text to Speech we will need an API key to use Cognitive Services. You can get one from Azure Cognitive Services here.

Source Audio File

I created an audio file in Audacity ¬†for testing purposes. In my real application it is direct spoken text, but that’s a topic for another time.¬† I set the project rate to 16000hz for the conversion source file then exported the file as a .wav file.

Capture Audio

The Script

The Script below needs to be updated for your input file (line 2) and your API Key (line 7). Run it liine by line in VSCode or PowerShell ISE.

Summary

That’s it. Pretty simple once you have a reference script to work with. Enjoy.

Converted

 

 

Utilising Azure Text to Speech Cognitive Services with PowerShell

Introduction

Recently I’ve been building an IoT Project that leverages Azure Cognitive Services. A couple of the services I needed to use were for converting Text to Speech and Speech to Text. The guides were pretty good from Microsoft, but not obvious for use with native¬†PowerShell. I’ve got it all working, so am documenting it for myself for the future but also to help anyone else trying to work it out.

Accessing the Cognitive Services Text to Speech API

Azure Cognitive Services Text to Speech is a great service that provides the ability as the name suggests, convert text to speech.

First you’ll need to get an API key. Head to the Cognitive Services Getting Started page and select Try Text to Speech¬†and Get API Key.¬†It will give you a trial key and 5000 transactions limited to 20 per minute. If you want to use it longer, provision a Speech to Text service using the Azure Portal.

The Script

I’m using a female voice in English for my output format. All the available output languages and genders are available here.

There are also 8 audio output formats. The two I’ve used most are raw 16khz pcm for .wav format and 16khz mp3 for MP3 output as highlighted below. The script further below is configured for MP3.

  • ssml-16khz-16bit-mono-tts
  • raw-16khz-16bit-mono-pcm
  • audio-16khz-16kbps-mono-siren
  • riff-16khz-16kbps-mono-siren
  • riff-16khz-16bit-mono-pcm
  • audio-16khz-128kbitrate-mono-mp3
  • audio-16khz-64kbitrate-mono-mp3
  • audio-16khz-32kbitrate-mono-mp3

The script below is pretty self-explanatory. Update Line 5 for your API Key, and Lines 11 and 13 if you want the output audio file to go to a different directory or filename.  The text to be converted is in line 59.

Step through it using VSCode or PowerShell ISE.

Summary

Using Azure Cognitive Services you can quickly convert text to audio. Enjoy.