Outputting data from an Azure Function to Power BI with PowerShell

PowerShell Azure Function to Power BI via Event Hub and Stream Analytics

Last week I wrote this post that detailed how to use the Azure Table Storage output binding in an Azure PowerShell Function. As part of the same solution I’m working on, I also need to get data/events into Power BI for reporting dashboards. An Azure Function (PowerShell) has the ability to obtain the data but the path to Power BI requires a number of steps that start with using the Azure Function Event Hub output binding.

The end to end process is;

  • Azure Function obtains information from an API (Brewery DB)
    • processes the data and;
  • Sends the data to an Azure Event Hub
  • A Stream Analytics Job picks up the data and puts it into a Power BI Dataset.

Azure Function to Event Hub to Power BI.png

This post will detail the process of configuring this process using the same Beer Styles example from the Azure Table Storage Azure Output Binding post here and PowerShell as the documentation doesn’t give a working example for PowerShell.

The inputs and Azure Function are the same. Just the output from the Azure Function to Azure Event Hub is added. You can have multiple output bindings, so this example will add the Event Hub output whilst keeping the Azure Table Service output as well.

Azure Event Hub

Before we can add and configure an Azure Event Hub Output Binding on an Azure Function we need to create an Azure Event Hub Namespace.

From the Azure Portal, create a resource (Event Hub).

Created Event Hub Namespace.PNG

Once the Event Hub Namespace has been created, we need to create an Event Hub. Select the Event Hub Namespace => Event Hubs => + Event Hub and give it a name.

Created Event Hub.PNG

Power BI

From the Azure Portal create a resource Power BI Embedded and create a Workspace if you don’t already have one.

Azure Stream Analytics

Now we need to create the Stream Analytics Job that will take the event data from the Event Hub and put it into a Power BI Dataset.

From the Azure Portal create a resource Stream Analytics Job. Give it a name and select create.

Stream Analytics Job.PNG

Once created select your Stream Analytics Job => Inputs => Add stream input => Event Hub. Provide a job Alias, select your Azure Subscription, the Event Hub Namespace and Event Hub created earlier and select Create.

Stream Analytics Input.PNG

Select Outputs from your Stream Analytics Job => + Add => Power BI => Authorize to authorise access to Power BI. Provide an output Alias, select your Group workspace and provide a Dataset name and Table name => Save.

Stream Analytics Output.PNG

Select Query and update the query to copy the input to the output. Select Save. If you weren’t filtering what you wanted into the Power BI Dataset on the Azure Function (or other input to the Event Hub) you could filter it with a query here.

Stream Analytics Query.PNG

Select the Job Overview and select Start => Now.

Stream Analytics Job Overview Start.PNG

Azure Function App Event Hub Output

With all the wiring in place to get our data out to Power BI we can now configure our Azure Function App to output to our Event Hub.

Select your Azure Function => Integrate => Outputs => New Output => Event Hub => New Connection => Event Hub and select your Event Hub Namespace, Event Hub => Select.

Azure Function Output Event Hub.PNG

Update the Event Hub name to the name of your Event Hub and select Save.

Output Event Hub Name.PNG

Selecting our Azure Function from the previous blog post adding in the following line will also copy our output object to our Event Hub Output Binding.

 $outputArray | convertTo-json | Out-File $outputEventHubMessage

Az Func Output to Event Hub.PNG

Processing our Events

Executing our Azure Function sends the events to our Output Bindings.

Azure Function Run.PNG

We can see the progress through looking at our Event Hub Metrics and Stream Analytics Overview (spike in events as they are processed)

Job Processing.PNG

After a minute we can see the dataset has been created in Power BI.

Power BI Dataset.PNG

We can then create a report from the dataset

New Report.PNG

using the data that was ingested based off our Azure Function.

Create Report Info.PNG

Summary

Using the Azure Function Event Hub Output Binding in conjunction with Stream Analytics we can easily get data to Power BI for reporting and dashboards.

Using AutoRest for PowerShell to generate PowerShell Modules

AutoRest for PowerShell

Recently the Beta of the AutoRest for PowerShell Generator has been made available. At the recent Microsoft MVP Summit in Seattle Garrett Serack gave those that were interested a 1 hr corridor session on getting started with AutoRest for PowerShell.

AutoRest is a tool that generates client libraries for accessing RESTful web services. Microsoft are moving towards using AutoRest to generate SDK’s for the API’s in the standard languages they provide SDK’s for. In addition the AutoRest for PowerShell generator aims to automate the generation of PowerShell Modules for Azure API’s.

In this post I’ll give an intro to getting started with AutoRest to generate PowerShell modules using an AutoRest example and then an Azure Function based API.

AutoRest for PowerShell Prerequisites

AutoRest is designed to be cross-platform. As such its dependencies are NodeJS, PowerShell Core and Dotnet Core.

AutoRest for PowerShell Installation

Installation is done via command line once you have NodeJS installed and configured

npm install -g autorest@beta

if you have previously installed AutoRest you will want to update to the latest version as updates and fixes are coming regularly

autorest --reset

AutoRest Update.PNG

Building your first AutoRest for PowerShell Module

The AutoRest documentation has a couple of examples that are worth using initially to get to know the process and the expected output from AutoRest.

From a PowerShell Core command prompt (type pwsh in a command window) make the following Invoke-WebRequest (IWR) to retrieve the XKCD Swagger/OpenAPI file. I pre-created a sub-directory named XKCD under the directory where I ran the following command.

iwr https://raw.githubusercontent.com/Azure/autorest/master/docs/powershell/samples/xkcd/xkcd.yaml -outfile ./xkcd/xkcd.yaml

Download XKCD Yaml.PNG

We can now start the process to generate the PowerShell module. Change directory to the directory where you put the xkcd.yaml file and run the following command.

autorest --powershell --input-file:./xkcd.yaml

AutoRest Generate Libs.PNG

Following a successful run a generated folder will be created. We can now build the PowerShell Module. We can add the -test flag so that at the end of Generating the Module, Pester will be used to test it. First run PowerShell Core pwsh  then build-module

pwsh 
./generated/build-module.ps1 -test

AutoRest - Build Module.PNG

With the module built and the tests passed we can run it. We need to load the module and look to see the cmdlets that have been built for the API.

.\generated\run-module.ps1 xkcd.psm1
get-command -module xkcd

AutoRest - Load Module.PNG

Using the Get-ComicForToday cmdlet we can query the XKCD API and get the Comic of the Day.

Get-XkcdComicForToday | fl

AutoRest - Comic of the Day.PNG

Taking it one step further we can download the comic of the day and open it, with this PowerShell one-liner.

invoke-webrequest (Get-XkcdComicForToday).img -outfile image.png ; & ./image.png

Download and Display XKCD Comic of the Day.PNG

Let’s Create a Simple API – BOFH Excuse Generator

I created an HTTP Trigger based PowerShell API Function that takes GET requests and returns a BOFH – Bastard Operator from Hell (warning link has audio) style excuse. For those that haven’t been in the industry for 20+ years or aren’t familiar with the Bastard Operator from Hell, it is essentially a set of (semi) fictional transcripts of user interactions from a Service Desk Operator (from Hells’) perspective set in a University environment. Part the schtick is an Excuse of the Day. My API when queried returns a semi plausible Excuse of the Day.

Here is my HTTP Trigger PowerShell Azure Function (v1). A simple random excuse generator.

I configured the function for anonymous, GET operations only and the route to be excuse

Testing the Function from my local machine showed it was working and returning an excuse.

Now it’s time to generate the OpenAPI Spec. Select the Azure Function App => Platform Features => API definition

Select Generate API definition template and the basics of the OpenAPI spec for the new API will be generated. I then updated it with the output application/json and what the API returns as shown in the highlighted sections below. Select Save 

Now we can test it out and we see that we have success.

Taking the OpenAPI Spec (YAML format file) for our BOFH Excuse API we go through the steps to successfully generate the PowerShell Module using AutoRest. Running the freshly baked cmdlet from the BOFH PowerShell Module returns a BOFH Excuse. Awesome.

Summary

Using AutoRest it is possible to generate PowerShell Modules for our API’s. Key to the successful generation is the definition of the OpenAPI Spec for our API. In my example above, obviously if you don’t define what the API call returns then the resulting cmdlet will query the API, but won’t return the result.