Batching Microsoft Graph API Requests with JSON Batching and PowerShell

Microsoft Graph JSON Batching

Late in 2018 it came to my attention new functionality with the Microsoft Graph API for batching API requests into a single request. As I predominantly use PowerShell for scripting into Microsoft Graph parallel requests historically required extra functions to  achieve something similar. Use of Invoke-Parallel for instance, that I’ve previously discussed in posts such as How to create an Azure Function App to Simultaneously Start|Stop all Virtual Machines in a Resource Group.

Fast forward to 2019 and I’ve been building a bunch of reports from Microsoft Graph that aggregate data from multiple API endpoints such as /users /auditLogs and /security . A perfect time to explore deeper JSON Batching with Microsoft Graph.

Getting Started

Essentially your Graph API requests are the same, but instead of calling the API endpoint directly, you make a POST request to the Batch endpoint …

https://graph.microsoft.com/v1.0/$batch

… with your request in the body that includes the suffix for the API URI for the request, the request method and an ID. The example below is to query the User API endpoint for users whose displayName starts with the string provided.

$myRequest = [pscustomobject][ordered]@{
   id=$requestID
   method="GET"
   url="/users?`$filter=(startswith(displayName,`'$($userSearchNameEncoded)`'))&`$top=10"
}

Additional requests are added as required to build the Request Body and passed as part of the Post request to the Batch endpoint. Each additional request in the batch requires a different ID and the respective particulars for the request(s).

Batching Requests using PowerShell

Below is an example for generating a batch in PowerShell with two requests.

To execute the request looks like this

$getBatchRequests = Invoke-RestMethod -Method Post -Uri $batchUrl -Body $batchBody -headers $requestHeaders

Responses

The responses come back under the Responses property as shown in the graphic below. The responses have the ID of the requests associated with them so you can associate a response with a request.

Microsoft Graph API JSON Batching 1.PNG

The request responses can then be parsed by iterating through the jobs.

Summary

Using Microsoft Graph JSON Batching we can submit multiple requests as a single job, have the requests performed on the backend in parallel and get the results back.

Keep in mind that the request can also use the DependsOn option to make a request dependent on another that must be completed prior. Also if the response contains a dataset that is paged. Check the nextLink property on the responses and make subsequent calls to return the remaining data.

Azure Self Service Password Reset Reporting using PowerShell

Just over 18 months ago I wrote this post on using PowerShell and oAuth to access the Azure AD Reports API to retrieve MIM Hybrid Report data.  This week I went to re-use that for Azure Password Reset Reporting and found out that the API had been deprecated.

API Deprecated.PNG

Using the error information that actually was informative I proceeded to the new API. Having authenticated as I had in the previous article, I executed the following to retrieve a list of the Audit Reports available.

$TenantDomain = "customer.com.au"
$url = 'https://graph.windows.net/' + $tenantdomain + "/activities/auditActivityTypes?api-version=beta"
# Returns a JSON document for the Audit Activity Types
$auditActivities= (Invoke-WebRequest-Headers $headerParams-Uri $url).Content |ConvertFrom-Json
$auditActivities.activityTypes

Before I go any further it’s a good time to mention you can easily access the Audit log events using the Azure Portal and retrieve the data you maybe looking for and download a report. There is also the Azure Audit logs content pack for PowerBI as detailed here. Those are awesome solutions, but if you want to do something a little more bespoke and programmatic then keep reading.

Azure AD Password Events Audit Log Data

For the record (as at 18 Dec 2018) there are 1023 different Activity Resource Types. The full list is available here. The ones I’m interested in right now though are a subset of the Self-service Password Management category;

SSPR Audit Reports
SSPR Audit Reports
After investigation, the API to get the information I was after is now available on the Microsoft Graph Beta Audit Logs endpoint auditLogs/directoryAudits
https://graph.microsoft.com/beta/auditLogs/directoryAudits 

My Azure AD Registered App needed to be updated to have the additional role (AuditLog.Read.All) which was done via the Registered Applications blade under Azure Active Directory in the Azure Portal;

AuditLog.Read.All

My script then needed to be updated to talk to the Microsoft Graph and the new scope;

$resource = "https://graph.microsoft.com"
$scope = "AuditLog.Read.All; Directory.Read.All"

So to get Azure Password Reset events for the last week the following calls can be made;

# Get Reset Password Events
$DirectoryAuditURL = "https://graph.microsoft.com/beta/auditLogs/directoryAudits"
$passwordMgmtAuditData = Invoke-RestMethod -Method Get -Uri "$($DirectoryAuditURL)?`$filter=category eq `'UserManagement`' and activityDateTime ge 2018-12-11T03:15:10.6837342Z and startswith(activityDisplayName%2C+`'Reset password`')" -Headers @{Authorization = "Bearer $($Global:accesstoken)"}

To retrieve the other events contained in the auditLog we just need to alter the event to retrieve events for and the timeframe of interest. The events from the table above associated with Azure Self Service Password Reset and Azure Change Password are;

Blocked from self-service password reset
Change password (self-service)
Credentials Registered and Password Reset Status of User
Reset password (by admin)
Reset password (self-service)
Security info saved for self-service password reset
Self-serve password reset flow activity progress
Self-service password reset flow activity progress
Unlock user account (self-service)
User completed security info registration for self-service password reset
User registered for self-service password reset
User started security info registration for self-service password reset

The Script

Here is the modified script from my previous post here that uses oAuth to retrieve Azure Password Reset events. As per the other script it enables the scopes required. If you’re not Global Admin get the script run initially by someone who has the Global Admin role or get them to assign the AuditLog.Read.All permission to the Azure AD Application you have created. You can then login and get an Access Token and a Refresh Token.

Update;

  • Line 5 for your Application ClientID
  • Line 6 for your Application Secret
  • Line 18 for where you want to store the Refresh Token
  • Line 97 can be un-remarked after getting a token to use the Refresh Token (and Line 96 commented out). To get a new Access Token using the stored Refresh Token (Line 18) call the Get-NewTokens function

If, due to your time filter you have more than the maximum (1000 events) that can be returned per call, you can use the skiptoken to get the next page. If you have more than 2000 use the subsequent skipToken in additional calls in a do { get data } while ($skipToken) loop with the addition of adding the returned data to a collection.

$skipToken = $passwordMgmtAuditData.'@odata.nextLink'
$results = Invoke-RestMethod -Method Get -Uri $skipToken -Headers @{Authorization = "Bearer $($Global:accesstoken)"}

If you want to automate the time period to retrieve events for then you can use the Get-Date function and set the time window.

# Date Time for the report. Last 14 Days
[string]$strDateTimeNow = Get-Date -Format yyyy-MM-ddThh:mm:ss
[datetime]$dateTimeNow = Get-Date -Format yyyy-MM-ddThh:mm:ss
[datetime]$minus14Days = $dateTimeNow.AddDays(-14)
[string]$14daysAgo = get-date($minus14Days) -Format yyyy-MM-ddThh:mm:ss

so then the call for the last 14 days of events would be

$passwordMgmtAuditData = Invoke-RestMethod -Method Get -Uri "$($DirectoryAuditURL)?`$filter=category eq `'UserManagement`' and activityDateTime ge $($14daysAgo)z and activityDateTime le $($strDateTimeNow)z and startswith(activityDisplayName%2C+`'Reset password`')" -Headers @{Authorization = "Bearer $($Global:accesstoken)"}

Summary

Using the Microsoft Graph auditLog API we can retrieve using PowerShell events of interest for reporting or other requirements.

Granfeldt PowerShell Management Agent Schema HRESULT: 0x80231343 Error

Yesterday I was modifying the Schema configuration on a Granfeldt PowerShell Management Agent on a Microsoft Identity Manager 2016 SP1 Server.

I was changing the Anchor attribute for a different attribute and on attempting to refresh the schema or view the configuration I got the following error;

Unable to retrieve schema. Error: Exception from HRESULT 0x80231343

Unable to retreive Schema.PNG

I knew I’d seen this before, but nothing was jumping to mind. And this was a particular large Schema script.

After some debugging I realized it was because the attribute I had changed the Anchor attribute too, was also listed in the Schema script. It wasn’t obvious as the attribute entry was multiple pages deep in the script.

Essentially if you are seeing the error Unable to retrieve schema. Error: Exception from HRESULT 0x80231343 there are two likely causes;

  • you haven’t declared an Object Class e.g User
    • $obj | Add-Member -Type NoteProperty -Name “objectClass|String” -Value “User”
  • the attribute you have as your anchor is also listed as an attribute in the schema script e.g
    • $obj | Add-Member -Type NoteProperty -Name “Anchor-ObjectId|String” -Value “333a7e07-e321-42ea-b0a5-820598f2adee”
    • $obj | Add-Member -Type NoteProperty -Name “ObjectId|String” -Value “333a7e07-e321-42ea-b0a5-820598f2adee”
      • you don’t need this entry, just the Anchor entry

Hopefully this helps me quickly find the reason next time I make a simple mistake like this in the Schema script.

 

Using SailPoint IdentityNow v3 API’s with PowerShell

The SailPoint IdentityNow SaaS product is evolving. I’ve previously posted about integrating with the IdentityNow API’s using PowerShell;

IdentityNow now has v3 API’s which are essentially the v2 and non-Published API’s with the added benefit of being able to obtain an oAuth token from a new oAuth Token endpoint. Unlike the v2 process for enabling API integration, v3 currently requires that SailPoint generate and provide you with the ClientID and Secret. This Compass document (at the very bottom) indicates that this will be the preferred method for API access moving forward.

The process to get an oAuth Token the process is;

  • Generate Credentials using your IdentityNow Admin Username and Password as detailed in my v1 Private API post
    • Lines 1-12
  • Use the credentials from the step above in the oAuth token request
  • Use the ClientID and Secret as Basic AuthN to the Token endpoint
  • Obtain an oAuth Token contained in the resulting $Global:v3Token variable

Authentication Script

Update:

  • Line 2 with your Org name
  • Line 5 with your Admin Login Name
  • Line 6 with your Admin Password
  • Line 15 with your SailPoint supplied ClientID
  • Line 16 with your SailPoint supplied Secret

Your resulting token will then look like this;

SailPoint v3 API oAuth Token.PNG

Using the v3 oAuth Access Token

So far I’ve found that I can use the oAuth Token to leverage the v2 and non-published API’s simply by using the JWT oAuth Token in the Header of the webrequest e.g

@{Authorization = "Bearer $($Global:v3Token.access_token)"}

Depending on which API you are interacting with you may also require Content-Type e.g

@{Authorization = "Bearer $($Global:v3Token.access_token)"; "Content-Type" = "application/json"}

Summary

Talk to your friendly SailPoint Support Rep and get your v3 API ClientID and Secret and discard this previous hack of scraping the Admin Portal for the oAuth Token saving a few hundred lines of code.

Enabling Requestable Roles in SailPoint IdentityNow using PowerShell

Recently I wrote this post about Retrieving, Creating, and Managing SailPoint IdentityNow Roles using PowerShell.

Last week SailPoint enhanced Roles with the ability to request them. The details are located on Compass here.

I had a number of Roles that we wanted to make requestable, so rather than opening each and using the Portal UI to enable them, I did it via the API using PowerShell.

As per my other Roles post, a JWT Bearer Token is required to leverage the Roles API’s. That is still the same. I covered how to obtain a JWT Bearer Token specifically for interacting with these API’s in this post here. I’m not going to cover that here so read that post to get up to speed with that process.

Enabling Roles to be Requestable

The following script queries to return all Roles, iterates through them to make them requestable. Update;

  • Line 2 for your IdentityNow Org Nam
  • after Line 9 you can refine the roles you wish to make requestable

Summary

Using the API we can quickly enable existing IdentityNow roles to be requestable.  When creating new Roles we can add in the attribute Requestable with the value True if we want them to be requestable.

Using Invoke-WebRequest calls within a Granfeldt PowerShell MA for Microsoft Identity Manager

MIM Sync Service Account IE Security Settings

If you use PowerShell extensively you should be familiar with the Invoke-RestMethod cmdlet and the ability for PowerShell to call API’s and receive information. The great thing about Invoke-RestMethod is the inbuilt conversion of the results to PowerShell Objects. However there are times when you need the raw response (probably because you are trying to bend things in directions they aren’t supposed to be; story of many of my integrations).

From within Granfeldt PowerShell Management Agent script(s) that use Invoke-WebRequest calls, these will in turn leverage the Internet Explorer COM API on the local machine. That means the account that is performing those tasks will need to have the necessary permissions / Internet Explorer configuration to do so.

Enabling Invoke-WebRequest for the FIM/MIM Synchronization Server Service Account

Logon to your Microsoft Identity Manager Synchronization Server using the Service Account associated with the Forefront Identity Manager Synchronization Service. If you’ve secured the service account properly you will need to temporarily allow that service account to Log on Locally.

FIM Sync Service Account.PNG

Open Internet Explorer and open Internet Options. Select the Security Tab, the Internet zone and select Custom Level.

Locate the Scripting section and set Active scripting to Enable. Set the Allow websites to prompt for information using scripted windows to Disable. Select Ok and Ok.

MIM Sync Service Account IE Security Settings.PNG

With the configuration completed, go back and change back the Forefront Identity Manager Service Accounts’ local permissions so it can’t logon locally.

Your Import / Export Granfeldt PowerShell Management Agent Scripts can now use Invoke-WebRequest where the requests/responses use the Internet Explorer COM API and respect the Internet Explorer security settings for the user profile that the requests are being made by.

Hopefully this helps someone else that is wondering why the scripts that work standalone fail when operating under the FIM/MIM Sync Service Account by the Granfeldt PowerShell Management Agent.

Speeding up PowerShell lookups across large Collections

This week I needed to create a report based on information returned from two queries. The query results where contained in two separate collections (50k+ objects each). Taking the smaller filtered collection and looking up the other collection for the additional information using PowerShell like this proved frustrating slow:

$extraData = $collection2 | Where-Object {$_.UserPrincipalName -eq $collection1.UserPrincipalName } | Select-Object

An alternative then was to query directly (via an API) for the additional information whilst iterating through the main collection rather than searching for it in the other collection e.g.

foreach ($obj in $collection1){ $extraData = Invoke-RestMethod -method GET ...... }

That too was way too slow and wasn’t really being a nice NET citizen for the API on the end of 50k+ queries.

Solution

My solution was to join the two collections of objects and then build my report based off just one collection. Step in the Join-Object function from Warren F.

Join-Object provides a lot of flexibility on how and what to join between collections. For my requirements I just needed to use Join-Object to join based on a common key and bring in all the data from the other collection. That then looked like this in PowerShell;

$reportData = Join-Object -Left $collection1 -Right $collection2 -LeftJoinProperty UserPrincipalName -RightJoinProperty UserPrincipalName -Type AllInLeft
Whilst this one line to join my two collections takes just over an hour to execute my entire report now completes in less than 90 minutes vs the 5 1/2 hours it was previously taking to run. Thx psCookieMonster

Searching and Retrieving your GitHub Gists using PowerShell

GitHub Gists, I love them. I treat them as my personal snippet storage as well as the repository for many posts on this blog. If you are new to them, it is important to know that you can have public and secret Gists. Secret Gists thereby give you your private little snippet storage environment.

At some point though you’re going to want to retrieve a snippet based on a fragment of knowledge, such as an API, keyword or similar. Through the Gist Webpage you can search and it will give you a list of your Gists where your keyword exists and if you hit enter it will give you all Gists.

But what if you want to search your Gists based on other or more complex criteria (public vs secret). Step in the Gist API, and my scripting tool of choice. This post will quickly detail authenticating to GitHub, retrieving all your Gists and then narrowing down the one(s) you’re after. Of course there are PowerShell Modules that can do some of this, but I needed something light and portable without needing to install another module.

If nothing more this is an easy post for me to find, to retrieve my script to then find my Gists.

Obtaining a Personal Access Token

Login to Github using your account and head to https://github.com/settings/tokens and select Generate new token. Give it a description and select gist.

GistToken.PNG

After selecting Generate Token you will be shown your access key. Copy it and store it safely. It won’t be displayed again and is your password to access GitHub.

Simple retrieval and Search Script

Update the following script;

  • Line 2 for your username
  • Line 4 for your token
  • Line 9 for your search term

As it stands the script is looking for Secret Gists (as per line 21). Change to $true for Public. It will find all that meet the search query (line 9) in the Description field. It will then get the Raw Gist contents and output it to the console. Update accordingly for what you want to happen after retrieval.

 

 

 

Nested Virtual PowerShell Desktop Environments on Windows 10 & Windows Server 2019 in Azure – Part 3

Docker Virtual PowerShell Desktop Env to Internet - SaaS

This is the third and likely last post in this series. In Part 1 I introduced the capability to have Virtual PowerShell Environments using Docker and the full Windows 10 / Server 2019 Build 1809 container images. In Part 2 I detailed remotely access the Azure RM Windows 10 / Server 2019 host that contains the Docker Container with our full Windows 1809 environment (and therefore PowerShell Desktop).

In this post I’ll detail building a Docker Image based off of the Windows 1809 Container image. The resulting Docker Image will;

  • create a base PowerShell environment with the necessary PowerShell Modules for performing common Azure based administrative activities
  • using the Docker image for administrative functions
  • be accessible to be started using SSH from other SSH Clients (e.g Putty)

Building a Docker Image based off Windows 1809 Container

Using the capabilities I showed in in Part 2 of this series I’m going to build the image from Azure Cloud Shell that I’ll use to SSH into the Windows 10 AzureRM hosted Virtual Machine.

Having logged in to Azure Cloud Shell and connected to my Azure VM via SSH as detailed in Part 2. I then created a new CMD file named NewImage1809.cmd that has the following command inside it.

docker run -it --name psNov2018 mcr.microsoft.com/windows:1809 powershell

New Image Start

Running the commands hostname and dir c:\program files\WindowsPowerShell\Modules shows that we are inside the Windows 1809 Container Image.

Hostname and Existing Modules.PNG

I used the Cloud Shell Upload option to upload a New Env Setup.ps1 script that contains the PowerShell commands to install a bunch of PowerShell Modules. Using the Cloud Shell Editor I opened the file.

New Env Setup Script

Here is that series of commands.

I can then select the block of commands and paste it into the PowerShell terminal console below and hit enter for it to execute them.

Excute

One by one the modules are installed

Modules Installed

When completed enter exit.

Exit after Module Install

Now we can stop our Container Image.

docker stop psNov2018

Docker stop.PNG

and commit our changes to a new container named ‘powershell-env-image-nov18’

docker commit psNov2018 powershell-env-image-nov18

Docker Commit.PNG

Listing the docker images with

docker image ls

shows our new Image.

Docker Image List.PNG

We can now Run our new image with

docker run -it powershell-env-image-nov18:latest powershell

Run Image.PNG

We can see the modules we installed previously.

Image Module List.PNG

and we can import them.

Import Modules.PNG

Putty to PowerShell Virtual Environment

As good as Azure Cloud Shell is, and as convenient as it is for quick tasks and execution, you’re going to want to use an SSH Client. I’ll show using Putty, but you can use whatever your favourite client is. To connect to the environment I;

  • using the Putty Key Generator I loaded the OpenSSH Private Key generated in Part 1 and saved it in Putty ppk format
  •  using Putty Pageant I can use the ppk formatted key for my SSH session to the Windows 10 1809 host
    • Note: WinSCP can also utilise the ppk key for authentication which makes getting files onto the Host very easy
  • if you find you don’t automatically get your elevated session that allows you to start the Docker Container/Image then create the following registry key on the Windows 10/Server 2019 host and reconnect. DWORD (32-bit) value of 1 for LocalAccountTokenFilterPolicy
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System]
"LocalAccountTokenFilterPolicy"=dword:00000001

I can then connect with Putty using my key, and run the DockerPS.cmd file I showed in Part 2 which outputs the version of PowerShell.

Summary

In this post I’ve shown how to customise a Windows 1809 Container for a Virtual PowerShell environment, along with using client based SSH and SCP tools to connect to and manage the base Host.

Searching & Returning all Objects/Users from a SailPoint IdentityNow Source

There are times when need to get an extract of all objects on an IdentityNow Source. Just a particular Source, not the object from the Identity Cube with attributes contributed from multiple sources.

I’ll cover how I do that in this post, which in turn also handles paging the results from IdentityNow as the SearchLimit is 2500 objects.

The basis of the logic is;

  • Define the Source to retrieve objects from
  • Define the number of results you wish to return per page (maximum is 2500)
  • Page results until you return the base object for all objects on the Source
  • Retrieve the Full Object details for each object

The Script

The following script has been written to run in VS Code and provide a Progress bar using the psInlineProgress PowerShell Module available from the PowerShell Gallery and here. If you are also running this via VSCode, after obtaining psInlineProgress update the psInlineProgress.psd1 file to change Line 36 as shown below. You should be able to find it in C:\Program Files\WindowsPowerShell\Modules\psInlineProgress\1.1

#PowerShellHostName = 'ConsoleHost'
PowerShellHostName = 'Visual Studio Code Host'

Update;

  • Line 3 for your IdentityNow API ClientID
  • Line 5 for your IdentityNow API ClientSecret
  • Line 9 for you IdentityNow Tenant name
  • Line 13 for the ID of the IdentityNow Source you want to retrieve entities from
  • Line 17 for the number of entries to return per page (2500 is the maximum)

Example

The output below shows using the script to return 2591 objects from an IdentityNow Source.

Search and return all objects on an IdentityNow Source

Summary

Using the v2/accounts IdentityNow API we can retrieve the base objects associated with an IdentityNow Source and then call it again with each objectID to retrieve the full object record. This can be useful if you want to then programatically extract and process the information rather than downloading a CSV via the IdentityNow Portal. Say for example ingestion into another system or Identity Management tool. But that’s a post for another time.