As I’m sure you are familiar (with my many posts on the topic), the Granfeldt PowerShell Management Agent is extremely flexible. When used to integrate Microsoft Identity Manager with modern REST API’s it is easy to retrieve pages of results from a REST API and process the objects through the Management Agent. However sometimes you need to integrate Microsoft Identity Manager with an API (e.g. a SOAP WebService) that doesn’t provide functionality to page results.
Workday is an example of this (but the concepts and approach detailed in this blog post are valid to similar API integration scenarios). As I’ve posted about previously using the WorkdayAPI PowerShell Module in conjunction with the Granfeldt PowerShell Management Agent (PSMA) we can easily get the base records of Workday Users. However, to get the full Workday Record for each user using the WorkdayAPI PowerShell Module, requires a SOAP WebService call (facilitated by the PS Module) for each user. If you have a large Workday Tenant that can be a large number of calls and very time consuming (to be executed serially) for a Full Import into MIM. To achieve this as efficiently as possible, whilst still using the Granfeldt PSMA, a solution is to multi-thread the API calls to retrieve Workday Users’ full records.
Here is the approach to implementing Multi-Threaded Granfeldt PowerShell Management Agent Imports using Workday as an example;
Retrieving the base records for all Workday User Records is the same as I detailed in this post here. Using the -Force switch will also return in-active records, so only do that if you really need to.
My method for splitting up the records into batches is a slight variation on the method I detailed in my PowerShell Snippets Volume 1 due to the way the collection is returned.
The excerpt below shows the Import Script taking the Run Profile PageSize and splitting the records into batches of that size. My recommendation (for Workday) is to configure a PageSize of 150 so that each page is returned and processed in ~3-6 minutes (dependant on your implementation of Workday and individual users record size).
Note: Make sure you configure your PSMA for Paging so that the PageSize is passed to the Import Script.
# Split the Workday Records into Batches based on PageSize from the Run Profile # Suggest setting the Run Profile for 150 which will allow importing from Workday in digestable pages $Global:groups = @() $parts = [math]::Ceiling($WDayRecordsBase.Length / $pagesize) for ($i = 0; $i -le $parts; $i++) { $start = $i * $pagesize $end = (($i + 1) * $pagesize) - 1 $Global:groups += , @($WDayRecordsBase[$start..$end]) }
Processing each batch is effectively passing each Batch to a PowerShell ScriptBlock that executes a specified number of the individual requests in parallel.
As I’ve done previously for processing parallel functions in PowerShell I’m using the Invoke-Parallel function from Warren Frame. It will take a little effort for each different API you are talking to, in order to determine the optimal number of requests to run in parallel. This is done with the -throttle parameter. For Workday the optimal is ~15. Going much higher will trigger the Workday API to reset your connection. And in my experience the connection reset could be triggered quickly or after a few hours of operation.
The following snippet shows defining a batch to process (batch of records to retrieve based off Run Profile PageSize) and multi-threading the calls (e.g. 15 maximum parallel requests) to retrieve the full records.
$global:wdBatch = $Global:groups[$global:groupsProcessed + 1] .... $wddetails = Invoke-Parallel -InputObject $global:wdBatch -throttle 15 -runspaceTimeout 60 -NoCloseOnTimeout -ImportVariables -ImportModules -ScriptBlock {...}
Process the results through the MA as you normally would then trigger the MA to reload for the next page (see full examp.
Here is an example Import Script implementing the multi-threaded approach detailed in this post (from Workday). You can find the previous example Workday PSMA I wrote here.
Multi-Threading Granfeldt PowerShell Management Agent Imports is achievable by leveraging PowerShell Runspaces (via the Invoke-Parallel function) and a little manipulation of a collection using PowerShell.
Another thing to keep in mind is that coming in PowerShell v7 is ForEach-Object -Parallel functionality. This will likely negate the need for the Invoke-Parallel function.
A few weeks back the Microsoft AI Tour was in Sydney Australia. There was a…
If you're anything like me you always have PowerShell open, and often both PowerShell and…
Decentralised Identity is a technology I'm passionate about and have written many posts and tools…
Over two years ago I authored a PowerShell Module that enabled the automation of 1Password.…
Buried in my PowerShell Snippets Vol 4 post from 2021 is the PowerShell script and…
Short post on how to recovery from "The Windows Subsystem for Linux instance has terminated"…
This website uses cookies.
View Comments