As the title suggests this is Part 3, and the final part in a three-part post on configuring FIM/MIM to synchronise users passwords from AD to the Domino ID Vault via PCNS and FIM/MIM. Part 1 here detailed the creation of a PowerShell Management Agent to join users from Domino to the MIM Sync Metaverse. Part 2 here detailed the creation and configuration of the Domino Agents to receive password changes via the PS MA into the ID Vault.
This post will wrap it all up with the details on calling the Domino Agents on password sync events (from PCNS via MIM)
Prerequisites
You will need the IBM Notes client installed and configured on your MIM Sync Server in order to put a document in the database we created in Part 2 and start the agent to process the document(s).
Overview
Essentially this is the process;
Password changed for a user (either by an admin, or by the user via their domain joined workstation, password reset or any other password change mechanism)
Password change is captured by the AD PCNS Filter installed and configured on each (writeable) Domain Controller
The DC using the PCNS Config in the domain locates the MIM Sync Server to send the password change too
The MIM Sync Server has the associated AD Domain configured as a Password Sync Source
Our new PowerShell ID Vault Notes MA is configured as a Password Target
MIM Sync passes off the password change event for MIM joined users to the ID Vault Password Change MA which initiates the Password.ps1 script (below)
The password.ps1 script creates a document (that contains the details for the password change) in our ID Vault Password Sync Database we created in Part 2 of this series and then tells the MIMPwdTrigger Agent to start
The MIMPwdTrigger Agent picks up the document, passes it to the MIMPasswordSync Agent which sends the password change to the ID Vault
Put this Password.ps1 script in the same location you put the Schema, Import and Export scripts earlier.
Testing Password Sync End to End (Active Directory to the ID Vault)
The following screen shots show me tracing through the logs for a password change as it makes it way from the AD Domain Controller to MIM Sync to the MA to the MA Password script to the Notes DB as a document triggered to be process by the Notes Agent and the user updated in the ID Vault.
First the password change event is initiated to the MIM Sync Service by the Domain Controller that captured the password change.
PCNS provides all the details for the password change.
The MIM Sync Server determines where to send the change which includes our PS Notes MA.
MIMPasswordSync|mimpasswordsync: 08/03/2017 02:56:23 PM: Removed User ID Vault change document from ‘xxxNotes1/xxxxx-Aus’
And finally we see the change reflected in the ID Vault. Looking at the time-stamps along the way we see that it all happened in approximately 2 seconds.
Summary
This three-part blog post has shown how to get passwords from Active Directory to the MIM Sync connected source across to IBM Domino and into the ID Vault using the Granfeldt PowerShell Management Agent and some configuration with a Database in Domino with two Domino Agents.
What have you synchronised passwords too using FIM/MIM ?
Earlier this week I posted this blog post that showed a working example of using a custom Pwned Password FIM/MIM Management Agent to flag a boolean attribute in the MIM Service to indicate whether a users password is in the pwned password dataset or not. If you haven’t read that post this won’t make a lot of sense, so read that then come back.
The solution when receiving a new password for a user (via Microsoft Password Change Notification Service) was checking against the Have I Been Pwned API. The disclaimer at the start of the blog post detailed why this is a bad idea for production credentials. The intent was to show a working example of what could be achieved.
This update post shows a working solution that you can implement internal to a network. Essentially taking the Pwned Password Datasets available here and loading them into a local network SQL Server and then querying that from the FIM/MIM Pwned Password Management Agent rather than calling the external public API.
Creating an SQL Server Database for the Pwned Passwords
On my SQL Server using SQL Server Management Studio I right-clicked on Databases and chose New Database. I gave it the name PwnedPasswords and told it where I wanted my DB and Logs to go to.
Then in a Query window in SQL Server Management Studio I used the following script to created a table (dbo.pwnedPasswords).
use PwnedPasswords;
CREATE TABLE dbo.pwnedPasswords
( password_id int NOT IDENTITY(1,1) NULL,
passwords varchar(max) NOT NULL,
CONSTRAINT passwords_pk PRIMARY KEY (password_id)
);
Again using a query window in SQL Server Management Studio I used the following script to create an index for the passwords.
USE [PwnedPasswords]USE [PwnedPasswords]
GO
SET ANSI_PADDING ON
GO
CREATE UNIQUE NONCLUSTERED INDEX [PasswordIndex] ON [dbo].[pwnedPasswords]( [password_id] ASC)INCLUDE ( [passwords]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
GO
The last thing I did on the DB was to take the MIM Sync Server Active Directory Service Account (that was already in the SQL Server Logins) and give that account Reader Access to my new PwnedPasswords Database. I gave this account access as I’m using Integrated Authentication for login to SQL and as the MA is initiated by the MIM Sync Service Account, that is the account that needs the access.
Getting the Pwned Password Datasets into the new Database
I’m far from a DBA. I’m an identity guy. So using tools I was most familiar with (PowerShell) I created a simple script to open the password dump files as a stream (as Get-Content wasn’t going to handle the file sizes), read in the lines, convert the format and insert the rows into SQL. I performed the inserts in batches of 1000 and I performed it locally on the SQL Server.
In order to get the content from the dump file, add another column and get it in a format quickly to insert into the SQL DB I used the Out-DataTable function available from here.
The script could probably be improved as I only spend about 20-30 minutes on it. It is opening and closing a connection to the SQL DB each time it inserts 1000 rows. That could be moved outside the Insert2DB Function and maybe the batch size increased. Either way it is a starting point and I used it to write millions of rows into the DB successfully.
This then is the only other change to the solution. The Password.ps1 script rather than querying the PwnedPasswords API queries the SQL DB and sets the pwned boolean flag accordingly.
Summary
This enhancement shows a working concept that will be more appealing to Security Officers within corporate organisations if you have an appetite to know what your potential exposure is based on your Active Directory Users Passwords.
Recently I wrote about getting started with the latest IBM/Lotus Notes/Domino Management Agent for Microsoft Identity Manager. In a recent engagement we are using that MA to provision and manage identities into Domino. We are also using the MA to synchronise passwords via PCNS and MIM to the Notes users’ Internet (HTTP) password.
What you may or may not be aware of is that IBM introduced a new feature with Domino 8.5 called the ID Vault. The ID Vault is a Domino based application that holds protected copies of Notes user IDs. Now here’s the twist. The Microsoft Domino MA only supports password synchronisation to the HTTP password, not to the ID Vault.
My customer is using the ID Vault and naturally we need to synchronise password changes to both the HTTP Password and the ID Vault (for users Notes IDs). This post is the first in a series that details how I recently accomplished synchronising passwords to the Domino ID Vault.
This post provides the introduction and the creation of a PowerShell Management Agent into Domino to join identities into the MIM Metaverse
Post two details Creating Domino Agents that will handle taking requests from the MIM PS MA to change users ID Vault password
Post three will detail calling the Domino Agents on password sync events (from PCNS via MIM)
Overview
The following diagram shows a high-level overview of password synchronisation using FIM/MIM from AD to Domino. Password changes/resets can be initiated using a number of methods. The FIM/MIM Self Service Password Reset functionality, users changing their password via their domain joined workstations as defined by AD Group Password Polic(y)ies, using the AD FS Password Change function, or even on behalf of users by a Service Desk/Administrator. In each scenario implementing Microsoft’s Password Change Notification Service will get the password change to FIM/MIM. I’m not going to cover PCNS as it is out of the box and straight forward to install and configure. This MS PFE PCNS implementation document covers it quite well.
Likewise I’m not going to go into any detail about password sync to the HTTP Password. That’s out of the box functionality, that is pretty much the same as any other MA configured as a Password Sync Target. That said in my environment I did have to configure the MS Domino MA like this to get password events out to Domino.
ID Vault FIM/MIM PowerShell Management Agent
First up, we are going to need a Management Agent to join Notes users to our users from Active Directory in the Metaverse. I’ve gone to my favourite PowerShell Management Agent (Granfeldt) for this.
The Granfeldt PS MA will be configured to;
Import and join Domino Users to the Metaverse. The MA will be slimline in the number of attributes it brings in. Enough to perform the join and have enough information about the users context in Domino to be able to perform the password sync event
be a target for Password Synchronisation
send the password change event to the Domino Agent we will build to perform the password change. A Domino Agent is required as the ID Vault will only accept password changes from a process run on the Domino Server(s). More on this in parts 2 and 3
The integration of the MIM Sync Engine with Domino with the PowerShell Management Agent is done using LDAP. The Name and Address Book is easily accessed via LDAP.
To get started I looked up the Server Document for the Domino Server I wanted to connect to that had the Name and Address Book. Selecting the Directory tab I could see that LDAP(S) 389/636 was enabled.
I then went to the Name and Address Book and looked up my Admin Notes ID to get its context so I could translate it to LDAP format. Darren Kloud Robinson/OrgU/Org-AUS translates to CN=Darren Kloud Robinson,OU=Org,O=Org-AUS“.
Knowing my Notes ID Password I used LDP from Windows Server to bind to the Domino Directory. You could use any LDAP Browser/Tool.
Once I validated I could connect and browse the tree, I knew I had my connection details sorted, I translated that to PowerShell.
That looked something like this;
I then wrapped this into a FIM/MIM PowerShell Management Agent. The Granfeldt PS MA Scripts are below.
Domino PowerShell Management Agent Schema Script
As described above the Schema Script is very light on the number of attributes specified. Basically the minimum required to get a join and give us the context of the user to process password sync events.
Domino PowerShell Management Agent Import Script
As detailed above the Import brings through enough metadata to perform the join and give us the attributes needed for the user context to be able to sync passwords through.
Domino PowerShell Management Agent Export Script
File just needs to exist. Not used in this scenario.
See Part 3 in this series for the Password.ps1 script. But if you are following sequentially, copy the empty Export.ps1 script for now and name it Password.ps1 and have it located in the same directory as the other PS MA scripts.
Wiring it all together
As for creating the PS MA, I’ve detailed this in-depth many times. Check out this post (or the many other similar I’ve posted) and the Getting Started section if you are new to the Granfeldt PowerShell Management Agent. Copy the above scripts to the directory you create, and when creating the MA provide the paths to the scripts (in 8.3 format).
A key item though is to configure the PS MA as a Password Sync Target as per the screenshot below. You will also need to configure where passwords are coming from to send to this new MA. If it is Active Directory, open the Properties of your AD MA select Configure Directory Partitions then under Password Synchronization enable the checkbox Enable this partition as a password synchronization source. Select Targets and select your newly created Notes ID Vault Password MA. Select Ok then Ok again.
After creating a Run Profile and doing a Stage and Import, based on your Join rule (probably email address) you should have a heap of connectors. In my environment displayName contains the context of the user. Eg. Full Name/OrgU/Org We’ll need this to send the password change event to the ID Vault.
Summary
Through the PowerShell MA as detailed above we have been able to enumerate users from Domino and join them to existing users in the MIM Sync Metaverse. We can now set about creating Domino Agents to take password sync events from this MA and change users passwords in the Domino ID Vault.
Part 2 in this series here details creating the Domino Agents and configuring Domino to accept the changes.
Update:An element of this solution details checking passwords online (using the Have I Been Pwned API). Troy explains succinctly in his blog-post announcing the pwned passwords list why this is a bad idea. If you are looking to implement the concept I detail in this post then WE STRONGLY recommend using a local copy of the pwned password list. THIS POST HERE details using a local SQL Database to hold the Pwned Passwords Datasets and the change to the Management Agent to query the SQL DB instead of the HIBP API.
Troy also extended his HaveIBeenPwned API to include the ability to query as to whether a password has been pwned and is likely to be used in a brute force attack.
Microsoft provide a premium license feature in Azure Active Directory (Azure Active Directory Identity Protection) whereby leaked credential sets are checked and Admins alerted via reports. But what if you aren’t licensed for the Azure AD Premium Features, or you want something a little more customised and you have Microsoft/Forefront Identity Manager? That is what this post covers.
Overview
The following diagram looks a little more complicated than what it really is. The essence though is that password changes can come from a multitude of different scenarios. Using Microsoft’s Password Change Notification Service (PCNS) we can capture password changes and send them to Microsoft Identity Manager so that we can synchronise the password to other systems, or for this use case we can lookup to see if the users new password is on the pwned password list.
This post will cover creating the Pwned Password FIM/MIM Management Agent and flagging a boolean attribute in the MIM Service to indicate whether a users password is on the pwned password or not.
Password change and sync architecture
Prerequisites
There are a few components to this solution depicted above. You will need;
FIM/MIM Synchronisation Server
with an Active Directory Management Agent configured (most likely you will have a Projection Rule on this MA to get your users into the Metaverse)
not shown in the diagram above you will also need the MIM MA configured to sync users from the Metaverse to the MIM Service
FIM/MIM Service and Portal Server (can be on the same server as above)
Microsoft Password Change Notification Service (PCNS). This MS PFE PCNS implementation document covers it quite well and you will need;
the PCNS AD Schema Extension installed
the PCNS AD Password Filters installed on all your (writeable) Domain Controllers
PCNS configured to send password changes to your FIM/MIM Sync Server
Granfeldt PowerShell Management Agent (that we will use to check users passwords against the Have I Been Pwned pwned password API)
Lithnet Resource Management PowerShell Module
download it from here and install it on your FIM/MIM Server as the Pwned Password MA will use this module to populate the Pwned Password Status for users in the MIM Service
Getting Started with the Granfeldt PowerShell Management Agent
If you don’t already have it go get it from here. Søren’s documentation is pretty good but does assume you have a working knowledge of FIM/MIM and this blog post is no different.
Four items of note for this solution;
You must have an Export.ps1 file. Even though we’re not doing exports on this MA, the PS MA configuration requires a file for this field. The .ps1 doesn’t need to have any logic/script inside it. It just needs to be present
The credentials you give the MA to run this MA are the credentials for the account that has permissions to the On Premise Active Directory where we will be importing users from to join to our Metaverse so we can pass password changes to this Management Agent
The same account as above will also need to have permissions in the MIM Service as we will be using the account to update the new attribute we are going to create
The path to the scripts in the PS MA Config must not contain spaces and be in old-skool 8.3 format. I’ve chosen to store my scripts in an appropriately named subdirectory under the MIM Extensions directory. Tip: from a command shell use dir /x to get the 8.3 directory format name. Mine looks like C:\PROGRA~1\MICROS~4\2010\SYNCHR~1\EXTENS~2\PwnedPWD
With the Granfeldt PowerShell Management Agent downloaded from Codeplex and installed on your FIM/MIM Server we can create our Pwned Password Management Agent.
Creating the Pwned PowerShell Management Agent
On your FIM/MIM Sync Server create a new sub-directory under your Extensions Directory. eg. PwnedPWD in C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Extensions then create a sub-directory under PwnedPWD named Debug. C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Extensions\PwnedPWD\Debug
Copy the following scripts (schema.ps1, import.ps1, export.ps1, password.ps1) and put them into the C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Extensions\PwnedPWD directory
Schema.ps1
The following schema.ps1 script sets up the object class (user) and a handful of attributes from Active Diretory that will be useful for logic that we may implement in the future based on users password status.
Import.ps1
The import.ps1 script connects to Active Directory to import our AD users into the Pwned Password Management Agent so we can join to the Metaverse object already present for users on the Active Directory Management Agent. The user needs to be joined to the Metaverse on our new MA so they are addressable as a target for PCNS.
Export.ps1
As detailed earlier, we aren’t using an Export script in this solution.
Password.ps1
The Password script receives password changes as they occur from Active Directory and looks up the Have I Been Pwned API to see if the new password is present on the list or not and sets a boolean attribute for the pwned password status in the MIM Service.
On your FIM/MIM Sync Server from the Synchronisation Manager select Create Management Agent from the right hand side pane. Select PowerShell from the list of Management Agents. Select Next.
create pwned passwords management agent
Give your MA a Name and a Description. Select Next.
create pwned passwords management agent
Provide the 8.1 style path to your Schema.ps1 script copied from the steps earlier. Provide an AD sAMAccountName and Password that also has permissions to the MIM Service as detailed in the Prerequisites. Select Next.
connectivity pwned passwords management agent
Provide the paths to the Import.ps1, Export.ps1 and Password.ps1 scripts copied in earlier. Select Next.
global parameters pwned passwords management agent
Select Next.
partitions and hierarchies pwned passwords management agent
Select the user checkbox. Select Next.
object types pwned passwords management agent
Select all the attributes in the list. Select Next.
attributes pwned passwords management agent
Select Next.
anchors pwned passwords management agent
Select Next.
connector filter pwned passwords management agent
Create a Join Rule for your environment. eg. sAMAccountName => person:Accountname Select Next.
join and projection rule pwned passwords management agent
Create an import flow rule for user:pwdLastSet => person:pwdLastSet. Select Next.
attribute flow pwned passwords management agent
Select Next.
deprovisioning pwned passwords management agent
Ensure that Enabled password management is selected, then select Finish.
With the Pwned Password MA created and configured we need to create at least a Stage (Full Import) and Full SyncRun Profiles and execute them to bring in the users from AD and join them to the Metaverse.
This should be something you’re already familiar with.
Run Profile pwned passwords MA
When running the Synchronisation we get the joins we expect. In my environment PwdLastSet was configured to sync to the MIM Service and hence the Outbound Sync to on the MIM Service MA.
MIM Synchronisation pwned passwords MA
MIM Service Configuration
In the MIM Service we will create a custom boolean attribute that will hold the pwned status of the users password.
Schema
Connect to your MIM Portal Server with Administrator privileges and select Schema Management from the right hand side menu.
Select All Attributes then select New
Provide an attribute name (System name) and a Display Name with a Data Type of Boolean. Provide a Description and select Finish
Select Submit
Search for User in Resource Types then select the User checkbox from the search results and select Binding then select New.
In the Resource Type box type User then click the validate field button (the one with the green tick). In the Attribute Type box type Pwned Password then click the validate field button (the one with the green tick). Select Finish
Select Submit
Configure the Active Directory MA to send passwords to the Pwned Passwords MA
On your existing Active Directory Management Agent select Properties. Select Configure Directory Partitions then under Password Synchronization enable the checkbox Enable this partition as a password synchronization source. Select Targets and select your newly created Pwned Password MA. Select Ok then Ok again.
Pwned Password MA
Testing the End to End Pwned Password Check
Now you should have configured;
PCNS including installation of the Active Directory filters
The existing Active Directory Management Agent as a Password Source
The existing Active Directory Management Agent to send password change events to the Pwned Password MA
Select a user in Active Directory Users and Computers, right click the user and select Reset Password.
Active Directory Users and Computers, right click the user and select Reset Password
I first provided a password I know is on the pwned list, Password1
password I know is on the pwned listPassword Changed
With PCNS Logging enabled on the MIM Sync Server I can see the password event come through.
PCNS Logging enabled on the MIM Sync Server
Checking in the Pwned Password MA debug log we can see in the debug logging for the user we changed the password for and that when it was checked against Have I Been Pwned the password is flagged as pwned.
Note:If you implement the solution in a production environment obviously remove the password from being logged.
Pwned Password MA debug log
In the MIM Portal search for and locate the user the password we just changed the password for.
MIM Portal search for and locate the user
Select the user. Scroll to the bottom and select Advanced View. Select the Extended Attributes tab. Scroll down and we can see the Pwned Password shows as checked.
Pwned Password shows as checked
Now repeating the process with a password that isn’t in the Pwned Password list. After changing the password in Active Directory Users and Computers the password went through its sync path. The log shows the password isn’t in the list.
log shows the password isn’t in the pwned password list
And the MIM Portal shows the Boolean value for Pwned Password is now not selected.
Pwned Password is now not selected
Summary
Using PCNS and FIM/MIM we can check whether our Active Directory users are using passwords that aren’t in the Pwned Password list.
What we can then do if their password is in the Pwned Password list is a number of things based on what the security policy is and even what type of user it is. You’ll notice that I’ve included additional attributes in the MA that we can flow through the Metaverse and into the MIM Service that may help with some of those decisions (such as adminCount which indicates if the user is an Administrator).
Potentially for Admin users we could create a workflow in the MIM Service that forces their account to change password on next logon. For other users we could create a workflow that sends them a notification letting them know that they should change their password.
Either way, we now have visibility of the state of users passwords. Big thanks to Troy for adding Pwned Passwords to his Have I Been Pwned API.
Reiterating:An element of this solution details checking passwords online (using the Have I Been Pwned API). Troy explains succinctly in his blog-post announcing the pwned passwords list why this is a bad idea. If you are looking to implement the concept I detail in this post then WE STRONGLY recommend using a local copy of the pwned password list.
This is Part Two in the two-part blog post on managing users profile photos with Microsoft FIM/MIM. Part one here detailed managing users Azure AD/Active Directory profile photo. This post delves deeper into photos, specifically around Office 365 and the reason why you may want to manage these via FIM/MIM.
Background
User profile photos should be simple to manage. But in a rapidly moving hybrid cloud world it can be a lot more complex than it needs to be. The best summary I’ve found of this evolving moving target is from Paul Ryan here.
Using Paul’s sound advice we too are advising our customers to let users manage their profile photo (within corporate guidelines) via Exchange Online. However as described in this article photos managed in OnPremise Active Directory are synchronized to Azure AD and on to other Office365 services only once. And of course we want them to be consistent across AD DS, Azure AD, Exchange Online and all other Office365 Services.
This post details synchronizing user profile photos from Exchange Online to MIM for further synchronization to other systems. The approach uses a combination of Azure GraphAPI and Exchange Remote PowerShell to manage Exchange Online User Profile Photos.
Current State
Users historically had a photo in Active Directory. DirSync/ADSync/AzureADConnect then synchronized that to Azure AD (and once only into Office 365).
Users update their photo in Office365 (via Exchange Online and Outlook Web Access)
the photo is synchronized across Office365 Services
Desired State
An extension of the Current State is the requirement to be able to take the image uploaded by users in Exchange Online, and synchronize it back to the OnPremise AD, and any other relevant services that leverage a profile photo
Have AzureADConnect keep AzureAD consistent with the new photo obtained from Office365 that is synchronized to the OnPrem Active Directory
Sync the current photo to the MIM Portal
Synchronizing User Profile Photos with Microsoft Identity Manager
Synchronizing Office365 Profile Photos
Whilst Part-one dealt with the AzureAD side of profile photos as an extension to an existing AzureAD PowerShell Management Agent for FIM/MIM, I’ve separated out the Office365 side to streamline it and make it as efficient as possible. More on that later. As such I’ve created a new PowerShell Management Agent specifically for Office365 User Profile Photos.
I’m storing the Exchange Online photo in the MIM Metaverse as a binary object just as I did for the AzureAD photo (but in a different attribute ). I’m also storing a checksum of the photos (as I did for the AzureAD Photo, but also in a different attribute) to make it easier for comparing what is in Azure AD and Exchange Online, to then be used to determine if changes have been made (eg. user updated their profile photo).
Photo Checksum
For generating the hash of the profile photos I’m using Get-Hash from the Powershell Community Extensions. Whilst PowerShell has Get-FileHash I don’t want to write the profile photos out to disk and read them back in just to get the checksum. That slows the process up by 25%. You can get the checksum using a number of different methods and algorithms. Just be consistent and use the same method across both profile photos and you’ll be comparing apples with apples and the comparison logic will work.
Some notes on Photos and Exchange Online (and MFA)
This is where things went off on a number of tangents. Initially I tried accessing the photos using Exchange Online Remote PowerShell.
CAVEAT 1: If your Office365 Tenant is enabled for Multi-Factor Authentication (which it should be) you will need to get the Exchange Online Remote PowerShell Module as detailed here. Chances are you won’t have full Office365 Admin access though, so as long as the account you will be using is in the Recipient Management Role you should be able to go to the Exchange Control Panel using a URL like https://outlook.office365.com/ecp/?realm=<tenantname>&wa=wsignin1.0 where tenantname is something like customer.com.au From the Hybrid menu on in the right handside pane you will then be able to download the Microsoft.Online.CSE.PSModule.Client.application I had to use Internet Explorer to download the file and get it installed successfully. Once installed I used a few lines from this script here to load the Function and start my RPS session from within PowerShell ISE during solution development.
CAVEAT 2: The EXO RPS MFA PS Function doesn’t allow you to pass it your account password. You can pass it the identity you want to use, but not the password. That makes scheduled process automation with it impossible.
CAVEAT 3: The RPS session exposes the Get-UserPhoto cmdlet which is great. But the RPS session leverages the GraphAPI. The RPS PS Module doesn’t refresh it’s tokens, so if the import takes longer than 60 minutes then using this method you’re a bit stuffed.
CAVEAT 4: Using the Get-UserPhoto cmdlet detailed above, the syncing of photos is slow. As in I was only getting ~4 profile photos per minute slow. This also goes back to the token refresh issue as for pretty much any environment of the size I deal with, this is too slow and will timeout.
CAVEAT 5: You can whitelist the IP Address (or subnet) of your host so MFA is not required using Contextual IP Addressing Whitelisting. At that point there isn’t really a need to use the MFA Enabled PREVIEW EXO RPS function anyway. That said I still needed to whitelist my MIM Sync Server(s) from MFA to allow integration into the Graph API. I configured just the single host. The whitelist takes CIDR format so that looks like /32 (eg. 11.2.33.4/32)
Performance Considerations
As I mentioned above,
using the Get-UserPhoto cmdlet was slow. ~4 per minute slow
using the GraphAPI into Exchange Online and looking at each user and determining if they had a photo then downloading it, was also slow. Slow because at this customer only ~50% of their users have a photo on their mailbox. As such I was only able to retrieve ~145 photos in 25 minutes. *Note: all timings listed above were during development and actually outputting the images to disk to verify functionality.
Implemented Solution
After all my trial and error on this, here is myfinal approach and working solution;
Use the Exchange Online Remote PowerShell (non-MFA version) to query and return a collection of all mailboxes with an image *Note, add an exception for your MIM Sync host to the white-listed hosts for MFA (if your Office365 Tenant is enabled for MFA) so the process can be automated
Use the Graph API to obtain those photos
with this I was able to retrieve ~1100 profile photos in ~17* minutes (after ~2 minutes to query and get the list of mailboxes with a profile photo)
Pre-requisites
There’s a lot of info above, so let me summarize the pre-requisties;
The Granfeldt PowerShell MA
Whitelist your FIM/MIM Sync Server from MFA (if your Office 365 environment is enabled for MFA)
Add the account you will run the MA as, that will in turn connect to EXO via RPS to the Recipient Management Role
Create a WebApp for the PS MA to use to access users Profile Photos via the Graph API (fastest method)
Powershell Community Extensions to generate the image checksum
Creating the WebApp to access Office365 User Profile Photos
Go to your Azure Portal and select the Azure Active Directory Blade from the Resource Menu bar on the left. Then select App Registrations and from the Manage Section of the Azure Active Directory menu, and finally from the top of the main pane select “New Application Registration“.
Give it a name and select Web app/API as the type of app. Make the sign-in URL https://localhost and then select Create.
Record the ApplicationID that you see in the Registered App Essentials window. You’ll need this soon.
Now select All Settings => Required Permissions. Select Read all users basic profiles in addition to Sign in and read user profile. Select Save.
Under Required Permissions select Add and then select 1 Select an API, and select Office 365 Exchange Online then click Select.
Choose 2 Select Permissions and then select Read user profiles and Read all users’ basic profiles. Click Select.
Select Grant Permissions
From Settings select Keys, give your key a Description, choose a key lifetime and select Save.RECORD the key value. You’ll need this along with the WebApp ApplicationID/ClientID for the Import.ps1 script.
Using the information from your newly registered WebApp, we need to perform the first authentication (and authorization of the WebApp) to the Graph API. Taking your ApplicationID, Key (Client Secret) and the account you will use on on the Management Agent (and that you have assigned the Recipient Management Role in Exchange Online) and run the script detailed in this post here. It will authenticate you to your new WebApp via the GraphAPI after asking you to provide the account you will use on the MA and Authorizing the permissions you selected when registering the app. It will also create a refresh.token file which we will give to the MA to automate our connection. The Authorization dialog looks like this.
Creating the Management Agent
Now we can create our Management Agent using the Granfeldt PowerShell Management Agent. If you haven’t created one before checkout a post like this one, that further down the post shows the creation of a Granfeldt PSMA. Don’t forget to provide blank export.ps1 and password.ps1 files on the directory where you place the PSMA scripts.
PowerShell Management Agent Schema.ps1
PowerShell Management Agent Import.ps1
As detailed above the PSMA will leverage the WebApp to read users Exchange Profile Photos via the Graph API. The Import script also leverages Remote Powershell into Exchange Online (for reasons also detailed above). The account you run the Management Agent as will need to be added to the Recipient Management Role Group in order to use Remote PowerShell into Exchange Online and get the information required.
Take the Import.ps1 script below and update;
Update lines 11, 24 and 42 for the path to where you have put your PSMA. Mine is under the Extensions directory in a directory named EXOPhotos.
copy the refresh.token generated when authenticating and authorizing the WebApp earlier into the directory you specified in line 42 above.
Create a Debug directory under the directory you specified in lines 11,24 and 42 above so you can see what the MA is doing as you implement and debug it the first few times.
I’ve written the Import to use Paged Imports, so make sure you tick the Paged Imports checkbox on the configuration of the MA
Update Lines 79 and 80 with your ApplicationID and Client Secret that you recorded when creating your WebApp
Running the Exchange User Profile Photos MA
Now that you have created the MA, you should have select the EXOUser ObjectClass and the attributes defined in the schema. You should also create the EXOPhoto (as Binary) and EXOPhotoChecksum (as String) attributes in the Metaverse on the person ObjectType (assuming you are using the built-in person ObjectType).
Configure your flow rules to flow the EXOPhoto and EXOPhotoChecksum on the MA to their respective attributes in the MV.
Create a Stage Only run profile and run it. If you have done everything correctly you will see photos come into the Connector Space.
Looking at the Connector Space, I can see EXOPhoto and EXOPhotoChecksum have been imported.
After performing a Synchronization to get the data from the Connector Space into the Metaverse it is time to test the image that lands in the Metaverse. That is quick and easy via PowerShell and the Lithnet MIIS Automation PowerShell Module.
The file is output to the directory with the filename specified.
Opening the file reveals correctly my Profile Photo.
Summary
In Part one we got the AzureAD/Active Directory photo. In this post we got the Office365 photo.
Now that we have the images from Office365 we need to synchronize any update to photos to Active Directory (and in-turn via AADConnect to Azure AD). Keep in mind the image size limits for Active Directory and that we retrieved the largest photo available from Office365 when synchronizing the photo on. There are a number of PowerShell modules for photo manipulation that will allow you to resize accordingly.
One of the foundations of Identity Management is the ability to join an identity between disparate connected systems. As we extend our management of identities into cloud services this adds a few twists.
A key concept is to use an anchor that is persistent. Something that doesn’t change through a users life-cycle. A user’s Security IDentifier (SID) in Active Directory is perfect. It doesn’t change when a user or group may get renamed. What gets interesting is how the SID is represented when returned using different methods. That is what I quickly cover in this post.
Overview
The defacto method for connecting your OnPremise Active Directory to Azure Active Directory is to use Azure Active Directory Connect. AADC will synchronise users and groups SID’s to the corresponding object in AAD into the onPremisesSecurityIdentifier attribute.
When the onPremisesSecurityIdentifier attribute is retrieved via the GraphAPI the format looks like this: S-1-5-21-3878594291-2115959936-132693609-65242
Using an Active Directory Management Agent for FIM/MIM and synchronizing the objectSID from the OnPremise AD represents the value in the Metaverse in binary format which when viewed as text looks like this: 15000005210002431664623112825230126105190232781820
Translating SID formats so we can join identities
What we need to do is translate the string representation of the SID returned from the GraphAPI and AzureAD so that we have it in a binary format. Then we can then use those attributes in join rules to match users/groups between AzureAD and our OnPremise Active Directory.
On my AzureAD PowerShell Management Agent I have an attribute named AADonPremiseSID configured with the format as Binary in my PSMA Schema.ps1 as shown below.
On my Azure AD PSMA I have the following lines in my Import.ps1 which essentially takes the value retrieved from the GraphAPI S-1-5-21-3878594291-2115959936-132693609-65242 and converts it to a binary array that in text looks something like 15000005210002431664623112825230126105190232721825400 and stores it in the AADonPremiseSID binary attribute in the connector space.
# Create SID .NET object using SID string from AAD S-1-500-........
$sid = New-Object system.Security.Principal.SecurityIdentifier $user.onPremisesSecurityIdentifier
#Create a byte array for the length of the users SID
$BinarySid = new-object byte[]($sid.BinaryLength)
#Copy the binary SID into the byte array, starting at index 0
$sid.GetBinaryForm($BinarySid, 0)
#Add the SID to the user in the connector space
$obj.Add("AADonPremiseSID",$BinarySid)
This then lets me join my users (and groups using the same method) between AD and AAD. Essentially a line to put it into Security Identifier format, two lines to convert it to a binary array and a line to store it in the connector space. Simple when you don’t over think it.
I’m posting this because I know I’m going to need to do this often. Hope it helps someone else too.
This post details using these concepts together specifically for AzureAD Groups.
Pre-Requisites
Read the two posts linked to above. They will detail Differential Queries and Paged Imports. My solution also utilises another of my favourite PowerShell Modules. The Lithnet MIIS Automation PowerShell Module. Download and install that on the MIM Sync Server where you be creating the MA.
What you need is the Schema and Import PowerShell Scripts. Here they are.
Schema.ps1
Two object classes on the MA as we need to have users that are members of the groups on the same MA as membership is a reference attribute. When you bring through the Groups into the MetaVerse and assuming you have an Azure AD Users MA using the same anchor attribute then you’ll get the reference link for the members and their full object details.
Import.ps1
Here is my PSMA Import.ps1 that performs what is described in the overview. Enumerate AzureAD for Groups, import the active ones along with group membership.
Summary
This is one solution for managing a large number of Azure AD Groups with large memberships via a PS MA with paged imports showing progress thanks to differential sync which then allows for subsequent quick delta-sync run profiles.
In the last 12 months I’ve lost count of the number of PowerShell Management Agents I’ve written to integrate Microsoft Identity Manager with a plethora of environments. The majority though have not been of huge scale (<50k objects) and the import of the managed entities into the Connector Space/Metaverse runs through pretty timely.
However this week I’ve been working on a AzureAD Groups PS MA for an environment with 40k+ groups. That in itself isn’t that large, but when you start processing Group Memberships as well, the Import process can take an hour for a Full Sync. During this time before the results are passed to the Sync Engine you don’t have any visual of where the Import is up to (other than debug logging). And the ability to stop the MA requires a restart of the Sync Engine Server.
I’ve wanted to mess with Paging the Imports for sometime, but it hadn’t been a necessity. Now it is, so I looked to working out how to achieve it. The background information on Paged Imports is available at the bottom of the PSMA documentation page here. However there are no working examples. I contacted Soren and he had misplaced his demo scripts for the time being. With some time at hand (in between coats of paint on the long weekend renovation) I therefore worked it out for myself. I detail how to implement Paged Imports in this blogpost.
This post uses an almost identical Management Agent to the one described in this post. Review that post to get an understanding of the AzureAD Differential Queries. I’m not going to cover those elements in this post or setting up the MA at all.
Getting Started
There are two things you need to do in preparation for enabling Paged Imports on your PowerShell Management Agent;
Enable Paged Imports (if your Import.ps1 is checking for this setting)
Configure Page Size on your Import Run Profiles
The first is as simple as clicking the checkbox on the Global Parameters tab on your PS MA as shown below.
Identity Manager Paged Imports
The 2nd is in your Run Profile. By default this will be 100. For my “let’s figure this out” process I dropped my Run Profiles to 50 on one Run Profile and 10 on another.
Paged Imports Page Size
Import Script
With Paged Imports setup on the MA the rest of the logic goes into your Import Script. In your param section at the start of the script $usepagedimport and $pagesize are the variables that reflect the items from the two enablement components you did above.
$usepageimport is either True or False. Your Import.ps1 script can check to see if it is set and process accordingly. In this example I’m not even checking if it is set and doing Paged Imports anyway. For completeness in a production example you should check to see what the intention of the MA is.
$pagesize is the pagesize from the Run Profile (100 by default, or whatever you changed your’s too).
An important consideration to keep in mind is that the Import.ps1 will be called multiple times (ie. #of_times = #ofObjects / pagesize).
So anything that you would normally expect in any other MA to only process once when the Import.ps1 runs you need to limit to only running once.
Essentially the way I’ve approached it is, retrieve all the objects that will be processed and put them in a Global variable. If the variable does not have any values/data then it is the first run, so go and get our source data. If the Global variable has values/data in it then we must be on a subsequent loop so no need to go process that part, just page through our import.
In my example below this appears as;
if (!$global:tenantObjects) {
# Authenticate
# Search and get the users
# Do some rationalisation on the results (if required)
# setup some global variables so we know where we are with processing the data
} # Finish the one time tasks
As you’ll see in the full import.ps1 script below there are more lines that could be added into this section so they don’t get processed each time. In a production implementation I would.
For the rest of the Import.ps1 script we are expecting it to run multiple times. This is where we do our logic and process our objects to send through to the Sync Engine/Connector Space. We need to keep track of where we are up to processing the dataset and continue on from where we left off. We also need to know how many objects we have processed in relation to the ‘pagesize’ we get from the Run Profile so we know when we’ve finished.
When we reach the pagesize but know we have more objects to process we set the $global:MoreToImport to $true and break out of the foreach loop.
When we have processed all our objects we set $global:MoreToImport = $false and break out of the foreach loop to finish.
With that explanation out of the way here is a working example. I’ve left in debugging output to a log file so you can see what is going on.
You can get the associated relevant Schema.ps1 from the Management Agent described in this post. You’ll need to update your Tenant name on line 29, your directory paths on lines 10 and 47. If you are using a different version of the AzureADPreview PowerShell Module you’ll need to change line 26 as well.
Everything else is in the comments within the example script below and should make sense.
Summary
For managing a large number of objects on a PS MA we can now see progress as the import processes the objects, and we can now stop an MA if required.
Licensing for Office365 has always been a moving target for enterprise customers. Over the years I’ve implemented a plethora of solutions to keep licensing consistent with entitlement logic. For some customers this is as simple as everyone gets say, an E3 license. For other institutions there are often a mix of ‘E’ and ‘K’ licenses depending on EmployeeType.
In this blog post I detail how I’m using Søren Granfeldt’s extremely versatile PowerShell Management Agent yet again. This time to import Office365 licensing information into Microsoft Identity Manager.
I’m bringing in the licenses associated with users as attributes on the user account. I’m also bringing in the licenses from the tenant as their own ObjectType into the Metaverse. This includes the information about each license such as how many licenses have been purchased, how many licenses have been issued etc.
Overview
I’m not showing assigning licenses. In the schema I have included the LicensesToAdd and LicensesToRemove attributes. Check out my Adding/Removing User Office365 Licences using PowerShell and the Azure AD Graph RestAPI post to see how to assign and remove licenses using Powershell. From that you can workout your logic to implement an Export flow to manage Office365 licenses.
Getting Started with the Granfeldt PowerShell Management Agent
If you don’t already have it, what are you waiting for. Go get it from here. Søren’s documentation is pretty good but does assume you have a working knowledge of FIM/MIM and this blog post is no different.
Three items I had to work out that I’ll save you the pain of are;
You must have a Password.ps1 file. Even though we’re not doing password management on this MA, the PS MA configuration requires a file for this field. The .ps1 doesn’t need to have any logic/script inside it. It just needs to be present
The credentials you give the MA to run this MA are the credentials for the account that has permissions to the Office365 Tenant. Just a normal account is enough to enumerate it, but you’ll need additional permissions to assign/remove licenses.
The path to the scripts in the PS MA Config must not contain spaces and be in old-skool 8.3 format. I’ve chosen to store my scripts in an appropriately named subdirectory under the MIM Extensions directory. Tip: from a command shell use dir /x to get the 8.3 directory format name. Mine looks like C:\PROGRA~1\MICROS~2\2010\SYNCHR~1\EXTENS~2\O365Li~1
Schema.ps1
My Schema is based around the core Office365 Licenses function. You’ll need to create a number of corresponding attributes in the Metaverse Schema on the Person ObjectType to flow the attributes into. You will also need to create a new ObjectType in the Metaverse for the O365 Licenses. I named mine LicensePlans. Use the Schema info below for the attributes that will be imported and the attribute object types to make sure what you create in the Metaverse aligns, so you can import the values. Note the attributes that are multi-valued.
As per the tips above, the format for the script paths must be without spaces etc. I’m using 8.3 format and I’m using an Office 365 account to connect to Office365 and import the user and license data.
As per the Schema script earlier in this post I’m bringing user licensing metadata as well as the Office365 Tenant Licenses info.
Attributes to bring through aligned with what is specified in the Schema file.
Flow through the attributes to the attributes I created in the Metaverse on the Person ObjectType and to the new ObjectType LicensePlans.
Wiring it up
To finish it up you’ll need to do the usual tasks of creating run profiles, staging the connector space from Office365 and importing into the Metaverse.
In the last couple of weeks I’ve been evaluating a number of different approaches/concepts for some upcoming MIM development projects. Some of these I’ve blogged about already.
Having an Identity Manager Metaverse with identity data is a key dependency to being able to validate ideas and concepts. So what’s a good source of some interesting and varied identity data with string, integer, reference, and boolean attributes? Twitter? Yeah why not. There’s an API. Should be pretty quick to get some sample data right?
In this blog post I’m going to give an overview of creating a PowerShell Management Agent to consume Twitter identities and their data into Microsoft Identity Manager. I’ll cover;
Obtaining Twitter user data from Twitter using Powershell and the Twitter RestAPI
Using Søren’s Powershell Management Agent to import Twitter user data obtained via the RestAPI
Manipulating the Twitter data into the MIM Synchronisation Engine
Twitter Data
Here’s an overview of my approach/rationale of what data I was looking for and how I got it from Twitter;
I don’t need real-time data. Just identity data
I need data of all different data-types
I need data with all the randomness that identity data often contains
I created a standalone script that took a seed Twitter identity (one of my accounts) and;
obtained the Twitter account info including the list of the Twitter accounts it followed
the Twitter accounts that follows it
The standalone script uses the Twitter RestAPI to obtain the data and respected the service and rate-limits
Here is what a sample of some of the data looks like in the Metaverse.
Pre-requisites
You need to enable your Twitter Account for API access. Follow the details here
Getting the Seed Twitter account info
By now you should have downloaded the Twitter PowerShell API Modules and installed them. If you haven’t get WMF5 installed and run the install-module command as shown below in Line 1.
Modify the script below to give the;
Seed Twitter Account you want to bring in the Friends and Followers for as users into the MIM Metaverse
The API keys asscoiated with your Twitter account(s) you’re going to use to query the Twitter API
The directory you want to dump the account info out to
….. and let it loose.
Now we have two XML files with a whole bunch of Twitter accounts and their metadata. There is an almost certainty that the seed account you’ve used is both followed by twitter accounts that you also follow. We’re going to need to remove the duplicates so that when we import the Twitter accounts into MIM we don’t have duplicates.
Basic, basic script to read in both files and spit out the unique Twitter Accounts is shown below.
Consuming data into the MIM Sync Engine obtained via PowerShell is quick and simple utilising Søren Granfeldt’s extremely versatile PowerShell Management Agent. I’m just going to cover importing the data from the XML file we generated above.
Getting Started with the Granfeldt PowerShell Management Agent
First up, you can get it from here. Søren’s documentation is pretty good but does assume you have a working knowledge of FIM/MIM and this blog post is no different.
A few items of note are;
You must have a Password.ps1 file. Even though we’re not doing password management on this MA, the PS MA configuration requires a file for this field. The .ps1 doesn’t need to have any logic/script inside it. It just needs to be present
Same for an Export.ps1 file. I’m not doing any exports on the MA, but an export script must be present.
The credentials you give the MA to run this MA are irrelevant as they aren’t used as part of the import as I’m bringing in data from files generated via separate PowerShell scripts
The path to the scripts in the PS MA Config must not contain spaces and be in old-skool 8.3 format. I’ve chosen to store my scripts in an appropriately named subdirectory under the MIM Extensions directory. Tip: from a command shell use dir /x to get the 8.3 directory format name. Mine looks like C:\PROGRA~1\MICROS~2\2010\SYNCHR~1\EXTENS~2\Twitter
Import Twitter Users into Microsoft Identity Manager
Using the guidance above on the Granfeldt PSMA here are the two key scripts for the Twitter MA.
The Schema Script to expose the core Twitter identity attributes.
Schema Script
Import Script
The Import Script that takes the rationalised XML file created earlier from the friends and followers queries and populates the connector space.
Password Script
Required by the PSMA but not used as detailed earlier
Export Script
Required by the PSMA but not used as detailed earlier
Creating the Management Agent
Path to the Schema Script in 8.3 format as detailed earlier.
Path to the Import, Export and Password scripts also in 8.3 format.
Select the attributes you want to bring in to the connector space.
Any Join logic, and a Projection Rule.
Import Flow Rules to bring in the Tweeters.
Create your Run Profiles, perform a Stage and Full Sync and BAM; Tweeters in the Metaverse. Real Word Identity Data Ahoy. Exception testing commences now.
