Wednesday, September 20, 2017

Announcing Lithnet AutoSync for Microsoft Identity Manager

One of the things that I've always believed was missing from MIM and its predecessors was the ability to automatically 'run' the sync engine. The supported way of 'automating' the sync engine is to develop scripts that call the MIIS WMI methods. However, those scripts usually just cycle through the management agents and run profiles in a predetermined order, at a predetermined interval.

Over the years, I've often thought that there must be a better way than this! When we consider the various operations that can be performed on each management agent, the clues to how to do this start to become clear.

Delta import Performed when a change occurs in a connected system
Delta synchronization Performed when an import operation stages changes in a connector space
Export Performed when a synchronization stages outbound changes in a connector space
Confirming import Performed when an export leaves unconfirmed imports in the connector space

In all cases, except the delta import, the operations occur within the connector spaces of the sync engine itself. This means that the sync engine has the information needed to know when a sync, export and confirming import is required - It's all right there in the run history. All we really have to do, is somehow trigger the initial delta import when we know that changes have been made in the connected system.

Today, I'm releasing my solution to this problem - Lithnet AutoSync.

AutoSync is designed to run the sync engine for you. All you need to do is tell it when to run the import operations. You can do this on a schedule (eg every 15 minutes), or in response to an event if your connected system supports change detection. Built-in triggers are provided for Active Directory, LDS, and the MIM service, and you can write your own triggers using PowerShell. I've put a community PowerShell trigger gallery page up where you'll eventually be able to find and share trigger scripts for various systems.

Once the import has been triggered, AutoSync takes care of the rest. If the import results in staged changes in the connector space, a delta sync is performed. If the delta sync stages outbound changes on other management agents, then export operations are performed. After those exports, confirming imports are performed, and the cycle continues.

AutoSync is fast. By ensuring run profiles are executed only when needed, AutoSync keeps the sync engine from doing work it doesn't need to do. When combined with event-based triggers, the sync engine can respond to changes in connected systems in as close to real-time as possible. AutoSync allows for the fastest propagation of changes possible, while respecting the sync engine rules for overlapping run profiles.

AutoSync follows the Microsoft guidelines when it comes to running multiple management agents simultaneously. While import and export operations are allowed to overlap, synchronizations must be run exclusively.

AutoSync has been in development for over 18 months, and I'm very happy to be sharing this with you today. A special thanks to Piyush Khandelwal is in order, as this project wouldn't have been possible without his many, many hours of diligent testing, bug-finding and feedback. A big thank you to those who have also helped test over the life of the project including Darren Robinson, Søren Granfeldt, and the whole team back at the office who have put up with untold numbers of pre-release builds.

Head over to github to read the getting started guide, and happy auto-syncing!

Monday, March 13, 2017

Assisted password reset add-on for the FIM/MIM portal

Microsoft Identity Manager and its predecessor, Forefront Identity Manager cater for self-service password reset (SSPR) scenarios with out of the box workflows that support SMS, email, and question/answer authentication. Self-service password reset is a very important capability for any organization, and when properly deployed, can significantly reduce calls to the service desk.

However, even when SSPR is available in an organization, there will always be a percentage of password resets that the service desk performs. It could be that the user is not enrolled in SSPR, that they didn't know SSPR was available, or their registered SSPR mechanisms were no longer available (eg they have a new phone number, or no longer have access to their registered email address). In these cases, the service desk is usually called and a manual password reset is performed. This is not a scenario that is current supported by MIM directly, which typically results in the service desk dropping back to the AD admin centre or the users and computers console to perform this task.

Today, I'm announcing the release of the Lithnet Assisted Password Reset (APR) add-on for the FIM/MIM portal. This tool integrates with the user RCDC to display a "Reset password" link, allowing quick access to reset the user's password using either a generated or user-specified password.

It's simple, easy to install, and highly customizable.

It supports
  • Automatically generated random passwords of a configurable length
  • Manually specified passwords
  • Includes the option to force the user to change their password at next logon
  • Optionally forces service desk staff member to re-authenticate in order to reset a password
  • Can unlock locked accounts

Visit the GitHub site to get started.

Wednesday, February 22, 2017

User verification add-on for the FIM/MIM Portal

Today I'm releasing a new add-on for the FIM/MIM portal. The Lithnet User Verification Module allows IT staff to use the MIM portal to send an SMS code sent to a user's mobile phone. This is useful in scenarios where a user calls the service desk and needs to be verified before the service desk can take an action such as resetting a password or asking for a change to a group that they own.

If you have your users registered for SMS-based self-service password reset, then this module is ready for you to use today. It will use the same SmsServiceProvider.dll you created to enable SSPR, and will get the user's mobile number from the msIdmOtpMobilePhone attribute.

There are lots of configuration options available, so if you want to get the mobile number from a different attribute, you can certainly do that. You can also customize the attributes displayed by the tool, change the length of the security code, and even restrict access to the tool to a particular set of users

You access the tool by adding a new UocHyperLink control to your RCDC as shown below.

A new window will open with a customizable list of user attributes shown. Clicking the Send Code button will generate a unique code and send it to the user's phone

The code is shown to the service desk operator, and the user receives the code on their phone

It's a simple, but useful tool for authenticating users over the phone.

Visit the GitHub site to download the latest release and read the installation instructions

Thursday, February 2, 2017

Announcing v2 of the Lithnet FIM/MIM Service REST API

In 2015, I released the first version of the REST API for the FIM/MIM Service. I designed it to abstract away the complexities of the native SOAP endpoint, and open up the possibilities of integrating with FIM from operating systems and libraries outside of the Windows/.NET ecosystem.

It's been used by many awesome public and private projects since then. Check out Peter Stapf's guide on using it to create PowerApps.

Features have been added over time, usually by request, which means now, using simple JSON calls, you can perform the following tasks
  • Create resources
  • Modify resources
  • Delete resources
  • Get a resource
  • Get the current user's permissions on a resource
  • Search for resources
  • Full localization support
  • Getting approval requests
  • Approving or rejecting requests
However, I needed to make some changes to the API that would have broken compatibility with existing versions, so I decided to add another endpoint to this API, and release a new version.

Both versions of the API are contained in the one installation package, and can safely run side-by-side. Any existing applications will continue to work with the v1 API version.

Here are a list of fixes, enhancements, and new features of the v2 API.


  • Fixes an issue where all attribute values were returned as strings, rather than their native JSON data type
  • Fixes an issue where multivalued attributes containing only a single value were serialized as a string, instead of an array


  • Provides flexibility in how resources are returned from the API by allowing the caller to specify
    • If null values should be returned or not
    • If values for single-valued attributes should be returned as arrays
    • If the resource should be rendered as a fixed structure, that does not depend on knowing the resource schema in advance
  • Aligns the API to a set of REST API guidelines published by Microsoft

New Features

  • Adds support for getting permission hints for each attribute on a resource
  • Adds paged search support

Check out the updated documentation, and download the new version today

Monday, January 23, 2017

AD, LDS and LDAP unauthenticated binds: A series of unfortunate security events

Update: March 2018: Microsoft have agreed to address this issue in an upcoming version of Windows Server :)

A caution to anyone that uses applications that rely on LDAP authentication against Active Directory, or Active Directory Lightweight Directory Services to do so. Both services will appear to accept a blank password for any users when performing a simple bind. While behind the scenes, that's not what is happening, if your application doesn't check for and reject a logon attempt with the blank password itself, it might incorrectly assume a successful authentication against LDAP. This post details how I came to learn about this behaviour, how wide spread the problem is, and what can be done about it.

The discovery

A few weeks ago, I was at my desk, enjoying my lunch, when I received a call from a customer in a panic. He told me that our AD LDS server was allowing people to access his application without typing in a password. I assumed he was talking about anonymous binds (binding with no username and password) and informed him that it's a normal LDAP thing, and it just means his application probably hadn't been coded properly.

But he insisted that was not what was happening. An anonymous bind requires a blank username and password, but he told me he was entering a username but had a blank password. I assumed he was going mad, but decided to fire up ldp.exe to test the scenario, confident that I'd be back to my sandwich and mindless web browsing in no time.

Imagine my surprise when I was greeted with the following response

res = ldap_simple_bind_s(ld, 'uid=ryan,ou=users,dc=lithnet,dc=local, <unavailable>); // v.3

Sorry... what?!

"Ummm..... Let me call you back..."

The investigation

We were transitioning away from an Oracle LDAP server at the time, so I tried to perform the same bind operation there:

res = ldap_simple_bind_s(ld, 'uid=ryan,ou=users,dc=lithnet,dc=local', <unavailable>); // v.3
Error <48>: ldap_simple_bind_s() failed: Inappropriate Authentication
Server error: binds with a dn require a password


Good. That's what I would have expected. What on earth was LDS doing? First I needed to make sure I didn't have an information disclosure issue. There is sensitive information in our directory, and it would be really bad news if this data was disclosed. I tried to perform a search in LDS after 'binding' with no password.

ldap_search_s(ld, "DC=lithnet,DC=local", 2, "uid=ryan", attrList,  0, &msg)
Error: Search: Operations Error. <1>
Server error: 000004DC: LdapErr: DSID-0C090752, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, v2580
Error 0x4DC The operation being requested was not performed because the user has not been authenticated.

Phew. Looking back at the logon result, it did indeed show I was authenticated as the built-in account 'NT AUTHORITY\ANONYMOUS LOGON'. This is the same result as performing an anonymous bind. Without granting permissions to this account, no data was going to be inadvertently disclosed. But it still didn't explain why the user could bind without a password. It looks like the user specified is being authenticated successfully without a password, although we are not being authenticating as that user at all.

Now, I want to talk about anonymous binds for a moment. There is some confusion out there in articles and posts I have read about this behaviour, with people saying that this is an anonymous bind and it is normal behaviour. Anonymous binds are indeed normal and required by the LDAP specification. Clients must be able to connect to the RootDSE anonymously, which contains information they need to understand the capabilities, configuration, and authentication types that the directory supports. LDS allows anonymous binds to the RootDSE, as does any RFC-compliant LDAP client. The fact that we were bound as ANONYMOUS LOGON isn't the issue. The fact that we provided a username and no password is an issue. This is NOT an anonymous bind as per the RFC specification.

Moving on, I decided to try to log into an application that I knew used LDS for authentication.

Phew, again. The application is obviously smart enough to detect the blank password and refuse to authenticate. I started wondering if this the responsibility of all applications to implement this check? It does sound like a reasonable safety precaution to implement when coding an application, but I've seen my fair share of code enough to know that not all developers are as prudent as this. I tried to login to the application the customer called me about, and sure enough, it let me into the application. I had varying results from other applications I tried. It really came down to the specific LDAP library and authentication implementation that these clients used. Suffice to say, it was not a one-off problem.

Rolling up the sleeves

It was time to dig into the LDAP RFC to see what is going on here. Section 5.1.2 of RFC 4513 revealed something very interesting that I did not know about LDAP before. It reads;

5.1.2. Unauthenticated Authentication Mechanism of Simple Bind

An LDAP client may use the unauthenticated authentication mechanism of the simple Bind method to establish an anonymous authorization state by sending a Bind request with a name value (a distinguished name in LDAP string form [RFC4514] of non-zero length) and specifying the simple authentication choice containing a password value of zero length

So this was apparently a legitimate thing, and it's called an Unauthenticated bind. Note this is distinct from an "Anonymous bind" where both a blank username and blank password must be provided. Reading further along reveals something even more interesting

The distinguished name value provided by the client is intended to be used for trace (e.g., logging) purposes only. The value is not to be authenticated or otherwise validated (including verification that the DN refers to an existing directory object). The value is not to be used (directly or indirectly) for authorization purposes.
It seems that this 'feature' exists only for logging purposes. According to the RFC, you don't even need a real DN. I tested this out.

res = ldap_simple_bind_s(ld, 'this is not a DN', <unavailable>); // v.3

Well, that worked, as the RFC said it should. At this point I started to get a bit worried. I've spent a long time working with LDAP directories and I didn't know this was a thing. What chance do we have that application developers, who don't spend their lives living and breathing identity and LDAP, are aware of this and implement appropriate mitigations? Reading the next paragraph did give me some hope.

Unauthenticated Bind operations can have significant security issues (see Section 6.3.1). In particular, users intending to perform Name/Password Authentication may inadvertently provide an empty password and thus cause poorly implemented clients to request Unauthenticated access.

Yes! This is exactly what I was concerned about.

Clients SHOULD be implemented to require user selection of the Unauthenticated Authentication Mechanism by means other than user input of an empty password. Clients SHOULD disallow an empty password input to a Name/Password Authentication user interface.

Urgh. Well, yes I agree 100% in principal.  They SHOULD do that. It's basic input validation. But how many of it actually even know about this curiosity though? The next line was the one I wanted to see.

Additionally, Servers SHOULD by default fail Unauthenticated Bind requests with a resultCode of unwillingToPerform.
By default, the server should not accept this! Perhaps we changed something in our build and accidentally enabled this? To rule this out, I built a brand new LDS instance on a fresh Windows Server 2012 R2 server. I fired up LDP.exe, and guess what?

res = ldap_simple_bind_s(ld, 'this is not a DN', <unavailable>); // v.3

Then, I had a thought, accompanied by a feeling of dread. AD LDS uses the same underlying engine as Active Directory. Could the same problem exist there?

res = ldap_simple_bind_s(ld, 'this is not a DN', <unavailable>); // v.3

Yep. :(

Mixed emotions

I've got mixed emotions at this point. I'm happy that we didn't do something wrong in our build, but also, part of me was wishing that we had. Trawling through the AD and LDS documentation made no mention of a setting to be able to turn this off. At this point we had over a hundred applications moving off the old Oracle LDAP server and onto LDS. Some of these were home grown, and could be fixed ourselves, others were written by 3rd parties, and getting code updates would be a lot more complicated.

At this point, I'd given up on lunch.

It is important to note, that this issue poses absolutely no risk to the security of your LDAP system. AD and LDS are secure and there is no risk of information disclosure (unless you deliberately grant rights to NT Authority\Anonymous logon). The security of your applications that unconditionally delegate responsibility for authentication to the LDAP directory however, is another question.

We logged a case with Microsoft, and after much deliberation about the intent of the RFC and whether it was the client or servers responsibility to fix this, we submitted a product change request, which is still under consideration.

The discussion we had centred around the fact that this issue stems from poorly coded applications. It's hard to disagree with this point. Especially since the RFC says that clients should not allow this to happen. But the RFC also says that the servers also should not allow this to happen. When it comes down to it, we have to consider which is easier to do? Find and update every bad application that was ever made, and keep an eye on every application to come in the future? That's going to be about as successful as herding cats.

Or do we prevent the problem at its source? Unauthenticated binds serve no useful purpose that I am aware of. The RFC mentions that it exists for trace purposes, and I doubt that many people even know such a thing exists. There is certainly no product I have ever come across that required unauthenticated binds turned on for it to work. The RFC states that it should be turned off by default, so we can consider that it isn't crucial to LDAP operations.

How wide spread is this problem?

There are only a handful of LDAP server vendors, but untold numbers of LDAP clients. I decided to investigate a few well-known LDAP providers, to determine what their support for disabling unauthenticated binds is.

ProductCan be disabledDisabled by default
Red Hat Directory Server Yes Yes
OpenLDAP Yes Yes
Novell eDirectory Yes No
Oracle/Sun Directory Server Yes Yes
Microsoft AD LDS/ADAM No No
Microsoft Active Directory No No

Red Hat Directory Server, Open LDAP and Oracle/Sun Directory all ship with unauthenticated binds disabled by default. As per the RFC. Novell has it enabled, but provide guidance on turning it off. Microsoft has the only product (in my admittedly limited research) that does not provide this option at all. Given how wide-spread the deployment of Active Directory is, this is really a concern.

It's a common enough problem that there are many publicly listed vulnerabilities in the CVE database. Keep in mind that these are only the ones that were publicly disclosed. There are no doubt many more not listed, or not yet even known. If well-established vendors such as VMWare and Apache can make these mistakes, it goes to show that anyone can. We're talking some serious issues here. Even the SSSD authentication module in Linux was vulnerable until 2010.

We have to ask the question, if unauthenticated binds are the source of so many critical security issues in other products, and they add little to no measurable value, why should that capability be turned on by default in any product? Why should that capability even exist?

So, who should fix it?

Discussion forums and bug reports are full of back-and-forthing between the developers saying that its a misconfiguration on the LDAP server side, and users being stuck with products like AD LDS that do not allow you to configure this, one way or another, begging the developers to fix it on the application side. The RFC compounds this problem by stating that it is both the client and server's responsibility to protect against the problem.

In my view, both parties have a responsibility here.

We can't reasonably expect every application developer to read every page of the LDAP RFCs. This is a counter-intuitive 'feature' hidden away in the depths of a protocol definition. One could argue that adequate testing would identify this, but that's assuming you are testing against a directory that has this setting turned on. You could also argue that applications should be checking their input values, and again you wouldn't be wrong. Ultimately, an application is responsible for its own security.

But server vendors have a responsibility to ship products in a secure configuration that protects the organizations using them. I'm a big believer in the fact that LDAP is NOT an authentication protocol (I've got a blog post coming about this), and yes technically, there are no security flaws in the LDAP server itself. However, we have to accept the reality, rightly or wrongly, that it's a very widely-used de facto authentication protocol. A default setting change here (or at minimum, the option to change the setting), protects the organizations who use these applications from easy to avoid cases of information disclosure, unauthorized access, and a potential raft of other serious security issues.

There is no discernible impact to turning off unauthenticated binds. If we were to see an LDAP v4 in the future, I would hope to see this confusing and unnecessary feature removed. The only better than that, would be if we stop using LDAP as an authentication protocol altogether. While that's a future that, at least for now, only exists in my dreams, Microsoft can and should do something now to provide the ability to turn off unauthenticated binds in AD and LDS, and set it to be off by default in a future version of the product. The world will be a better (safer) place.

Saturday, January 21, 2017

The FIM/MIM Synchronization engine stops responding


There is a known issue with FIM/MIM that causes the synchronization service to stop responding, requiring you to kill miiserver.exe with task manager and restart the service.


This is triggered when the following two conditions occur
  1. A delta import on the FIM MA finishes
  2. A synchronization run on another MA is in progress


When the FIM MA goes to write its delta watermark, it does so by updating the value in its MA configuration. Unlike other MAs, this requires a full MA config update (the same as if you changed a flow rule or other setting in the MA config), which increments the version number, and requires an exclusive database lock.

A synchronization running at the same time reads the same database table and causes a deadlock situation that is never resolved.

Evidence that this behaviour is different from other MAs can be seen by running the following command using Lithnet MIIS PowerShell. The FIM MA will always have a much higher version number that any other MA, and will increment after every import operation. Other MAs only update their version number when their configuration is changed by an admin in the UI.

PS C:\Windows\system32> Get-ManagementAgent | select Name, Version, Category
Name                   Version Category  
----                   ------- --------  
AD                     59      AD        
_IAM_ReleaseVersion    42      Extensible2
FIMService             113951  FIM       
_IAM_SyncEngineVersion 41      Extensible2
Sap                    35      Extensible2
GoogleApps             83      Extensible2
LDS                    23      ADAM      
ACMA                   76      Extensible2
StudentDB              93      Extensible2
ZAVip                  99      MSSQL     

Interestingly, the UI prevents you from making changes to an MA when other MAs are running, presumably to protect against this kind of situation. However a FIM MA delta import ultimately does exactly the same thing.


Unfortunately, the only way to prevent this from happening is to not run any synchronization profiles, when an import is running on the FIM MA.

This issue was logged with Microsoft, however, this will not be fixed, as their support statement is that running multiple run profiles simultaneously is not supported.

Tuesday, January 17, 2017

"The cause of the error is not clear" - User will not sync into Azure AD with AAD Sync or AD Connect

Recently, we had an issue where four specific users would not sync into Azure AD. There were no noticeable differences in attributes between these users and ones that were working.

Compounding the issue was a rather unhelpful error message

The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support

With a little help from Microsoft support, we were able to resolve the issue using the following steps
  1. First, create a new user in Office 365 with a default domain UPN (eg
  2. Get the users ObjectGUID from AD
  3. Set the ImmutableID attribute on the new account to be the ObjectGUID of the AD account
  4. Run a delta sync or wait for next scheduled sync. At this point, the AD user will be joined with the Azure user account, and the user's attributes will be updated appropriately.
For example, if you receive the following error in an email
The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support.

You can use the following PowerShell command to resolve the issue

Credit goes to Cameron Duck for the troubleshooting process and coming up these resolution steps