- Part 1 – Setting up the ADFS Server
- Part 2 – Configuring VIO to consume ADFS
- Part 3 – Multi group mappings
This is the final part of the series covering how to configure ADFS Federation with VMware Integrated Openstack (VIO). Parts 1 and 2 of the series focused on the steps required to configure the base integration between ADFS and VIO, finishing with a look at the json files VIO uses to map users/groups from ADFS.
This part of the series looks at a more advanced use case from our customer deployment:
- Users should be able to belong to more than one security group. Where this occurs the user should be granted access to all matching projects within VIO
The Issue
Using the setup described in parts 1 and 2 is sufficient to allow users to log in to VIO using AD backed credentials, be mapped to a project based on membership of an AD group and be assigned to the member role within the project.
Everything works as expected when the user is a member of only one of the AD groups listed within the mapping.json file. During testing we found that if a user is a member of two groups e.g. VIO-Project1 and VIO-Project2 which are both specified within the mapping.json file, VIO would only assign the user to a single project, and did not honour all groups the user belonged to.
Reviewing the OpenStack documentation it states the following for how rules are applied:
A mapping is selected by IdP and protocol. Then keystone takes the mapping and processes each rule sequentially stopping after the first matched rule. A rule is matched when all of its conditions are met.
Based on this statement it was expected that if only a single mapping was going to be applied, then it would be the first one in the mapping.json file. In our example configuration this would be VIO-Project1. In fact what we saw during testing was that the user was always assigned to VIO-Project2, the last rule in the file.
Identifying the Root Cause
As always the first step was to look at the log files for additional information. VIO logs messages about user authentication to the keystone.log file in /var/log/keystone on the controller nodes.
Looking within the file for messages containing our test user account name we see some useful looking messages like the following extract:
2019-08-15 14:16:58.587 1829 DEBUG keystone.federation.utils [req-26599e0b-7d94-42b7-927c-8bd211b39bcd - - - - -] identity_values: [{u'group': {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, u'user': {u'name': u'adfsuser03@corp.local'}, u'projects': [{u'name': u'VIO-Project1', u'roles': [{u'name': u'member'}]}]}, {u'group': {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, u'user': {u'name': u'adfsuser03@corp.local'}, u'projects': [{u'name': u'VIO-Project2', u'roles': [{u'name': u'member'}]}]}] process /usr/lib/python2.7/dist-packages/keystone/federation/utils.py:538
2019-08-15 14:16:58.589 1829 WARNING keystone.federation.utils [req-26599e0b-7d94-42b7-927c-8bd211b39bcd - - - - -] Ignoring user name
2019-08-15 14:16:58.590 1829 DEBUG keystone.federation.utils [req-26599e0b-7d94-42b7-927c-8bd211b39bcd - - - - -] mapped_properties: {'group_ids': [], 'user': {'domain': {'id': 'Federated'}, 'type': 'ephemeral', u'name': u'adfsuser03@corp.local'}, 'projects': [{u'name': u'VIO-Project2', u'roles': [{u'name': u'member'}]}], 'group_names': [{u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}]} process /usr/lib/python2.7/dist-packages/keystone/federation/utils.py:540
These are interesting as the first line shows both VIO-Project1 and VIO-Project2 listed in the value of the projects attribute. This is what we expect as the user belongs to both of the VIO-Project1 and VIO-Project2 groups in AD and both groups are included in the mapping.json file.
Continuing down we see a warning about a username being ignored, but no further details such as the username or the reason it is being ignored. So, not much we can with that warning at the moment.
The final line shows the mapped properties that VIO is applying for our user, notice how suddenly the projects attribute value only lists VIO-Project2, VIO-Project1 has disappeared. The log file doesn’t tell us why it has dropped VIO-Project1, it does however tell us that the file generating the log messages is /usr/lib/python2.7/dist-packages/keystone/federation/utils.py so the next step was to look at that python file.
utils.py
The utils.py file is located in /usr/lib/python2.7/dist-packages/keystone/federation on the controller node.
It’s quite a long file and contains all of the steps VIO performs to read the content from the mapping.json file and process the rules to determine what actions it needs to take.
Looking back at the keystone.log file, each message contains the line number of code within the utils.py file that generated the log message, this is shown on the end of the file name e.g. /usr/lib/python2.7/dist-packages/keystone/federation/utils.py:540 indicating line 540 within utils.py.
That’s a great start, unfortunately the way a python script is structured means you can’t just start at line 1 and read down till you reach the end of the file. Personally I don’t know Python, it’s not a language I have any experience with, and it has its differences to the javascript language which I am very familiar with.
While I could read most of the code as it was obvious what the purpose was, there were bits that had me referring to tutorial sites on google. A few hours later and I had consumed as much of the file as I was ever going to understand and identified the offending line as line 663:
projects = identity_value['projects']
In this line the attribute identity_value refers to a single entry within the identity_values list, which contains the information read in from the mapping.json file. Remember that first line in the keystone.log file extract from earlier, well that shows us that identity_values has the following value:
[{u'group': {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, u'user': {u'name': u'adfsuser03@corp.local'}, u'projects': [{u'name': u'VIO-Project1', u'roles': [{u'name': u'member'}]}]}, {u'group': {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, u'user': {u'name': u'adfsuser03@corp.local'}, u'projects': [{u'name': u'VIO-Project2', u'roles': [{u'name': u'member'}]}]}]
So, separating this into individual entries you get the following two json string values:
{u'group': {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, u'user': {u'name': u'adfsuser03@corp.local'}, u'projects': [{u'name': u'VIO-Project1', u'roles': [{u'name': u'member'}]}]}
{u'group': {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, u'user': {u'name': u'adfsuser03@corp.local'}, u'projects': [{u'name': u'VIO-Project2', u'roles': [{u'name': u'member'}]}]}
Within the context of the entire utils.py file each one of these values is evaluated in turn as part of a loop containing line 663.
So, line 663 assigns the projects attribute the value found within identity_value json string against the projects entry. For the first identity_value entry this is :
[{u'name': u'VIO-Project1', u'roles': [{u'name': u'member'}]}]
For the second entry the value is:
[{u'name': u'VIO-Project2', u'roles': [{u'name': u'member'}]}]
Line 663 means the projects attribute is only ever going to be assigned a single project name based on the mapping.json file structure we have specified.
Also, since each of the identity_value entries are evaluated within a loop, the projects attribute will always contain the last value evaluated, which is VIO-Project2 in our example. This backs up the final log entry message from keystone.log which shows only VIO-Project2 being used:
2019-08-15 14:16:58.590 1829 DEBUG keystone.federation.utils [req-26599e0b-7d94-42b7-927c-8bd211b39bcd - - - - -] mapped_properties: {'group_ids': [], 'user': {'domain': {'id': 'Federated'}, 'type': 'ephemeral', u'name': u'adfsuser03@corp.local'}, 'projects': [{u'name': u'VIO-Project2', u'roles': [{u'name': u'member'}]}], 'group_names': [{u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}, {u'domain': {u'name': u'adfs-users'}, u'name': u'Federated Users'}]} process /usr/lib/python2.7/dist-packages/keystone/federation/utils.py:540
Resolving the issue
Now that we have the root cause identified what can be done to fix it?
Well I had a plan, and made a change to the utils.py file. While testing my change my lab environment crashed preventing me from logging in via ADFS, displaying a series of error messages. The messages in the keystone.log file referenced several other .py files on the controller. This was not looking good. That sent me down a rabbit hole of checking the multiple other python files mentioned in the error messages. None of those contained any references to projects or other values from the utils.py file though, so shouldn’t have been related to my change.
By this point my environment had became so unusable I was forced to stop and restart my VIO deployment via the vSphere Client plugin to try and revive it.
This is when the moment of sheer joy happened. As I logged into VIO post restart to check the status of the ADFS integration which appeared to be working again, I discovered my user was a member of multiple projects. Not only that it had the correct role assigned to it within each project and my other users still seemed to have the correct mappings.
Had I just fixed the issue with a really simple change to the python script? Doubting myself I created a second AD user, adding it to both the VIO-Project1 and VIO-Project2 groups. Logging in as the new user I was amazed to see that yes it had worked, the user was a member of both projects and none of the other configuration had been adversely impacted.
I had indeed fixed it, so what was the really simple change I made to the python script?
On line 663 of utils.py I changed the code so that the projects attribute was treated as the list object that it had been declared as earlier in the code. Instead of assigning a value directly to projects like it was a simple string, I amended it’s value to add the returned value to the existing value already assigned to projects.
Effectively treating projects like you would do with an array in javascript when you append a value to a list of values rather than replacing the entire list. The updated line now looks like this:
projects.extend(identity_value['projects'])
Now when the utils.py file is executed all values found for projects within the mapping.json file are returned and stored. The extend method is used rather than append as these methods mean different things in the python language.
Cautionary Note
This fix has not been validated by VMware as suitable for use within Production environments with a valid VMware support contract at this time. That process will be kicked off internally and hopefully the fix will be included in an upcoming future VIO release.
For now please use this at your own risk and seek advice from VMware Support before making any changes to your environment.
Closing Note
Finally, why did I describe this simple fix as causing a moment of pure joy?
Well for this particular customer ADFS integration was a deal breaker. If we couldn’t get the group based mapping to work with automatic creation of projects and role assignment, their planned use cases for the platform would be incredibly difficult to achieve.
Add to that the fact that myself and my colleague had between us spent several months trying to resolve the issues and had been unable to find any examples internally or externally that helped us. We had begun to believe it was not possible to achieve this level of integration with additional development being performed on the VIO product, or even upstream OpenStack.
VIO may not be one of VMware’s more popular products. If this series of blog posts can help one other person get ADFS set up to fit their needs, then all of the effort that went into finding a resolution will have been worth it.