RBAC in a modern DevOps world

Howdy folks!

Infrastructure-as-code tools like ARM templates and Terraform are more and more used to deploy infrastructure solutions. In general, these tools run within the context of a service principal so there needs to be an account with high privileges – at least high enough to deploy a given type of resource in your cloud environment. However, in terms of security, it seems to be hard to adhere to the principles security experts tend to suggest, like segregation of duties, role-based access control, the principle of least privilege, and so on. Really? Well, let’s first have a closer look at some of these principles.

The principle of least privilege

In customers’ environments, I still often see administrative user accounts being members of the Active Directory Domain Admins user group. And Enterprise Admins. Of course, they are also members of the Schema Admins and Exchange Organization Administrators groups at the same time. Administrators tend to put their user accounts into all these groups so they can do their jobs. There won’t be an error message stating that you do not have the necessary privilege to conduct a task. Well, that might be one way to go, but it’s definitely not our way in terms of security! But why is that? Well, because they can – simple as that. It makes life easier – according to the administrators who do not want to cope with more or less difficult principles only made for blocking them from doing their jobs. Wait, these are exactly the same arguments I hear from developers not interested in security!

“You security guys just want to block me from doing my job! If I’m not allowed to have local administrator rights on my machine I cannot work as expected”

The more, the better. Well, no! The principle of least privilege, or POLP, says that accounts must be able to access only the information and resources that are necessary for its legitimate purpose (Saltzer & Schroeder 1975). In other words: if it is your job to manage user accounts you might end up having a user account being part of the account operators user group instead of being member of domain admins. For developers on local machines this can but does not necessarily have to mean that they get elevated access on this particular machine. But not on all machines in the corporate network. It’s not about blocking anyone from doing anything. It’s about protecting your environment from being attacked.

Segregation of duties

If one person has access rights on several instances of complementary  systems, he or she could either try to elevate their own access rights or to use the existing rights to do any kind of harm against the existing environment. Separation/Segregation of duties, or SoD, is the concept of having more than one person required to complete a task. For example, one administrator is responsible for creating new user accounts and managing group memberships. A second administrator manages file server and SharePoint access, and a third administrator is responsible for anything related to the messaging environment. If everything is setup properly, the file server admin will never be able to elevate his or her own rights to access the Exchange environment.

Role-based access control

Role-based access control, or RBAC, is one technical possibility to adhere to both, the principle of least privilege and the segregation of duties. In Microsoft Azure, you need three things:

1. A security principal. This can be a user account, a group, or a service principal. The service principal is a machine account that is used by applications to authenticate against Azure Active Directory. Terraform, for example, runs in the context of a service principal.
2. A role definition. The role is the set of management rights you get in Azure. For example, the contributor role gives you management access to all resources but you do not have the right to change access rights.
3. A scope. The scope defines the level your management rights apply to. In Azure this can be a single resource, a resource group, a subscription, or a management group which contains several subscriptions. Access rights are always inherited so if you have access rights on a subscription, you have it on all resources in that subscription.

With RBAC you technically make sure that your security principals have access to only to resources they need to have access to with only the privileges needed. The best way is to grant access rights to a security group only and to then manage group membership for  privileged accounts.

Privileged Identity Management

Now, that you can make sure that your accounts only have appropriate access rights to only resources they are meant to have access to, you can go one step further. With Privileged Identity Management, or PIM, you can make sure that your principals also only have their access rights when they need them. Accounts are made eligible for role assignments but access rights need to be requested when needed and they are only given for a specific timeframe.

If everything is setup properly, an attacker, coming from within your company or being an external, who gets hold of a user account will only have access to a subset of resources and information. If PIM is in place and you use manual approval, the attacker won’t even get access to any information or resource.

Multi-factor authentication

With multi-factor authentication, or MFA, you raise the bar for attackers who want to get hold of your privileged accounts. It’s no longer enough to know a username and guess the password, but you are also challenged to prove who you are by entering a one-time password or approve a sign-in request in your smartphone app or via phone call. You can even use password-less sign-in with Windows Hello or FIDO2-compliant security keys such as the YubiKey 5 made by Yubico.

My 9 most important security principles in a modern DevOps world

As I’ve mentioned before, infrastructure-as-code tools need to run within the context of a security principal. If you do not want to use interactive sign-ins with every deployment, you cannot rely on PIM or MFA. However, you still need to plan for security in both, your pipeline tool and your Azure tenant. The principles that have worked best for me so far  in a DevOps world with Microsoft Azure are as follows:

1. Azure Monitor, Azure Security Center and Azure Sentinel are used to get visibility into attack attempts against your identities so you can react at a very early stage. Without monitoring you are blind and being blind means you are not able to make the right decisions. In short: monitoring is essential!
2. The privileged service principal used in a pipeline tool is either authenticated using a client certificate or a very strong passphrase. Azure Active Directory allows you to use passphrases with a maximum of 256 (!) characters in length. Make use of that!
3. The service principal is created in Azure Active Directory and authorized on particular Azure subscriptions or resource groups using Role-Based Access Control. You can either use built-in roles such as Contributor or Virtual Machine Contributor, or create your own RBAC roles depending on your needs.
4. Interactive user accounts are protected using Multi-Factor Authentication and Privileged Identity Management.
5. User accounts for daily business and administrative accounts for privileged tasks have to be separated.
6. Access to your CI/CD tool such as Azure DevOps is granted based on job roles. It is a privileged access right so authorization needs to comply with your rules for RBAC.
7. In productive subscriptions, DevOps engineers don’t have management access!
8. That said, if there is a change needed in an existing infrastructure, someone has to create a pull request which then is reviewed, commented, approved (and merged), or declined. Since there is no direct human interaction possible on Azure infrastructure level, all changes have to pass the pipeline where they are comprehensibly archived.
9. Since things can go terribly wrong, you have at least two cloud-based emergency accounts that are highly privileged to both, Azure Active Directory (Global Administrator) and all your Azure Subscriptions (Owner). If there is something wrong with your hybrid identity synchronisation and authentication environment, you can still log on with a cloud-based user account. At least one of the accounts is excepted from phone-based MFA in case there is an issue with the service that prevents you from authenticating. This is true for conditional access policies, as well. Make sure to store devices (e.g. the accounts’ security keys) and credentials in safe locations. Last but not least, don’t forget to monitor your sign-in and audit logs. There should not be an event related to an emergency account unless you face an emergency!

If you know more principles that are related to access and identity security in a DevOps world, just drop me a line – I will be more than happy to add them to this article.

Frequently-asked questions

You might still have some further questions that have not been fully answered by now. Let me try to sort some of them out:

1. “If you want to have a highly privileged account for the pipeline, who can create this service principal? How can you make sure to adhere to Segregation of Duties and the Principle of Least Privilege?”
   An Azure AD service principal is created by someone who has privileged access to Azure Active Directory. He or she then adds the service principal to a security group which is granted the respective access rights on subscription or resource group level. Another user account with owner rights on Azure Subscription level is used to grant access to the security group. This is a one-time job that usually is done at the beginning of a DevOps to cloud journey. Afterwards, you just have to manage group membership for granting or revoking access to particular accounts. The user account used for granting access to the security group either can be an elevated interactive administrative account or you use one of your emergency accounts to prove they are working as expected. The elevated Azure admin is not one of the DevOps engineers but rather someone who is responsible for managing Azure governance and security in your company.
2. “So you say, that there still is someone who is highly privileged?”
   You will still need to have privileged accounts that are able to conduct privileged tasks. You need administrators who are responsible for Azure Active Directory, as you need your Windows AD administrators. You will also have to have other administrators that are responsible for backup and recovery tasks, and again others who are responsible for governance and security. However, you should plan for SoD and POLP. Of course, emergency accounts are highly privileged because of obvious reasons but they are secured, monitored, and validated at regular intervals.
3. “If the pipeline’s service principal is highly privileged, someone who has access to it can either use the pipeline, or the service principal itself to screw things up. How to you cover this?”
   Everything which passes the pipeline is comprehensibly logged so you always know who has requested what change at what time. Pull requests need to be generated, reviewed and, eventually, approved and merged, or declined. Administrative access to Azure AD and subscription is privileged and regimented; and so is pipeline access. Make sure to monitor sign-in and audit logs for your service principals so you are informed if someone uses credentials out of the pipeline’s scope.

If you still have other questions regarding these topics, please feel free to ask. I will try to answer them and add them to this post, as well.

By for now and thanks for reading,
Tom