Create ARM Service Connection in Azure DevOps with Azure Cloud Shell

ARM Service Connections

When deploying your Azure Resources using Azure DevOps Pipelines you typically need to create an Azure Resource Manager (ARM) Service Connection in your Azure DevOps project to connect it to your Azure Subscription.

 

azure devops arm service connection

 

ARM Service connections use Azure Active Directory (AD) Service Principals to authenticate to Azure Subscriptions and specify the resources and access levels available through the connection. Azure DevOps ARM Service Connections support different authentication methods. Including Service principal (automatic), Service principal (manual), Managed identity and Publish Profile.

When creating an ARM Service Connection in Azure DevOps via the GUI you will notice that the Service principal (automatic) authentication option is recommended. Because it makes the process easier by automatically creating a Service Principal in Azure AD.

The Service Principal (automatic) authentication method uses the credentials of the user currently signed into Azure DevOps to create the Service Principal in Azure AD, so the account needs to have the appropriate access to the Azure Subscription and Resource Groups. In addition, this option does not let you specify the name of the Service Principal or the authentication option such as the Client Secret or Certificates and their values.

So while the Service principal (automatic) authentication option is quicker, it does scale well when you have a lot of connections to manage.I would recommend manually creating the Service Principal in Azure AD to maximise security and limit access and only using the automatic option in personal projects. You can learn more about Azure DevOps Service Connection Types here

 

authentication arm service connection

 

In this article, I am going to show you how to use Azure Cloud Shell to create an ARM Service Connection in Azure DevOps. Including creating a Service Principal that uses a Client Certificate for authentication. This approach provides a scalable foundation that you can use to maximise security, reduce complexity by not having passwords to manage.

Specifically, we will be covering the following.

 

  1. Create a new Resource Group and Key Vault.
  2. Create a new Self-Signed Certificate and add it to Key Vault.
  3. Create a Service Principal with Self-Signed Certificate from Key Vault.
  4. Assign Service Principal access to new Resource Group.
  5. Create an ARM Service Connection.

Prerequisites

To follow along with this article, you will need the following:

 

  • An Azure subscription. If you don’t have an Azure Subscription, you can create a free account.
  • User with Global Administrator role in Azure AD and Owner role in Azure Subscription.
  • An Azure DevOps Organization, project and git repository.
  • Project Collection Administrator Group in Azure DevOps Organisation.

Create Resource Group and Key Vault

The first thing that we need to do is create the new Resource Group (RG) and Key Vault. Go to the Azure Portal and launch Cloud Shell with a PowerShell Environment so we can get started.

Note: Cloud Shell requires a Storage Account to store configuration. So the first time that you launch it, you will be prompted to specify the Subscription and create the storage account. You can select the Show advanced settings options to set the details of an existing RG and Storage Account or create new ones.

 

launch azure cloud shell

 

1 – Create a new Resource Group by running New-AzResourceGroup -Name <RG Name> – Location <Azure Location for RG>.

Update the values between the <> with the name and location of your choosing.

 

create new resource group

 

2 – Create a Key Vault in our Resource Group by running New-AzKeyVault -VaultName <KV Name> -ResourceGroupName <RG Name> -Location <Azure Location for RG>.

Update the values between the <> with the values for <RG Name> and <Azure Location for RG> that you specified in the previous step.

 

create new key vault

 

3 – Add an Access Policy on Key Vault so that you can add the Self-Signed Certificate to it. Run the Set-AzKeyVaultAccessPolicy -VaultName <KV Name> -UserPrincipalName <Your User Principal Name> -PermissionsToCertificates get,list,update,create -PermissionsToSecrets get.

Update the values between the <> with the value of <KV Name> that you specified in the previous step. As well as your UPN name i.e. someone@example.com.

Note:  Cloud Shell uses it’s own token to authenticate users rather than user tokens. So you will need to login to Azure using Azure PowerShell to be able to add the Self-Signed Certificate to the Key Vault.

 

set key vault access policy

 

4 – Log in with the User Principal Name (UPN) that you specified when adding the Access Policy to the Key Vault and run the following command to create Self-Signed Certificate. And then add it to the Key Vault that you previously created.

Login

Connect-AzAccount -UseDeviceAuthentication

You will need to follow the instruction in the Cloud Shell console to log in i.e. open the URL displayed (https://microsoft.com/devicelogin) in a browser and enter the code provided.

Create Certificate

$Policy = New-AzKeyVaultCertificatePolicy -SecretContentType “application/x-pkcs12” -SubjectName “CN=<Certificate Common Name>” -IssuerName “Self” -ValidityInMonths 6 -ReuseKeyOnRenewal

Update the values between the <> with the value that you would like to use for the Common Name of your certificate. In my example I used “CN=test.com”.

Add Certificate to Key Vault

Add-AzKeyVaultCertificate -VaultName <KV Name> -Name <Certificate Name> -CertificatePolicy $Policy

Update the values between the <> with the value of the <KV Name> that you specified in the previous steps and specify a name for your certificate. In my example, I used “TestCert01”.

 

add self-signed certificate to key vault

 

5 – Now you can create a second Resource Group that you can deploy resources to. Running New-AzResourceGroup -Name <RG Name> – Location <Azure Location for RG>.

Update the values between the <> with the name and location of your choosing. In my example I have called the Resource Group azdevops-aes-dev-deployment-rg and specified the australiaeast location.

 

create deployment resource group

Create Service Principal and Assign RBAC

In the steps below, you will be retrieving the certificate from the Key Vault and using it to create your Service Principal in Azure AD. After which, you will assign an Azure RBAC role to your Service Principal so that it has the necessary permissions to deploy resources to your deployment Resource Group.

 

1 – Now you need to retrieve the Certificate from the Key Vault and then use it to create the Service Principal.

Get the Certificate from the Key Vault

$cert = Get-AzKeyVaultCertificate -VaultName <KV Name> -Name <Certificate Name>

Update the values between the <> with the value of the <KV Name> and the <Certificate Name> that you specified in the previous steps.

Get the Secret from the Key Vault

$secret = Get-AzKeyVaultSecret -VaultName <KV Name> -Name $cert.Name -AsPlainText

Update the values between the <> with the value of the <KV Name> that you specified in the previous steps.

Create the Service Principal 

$sp = New-AzADServicePrincipal -DisplayName <SP Name> -CertValue $secret -EndDate $cert.Expires

Update the values between the <> with the value that you would like to use for the Service Principal. In my example, I used “AADAZDEVOPSDEV01”.

 

create service principal with certificate

 

In the Azure Portal, you should see the Service Principal that you have created under the App registrations in Azure Active Directory. The Service Principal should also see the Self-Signed certificate associated with the Service Principal under Certificates & secrets.

 

azure ad service principal portal

 

2 – Now you can assign the Azure RBAC role to your Service Principal over the deployment Resource Group that you created in the previous steps. To make things easier you can use the AzureAD modules to get the application Id of your Service Principal and set as variable.

 

Set Service Principal Application Id Variable

Connect-AzureAD

$AppRegName = ‘<Service Principal Name>’

$appId = Get-AzureADApplication -Filter “DisplayName eq ‘$AppRegName'” | Select-Object -ExpandProperty AppId

Update the values between the <> with the value the Service Principal Name. In my example I used “AADAZDEVOPSDEV01”.

Assign Contributor Role to Service Principal

New-AzRoleAssignment -ApplicationId $appId -RoleDefinitionName “Contributor” -Scope /subscriptions/<Subscription ID>/resourceGroups/azdevops-aes-dev-deployment-rg

Update the values between the <> with the value of your Azure Subscription.

 

assign-azure-rbac

Create Personal Access Token Azure DevOps

Before you can create the ARM Service Connection, you will need to create a Personal Access Token (PAT) to use the Azure DevOp CLI from Azure Cloud Shell. Learn more about PATs here.

1 – Log in to your Azure DevOps Organisation and select Personal access tokens under User settings in the top left-hand navigation bar.

 

azure devops personal access tokens

 

2 – In the Personal Access Tokens screen select New Token and enter the appropriate details in the Create a new personal access token modal. Click Create when finished.

Note: For the purpose of this article, set the Scope to Full Access and the Expiration value to a time in the future. This setting is only recommended for learning purposes. The best practice is to restrict the scope of the PAT to only what is required for the specific interaction.

 

personal access token details

 

3 – You should now see the token details appear. Azure DevOps does not store the details of the token. So please copy these details into a safe location. You will need to re-generate the token if you misplace these details.

Tip – To make the solution more robust, consider storing the token in Azure Key Vault and retrieving the details at run time.

 

copy pat details

Create ARM Service Connection

Now that you have created your Service Principal in Azure AD and your PAT in Azure DevOps, you are ready to head back to Azure Cloud Shell and create your ARM Service Connection.

Note: To complete these steps below, you will need the Application Id of the Service Principal, Azure Subscription Name, Azure Subscription Id and Azure Tenant Id.

 

1 – You now need to set the AZURE_DEVOPS_EXT_PAT environment variable in Azure Cloud Shell to allow the Azure DevOps CLI to login into your Azure DevOps Organisation. Run the following command, $env:AZURE_DEVOPS_EXT_PAT = ‘<PAT Token>’ .

Update the values between the <> with the details of your PAT that you created in the previous steps.

 

2 – Configure the default Organisation for Azure DevOps CLI by running the following command, az devops configure -d organization=<ORG URL>. Once you have configured Azure DevOps CLI run the az devops project list command to check that there are no connectivity issues.

Update the values between the <> with the details of your Azure DevOps Organisation i.e. https://dev.azure.com/example.

Note: The first time you run an Azure DevOps CLI command in Cloud Shell you will be prompted to install the extension.

azure cloud shell azure devops cli

3 – To be able to create the ARM Service Connection using the Self-Signed Certificate stored in the Key Vault that you created earlier you will need to convert it from a PKCS#12 format to a PEM format. In the steps below, you will retrieve the Certificate from the Key Vault, store it on your Cloud Shell drive as a PFX file then use OpenSSL to convert it to a PEM file that you can use when creating the ARM Service Connection.

Retrieve Certificate from Kay Vault and Store as PFX file

$cert = Get-AzKeyVaultCertificate -VaultName <KV Name> – Name <Certificate Name>

$secret = Get-AzKeyVaultSecret -VaultName <KV Name> – Name $cert.Name -AsPlainText

$secretByte = [Convert]::FromBase64String($secret)

[System.IO.File]::WriteAllBytes(“cert.pfx”, $secretByte)

Convert Certificate from PFX to PEM

openssl pkcs12 -in cert.pfx -out cert.pem -nodes -password pass:

 

get and convert certificate from azure key vault

4 – You are now finally ready to use the Azure DevOps CLI to create your ARM Service Connection in your Azur DevOps project with certificate-based authentication. Run the following command.

Create ARM Service Connection

az devops service-endpoint azurerm create `

–azure-rm-service-principal-id <Service Principal Application Id>  `

–azure-rm-subscription-name ‘<Azure Subscription Name>’ `

–azure-rm-subscription-id <Azure Subscription Id> `

–azure-rm-tenant-id <Azure Tenant Id>`

–azure-rm-service-principal-certificate-path ./cert.pem `

–project <Azure DevOps Project Name> `

–name <ARM Service Connection Name>

Update the values between the <> with the details of your Service Principal Application, Azure Subscription Name, Azure Subscription Id, Azure Tenant Id, Azure DevOps Project Name and specify a name for your Service Connection.

  azure devops cli service endpoint

 

If you head over to your Azure DevOps Project, you should be able to see the Service Connection that you just created.

  new arm service connection azure devops

In Summary 

This article provided a step by step guide on how to create an ARM Service Connection in Azure DevOps with Azure Cloud Shell. Including how to create a Service Principal in Azure AD that uses a Self-Signed certificate for authentication.

While this article focused on the technical steps for creating an ARM Service Connection in Azure DevOps. It also provides a scalable and secure foundation that you can use to build an automated approach for creating and managing Service Principals in Azure AD and creating and managing ARM Service Connection in Azure DevOps.

Thank you for reading. I hope that this article has provided you with some useful information.

Keith Jenneke

Keith Jenneke

Managing Director | DevOps & Cloud Lead