Azure Resource Manager Storage Accounts: How to plan your storage architecture V2

Hey folks,

in spring 2016 I’ve published a blog post on how to automatically create ASM storage accounts and what to consider in order to speed up blob transfers. Today I invite you to learn how to automatically create ARM Storage Accounts and what’s the difference.

---

First of all let’s recap the most important terms and learn some facts that have changed since my spring 2016 blog post:

1. Storage Account: Storage Accounts are management namespaces giving you access to your storage resources such as blobs or files. Ressources can be accessed via http-endpoints such as https://storageAccountName.blob.core.windows.net/container/blob for a blob stored in your Storage Account. You can have up to 100 200 uniquely named Storage Accounts with a single Azure subscription by default. As you are billed for Azure storage usage based on a respective Storage Account but not for the Storage Account itself, you could create up to 100 200 Storage Accounts simultaneously. Why that? Continue reading below.
2. Availability Set: An Azure Availability Set is an assignment of two or more VMs to make sure that at least one of your VMs is available in case of planned or unplanned maintenance events in an Azure data center. Each VM is assigned an update or fault domain by the Azure platform. Azure VMs have only met the 99.95% SLA when they were deployed in an Availability Set in the past. As of November 2016 you are guaranteed an availability of 99.9% for single-instance VMs using premium storage for all disks, too.
3. Storage Stamps: Storage Stamps in Azure are clusters of several racks of storage nodes in an Azure data center. They are the physical layer your Storage Accounts are created on.

Everything is working fine – why should I focus on Azure Resource Manager (ARM) nevertheless?

Azure Service Manager (ASM) is the legacy management model for Azure platform focussing on single services. Azure Resource Manager offers more possibilites to devs and admins such as resource pooling in Resource Groups. In addition to that, new features such as cool blob storage or auto-shutdown scheduler for VMs are only implemented in Azure Resource Manager. You can use Azure Primary Portal to manage both, ASM- and ARM-based resources whereas Azure Legacy Portal can only handle ASM.

The following script block shows an example script for Azure RM Storage Account creation. You will realize that there are way more (and different) options to configure compared to Storage Accounts in ASM:

param(
[string]$resourceGroupName,
[string]$storageAccountName
)
Import-Module Azure
New-AzureRmStorageAccount -ResourceGroupName $ResourceGroupName `
-Name $storageAccountName `
-SkuName <Premium_LRS, Standard_GRS, Standard_LRS, Standard_RAGRS, Standard_ZRS> ## Select the Storage Account type`
-Kind <BlobStorage, Storage> ## BlobStorage can have cool or hot access tier `
-AccessTier <Cool, Hot> ## in combination with BlobStorage `
-Location xxx ## Azure datacenter`
-EnableEncryptionService Blob ## BlobStorage can be encrypted`

You can define if you want to use hot or cool blob storage or storage encryption at rest, features that are only available via ARM. Storage Account types, e.g. geo-redundant standard storage (Standard_GRS) or local-redundant premium storage (Premium_LRS), are defined using the -SkuName parameter; -Type as it was used in ASM does not work in ARM. So there are some differences that make it necessary to re-create older PowerShell scripts instead of only putting an “Rm” into the CMDLET’s name.

I don’t need 200 Storage Accounts at all. Why should I blow up my subscription with empty placeholder accounts?

Well, you shouldn’t. As I’ve already explained before, Storage Accounts are nothing else but management namespaces pointing to Azure Stamps. Every Storage Account, no matter if it was created via ASM or ARM, has a unique DNS name, e.g. storageAccountName.blob.core.windows.net – and DNS names refer to IP addresses.

Those IP addresses, however, don’t belong to the Storage Account’s DNS name but to the underlying storage stamp. Why? Well, the Storage Account’s DNS name is an alias (DNS CNAME record) for the stamp’s DNS name. If you remember my spring blog post you already know what’s next to come: Storage Accounts that are created simultaneously have a good chance to be created on the same stamp. And blob transfers between Storage Accounts on the same stamp are pretty quick compared to blob transfers between different stamps.

I don’t tell you to blow up your subscription with 200 Storage Accounts but Storage Accounts are limited, for example to 500TB maximum storage or 20,000 IOPS which is 40 standard tier VM disks. These are, indeed, enormous numbers. But however, if you have to deploy some high performance SQL VMs into one subscription; for example because you’re building a dev environment for a conference or something like that you can take advantage of knowing about stamps and how they’re related to Storage Accounts.

Okay, so, let’s create 10 ARM Storage Accounts:

### Import Azure module
import-Module Azure
### Define parameters
$baseName = 'azureandbeyondsa00'
$resourceGroupName = 'azureandbeyondtestrg'
$location = 'NorthEurope'
$creationScript = '<path to creation script>'
### Run 10 times
For ($i=0; $i -lt 10; $i++) {
$storageAccountName = $baseName + [string]$i
Start-Process powershell.exe -argumentList ($creationScript, $storageAccountName, $resourceGroupName, $location)
}

My creation script and result look like this:

param (
[string]$storageAccountName,
[string]$resourceGroupName,
[string]$location
)
Import-Module Azure
Write-Host "Testing Resource Group $resourceGroupName" -ForegroundColor Green
Try {
Get-AzureRmResourceGroup -Name $resourceGroupName -ErrorAction Stop
}
Catch {
Write-Warning "Resource Group $resourceGroupName doesn't exist. Creating resource group $resourceGroupName in $location..."
New-AzureRmResourceGroup -Name $resourceGroupName -Location $location
Wait-Event -Timeout 30
}
Write-Host "Creating ARM StorageAccount $storageAccountName" -ForegroundColor Green
New-AzureRmStorageAccount -Name $storageAccountName `
-SkuName Standard_LRS `
-Location $location `
-ResourceGroupName $resourceGroupName `
-Kind BlobStorage `
-AccessTier Cool

Let’s take a look at the stamps. We need to get the Storage Accounts we’d just created and to get the IP addresses of the respective blob url. Therefore we can use the following PowerShell script:

$baseName = 'azureandbeyondsa00'
$storageAccounts = @(Get-AzureRmStorageAccount | where {$_.StorageAccountName -like $BaseName + "*"})
Foreach ($storageAccount in $storageAccounts) {
$bloburl = $storageAccount.StorageAccountName + ".blob.core.windows.net"
$StorageAccountName = $storageAccount.storageaccountname
$dnslookup = [System.Net.Dns]::GetHostAddresses($bloburl)
$stampIp = $dnslookup.IpAddressToString
$StampIPList += @{$StorageAccountName = $stampIp}
}
$StampIPList

 

As you can see above, two Storage Accounts have been created on the same stamp whereas all the others use different stamps. Now back to the SQL example: You need to deploy several of those dev environments and you want to do it quickly. What you need are several Storage Accounts on the same stamp so make sure to use some kind of automation in order to have your Storage Accounts created simultaneously. After that you create your dev environment in a first Resource Group on the first Storage Account, create a second Resource Group on the second Storage Account, copy your VM disks to the second account and build your VMs around them. If your Storage Accounts are created on the same stamp you are likely to achieve your goal in very little time.

One thing to keep in mind: Storage Accounts can be moved from one stamp to another after they were created. So don’t create 200 Storage Accounts when you start using Azure because this will surely blow up your subscription with placeholder accounts you will not use any time later. But when you need several accounts because of limitations and you need to exchange large blobs between them you are now able to do it quickly.

That’s all for today. Have a nice week end and stay tuned!

Kind regards,
Tom