Azure Storage Accounts: How to plan your architecture to speed up blob transfers

Storage Accounts, Availability Sets and Storage Stamps are terms you will often be confronted with when you deal with Azure Storage. But what do these terms mean and how can you use your knowledge of them to speed up your cloud storage and enhance the availability of your systems?

First of all, what do these terms mean?

1. Storage Account: Storage Accounts are management namespaces giving you access to your storage resources such as blobs or files. Ressources can be accessed via http-endpoints such as https://storageAccountName.blob.core.windows.net/container/blob for a blob stored in your Storage Account. You can have up to 100 uniquely named Storage Accounts with a single Azure subscription. As you are billed for Azure Storage usage based on a respective Storage Account but not for the Storage Account itself, you could create up to 100 Storage Accounts at the beginning of planning your Azure infrastructure. Why that? You’ll see later.
2. Availability Set: An Azure Availability Set is an assignment of two or more VMs to make sure that at least one of your VMs is available in case of planned or unplanned maintenance events in an Azure data center. Each VM is assigned an update or fault domain by the Azure platform. By the way: Azure VMs only meet the 99.95% SLA when they are deployed in an Availability Set.
3. Storage Stamps: Storage Stamps in Azure are clusters of several racks of storage nodes in an Azure data center. They are the physical layer your Storage Accounts are created on.

As I’ve mentioned above, you can access your Storage Accounts, containers or blobs via http-endpoints. Such an endpoint is made up of a DNS name and subdirectories representing the storage containers within a Storage Account. If you resolve the DNS names using nslookup you will find out that each DNS name which represents a Storage Account actually is a DNS alias. And if you take a closer look at the screen shot I have posted below you will realize that two of my Storage Accounts (azuretttsa000 and azuretttsa002) point to the same IP address (40.68.232.56).

Why that?

The IP addresses represent Storage Stamps, simple as that. So Storage Accounts that are created on the same Storage Stamp are accessed via the same IP address.

I’ve written a little PowerShell script to automate Storage Account creation. The script creates one Azure Storage Account with the parameters -storageAccountName, -label, -type and -affinityGroup.

param (
    [string]$storageAccountName,
    [string]$affinityGroupName,
    [string]$storageAccountType
)
import-Module Azure
Write-Host "Creating StorageAccount $storageAccountName"
New-AzureStorageAccount -storageAccountName $storageAccountName -label $storageAccountName -type $storageAccountType -affinityGroup $affinityGroupName
Wait-Event -Timeout 10

In order to really automate and parallelize Storage Account creation I’ve written a second script that calls the creation script up to 100 times simultaneously. The sample script below calls the creation script 10 times.

### Import Azure module
import-Module Azure
### Define parameters
$baseName = ''
$affinityGroupName = ''
$creationScript = ''
### Run 10 times
For ($i=0; $i -lt 10; $i++) {
    $storageAccountName = $baseName + [string]$i
    Start-Process powershell.exe -argumentList ($creationScript, $storageAccountName, $storageAccountType, $affinityGroupName)
}

Now, what is this good for?

Well, keeping in mind that you can create up to 100 uniquely named Storage Accounts in a single Azure subscription and assuming that Storage Accounts are created nearby each other when they are created almost simultaneously, some of your Storage Accounts might share one or several Storage Stamps. To prove this you can check the endpoint IP addresses of all Storage Account that have been created previously:

$storageAccount = @(Get-AzureStorageAccount | where {$_.StorageAccountName -like $baseName + "*"})
Foreach ($account in $storageAccount) {
    $storageAccountName = $account.storageaccountname
    $endpoint = $storageAccountName + '.blob.core.windows.net'
    $dnslookup = [System.Net.Dns]::GetHostAddresses($endpoint)
    $IPAddress = $dnslookup.IpAddressToString
    Write-Host $StorageAccountname, $IPAddress
}

I’ve created 10 Storage Accounts simultaneously in West Europe datacenter and they have been created on 3 different Storage Stamps:

OK, but how does it help me?

You have learned that a Storage Stamp is a kind of storage cluster. Copying files from one Storage Account to another one on the same Storage Stamp is really quick. To prove it just take a VHD file from one of your Storage Accounts and copy it to two other Storage Accounts: One on the same Storage Stamp and one on another one. You can use one of the Azure Storage Explorers I’ve mentioned in one of my previous blog posts to accomplish this task. I’ve used Microsoft Azure Storage Explorer to copy my AzureTTTLab.vhd file to both Storage Accounts azuretttsa001 and azuretttsa002 simultaneously.

As you can compare with the IP table above, the Storage Account azuretttsa002 is placed on the same Storage Stamp as the source account azuretttsa000. While the copy job to the same stamp finished in seconds, the second job has only just begun with the data transfer (364 MB).

So if you want to duplicate one of your VMs in order to build an auto-scaling cloud service or if you want to clone an entire dev/test environment and you want to do it quickly, make sure to use destination Storage Accounts on the same Storage Stamp as the source account resides on. If there are any questions left feel free to contact me.

Happy testing and stay tuned,
Tom