Azure Automation and parallel processing

RunbooksFor a customer I was working on some Azure Automation for starting and stopping virtual machines on a schedule. Many resources are available with examples on how to stop and start virtual machines from Azure Automation runbooks. Unfortunately many examples are based on sequential execution of code and are nice when you have only a few VMs but result in an unacceptable long execution time in larger environments.

Execution in parallel is essential for larger environments. PowerShell Workflow can perform code in parallel using ForEach -Parallel, however Workflows may be undesirable since it has some caveats that are confusing for those who are used to Native PowerShell (PowerShell Workflows: Restrictions). And although Graphical Runbooks look nice, they do not support parallel execution.

What options do we have for parallel execution in Native PowerShell? And what are the differences in execution time when we compare different approaches?

I created 10 Azure Virtual Machines, stopped the VMs (deallocated) and created some runbooks to test the processing time for starting the VMs.

Code_Graphical_RunbookGraphical Runbook
They look cool, but execution of the ‘Start VM’ code is performed in sequence for each VM that comes from the ‘Get all VMs in RG’. Execution takes around 40 minutes; 4 minutes per VM on average. The same is true for the Graphical PowerShell Workflow Runbook.

PowerShell Native and PowerShell Workflow ForEach
The well known ForEach script statement processes every object in the collection one-by-one. Execution once again takes a long time. Wrapping ForEach in a workflow does not make a difference and execution is still sequential. Again around 40 minutes for just 10 VMs to be started.

Code_ForEach

PowerShell Workflow ForEach -Parallel
For parallel processing using PowerShell Workflows the For-Each with the -Parallel option is the way to go, since this is what it was designed for. Execution time is short and one would expect 4 minutes plus some overhead. Around 5 minutes is required to start the 10 VMs.

Code_PowerShell-WorkFlow_ForEach_Parallel

PowerShell Native Jobs
In Native Powershell actions can be performed in parallel by using the Start-Job command. When we put each individual Start-AzureRmVM commands in their own job we get PowerShell to execute them in parallel.

Code_Native_PowerShell_Jobs

The Start-Sleep 30 is a delay between each start-job and seems to provide enough time to get the job started and perform the scriptblock that was passed to it. Without the sleep I was unable to get any reliable results. This approach worked but the resulting code is a bit more complex and we have to deal with passing information into the scriptblock and the code needs to login. We cannot use the Azure Automation connection credential here since it is not available.

PowerShell Native Invoke-Parallel
Based on the post ‘Simultaneously Start|Stop all Azure Resource Manager Virtual Machines in a Resource Group‘ we could use the Invoke-Parallel function. As states in the blog post “The brilliant Invoke-Parallel Powershell Script from Cookie.Monster can be used to in essence multi-thread and run in parallel the Virtual Machine ‘start’ and ‘stop’ requests. Execution time comes in at around 5 minutes to start the 10 VMs.

Code_Native_PowerShell_Invoke-Parallel_Function

I ran the scripts multiple times. The results are shown in the table below:

Parallel Run1 Run2 Run3 Run4 Avg Reliable
Native PowerShell
ForEach No 39.54 42.46 51.84 35.83 42.42 Yes
Jobs (Start-Job) Yes 10.15 10.88 11.48 12.87 11.35 Yes*
Invoke-Parallel Yes 5.14 4.87 3.66 5.42 4.77 Yes
PowerShell WorkFlow
[1] ForEach No 39.68 ±40.00 Yes
ForEach -Parallel Yes 6.23 5.13 5.09 4.14 5.15 Yes
Graphical Runbook
[2] Graph.Runbook No ±36.00 –  –  –  ±40.00 Yes
[3] Graph.Workflow No –  –  –  –  ±40.00 Yes

*30 second start-sleep needed to be added after each start-job to prevent multiple jobs starting in parallel causing the runbook to fail.
[1] The PowerShell Workflow test was performed once and returned the same time as the Native PowerShell code. Both use the simple ForEach. An average time of around 40 minutes is assumed.
[2] The Graphical Runbook was tested once but resulted in the same execution time as the other non-sequential tests. An average time of around 40 minutes is assumed.
[3] The graphical PowerShell Workflow runbook was not tested since it does not support parallel processing and would also result in sequential execution of the start VM code. An average time of around 40 minutes is assumed.

Conclusions
So… for something as simple as performing one command on a bunch of Azure resource manager resources you can go with PowerShell workflow. This keeps the code short and simple and performance is great.

If you want to go Native PowerShell you have two options:
a. Use the Invoke-Parallel function
b. Use PowerShell Jobs.

I prefer to go the invoke-parallel way since this keeps the code easy to understand, is easy to implement, has many options and performs just as fast as ForEach -Parallel in PowerShell Workflow. The jobs (Start-Job) does also work, but I find the coding is unnecessarily complicated and you have to deal with scopes, credentials, logins and potential job conflicts.

Last thing to point out is that I do not know how this all scales if you go from 10 to 100 or even a 1000 servers. Anyway going the parallel route is always the way to go!

Invoke-Parallel download

GitHub for Invoke-Parallel by RamblingCookieMonster
TechNet Gallery: Invoke-Parallel – Parallel PowerShell

Resources on VM Start/Stop

Simultaneously Start|Stop all Azure Resource Manager Virtual Machines in a Resource Group
Stop Microsoft Azure Virtual Machines in parallel with PowerShell
Stop or Start VMs in an Windows Azure Subscription Using PowerShell Loop
Scheduled VM Shutdown/Startup with Azure Automation and Tags
Shutting down VMs on schedule in Azure
Scheduled Virtual Machine Shutdown/Startup – Microsoft Azure
Sequentially Start & Stop Azure VMs
Sequentially Start & Stop Azure VMs – Azure Automation
Start and Stop Azure VM’s (and more) in Parallel from Powershell

Resources on PowerShell WorkFlow

PowerShell Workflows: Restrictions
Syntactic Differences Between Script Workflows and Scripts
PowerShell workflow: everything you ever needed to know about it!
PowerShell Workflow for Mere Mortals: Part 1
PowerShell Workflow for Mere Mortals: Part 2
PowerShell Workflow for Mere Mortals: Part 3
PowerShell Workflow for Mere Mortals: Part 4
PowerShell Workflow for Mere Mortals: Part 5

Share this:

2 thoughts on “Azure Automation and parallel processing

  1. Thanks for sharing. I was looking for a way to improve an existing script to toggle power for approx. 60 Azure VM’s. Currently the script runs for almost 2 hours, so when I start refactoring I will give Invoke-Parallel a try.

  2. Hi there,I read your blogs named “Azure Automation and parallel processing – Thinking Cloud” on a regular basis.Your humoristic style is awesome, keep doing what you’re doing! And you can look our website about اغانى 2018.

Leave a Reply

Your email address will not be published. Required fields are marked *