Automate Classes with Kyle and Mendy Ep1

[MendyGreen]

See the recording for this cast, start at around 12 minutes in order to skip the countdown until 8:30

https://www.youtube.com/watch?v=Lv5ZVAJIFRk&t=1s

 

There was some confusion as I quickly ballooned from simple Searches and Groups to a Remote Monitor that changed into a complex Powershell script that then turned into a confusing State Based monitor.

I'll break it down here for people who want  a quick review without watching the entire show.

The goal: Determine which Virtual Machines are supposed to be running on the Host that are not actually running right now, and start them or make a ticket if they fail to start.

To identify these virtual machines I have to make sure I find machines that are either A) Not a replica B) If it is a replica, its the PRIMARY Replica. Additionally I have to confirm that the machine is supposed to be running. I use two methods to do this 1) I confirm that the VHD(x) was recently written to, and also confirm that the VM Settings itself is configured to start automatically.

I use the following Powershell Modules to determine what I listed above..

Get-VM

Get-VMReplication

Get-VMHardDiskDrive

Get-ChildItem

I use the following PowerShell parsing methods and logical processing to properly handle the data received from the above modules.

where

For-Each

If () { } else {}

My first step is to get a list of all Virtual Machines that are NOT in a running state. Specifically because there are other states (Starting, Shutting down) I limit my search to those that are either Saved or Off.

Get-VM | where {$_.State -in "Off","Saved"}

I'm using the "-in" logical comparison to match the value of the VM State retrieved from the Get-VM command to either Off or Saved. This will return any VM that isn't running. This is the base check.

Next we want to check for machines that are meant to be on. This is just an additional condition of looking for Virtual Machines that are configured to start automatically.

Get-VM | where {$_.State -in "Off","Saved" -and  $_.AutomaticStartAction -ne "Nothing"}

Using "-and" and "-ne" (not equals) I'm able to string a second check on the same VM object to exclude anything that is configured to DO NOTHING on reboot. 

My next  step is to get the VM Replication status, using the powershell module Get-VMReplication I will be returned with a list of all Virtual Machines that are replicas.This is similar to the Get-VM module except that I'm specifically getting ONLY Replicas. Using similar logic to above I know that I only want to get Virtual Machine replicas where the Primary Server is the same as the server that the name itself is running under.

Get-VMReplication -Mode Primary|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:computerName).HostName)}

If you look at Get-VMReplication you'll see a property "PrimaryServer" that has the full FQDN of the servername. The only way to get the FQDN in Powershell without using WMI is by tapping into .NET (which is faster) and that is the second part of the command. Combining addressing the DNS Namespace in .NET to retrieve the FQDN we combine it with the environment variable computername to generate the same string that would match within the PrimaryServer attribute. We now have a full list of VMs that exist on the VM as the primary replica copy.

Combining the first two modules together so we can get a single list of Virtual Machines the command will look like this.

Get-VM | where {($_.State -in "Off","Saved" -and  $_.AutomaticStartAction -ne "Nothing") -and ($_.Name -in (Get-VMReplication -Mode Primary|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:computerName).HostName)}).VMName)}

Note how we created separate groups in the logic, because two statements are AND but the second statement (the replica primary server) is going to be an OR against a final check -if Replica is even enabled at all. The final piece to point out is how I'm specifically selecting a specific property from the output by doing ".VMName" which is the value I want to compare against the $_.Name from Get-VM.

Adding in the final and last condition check to confirm that we're getting NON REPLICA Vm's as well as Replica VMs where the primary server is itself I'm going to adjust the code as follows.

Get-VM | where {($_.State -in "Off","Saved" -and  $_.AutomaticStartAction -ne "Nothing") -and ($_.Name -in (Get-VMReplication|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:computerName).HostName)}).VMName -or $_.ReplicationState -eq "Disabled")}

We now have  a list of VM's that are Off/Not Running, Configured to turn on automatically, will run on this host if it was on however we still don't know if the VM was recently used. Using Get-ChildItem and Get-VMHardDiskDrive I can pull out the path to the first VHD on the VM (which will always be the boot disk) and check the last write time. However note, that this can take some time and I only want to do this for the Virtual Machines that we know are supposed to be on. This means we need to create an If statement to see IF results are returned then we'll check for the Last Write TIme.

#Create variable containing all the Virtual Machines.
$VirtualMachines = Get-VM | where {($_.State -in "Off","Saved" -and  $_.AutomaticStartAction -ne "Nothing") -and ($_.Name -in (Get-VMReplication -Mode Primary|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:computerName).HostName)}).VMName)}

#Create If Statement and check if the variable is NULL
if ($null -ne $VirtualMachines) {
	#Loop through the virtual machines and check last write time
		$VirtualMachines|foreach {
			#create the actual check for the last write time, follow along as we nest additional modules within the IF Check
			if ( (Get-ChildItem (Get-VMHardDiskDrive -VMName $_.Name -ControllerLocation 0).Path).LastWriteTime -gt (Get-Date).AddDays(-2) ) {
				#Start VM
				Start-VM $_
				#Output Hostnames or VM Object
				$_
			} #end If statement for last write time
		} #end Loop statement
} # End If statement for Null variable, create else statement that no issues found.

else {Write-Host "No issues detected."}

The above script has been written in long hand with lots of comments to indicate what the script is doing. As you can see we're still sticking to the basics explained in previous examples of pulling information as we just overlay commands over with themselves to do more complex comparisons and matching.

The above script will do everything we need, you can remove the Start-VM line and just have it echo out the results or you can remove the $_ line by itself to remove the output from occurring in the event the machines start successfully.

Keep in mind about the Escape Characters, when using the above script in Automate Remote Monitor you will need it to be one line, and will need to call it in by executing the poewrshell interpreter directly. The full command to use in the remote monitor is as follows.

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -noprofile -command " & {$a = get-vm|? {($_.State -in \"Saved\",\"Off\" -and $_.AutomaticStartAction -ne \"Nothing\") -and ($_.ReplicationState -eq \"Disabled\" -or ($_.Name -in (Get-VMReplication -Mode Primary|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:computerName).HostName)}).VMName))}; if ($a){$a|% {if ((gci (Get-VMHardDiskDrive -VMName $_.Name -ControllerLocation 0).Path).LastWriteTime -gt (Get-Date).AddDays(-2)){Start-VM $_} }}else {Write-Host \"No issues detected.\"} }"

Using ";" to indicate line breaks to allow multi line scripts to be executed on a single line.

Regarding the remote monitor itself you can set it to be State Based by following the below screenshot.

The details of the state based conditions are covered more inside the video, and will not be covered in detail here. This post was just to cover the areas that I "rushed" through so that I could focus on Automate.

Please hit me up if you have any questions.

Edited March 15, 2019 by MGreen
Updated with '-Mode Primary' mentioned by Hikato for Extended Replicas

[DNA3e8]

Do you have a different source for the recording? it is in plying and the link is broken:

 

https://www.twitch.tv//v/376361931?t=

[MendyGreen]

We're looking into this! Just waiting on @kspooner

[kspooner]

19 hours ago, DNA3e8 said:

Do you have a different source for the recording? it is in plying and the link is broken:

 

https://www.twitch.tv//v/376361931?t=

Fixed, here is the link

 

[DNA3e8]

thank you!

[marmfield]

Thank you for this information!

 

@MGreen How do I modify this to not generate a failure on a server that has the Hyper-V role installed but isn't hosting any VMs?  I know that best practice would be to remove the Hyper-V role but that isn't always an option for some of our clients. 

 

Thank you in advance,

 

Matt

[MendyGreen]

I'm not completely sure @marmfield because it depends which command the error comes from. I know Get-VM returns nothing if there are no VMs so the error is likely to be coming from the second statement when I try looping through the variable.

You can wrap the entire statement in an IF check to make sure Get-VM isn't nul

if (Get-VM) { <paste the original code> }

This will return a blank result when run which will put the monitor in a warning state. you can add on an Else at the end of the above to spit out "No issues detected." just like the else in the original code post if you want the monitor to succeed.

[marmfield]

@MGreen This is the error I getting.  At first I thought it was just Server 2008 R2 machines but it is happening on a few Windows Server 2012 R2 as well.  The strange thing is out of 69 servers of various OS levels, it fails on 12 with the below message or an error about The 'get-vm' command was found in the module 'HyperV', but the module 
could not be loaded.

 

You must provide a value expression on the right-hand side of the '-' operator.
At line:1 char:31
+  & {$a = get-vm|? {($_.State - <<<< in "Saved","Off" -and $_.AutomaticStartAc
tion -ne "Nothing") -and ($_.ReplicationState -eq "Disabled" -or ($_.Name -in (
Get-VMReplication|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:
computerName).HostName)}).VMName))}; if ($a){$a|% {if ((gci (Get-VMHardDiskDriv
e -VMName $_.Name -ControllerLocation 0).Path).LastWriteTime -gt (Get-Date).Add
Days(-2)){Start-VM $_} }}else {Write-Host "No issues detected."} }
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordEx 
   ception
    + FullyQualifiedErrorId : ExpectedValueExpression

 

Edited March 11, 2019 by marmfield

[Hikato]

@MGreen great write-up, thanks for sharing!  There is a small issue with the results of Get-VMReplication.  If you have Extended replication set up, the host can be the replica copy, but be the primary server for the extended replica.

 

I fixed it by adding '-Mode Primary' to the Get-VMReplication cmdlet.  All is well now!

Edited March 14, 2019 by Hikato

[MendyGreen]

On 3/11/2019 at 1:18 PM, marmfield said:

@MGreen This is the error I getting.  At first I thought it was just Server 2008 R2 machines but it is happening on a few Windows Server 2012 R2 as well.  The strange thing is out of 69 servers of various OS levels, it fails on 12 with the below message or an error about The 'get-vm' command was found in the module 'HyperV', but the module 
could not be loaded.

 

You must provide a value expression on the right-hand side of the '-' operator.
At line:1 char:31
+  & {$a = get-vm|? {($_.State - <<<< in "Saved","Off" -and $_.AutomaticStartAc
tion -ne "Nothing") -and ($_.ReplicationState -eq "Disabled" -or ($_.Name -in (
Get-VMReplication|?{$_.PrimaryServer -eq ([System.Net.Dns]::GetHostByName($env:
computerName).HostName)}).VMName))}; if ($a){$a|% {if ((gci (Get-VMHardDiskDriv
e -VMName $_.Name -ControllerLocation 0).Path).LastWriteTime -gt (Get-Date).Add
Days(-2)){Start-VM $_} }}else {Write-Host "No issues detected."} }
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordEx 
   ception
    + FullyQualifiedErrorId : ExpectedValueExpression

 

@marmfield This error is basically telling you that $a (or Get-VM) is null. This will require some troubleshooting to be done on one of the problem hosts directly to see how the command behaves interactively vs through LT Command etc. We have this deployed on over a hundred hosts without an issue.

[marmfield]

8 hours ago, MGreen said:

@marmfield This error is basically telling you that $a (or Get-VM) is null. This will require some troubleshooting to be done on one of the problem hosts directly to see how the command behaves interactively vs through LT Command etc. We have this deployed on over a hundred hosts without an issue.

I get the same error on the host itself.  We have about 68 HyperV hosts and only 14 get this error.  I'll keep plugging away at it to see if I can figure out why the error, unless it is related to the fact that these HyperV hosts don't have any VMs on it and that would be the reason for the null value.

Is there a way to check for this?  I really need to learn Powershell so I don't have to ask all these questions! 

M

[MendyGreen]

3 minutes ago, marmfield said:

I get the same error on the host itself.  We have about 68 HyperV hosts and only 14 get this error.  I'll keep plugging away at it to see if I can figure out why the error, unless it is related to the fact that these HyperV hosts don't have any VMs on it and that would be the reason for the null value.

Is there a way to check for this?  I really need to learn Powershell so I don't have to ask all these questions! 

M

It can definitely be related to no VMs. Hit me up on slack after 5pm EST or over the weekend and I can help you out and at least see if I can give you enough info to assist you in learning.

[Namik]

This video was awesome thank you! Definitely easier to listen and learn from you guys than reading horrible documentation from Connectwise.