Problem Monitoring disk performance on virtual machines

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

Users have been reporting performance issues with some of our applications. Ive been asked to look at disk performance to see if this is the issue.

We have 4 host servers in a Dell blade system using HyperV to run the guest servers which act as the citrix xenapp servers. The citrix servers have a single vhd which resides on a SAN. Each host has its own virtual disk on the SAN in raid 5.

A quick check on the virtual Citrix servers using the PhysicalDisk object and the Average Disk queue length counter. Gives an average of around 110. Is this a possible cause or an erroneous result?

Any tips on the best way to monitor disk performance in this setup?

The host servers are 2008 and the guest servers are 2003.

 

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

You're obviously going about the right way in trying to diagnose this issue, it could well be a hard disk is on its way out. I am assuming you are managing your virual environment using XenApp Server? If this is the case there should be some more performance tools to run etc.

All the best and lets us know how you get on

 

No sorry there not XenApp servers (although we have that running on another blade ) the servers im on about have presentation server on with Office, a SQL ticketing system and a few other reporting tools.

The performance issue is a complete ball ache. It appears to be random although im sure when we find out whats causing it we'll realise it isnt random.

We've ruled out the WAN after using some monitoring tools.
A desktop refresh of most users PCs saw some improvement.
Rebooting the virtual machines usually resolves the issue for a short time (this is big hint i guess)

It will affect one person but not another on the same server.
The SAN doesn't report any errors. The general spec of the host servers (memory & processor) is fine for the amount of users.

 

What about the vm's themselves any signs of memory leaks or anything? I'm not familiar with hyper-V but you can get performance/usage graphs out of vmware for each vm can you do that on hyper-v, anything steadily climbing.

What sort of disks are the affected vm's running on 15k SAS, 10K, SATA?
Are there any AV scans/network scans backup schedules that run at these times?

 

First thing I would be doing is loading up the resource monitor and checking the response times on your disks. To give you an idea of what you should be expecting, anything over 20ms is indicitive of issues on your back end storage.

Things to look out for on your storage is whether they are properly aligned to the VHD, if your SAN storage isn't aligned correctly then what will happen is that your data will be read\written over multiple blocks instead of single blocks. Have a chat with your storage team to try and discover how the disks have been aligned (you haven't mentioned your SAN provider so it's difficult to tell you exactly how it should be configured).

For NetApp have a look here for a PS cmdlet to check alignment.

 

Dales
Just started reading through this article i think the correct way to monitor hyper v performance is from the host using the standard performance monitor. Im gonna follow the articles counter setup.
Monitoring Hyper-V Performance - All topics fundamental - Site Home - MSDN Blogs
The SAN is made up of 10k SAS drives.
On the servers we have endpoint which runs overnight.
Our backup finishes in the morning before most people get in.
We also have trendmicro deep security but im 95% sure thats only scanning our SQL cluster.

SimonD
ResourceMon on the host servers right?
The SAN is from Dell and fairly old and light weight. Sorry its an EMC SAN.

Thanks for the help guys.

 

In any storage system disk is the most likely place for a bottleneck to occur.

 

Actually, the guests rather than the host because the VHD for the guest OS is housed on the SAN whereas if you were doing it on the host you would be getting performance data from the local server (assuming that it's not boot from SAN for the blade servers as well).

Oh and Dell rebadge their EMC gear anyway, you may want to see if there are any outstanding firmware patches that need applying to the storage. Also you should be able to get some kind of performance data from the storage controllers.

I would probably go so far as to say that your running the OS' on mis or un-aligned disks which will be the root cause for your performance issues, unfortunately there are no easy fixes (you usually have to vacate the storage, collapse it and start again, not easy if it's your only storage platform).

 

Ive setup some Data Collector sets using the Hyper-V counters in 2008. Now i just wait
Each host has its own local c:/ drive that holds the OS.

Ive not come across mis or unaligned disks before what is it exactly?