1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Problem Monitoring disk performance on virtual machines

Discussion in 'Hardware' started by mcbro, Nov 28, 2011.

  1. mcbro

    mcbro Byte Poster

    136
    0
    23
    Users have been reporting performance issues with some of our applications. Ive been asked to look at disk performance to see if this is the issue.

    We have 4 host servers in a Dell blade system using HyperV to run the guest servers which act as the citrix xenapp servers. The citrix servers have a single vhd which resides on a SAN. Each host has its own virtual disk on the SAN in raid 5.

    A quick check on the virtual Citrix servers using the PhysicalDisk object and the Average Disk queue length counter. Gives an average of around 110. Is this a possible cause or an erroneous result?

    Any tips on the best way to monitor disk performance in this setup?

    The host servers are 2008 and the guest servers are 2003.
     
    Certifications: MCITP:EA, CCNA
  2. onoski

    onoski Terabyte Poster

    3,120
    51
    154

    You're obviously going about the right way in trying to diagnose this issue, it could well be a hard disk is on its way out. I am assuming you are managing your virual environment using XenApp Server? If this is the case there should be some more performance tools to run etc.

    All the best and lets us know how you get on:)
     
    Certifications: MCSE: 2003, MCSA: 2003 Messaging, MCP, HNC BIT, ITIL Fdn V3, SDI Fdn, VCP 4 & VCP 5
    WIP: MCTS:70-236, PowerShell
  3. mcbro

    mcbro Byte Poster

    136
    0
    23
    No sorry there not XenApp servers (although we have that running on another blade :dry) the servers im on about have presentation server on with Office, a SQL ticketing system and a few other reporting tools.

    The performance issue is a complete ball ache. It appears to be random although im sure when we find out whats causing it we'll realise it isnt random.

    We've ruled out the WAN after using some monitoring tools.
    A desktop refresh of most users PCs saw some improvement.
    Rebooting the virtual machines usually resolves the issue for a short time (this is big hint i guess)

    It will affect one person but not another on the same server.
    The SAN doesn't report any errors. The general spec of the host servers (memory & processor) is fine for the amount of users.
     
    Certifications: MCITP:EA, CCNA
  4. dales

    dales Gigabyte Poster

    1,998
    46
    97
    What about the vm's themselves any signs of memory leaks or anything? I'm not familiar with hyper-V but you can get performance/usage graphs out of vmware for each vm can you do that on hyper-v, anything steadily climbing.

    What sort of disks are the affected vm's running on 15k SAS, 10K, SATA?
    Are there any AV scans/network scans backup schedules that run at these times?
     
    Certifications: vExpert 2014+2015+2016,VCP-DT,CCE-V, CCE-AD, CCP-AD, CCEE, CCAA XenApp, CCA Netscaler, XenApp 6.5, XenDesktop 5 & Xenserver 6,VCP3+5,VTSP,MCSA MCDST MCP A+ ITIL F
    WIP: Nothing
  5. SimonD

    SimonD Terabyte Poster Moderator

    3,463
    397
    199
    First thing I would be doing is loading up the resource monitor and checking the response times on your disks. To give you an idea of what you should be expecting, anything over 20ms is indicitive of issues on your back end storage.

    Things to look out for on your storage is whether they are properly aligned to the VHD, if your SAN storage isn't aligned correctly then what will happen is that your data will be read\written over multiple blocks instead of single blocks. Have a chat with your storage team to try and discover how the disks have been aligned (you haven't mentioned your SAN provider so it's difficult to tell you exactly how it should be configured).

    For NetApp have a look here for a PS cmdlet to check alignment.
     
    Certifications: CNA | CNE | CCNA | MCP | MCP+I | MCSE NT4 | MCSA 2003 | Security+ | MCSA:S 2003 | MCSE:S 2003 | MCTS:SCCM 2007 | MCTS:Win 7 | MCITP:EDA7 | MCITP:SA | MCITP:EA | MCTS:Hyper-V | VCP 4 | ITIL v3 Foundation | VCP 5 DCV | VCP 5 Cloud | VCP6 NV | VCP6 DCV | VCAP 5.5 DCA
    WIP: VCP6-CMA, VCAP-DCD and Linux + (and possibly VCIX-NV).
  6. mcbro

    mcbro Byte Poster

    136
    0
    23
    Dales
    Just started reading through this article i think the correct way to monitor hyper v performance is from the host using the standard performance monitor. Im gonna follow the articles counter setup.
    Monitoring Hyper-V Performance - All topics fundamental - Site Home - MSDN Blogs
    The SAN is made up of 10k SAS drives.
    On the servers we have endpoint which runs overnight.
    Our backup finishes in the morning before most people get in.
    We also have trendmicro deep security but im 95% sure thats only scanning our SQL cluster.

    SimonD
    ResourceMon on the host servers right?
    The SAN is from Dell and fairly old and light weight. Sorry its an EMC SAN.

    Thanks for the help guys.
     
    Last edited: Nov 28, 2011
    Certifications: MCITP:EA, CCNA
  7. onoski

    onoski Terabyte Poster

    3,120
    51
    154
    In any storage system disk is the most likely place for a bottleneck to occur.
     
    Certifications: MCSE: 2003, MCSA: 2003 Messaging, MCP, HNC BIT, ITIL Fdn V3, SDI Fdn, VCP 4 & VCP 5
    WIP: MCTS:70-236, PowerShell
  8. SimonD

    SimonD Terabyte Poster Moderator

    3,463
    397
    199
    Actually, the guests rather than the host because the VHD for the guest OS is housed on the SAN whereas if you were doing it on the host you would be getting performance data from the local server (assuming that it's not boot from SAN for the blade servers as well).

    Oh and Dell rebadge their EMC gear anyway, you may want to see if there are any outstanding firmware patches that need applying to the storage. Also you should be able to get some kind of performance data from the storage controllers.

    I would probably go so far as to say that your running the OS' on mis or un-aligned disks which will be the root cause for your performance issues, unfortunately there are no easy fixes (you usually have to vacate the storage, collapse it and start again, not easy if it's your only storage platform).
     
    Last edited: Nov 28, 2011
    Certifications: CNA | CNE | CCNA | MCP | MCP+I | MCSE NT4 | MCSA 2003 | Security+ | MCSA:S 2003 | MCSE:S 2003 | MCTS:SCCM 2007 | MCTS:Win 7 | MCITP:EDA7 | MCITP:SA | MCITP:EA | MCTS:Hyper-V | VCP 4 | ITIL v3 Foundation | VCP 5 DCV | VCP 5 Cloud | VCP6 NV | VCP6 DCV | VCAP 5.5 DCA
    WIP: VCP6-CMA, VCAP-DCD and Linux + (and possibly VCIX-NV).
  9. mcbro

    mcbro Byte Poster

    136
    0
    23
    Ive setup some Data Collector sets using the Hyper-V counters in 2008. Now i just wait :tune
    Each host has its own local c:/ drive that holds the OS.

    Ive not come across mis or unaligned disks before what is it exactly?
     
    Certifications: MCITP:EA, CCNA

Share This Page

Loading...