Sean Crookston

VCAP-DCA Objective 6.4 : Troubleshoot Storage Performance and Connectivity

August 18, 2010

Sean Crookston

4 Comments

Another VCAP-DCA objective here. For a complete list of study objectives for the VCAP-DCA ( VDCA-410) browse to http://www.seancrookston.com/vcap-dca/.

This is the first pass through the objective for troubleshooting storage performance and connectivity. If you haven’t already done so, download the zip file full of additional study resources from Kendrick Coleman’s blog. These can be found at his VCAP exam landing page. He has compiled alot of resources and broken them down by objective, and these make great additional reading if you are not greatly familiar with any of the following.

Knowledge

Recall vicfg-* commands related to listing storage configuration

Refer to the vSphere Command-Line Interface Reference for more information on the below commands.

vicfg-iscsi	vicfg-iscsi supports configuration and property retrieval for software or hardware iSCSI initiators.
vicfg-mpath	The vicfg-mpath command supports listing information about Fibre Channel or iSCSI LUNs and changing a path’s state.
vicfg-mpath35	vicfg-mpath35 provides an interface to configure multipath settings for Fibre Channel or iSCSI LUNs on ESX/ESXi version 3.5 hosts.
vicfg-nas	The vicfg-nas command manipulates NAS file systems associated with ESX/ESXi hosts.
vicfg-rescan	Perform a rescan operation each time you reconfigure your storage setup
vicfg-scsidevs	The vicfg-scsidevs command displays information about available LUNs on ESX/ESXi 4.x hosts.
vicfg-volume	The vicfg-volume command supports resignaturing a snapshot volume and mounting and unmounting the volume.
vicfg-module	The vicfg-module command supports setting and retrieving VMkernel module options.

Recall vSphere 4 storage maximums

Refer to the storage section of the vSphere Configuration Maximums guide.

Identify logs used to troubleshoot storage issues

Describe the VMFS file system

Skills and Abilities

Use vicfg-* and esxcli to troubleshoot multipathing and PSA-related issues

Use the vicfg-mpath command to list information about FC or iSCSI luns. Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.

 vicfg-mpath [<conn_options>]
   [--help |
    --list [--path <path> |--device <device>] |  --list-compact [--path <path> |--device <device>] |
    --list-map [--path <path> |--device <device>] |
    --list-paths [--device <device>] |        --list-plugins |
    --state  [active|off] ]

You use the esxcli command to to set path policy like below, where VMW_PSP_xxx will either be Fixed, Fixed_AP, MRU, or RR

esxcli <connection options> nmp device setpolicy –device naa.xxx –psp VMW_PSP_xxx

To check the preferred path:

esxcli <connection options> nmp fixed getpreferred –device naa.xxx

To change the preferred path:

esxcli <connection options> nmp fixed setpreferred –device naa.xxx vmhbax:Cx:Tx:Lx

To retrieve and set round robin path options on a device controlled by the roundrobin psp

esxcli <connection options> nmp roundrobin

To retrieve path selection settings for a device using the roundrobin PSP

esxcli <connection options> nmp roundrobin getconfig –device na.xxx

To specify when the path should change(for load balancing)

esxcli <connection options> nmp roundrobin setconfig –type “bytes” –B 12345 –device naa.xxx

Switches after 12345 bytes have been sent along the current path.

esxcli <connection options> nmp roundrobin setconfig –type=iops –ipos 4000 –device naa.xxx

switches after 4000 I/O operations have been performed

You can use the Mask_Path plugin to mask paths. To view current list of rules

esxcli corestorage claimrule list

To Add

esxcli corestorage claimrule add –plugin MASK_PATH –rule <ruleid> –type xxx –A <adapter>

To Load path claiming rules

esxcli corestorage claimrule load

To release a device from the current plugin

esxcli corestorage claiming unclaim –t location –A vmhba0 –C 0 –T 0 –L 149

To run the path claiming rules

esxcli corestorage claimrule run

To delete a claimrule

esxcli corestorage claimrule delete –r rule#

Also refer to the KB on masking a lun from ESX and ESXi 4.0 using Mask_Path plug-in.

Use vicfg-module to troubleshoot VMkernel storage module configurations

The vicfg-module command supports setting and retrieving VMkernel module options. Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.

vicfg-module [<connection_options>]
    [--get-options <module_name> |
     --help |
     --list |
     --set-options "<option> <value>" <module_name> |
     --vihost <esx_host> ]

Use vicfg-* and esxcli to troubleshoot iSCSI related issues

 vicfg-iscsi [<connection_options>] [option] [suboption] [parameter] [<adapter_name>]

Option is one of --discovery, --static,--authentication, --phba, --target, --lun, --network (Hardware iSCSI only), --pnp (Hardware iSCSI only), --iscsiname, --parameter, --swiscsi, --adapter.

Suboption is one of --list, --add, --remove.

Parameters differ depend on the suboption used.

<adapter_name> is required unless you specify the --help, --swiscsi, or --adapter option.

A couple of examples.

Enable software iSCSI

vicfg-iscsi –-swiscsi –enable

Determine HBA Type/Retrieve ID

vicfg-iscsi –-adapter –list

Check Status of software iSCSI

vicfg-iscsi –-swiscsi –list

Add Dynamic or static discovery

vicfg-iscsi –-static -–add –-ip –name <iscsi name> <adapter name>

vicfg-iscsi –-discovery –-add –ip –-name <iscsi name> <adapter name>

To setup ports for iSCSI Multipathing.

List available uplinks for use with iSCSI adapters

esxcli swiscsi vmnic list –d <vmhba>

Connect iSCSI initiator to the VMkernel ports

esxcli swiscsi nic add –n <port_name> –d <vmhba>

To disconnect iSCSI initiator from VMkernel ports

esxcli swiscsi nic remove –n <port_name> –d <vmhba>

To list all sw iSCSI sessions at the adapter level or target level

esxcli swisci session list –d <iscsi_adapter>

esxcli swiscsi session list –d vmhba36

Remove iSCSI sw sessions

esxcli swiscsi session remove -d

Troubleshoot NFS mounting and permission issues

Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.

 vicfg-nas [<conn_options>]
     [--add --nasserver <server_name> --share <share_name> |
      --delete <share_name> |
      --help |
      --list |
      --nasserver <server_name> |
      --readonly |
      --vihost <esx_host ]

Use esxtop/resxtop and vscsiStats to identify storage performance issues

‘d’ for disk adapter view

‘u’ for disk device view

‘v’ for disk VM view

From Duncan Epping’s ESXTOP section on his blog, the following counters are critical to troubleshooting performance specificially related to storage.

DISK	GAVG (H)	25	Look at “DAVG” and “KAVG” as the sum of both is GAVG.
DISK	DAVG (H)	25	Disk latency most likely to be caused by array.
DISK	KAVG (H)	2	Disk latency caused by the VMkernel, high KAVG usually means queuing. Check “QUED”.
DISK	QUED (F)	1	Queue maxed out. Possibly queue depth set to low. Check with array vendor for optimal queue depth value.
DISK	ABRTS/s (K)	1	Aborts issued by guest(VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason.
DISK	RESETS/s (K)	1	The number of commands reset per second.

Configure and troubleshoot VMFS datastores using vmkfstools

You can do a lot with this command so refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.

vmkfstools <conn_options> <options> <target>

If <target> is a file system, <options> can be one of the following:

--createfs [blocksize]kK|mM --setfsname <fsname> --queryfs --extendfs <span_partition> <head_partition>

If <target is a virtual disk, <options> can be one of the following:

--clonevirtualdisk --createdrm --createdrmpassthru --createvirtualdisk <size>kK|mM|gG --adaptertype <type> --diskformat <format> <location> --deletevirtualdisk --diskformat --extendvirtualdisk --geometry --inflatedisk --querydrm --renamevirtualdisk <oldName> <newName> --writezeros

Troubleshoot snapshot and resignaturing issues

The vicfg-volume command supports resignaturing a snapshot volume and mounting and unmounting the volume. Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.

vicfg-volume [<connection_options>]
    [--help |
     --list |
     --persistent-mount <VMFS-UUID|label> |
     --resignature <VMFS-UUID|label> |
     --umount <VMFS-UUID|label> |
     --vihost <esx_host>]

Check out this blog for a some more information and a discussion on using the vicfg-volume command.

Analyze log files to identify storage and multipathing problems