Tag Archives: vsphere

14 Aug

VCAP-DCA Objective 6.3- Troubleshoot Network Performance and Connectivity

Another  VCAP-DCA objective here.

One thing I want to highlight that was recently blogged about by VMware is VMware’s Resolution Paths they have published online. Going through these you will be able to tackle issues you’ve never encountered before and better plan for possible troubleshooting scenarios.

Another thing is I have created a new tabbed page for VCAP-DCA notes and links found at http://www.seancrookston.com/blog/vcap-dca/

I will add links to other sites or resources for studying for the VCAP as I come along them. The first site I’ve added is Kendrick Coleman’s VCAP exam landing page. This page contains the complete blueprint along with a very nice link to a zip file full of PDF’s relevant to the exam. In total it is 95.1 MB in size and contains 232 documents.

Knowledge

Identify virtual switch entries in a Virtual Machine’s configuration file

Best thing to do here is open up a vmx file and learn what is configured. Below is a trimmed down vmx from my lab with just the network setting showing.

virtualHW.version = “7″

ethernet0.present = “true”
ethernet0.wakeOnPcktRcv = “true”
ethernet0.networkName = “VM Network”
ethernet0.addressType = “vpx”
ethernet0.generatedAddress = “00:50:56:a4:52:92″

ethernet1.present = “true”
ethernet1.virtualDev = “e1000″
ethernet1.networkName = “VM Network”
ethernet1.addressType = “vpx”
ethernet1.generatedAddress = “00:50:56:a4:34:04″
ethernet2.present = “true”
ethernet2.virtualDev = “e1000″
ethernet2.networkName = “VM Network”
ethernet2.addressType = “vpx”
ethernet2.generatedAddress = “00:50:56:a4:74:e9″

ethernet0.startConnected = “true”
ethernet2.startConnected = “false”

ethernet0.pciSlotNumber = “32″
ethernet1.pciSlotNumber = “33″
ethernet2.pciSlotNumber = “35″

ethernet0.virtualDev = “e1000″

ethernet1.startConnected = “false”

Identify virtual switch entries in the ESX/ESXi Host configuration file

load up /etc/vmware/esx.conf and check it out

Identify CLI commands and tools used to troubleshoot vSphere networking configurations

Identify logs used to troubleshoot network issues

Skills and Abilities

Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations

There is not a ton of information out there on using the net-dvs command. One blog that contains some relevant information can be found at http://geeksilver.wordpress.com/2010/05/21/vds-vnetwork-distributed-switch-my-understanding-part-2/

Something I did not know, this command is listed as an unsupported command. It will not run(to my knowledge) from the vMA and I ran it when locally logged into the host. The syntax of the command can be found below

Warning: This is an unsupported command. Use at your own risk.
net-dvs -a [ -P maxPorts] switch_name
net-dvs -d switch_name
net-dvs [ -A | -D ] -p port switch_name
net-dvs [ -s name=value | -u name ] -p port switch_name
net-dvs -l [ switch_name ]
net-dvs -i (init database)
net-dvs [-S | -R | -G ]
net-dvs -T
net-dvs -v “vlanID[;t|p[0-7][;min-max,min-max...]]
net-dvs -V “primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p…”
net-dvs -m “sid;dname;snaplen;[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,…;srcInPort1,srcInport2,…;srcOutPort1,srcOutPort2,…;:sid2;dname2…”
net-dvs dvswitch -k “respool1_id;respool2_id;…”
net-dvs dvswitch -p dvport -K “respool1_id:shares:limit;respool2_id:shares:limit;…”
net-dvs dvswitch -p dvport -z “respool_id”
net-dvs dvswitch -j [activate|deactivate]
net-dvs -L uplink_name1[,uplink_name2,...] -t team_policy_type -p port switch_name
net-dvs dvswitch -H “red|yellow|green:some message” switch_name
net-dvs -o “depth,param|classname;depth,param|classname;… -p port|globalPropList switch_name
net-dvs –mtu mtu_value [-p dvport] switch_name
net-dvs –x 0|1 -p dvport switch_name
net-dvs –vlan vlanID -p dvport switch_name
net-dvs –reset -p dvport switch_name
net-dvs –cap cap_value -p dvport switch_name
net-dvs –states -p dvport switch_name
net-dvs –miscInfo  ;# Dumps cpu/meminfo
net-dvs –vmknicIp <vmknic> ;# Displays IPv4 address on <vmknic>

Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations

Below are the commands I’d consider relevant for troubleshooting in this section. You can use the vSphere Command Line Reference to gain more information on each of these commands and others.

vicfg‑authconfig(4.1 only)
Manages Active Directory authentication.

vicfg‑dns.pl
Specifies an ESX/ESXi host’s DNS (Domain Name Server) configuration.

vicfg‑ipsec
Supports setup of IPSec.

vicfg‑nics
Manages the ESX/ESXi host’s NICs (uplink adapters).

vicfg‑ntp
Specifies the NTP (Network Time Protocol) server.

vicfg‑route
Lists or changes the ESX/ESXi host’s route entry (IP gateway).

vicfg‑snmp
Manages the Simple Network Management Protocol (SNMP) agent.

vicfg‑vmknic
Adds, deletes, and modifies virtual network adapters (VMkernel NICs).

vicfg‑vswitch
Adds or removes virtual switches or vNetwork Distributed Switches, or modifies switch settings.

Configure a network packet analyzer in a vSphere environment

Too much to put in words on this one. Check out the blog below for assistance. I’d reccomend using wireshark as this is what was used in the troubleshooting course offered by VMware.

http://itknowledgeexchange.techtarget.com/it-consultant/packet-sniffing-is-your-best-friend/

http://www.petri.co.il/wireshark-ethereal.htm

Troubleshoot Private VLANs

Great source of PVLAN information at http://professionalvmware.com/2010/04/private-vlan-resources/

Free video(nearly 40 minutes!) detailing PVLAN’s from Eric Sloof at http://www.ntpro.nl/blog/archives/1465-Online-Training-Configure-Private-VLAN-IDs.html

Complete definition of what is a PVLAN from VMware

How to configure PVLAN’s from VMware

Troubleshoot Service Console and vmkernel network configuration issues

Using  VMware’s Resolution Paths a good starting point is the KB for troubleshooting service console issues.

Troubleshoot DNS and routing related issues

This VMware KB is probably a good start for troubleshooting DNS/routing.

To change/check default gateway settings

etc/opt/vmware/vpxa/vpxa.cfg

To change/update dns

/etc/resolv.con

Use esxtop/resxtop to identify network performance problems

Run esxtop and hit ‘n’ to enter the networking view

Again the best resource I’ve found so far on troubleshooting using esxtop as a whole is Duncan Epping’s Blog and I’ve included the two counters for networking in the table below.

Two key performance counters you will need to know when troubleshooting network issues are below for both received and transmitted dropped packets. This goes without saying, but you are looking for no dropped packets here. The default view for networking will also show current and peak transmission stats to assist in your troubleshooting.

NETWORK %DRPTX 1 Dropped packages transmitted, hardware overworked. Possible cause: very high network utilization
NETWORK %DRPRX 1 Dropped packages received, hardware overworked. Possible cause: very high network utilization

Use CDP and/or network hints to identify connectivity issues

Cisco Discovery Protocol CDP Information via the ESX Command Line and Virtual Center (note replace vmware-vim-cmd with vim-cmd)

The command below will query and show network hints

vim-cmd hostsvc/net/query_networkhint

Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment

Tools

ESX Configuration Guide
ESXi Configuration Guide
vSphere Command-Line Interface Installation and Scripting Guide
Product Documentation
vSphere Client
vSphere CLI
vicfg-*
net-dvs
resxtop/esxtop

Other relevant blogs and websites related to this section

http://www.virtualinsanity.com/index.php/2010/03/29/performance-troubleshooting-vmware-vsphere-network/

http://vmware.com/files/pdf/VMware_NFS_BestPractices_WP_EN.pdf

http://blogs.vmware.com/files/network-1.htm

http://www.tuxyturvy.com/blog/index.php?/archives/37-Troubleshooting-VMware-ESX-network-performance.html

vSphere Command Line Reference

http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/

http://www.vmware.com/pdf/vsphere4/r41/vsp4_41_vcli_inst_script.pdf

08 Aug

VCAP-DCA Objective 8.3 ; Administer vSphere Using the vSphere Management Assistant

Another section of the VCAP covered here briefly. It’s no secret as noted here that the service console is on its way out, so knowledge of command-line based administration and the vMA has become even more important.

The vMA is a CentOS based virtual machine packaged with the vSphere CLI and vSphere SDK for Perl. It allows the ability to run scripts against ESX/ESXi as well as vCenter and now includes active directory authentication with the 4.1 release. The vMA can also be used as a syslog server, which is a necessity when using ESXi as the logs are not retained after a reboot.

Before reading through any of the blueprint outline below I would recommend giving the vMA 4.1 guide a quick read. It details all the steps to setup a working vMA appliance.

You can download the appliance here

You should also check out these two blog posts

  1. Blog on setting up for AD authentication
  2. Blog on setting up your vMA appliance as a log host (not necessarily relevant to this section of the blueprint but important nonetheless)

Knowledge

• Identify vMA prerequisites

• Identify vMA specific commands

• Determine when vMA is needed

Skills and Abilities

• Install and Configure vMA

Pretty straight forward, import the .OVF and follow the prompts.

Read the blogs mentioned above for setting up AD authentication, which can also be found in the admin guide. AD authentication is new to 4.1 so it is not currently on the blueprint but I imagine it will be added soon.

• Add/Remove target servers

Sudo vifp addserver <FQDN ESX host>

Sudo vifp remove <FQDN ESX host>

Sudo vifp reconfigure <FQDN ESX host>

• Perform updates to the vMA

sudo vma-update info

sudo vma-update scan

sudo vma-update

Use vmkfstools to manage VMFS datastores

From the vSphere Command-Line Interface Reference:

You use the vmkfstools vSphere CLI to create and manipulate virtual disks, file systems, logical volumes, and physical storage devices on an ESX/ESXi host. You can use vmkfstools to create and manage a virtual machine file system (VMFS) on a physical partition of a disk and to manipulate files, such as virtual disks, stored on VMFS-3 and NFS. You can also use vmkfstools to set up and manage raw device mappings (RDMs).

You will want to use this command extensively and go through all of its options.

 vmkfstools <conn_options> <options> <target>

If <target> is a file system, <options> can be one of the following:

 --createfs [blocksize]kK|mM --setfsname <fsname>
 --queryfs
 --extendfs <span_partition> <head_partition>

If <target is a virtual disk, <options> can be one of the following:

 --clonevirtualdisk
 --createdrm
 --createdrmpassthru
 --createvirtualdisk
     <size>kK|mM|gG
     --adaptertype <type>
     --diskformat <format> <location>
 --deletevirtualdisk
 --diskformat
 --extendvirtualdisk
 --geometry
 --inflatedisk
 --querydrm
 --renamevirtualdisk <oldName> <newName>
 --writezeros

vmkfstools –C vmfs3 –b 2m vmhba1:3:0:1

• Use vmware-cmd to manage VMs

From the vSphere Command-Line Interface Reference:

vmware-cmd provides an interface to perform operations on a virtual machine. You can retrieve information such as the power state, register and unregister the virtual machine, set configuration variables, and manage snapshots.

Again you will need to go through this command and the options extensively. A good example that every VMware admin has encountered at some point, resolving a stuck virtual machine, can be found here.

General Options

  vmware-cmd [--help | -q | -v]

Server Operations

 vmware-cmd -s <conn_option>
   [-l |
    -s register <config_file_path> [<datacenter>] [<resource_pool>] |
    -s unregister <config_file_path> ]

Virtual Machine Operations

 vmware-cmd <conn_option> <config_file_path>
    [answer |
     connectdevice <device_name> |
     createsnapshot <name> <description> quiesce [0|1] memory [0|1] |
     disconnectdevice <device_name> |
     getconfigfile |
     getguestinfo <variable> |
     getproductinfo [product|platform|build|majorversion|minorversion] |
     getstate |
     gettoolslastactive |
     getuptime |
     hassnapshot |
     reset [soft|hard] |
     removesnapshots
     revertsnapshot |
     setguestinfo <variable> <value> |
     start |
     suspend [soft|hard] ]

• Use esxcli to manage Storage Multipathing

Use the vicfg-mpath and vicfg-mpath35(for esx 3.5 hosts) to configure and manage storage multipathing.

This blog entry shows off a little powershell for setting a preferred path. Most of the reference for storage multipathing will be from the below. Additionally the vSphere Troubleshooting course is a good target for this type of exercise.

From the vSphere Command-Line Interface Reference:

The vicfg-mpath command supports listing information about Fibre Channel or iSCSI LUNs and changing a path’s state. Use vicfg-mpath35 when running against ESX/ESXi 3.5 systems. Use the esxcli command for managing pluggable storage architecture (PSA) and native multipathing (NMP), including path policy modification.

vicfg-mpath [<conn_options>]
   [--help |
    --list [--path <path> |--device <device>] |
    --list-compact [--path <path> |--device <device>] |
    --list-map [--path <path> |--device <device>] |
    --list-paths [--device <device>] |
    --list-plugins |
    --state  [active|off] ]

• Troubleshoot common vMA errors and conditions

Refer to the Troubleshooting vMA section of the vMA Administration Guide

By far the best resource for vMA information can be found at the blog virtuallyGhetto

Tools

• vSphere Management Assistant Guide

• vSphere Command‐Line Interface Installation and Scripting Guide

• Product Documentation

• vSphere Management Appliance

• vifp

• vima-update

• vSphere CLI o vicfg-* o vmkfstools o esxcli o vmware-cmd

• vSphere Client

Other relevant blogs and websites related to this section

vMA Setup

http://communities.vmware.com/community/vmtn/vsphere/automationtools/vima

http://www.vmware.com/support/developer/vima/vma41/doc/vma_41_guide.pdf

http://geeksilver.wordpress.com/2010/07/22/how-to-use-vma-4-1-installation-configuration/

http://www.simonlong.co.uk/blog/2010/05/28/using-vma-as-your-esxi-syslog-server/

http://www.virtuallyghetto.com/2010/05/getting-started-with-vma.html

vmkfstools

http://pubs.vmware.com/vi35/wwhelp/wwhimpl/common/html/wwhelp.htm?context=server_config&file=sc_appb.21.1.html

http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/vmkfstools.html

vmware-cmd

http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/vmware-cmd.html

http://www.vmware.com/support/esx2/doc/vmware-cmd.html

http://www.yellow-bricks.com/2009/04/15/the-basics-how-to-kill-a-vm-thats-stuck-during-shutdown/

vicfg-mpath

http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/vicfg-mpath.html

http://virtuallynil.com/2010/02/12/setting-preferred-paths-in-esxi/

01 Aug

VCAP-DCA Objective 6.2 – Troubleshoot CPU and Memory Performance

This is the first pass at studying Objective 6.2 – Troubleshoot CPU and Memory Performance  . I have hit most of the areas outlined in this section of the objective and this section as the others will be a work in progress over time with more detailed information to come as I progress in my studying.

Knowledge

• Identify resxtop/esxtop metrics related to memory and CPU

Esxtop does not exist in vMA, you must use resxtop

One limitation of resxtop is the lack of replay mode.

• Identify vCenter Server Performance Chart metrics related to memory and CPU

This can be checked at the cluster or server level and you will have more granular options to check at the server level.

Memory

You must first have a firm understanding of terminology when it comes to memory in the VMware world. This blog from Scott Sauer is a must read if you are not familiar with the below terms.

  • Transparent Page Sharing
  • Memory Overcommitment
  • Memory Overhead
  • Memory Balloon Driver
    With the ability to overcommit memory you will want to make sure excessive swapping is not occurring at both the host and virtual machine level.

Use memory reservations cautiously. Memory that is reserved cannot be used by another virtual machine that may need it.

Memory ballooning relies on drivers installed in the guest with VMware tools. No VMware tools means potential performance impacts to your server.

CPU

There are many counters that can be added and checked including core usage and reservations. Again reservations should be used cautiously.

Watch out for virtual machines that are consistently using a large percentage of cpu resources. A typical server is idle most of the time so check and see if something out of the ordinary is occurring. If in fact the server is using these resources then allocate another vCPU.

High CPU ready times are a dead giveaway for other issues that may be going on.

Virtual machines that have multiple CPU’s installed but the incorrect HAL will not help the virtual guest out.

Skills and Abilities

• Troubleshoot ESX/ESXi Host and Virtual Machine CPU performance issues using appropriate metrics

From the blog of Duncan Epping, these are four commonly needed values to look at when taking into account CPU performance issues. His blog entry is one that is updated over time based on the community so read the comments there and check if any of these thresholds are changed over time. Ultimately performance is relative to the environment so some of this may not always apply.

CPU %RDY 10 Overprovisioning of vCPUs, excessive usage of vSMP or a limit(check %MLMTD) has been set. See Jason’s explanation for vSMP VMs
CPU %CSTP 3 Excessive usage of vSMP. Decrease amount of vCPUs for this particular VM. This should lead to increased scheduling opportunities.
CPU %MLMTD 0 If larger than 0 the world is being throttled. Possible cause: Limit on CPU.
CPU %SWPWT 5 VM waiting on swapped pages to be read from disk. Possible cause: Memory overcommitment.

If you want to move historical data over to a gui based format you can use esxplot or Windows’ Perfmon to interpret the data. To gather this data you would use batch mode as shown below.

esxtop -b -d delay in seconds -n iterations > capturefile.csv
 

• Troubleshoot ESX/ESXi Host and Virtual Machine memory performance issues using appropriate metrics

Again from the blog of Duncan Epping, five commonly needed values to look at when troubleshooting memory performance. Same applies as above.

MEM MCTLSZ (I) 1 If larger than 0 host is forcing VMs to inflate balloon driver to reclaim memory as host is overcommited.
MEM SWCUR (J) 1 If larger than 0 host has swapped memory pages in the past. Possible cause: Overcommitment.
MEM SWR/s (J) 1 If larger than 0 host is actively reading from swap(vswp). Possible cause: Excessive memory overcommitment.
MEM SWW/s (J) 1 If larger than 0 host is actively writing to swap(vswp). Possible cause: Excessive memory overcommitment.
MEM N%L (F) 80 If less than 80 VM experiences poor NUMA locality. If a VM has a memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM and “remotely” uses memory via “interconnect”.

• Use Hot‐Add functionality to resolve identified Virtual Machine CPU and memory performance issues

A couple of good blogs by David Davis and Jason Boche outline what and how to use Hot-Add/Hot-Plug. The ability to use this without having to reboot the guest virtual machine is extremely limited. ON the Microsoft side Windows 2008 Server Datacenter is neccessary to support both features without a reboot while Windows 2008 Server Enteprise edition does not require a reboot for Hot Adding memory. When it comes to removing either hot added memory or hot plugged cpu’s a reboot is required for all Windows guest operation systems.

Tools

• vSphere Resource Management Guide • vSphere Command‐Line Interface Installation and Scripting Guide • Product Documentation • vSphere Client • vSphere CLI o resxtop/esxtop

Other relevant blogs and websites related to this section

http://communities.vmware.com/docs/DOC-10352 http://communities.vmware.com/docs/DOC-11812 http://www.boche.net/blog/index.php/2009/01/28/esxtop-drilldown/ http://www.vreference.com/public/vReference-esxtop1.2.pdf http://labs.vmware.com/flings/esxplot http://www.simonlong.co.uk/blog/2010/03/24/using-esxtop-with-vmware-esxi/

http://www.yellow-bricks.com/esxtop/

http://pubs.vmware.com/vsp40u1_i/resmgmt/c_using_the_esxtop_utility.html#1_7_6_22_1

http://www.boche.net/blog/index.php/2009/05/10/vsphere-memory-hot-add-cpu-hot-plug/

http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1367631_mem1,00.html

http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-articles/general/understanding-and-customizing-vmware-esx-server-performance-charts.html

http://www.virtualinsanity.com/index.php/2010/02/19/performance-troubleshooting-vmware-vsphere-memory/

http://www.virtualinsanity.com/index.php/2010/02/15/performance-troubleshooting-vmware-vsphere-cpu/