Sean Crookston

VCAP-DCA Objective 6.3- Troubleshoot Network Performance and Connectivity

August 14, 2010

Sean Crookston

1 Comment

Another VCAP-DCA objective here.

One thing I want to highlight that was recently blogged about by VMware is VMware’s Resolution Paths they have published online. Going through these you will be able to tackle issues you’ve never encountered before and better plan for possible troubleshooting scenarios.

Another thing is I have created a new tabbed page for VCAP-DCA notes and links found at http://www.seancrookston.com/blog/vcap-dca/

I will add links to other sites or resources for studying for the VCAP as I come along them. The first site I’ve added is Kendrick Coleman’s VCAP exam landing page. This page contains the complete blueprint along with a very nice link to a zip file full of PDF’s relevant to the exam. In total it is 95.1 MB in size and contains 232 documents.

Knowledge

Identify virtual switch entries in a Virtual Machine’s configuration file

Best thing to do here is open up a vmx file and learn what is configured. Below is a trimmed down vmx from my lab with just the network setting showing.

virtualHW.version = “7″

ethernet0.present = “true”
ethernet0.wakeOnPcktRcv = “true”
ethernet0.networkName = “VM Network”
ethernet0.addressType = “vpx”
ethernet0.generatedAddress = “00:50:56:a4:52:92″

ethernet1.present = “true”
ethernet1.virtualDev = “e1000″
ethernet1.networkName = “VM Network”
ethernet1.addressType = “vpx”
ethernet1.generatedAddress = “00:50:56:a4:34:04″
ethernet2.present = “true”
ethernet2.virtualDev = “e1000″
ethernet2.networkName = “VM Network”
ethernet2.addressType = “vpx”
ethernet2.generatedAddress = “00:50:56:a4:74:e9″

ethernet0.startConnected = “true”
ethernet2.startConnected = “false”

ethernet0.pciSlotNumber = “32″
ethernet1.pciSlotNumber = “33″
ethernet2.pciSlotNumber = “35″

ethernet0.virtualDev = “e1000″

ethernet1.startConnected = “false”

Identify virtual switch entries in the ESX/ESXi Host configuration file

load up /etc/vmware/esx.conf and check it out

Identify CLI commands and tools used to troubleshoot vSphere networking configurations

Identify logs used to troubleshoot network issues

Skills and Abilities

Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations

There is not a ton of information out there on using the net-dvs command. One blog that contains some relevant information can be found at http://geeksilver.wordpress.com/2010/05/21/vds-vnetwork-distributed-switch-my-understanding-part-2/

Something I did not know, this command is listed as an unsupported command. It will not run(to my knowledge) from the vMA and I ran it when locally logged into the host. The syntax of the command can be found below

Warning: This is an unsupported command. Use at your own risk.
net-dvs -a [ -P maxPorts] switch_name
net-dvs -d switch_name
net-dvs [ -A | -D ] -p port switch_name
net-dvs [ -s name=value | -u name ] -p port switch_name
net-dvs -l [ switch_name ]
net-dvs -i (init database)
net-dvs [-S | -R | -G ]
net-dvs -T
net-dvs -v “vlanID[;t|p[0-7][;min-max,min-max...]]
net-dvs -V “primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p…”
net-dvs -m “sid;dname;snaplen;[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,…;srcInPort1,srcInport2,…;srcOutPort1,srcOutPort2,…;:sid2;dname2…”
net-dvs dvswitch -k “respool1_id;respool2_id;…”
net-dvs dvswitch -p dvport -K “respool1_id:shares:limit;respool2_id:shares:limit;…”
net-dvs dvswitch -p dvport -z “respool_id”
net-dvs dvswitch -j [activate|deactivate]
net-dvs -L uplink_name1[,uplink_name2,...] -t team_policy_type -p port switch_name
net-dvs dvswitch -H “red|yellow|green:some message” switch_name
net-dvs -o “depth,param|classname;depth,param|classname;… -p port|globalPropList switch_name
net-dvs –mtu mtu_value [-p dvport] switch_name
net-dvs –x 0|1 -p dvport switch_name
net-dvs –vlan vlanID -p dvport switch_name
net-dvs –reset -p dvport switch_name
net-dvs –cap cap_value -p dvport switch_name
net-dvs –states -p dvport switch_name
net-dvs –miscInfo ;# Dumps cpu/meminfo
net-dvs –vmknicIp <vmknic> ;# Displays IPv4 address on <vmknic>

Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations

Below are the commands I’d consider relevant for troubleshooting in this section. You can use the vSphere Command Line Reference to gain more information on each of these commands and others.

vicfg‑authconfig(4.1 only)
Manages Active Directory authentication.

vicfg‑dns.pl
Specifies an ESX/ESXi host’s DNS (Domain Name Server) configuration.

vicfg‑ipsec
Supports setup of IPSec.

vicfg‑nics
Manages the ESX/ESXi host’s NICs (uplink adapters).

vicfg‑ntp
Specifies the NTP (Network Time Protocol) server.

vicfg‑route
Lists or changes the ESX/ESXi host’s route entry (IP gateway).

vicfg‑snmp
Manages the Simple Network Management Protocol (SNMP) agent.

vicfg‑vmknic
Adds, deletes, and modifies virtual network adapters (VMkernel NICs).

vicfg‑vswitch
Adds or removes virtual switches or vNetwork Distributed Switches, or modifies switch settings.

Configure a network packet analyzer in a vSphere environment

Too much to put in words on this one. Check out the blog below for assistance. I’d reccomend using wireshark as this is what was used in the troubleshooting course offered by VMware.

http://itknowledgeexchange.techtarget.com/it-consultant/packet-sniffing-is-your-best-friend/

http://www.petri.co.il/wireshark-ethereal.htm

Troubleshoot Private VLANs

Great source of PVLAN information at http://professionalvmware.com/2010/04/private-vlan-resources/

Free video(nearly 40 minutes!) detailing PVLAN’s from Eric Sloof at http://www.ntpro.nl/blog/archives/1465-Online-Training-Configure-Private-VLAN-IDs.html

Complete definition of what is a PVLAN from VMware

How to configure PVLAN’s from VMware

Troubleshoot Service Console and vmkernel network configuration issues

Using VMware’s Resolution Paths a good starting point is the KB for troubleshooting service console issues.

Troubleshoot DNS and routing related issues

This VMware KB is probably a good start for troubleshooting DNS/routing.

To change/check default gateway settings

etc/opt/vmware/vpxa/vpxa.cfg

To change/update dns

/etc/resolv.con

Use esxtop/resxtop to identify network performance problems

Run esxtop and hit ‘n’ to enter the networking view

Again the best resource I’ve found so far on troubleshooting using esxtop as a whole is Duncan Epping’s Blog and I’ve included the two counters for networking in the table below.

Two key performance counters you will need to know when troubleshooting network issues are below for both received and transmitted dropped packets. This goes without saying, but you are looking for no dropped packets here. The default view for networking will also show current and peak transmission stats to assist in your troubleshooting.

NETWORK	%DRPTX	1	Dropped packages transmitted, hardware overworked. Possible cause: very high network utilization
NETWORK	%DRPRX	1	Dropped packages received, hardware overworked. Possible cause: very high network utilization

Use CDP and/or network hints to identify connectivity issues

Cisco Discovery Protocol CDP Information via the ESX Command Line and Virtual Center (note replace vmware-vim-cmd with vim-cmd)

The command below will query and show network hints

vim-cmd hostsvc/net/query_networkhint

Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment

Tools

ESX Configuration Guide
ESXi Configuration Guide
vSphere Command-Line Interface Installation and Scripting Guide
Product Documentation
vSphere Client
vSphere CLI
vicfg-*
net-dvs
resxtop/esxtop