VCAP-DCA Objective 6.3- Troubleshoot Network Performance and Connectivity
Another VCAP-DCA objective here.
One thing I want to highlight that was recently blogged about by VMware is VMware’s Resolution Paths they have published online. Going through these you will be able to tackle issues you’ve never encountered before and better plan for possible troubleshooting scenarios.
Another thing is I have created a new tabbed page for VCAP-DCA notes and links found at http://www.seancrookston.com/blog/vcap-dca/
I will add links to other sites or resources for studying for the VCAP as I come along them. The first site I’ve added is Kendrick Coleman’s VCAP exam landing page. This page contains the complete blueprint along with a very nice link to a zip file full of PDF’s relevant to the exam. In total it is 95.1 MB in size and contains 232 documents.
Identify virtual switch entries in a Virtual Machine’s configuration file
Best thing to do here is open up a vmx file and learn what is configured. Below is a trimmed down vmx from my lab with just the network setting showing.
virtualHW.version = “7″
ethernet0.present = “true”
ethernet0.wakeOnPcktRcv = “true”
ethernet0.networkName = “VM Network”
ethernet0.addressType = “vpx”
ethernet0.generatedAddress = “00:50:56:a4:52:92″
ethernet1.present = “true”
ethernet1.virtualDev = “e1000″
ethernet1.networkName = “VM Network”
ethernet1.addressType = “vpx”
ethernet1.generatedAddress = “00:50:56:a4:34:04″
ethernet2.present = “true”
ethernet2.virtualDev = “e1000″
ethernet2.networkName = “VM Network”
ethernet2.addressType = “vpx”
ethernet2.generatedAddress = “00:50:56:a4:74:e9″
ethernet0.startConnected = “true”
ethernet2.startConnected = “false”
ethernet0.pciSlotNumber = “32″
ethernet1.pciSlotNumber = “33″
ethernet2.pciSlotNumber = “35″
ethernet0.virtualDev = “e1000″
ethernet1.startConnected = “false”
Identify virtual switch entries in the ESX/ESXi Host configuration file
load up /etc/vmware/esx.conf and check it out
Identify CLI commands and tools used to troubleshoot vSphere networking configurations
Identify logs used to troubleshoot network issues
Skills and Abilities
Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
There is not a ton of information out there on using the net-dvs command. One blog that contains some relevant information can be found at http://geeksilver.wordpress.com/2010/05/21/vds-vnetwork-distributed-switch-my-understanding-part-2/
Something I did not know, this command is listed as an unsupported command. It will not run(to my knowledge) from the vMA and I ran it when locally logged into the host. The syntax of the command can be found below
Warning: This is an unsupported command. Use at your own risk.
net-dvs -a [ -P maxPorts] switch_name
net-dvs -d switch_name
net-dvs [ -A | -D ] -p port switch_name
net-dvs [ -s name=value | -u name ] -p port switch_name
net-dvs -l [ switch_name ]
net-dvs -i (init database)
net-dvs [-S | -R | -G ]
net-dvs -v “vlanID[;t|p[0-7][;min-max,min-max...]]
net-dvs -V “primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p…”
net-dvs -m “sid;dname;snaplen;[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,…;srcInPort1,srcInport2,…;srcOutPort1,srcOutPort2,…;:sid2;dname2…”
net-dvs dvswitch -k “respool1_id;respool2_id;…”
net-dvs dvswitch -p dvport -K “respool1_id:shares:limit;respool2_id:shares:limit;…”
net-dvs dvswitch -p dvport -z “respool_id”
net-dvs dvswitch -j [activate|deactivate]
net-dvs -L uplink_name1[,uplink_name2,...] -t team_policy_type -p port switch_name
net-dvs dvswitch -H “red|yellow|green:some message” switch_name
net-dvs -o “depth,param|classname;depth,param|classname;… -p port|globalPropList switch_name
net-dvs –mtu mtu_value [-p dvport] switch_name
net-dvs –x 0|1 -p dvport switch_name
net-dvs –vlan vlanID -p dvport switch_name
net-dvs –reset -p dvport switch_name
net-dvs –cap cap_value -p dvport switch_name
net-dvs –states -p dvport switch_name
net-dvs –miscInfo ;# Dumps cpu/meminfo
net-dvs –vmknicIp <vmknic> ;# Displays IPv4 address on <vmknic>
Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations
Below are the commands I’d consider relevant for troubleshooting in this section. You can use the vSphere Command Line Reference to gain more information on each of these commands and others.
Manages Active Directory authentication.
Specifies an ESX/ESXi host’s DNS (Domain Name Server) configuration.
Supports setup of IPSec.
Manages the ESX/ESXi host’s NICs (uplink adapters).
Specifies the NTP (Network Time Protocol) server.
Lists or changes the ESX/ESXi host’s route entry (IP gateway).
Manages the Simple Network Management Protocol (SNMP) agent.
Adds, deletes, and modifies virtual network adapters (VMkernel NICs).
Adds or removes virtual switches or vNetwork Distributed Switches, or modifies switch settings.
Configure a network packet analyzer in a vSphere environment
Too much to put in words on this one. Check out the blog below for assistance. I’d reccomend using wireshark as this is what was used in the troubleshooting course offered by VMware.
Troubleshoot Private VLANs
Great source of PVLAN information at http://professionalvmware.com/2010/04/private-vlan-resources/
Free video(nearly 40 minutes!) detailing PVLAN’s from Eric Sloof at http://www.ntpro.nl/blog/archives/1465-Online-Training-Configure-Private-VLAN-IDs.html
Complete definition of what is a PVLAN from VMware
How to configure PVLAN’s from VMware
Troubleshoot Service Console and vmkernel network configuration issues
Troubleshoot DNS and routing related issues
This VMware KB is probably a good start for troubleshooting DNS/routing.
To change/check default gateway settings
To change/update dns
Use esxtop/resxtop to identify network performance problems
Run esxtop and hit ‘n’ to enter the networking view
Again the best resource I’ve found so far on troubleshooting using esxtop as a whole is Duncan Epping’s Blog and I’ve included the two counters for networking in the table below.
Two key performance counters you will need to know when troubleshooting network issues are below for both received and transmitted dropped packets. This goes without saying, but you are looking for no dropped packets here. The default view for networking will also show current and peak transmission stats to assist in your troubleshooting.
|NETWORK||%DRPTX||1||Dropped packages transmitted, hardware overworked. Possible cause: very high network utilization|
|NETWORK||%DRPRX||1||Dropped packages received, hardware overworked. Possible cause: very high network utilization|
Use CDP and/or network hints to identify connectivity issues
Cisco Discovery Protocol CDP Information via the ESX Command Line and Virtual Center (note replace vmware-vim-cmd with vim-cmd)
The command below will query and show network hints
Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment
ESX Configuration Guide
ESXi Configuration Guide
vSphere Command-Line Interface Installation and Scripting Guide
Other relevant blogs and websites related to this section
vSphere Command Line Reference