Sometimes it happens to troubleshoot an ESXi host for network problems.
Over time I created a small guide to help me remember the various commands, I share it hoping it will be useful to everyone 🙂
esxcli network (here the complete list)
Check the status of firewall
esxcli network firewall get Default Action: DROP Enabled: true Loaded: true
Enabling and disabling firewall
esxcli network firewall set --enabled false (firewall disabled) esxcli network firewall set --enabled true (firewall enabled)
TCP/UDP connection status
esxcli network ip connection list Proto Recv Q Send Q Local Address Foreign Address State World ID CC Algo World Name ----- ------ ------ ------------------------------- --------------------- ----------- -------- ------- ---------- tcp 0 0 127.0.0.1:80 127.0.0.1:28796 ESTABLISHED 2099101 newreno envoy tcp 0 0 127.0.0.1:28796 127.0.0.1:80 ESTABLISHED 28065523 newreno python tcp 0 0 127.0.0.1:26078 127.0.0.1:80 TIME_WAIT 0 tcp 0 0 127.0.0.1:8089 127.0.0.1:60840 ESTABLISHED 2099373 newreno vpxa-IO <line drop>
Configured DNS servers and search domain
esxcli network ip dns server list DNSServers: 10.0.0.8, 10.0.0.4 esxcli network ip dns search list DNSSearch Domains: scanda.local
List of vmkernel interfaces
esxcli network ip interface ipv4 get Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS ---- -------------- ------------- -------------- ------------ ------------ -------- vmk0 172.16.120.140 255.255.255.0 172.16.120.255 STATIC 172.16.120.1 false vmk1 172.16.215.11 255.255.255.0 172.16.215.255 STATIC 172.16.215.1 false
Netstacks configured on host (used on vmkernel interfaces)
esxcli network ip netstack list defaultTcpipStack Key: defaultTcpipStack Name: defaultTcpipStack State: 4660 vmotion Key: vmotion Name: vmotion State: 4660
List of physical network adapters
esxcli network nic list Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description ------ ------------ ------- ------------ ----------- ----- ------ ----------------- ---- ----------- vmnic0 0000:04:00.0 ntg3 Up Down 0 Half ec:2a:72:a6:bf:34 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet vmnic1 0000:04:00.1 ntg3 Up Down 0 Half ec:2a:72:a6:bf:35 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet vmnic2 0000:51:00.0 bnxtnet Up Up 25000 Full 00:62:0b:a0:b2:c0 1500 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter vmnic3 0000:51:00.1 bnxtnet Up Up 25000 Full 00:62:0b:a0:b2:c1 1500 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter vmnic4 0000:51:00.2 bnxtnet Up Up 25000 Full 00:62:0b:a0:b2:c2 1500 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter vmnic5 0000:51:00.3 bnxtnet Up Up 25000 Full 00:62:0b:a0:b2:c3 1500 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter
vmkping (KB reference)
command to send ICMP packets through vmkernel interfaces, very useful for checking MTU 🙂
usage examples
ping an host vmkping -I vmk0 192.168.0.1 check MTU and fragmentation vmkping -I vmk0 -d -s 8972 172.16.100.1 ping an host using vmotion netstack vmkping -I vmk2 -S vmotion 172.16.115.12
iperf ( good article here)
Very useful tool to check the actual usable bandwidth between 2 hosts, one host uses server mode and one uses client mode
the tool is located at this path
/usr/lib/vmware/vsan/bin/iperf3
NOTE: in vSphere 8 you may get ” Operation not permitted” error at runtime, you can enable the execution with the command
esxcli system secpolicy domain set -n appDom -l disabled then enforcing with esxcli system secpolicy domain set -n appDom -l enforcing
it is also necessary to disable the firewall to perform the tests
esxcli network firewall set --enabled false
usage example:
host server mode, the -B option allows a specific address and interface to be used for testing
/usr/lib/vmware/vsan/bin/iperf3 -s -B 172.16.100.2
client mode host, the -n option specifies the amount of data to be transferred for testing
/usr/lib/vmware/vsan/bin/iperf3 -n 10G -c 172.16.100.2
25G interface test result
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-4.04 sec 10.0 GBytes 21.3 Gbits/sec 0 sender [ 5] 0.00-4.04 sec 10.0 GBytes 21.3 Gbits/sec receiver
NOTE : at the end of the test remember to re-enable the firewall and enforcing 🙂
nslookup e cache DNSÂ (KB reference)
Sometimes it is necessary to verify that DNS name resolution is working properly on a host.
Use the nslookup command followed by the name to resolve
nslookup www.scanda.it
It may happen that changes to DNS records are not immediately received by esxi hosts, this is due to the DNS query caching mechanism.
To clear the DNS cache, use the following command (KB reference)
/etc/init.d/nscd restart
TCP/UDP connectivity test
On the esxi hosts, netcat (nc) tool is present to verify TCP/UDP connectivity to another host.
nc usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port] [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version] [-x proxy_address[:port]] [hostname] [port[s]]
If you need to verify access to an HTTPS service and the validity of its SSL certificate, you can use the command
openssl s_client -connect www.dominio.it:443
pktcap-uw (KB reference)
another very useful tool is pktcap-uw, which allows you to capture network traffic in full tcpdump style. The tool differs from tcpdump-uw in that it can capture traffic not only from vmkernel interfaces, but also from physical interfaces, switchports, and virtual machines.
let’s look at a few examples
capturing traffic from the vmkernel vmk0
pktcap-uw --vmk vmk0
traffic capture from physical uplink vmnic3
pktcap-uw --uplink vmnic3
Capturing traffic from a virtual switch port
pktcap-uw --switchport <switchportnumber>
NOTE: To get the port number mapping and virtual nic of a VM use the command net-stats -l
It is also possible to retrieve information from the LLDP protocol from uplinks used by a VSS ( do not support LLDP) with the following command
pktcap-uw --uplink vmnic1 --ethtype 0x88cc -c 1 -o /tmp/lldp.pcap > /dev/null && hexdump -C /tmp/lldp.pcapThe output will be in hexadecimal format and may be useful for performing port mapping of a host even on a Virtual Standard Switch.
I will not fail to update the list with other useful commands.
