VMware vSphere Network Design Basics

1. VICs will have vNICs. These vNICs are presented to ESXi and on ESXi these are called VMNICs. These VMNICs are presented to the vswitch as the uplink ports. The downlink ports of the vSwitch will grouped into different port groups. These port groups are presented to the Virtual Machines. The Virtual Machines will have vNICs which will connect to the portgroups and the traffic goes out via the VMNICs to the physical world.

2. When you have standard vswitch and if you are doing vMotion from one ESXi to another ESXi then the portgroups name should match (case sensitive) on those ESXi. This is a tedious task and this is where the we Distributed Virtual Switch will be useful. DVS supports NIOC.

3. The DVS is spread across the Datacenter, meaning all the ESXi hosts inside all the clusters inside that datacenter will see this DVS. so when you create a port group on the DVS the port group is seen by all ESXi.The portgroup in DVS is called Distributed Portgroup.

4. When you create the DVS, the unused VMNICs of the ESXi hosts are connected to the DVS.

5. You can create multiple DVS for specific function and you can control the access to these DVS individually. When you create multiple DVS you assign different unused VMNICs to different DVS so that the traffic is kept separate. It is recommended to have different DVS for different type of traffic. For Mgmt you create a DVS, for IP Storage you create a different DVS. VMkernel NIC/VMkernel port/VMkernel port group are needed for IP storage.

6. The VMNICs can be configured for static Etherchannel on standard vSwitch. The VMNICs can be configured for static or LACP negotiated etherchannel when you use DVS. But be aware that when you have Cisco VIC card presenting different vNICs as VMNICs to ESXi and those vNICs are connected to different FI, then you cannot have LAG constituting those VMNICs going to different FI. Because FIs do not support vPC.

7. VMware NSX does not support the “Route based Physical NIC Load” type of Load Balancing.

8. For iSCSI initiator, it should be VMKernel port group. Also only one VMNIC should be used and all others VMNIC should be unused. If this is not met then iSCSI port binding will not be met.

9. In vSphere 6 there are 3 different TCPIP stack, default TCPIP stack for regular VM Network Traffic, another TCPIP stack for VMotion because it requires higher bandwidth so it can have different gateway, another TCPIP stack for Provisioning (Cloning of VMs).
To add new stack, on the CLI use “esxcli network ip netstack add -N <name of stack>. The custom created stack can be chosen when creating a new VMKernel adapter.

10. If you have only two 10G adapter on the physical server then Bandwidth can be controlled by Network IO Control (NIOC).

11. For iSCSI you create the datastore as VMFS but for NFS you create it as NFS type datastore. When there is lot of paths available to the storage, it is advised to create NFS based Storage. This is the reason HyperConverged vendor like Nutanix are using the NFS based datastore. vSphere 6 can support upto 4 ISCSI VMkernel port and you can have have upto 4 different paths.

12. vSphere6 supports NFS4.1. NFS4.1 has the support for Kerberos Authentication. Also NFS4.1 supports Multipathing IO.NFS3 does not support Multipathing IO.

13. VSAN is supported both on Standard and DVS. On DVS there is much more capability. For VSAN to work properly, there underlying physical network should have low latency, BW guarantee and Resiliency. VSAN uses VMKernel ports. There need to be a VSAN Network need to created and the VMKernel ports talk to the VSAN Network.

For stretched VSAN cluster (support in VSAN 6.1) the underying Network Infra should be more reliable and high speed (Metro Ethernet). Do not do strectched cluster over WAN. Earlier versions of VSAN used to work with hybrid mode (Disks and PCIe Flash). With VSAN 6.1 there is a support for DIMM Flash.

14. VSAN 6.1 support clustering like Oracle RAC, Microsoft Clusters, DIMM Flash and vSphere Replication (Replication of data to a Replication Site).

15. VSAN uses multicasting for the heartbeat and to identify the VSAN nodes that constitutes the VSAN cluster(other hosts that has VMKernel ports that is connected to the VSAN Network). Ensure the Physical Network support/configured for IGMP Snooping.

16. Both DVS and Standard Switch supports Basic Multicast filtering but only DVS supports IGMP Snooping and MLD Snooping. Basic Multicast filtering depends on the MAC filtering. The issue is 32 Multicast IP is mapped to single Multicast MAC which makes Basic filtering less useful. IGMP Snooping is based on the IP meaning whenever there is Join message from the VM, Virtual switch knows the MAC/IP/port of the VM that sends the Join so any multicast traffic for that specific multicast group will be sent only to the VM.

17. For adding a customized services port so that it can be allowed on the vsphere firewall, this needs to be done on the CLI. on GUI you cannot do it.
a. change the directory
cd /etc/vmware/firewall
b. create a new service.xml file
vi service.xml newservice.xml
First it opens the existing service.xml file and from that choose respective and yank it. To search for a specific service do slash followed by the service (eg:/dhcp). To yank, first count the number of lines in the service definition, for eg there is 17 lines for the DHCP service in the service.xml file, from the start of the line for that service type 17yy, this will copy those 17lines to the buffer. Then type :n, this will take you the new file, then enter p this will paste the lines in the copied buffer. On the top line of the new file do <ConfigRoot> and in the bottom of the file </ConfigRoot>. To change the name of the copied Service ID and name of the service, service port number press w, this will delete the entire word. To save and quit :wq!
c. do esxcli network firewall refresh. This will add the service to the list. You can do “esxcli network firewall ruleset list” or “esxcli network firewall ruleset rule list” to verify if the new service is added.

18. To check the iscsi adapter and portal list, use the command “esxcli iscsi adapter list” and “esxcli iscsi physicalnetworkportal list” and “esxcli iscsi logicalnetworkportal list”. Also to check the networkportal details use “esxcli iscsi networkportal list”. To check iscsi sessions, use “esxcli iscsi session list | more” and “esxcli iscsi session connection list”.

19. To check the network related Tx/Rx, use esxtop and then hit ? and then hit n. To see new fields which were not shown, you can add them to the displayed list by doing esxtop and then hit f. If you use capital letters it adds the respective field to the output and if you type lower case it will remove the respective field from the esxtop output.

20. To do packet capture, first from the esxtop output (network related output) find the port id for the VM of which the packet capture is needed. Then do “pktcap-uw –switchport <port id> -c <count of packets> -s <size of packets> -o <filename.pcap>”. To see the packet capture, use “tcpdump-uw -r filename.pcap”. If you want to capture the traffic on the uplink “pktcap-uw -c <count of packets> -s <size of packets> -o <filename.pcap> –uplink <vmnic1> –direction <0 for receive and 1 for send>”. To capture mgmt traffic of ESXi, use
“pktcap-uw –vmk vmk0”. To trace the flow of traffic you can use “”pktcap-uw -c <count of packets> -s <size of packets> -o <filename.pcap> –uplink <vmnic1> –direction <0 for receive and 1 for send> –trace –console”. To check various options of capture, use “pktcap -h” and check out the flow filter option.

21. By default NIOC is enabled in vSphere 6.x. NIOC is available only on DVS With NIOC, you can allocate Reservation quota of Bandwidth per Network Resource Pool. Network Resource Pool will be assigned per Port-Group. Based on the Allocated Reservation quota of Bandwidth per Network Resource Pool the VMs in those port group share the BW.
If you have three 1G Adapters assigned to the DVS (3G of available BW), first you need to reserve the BW for the Virtual Machine traffic (for example 750Mbps). And then the rest of the available BW (2.25g) is allocated as reservation quota for the Network Resource Pool (for example create VM prod Network Resource Pool and allocate 1500Mbps and VM Dev Network Resource Pool and allocate 500Mps). After that assign these Network Resource Pools to the respective port groups. It is advised not to play with Network Resource Pool instead stick with the “system traffic” and edit the shares value.


Troubleshooting MRA Notes

1. Expressway X8.1-X8.8 -> CUCM 9.1(2)SU4 and IM&P 9.1(1)SU6a
Expressway X8.9 -> CUCM 10.X and IM&P 10.X
IM&P 11.5 Requires Expressway X8.8 Due to AXL schema changes in IM&P 11.5

2. When NTP is not configured and synchronized on ExpressWay-C and ExpressWay-E, Jabber Telephony registration to CUCM may not succeed. Security mechanism based on SIP SERVICE messages.
– Expressway-E time-stamps a SERVICE message
– Expressway-E sends the SERVICE message to Expressway-C
– Expressway-C verifies the SERVICE is received within 60 secs error margin

3. With X8.8+ : Expressway E must have forward and reverse DNS entries

4. Dual NIC Recommended Deployment. Cons for Single NIC :NAT Reflection and Higher Bandwidth usage.

5. DX650/70/80 and 88XX/78XX requires Public Root CA. List of public Root CA can be found in the CA-Trust-List.docx

6. TFTP encrypted configuration file is not supported because Jabber, DX, 78XX/88XX needs LSC. For LSC you need the CAPF. Out-of-the-box enrollment for CAPF over MRA is not supported.

7. MRA is supported only with CA signed certificate. Self-signed certs are not supported for MRA.

8. If you want to run encrypted config end to end, meaning MRA endpoint sending encrypted signaling and media to the CUCM running in cluster mode then EXP-C SAN name should contain the Unified CM Security Profile names. If there are different type of devices registering over MRA then it is recommended to use the Universal CM security profile.

9. If you are having 2 different domains, externally and internally then on the Internal DNS you need to have/maintain a forward lookup zone for the external domain.Instead of that you can use the pin-point subzone so you do not need to maintain the forward lookup zone.

10. External and Internal domain details are communicated between EXP-E & C using the SSH tunnel/ClusterDB exchange. This means if the SSH connection fails between EXP-C & EXP-E, then MRA will not work.

11. Use Collaboration Solution Analyzer tools to troubleshoot MRA issues.

Note: Use Cisco Live session and presentation to have more info. This Cisco Live session is so good.

debugs to collect for a FAX issue

debug vpm all (in case of FXS)
debug isdn q931 (in case of PRI)
debug voice ccapi inout
debug ccsip all/messages/verbos
debug voip vtsp all
debug voip dsmp all
debug voip hpi all
debug dsp-resource flex all
debug voip dspapi
debug fax relay t30 all-level-1
debug voip rtp session named-event (in case of NSE based switchover)




SIP calls/SIP debugs/SIP Debug Output Filtering Support

The following is a good reference material from Cisco support forum. Thanks to the creator.

#Show call active voice compact
#Show voip rtp connections
#show call active voice brief | incl call-legs
#sh sip-ua call summary
#sh call active voice summary


SIP debugging overview:

debug ccsip all: This command enables all ccsip type debugging. This debug command is very active, you should use it sparingly in a live network
debug ccsip calls: This command displays all SIP call details as they are updated in the SIP call control block. You can use this debug command to monitor call records for suspicious clearing causes.
debug ccsip errors: This command traces all errors that are encountered by the SIP subsystem.
debug ccsip events: this command traces event, such as call setups, connections and disconnections. An events version of a debug command is often the best place to start because detailed debugs provide much useful information.
debug ccsip info: This command enables tracing of general SIP security parameter index (SPI) information, including verification that call redirection is disabled.
debug ccsip media: This command enables tracing of SIP media streams
debug ccsip messages: This command shows the headers of SIP messages that are exchanged between a client and a server.
debug ccsip preauth: This command enables diagnostic reporting of authentication, authorization, accounting (AAA) for SIP calls.
debug ccsip states: This command displays the SIP states and state changes for sessions within the SIP subsytem.
debug ccsip transport: This command enables tracing the SIP transport handler and the TCP or UDP process

debug voip ccapi inout: This command shows every interaction with the call control application programming interface (API) on both the telephone interface and on the VOIP side. By monitoring the output, you can follow the progress of a call from the inbound interface or VOIP peer to the outbound side of the call. This debug command is very active. you should use it sparingly in a live network.

debug voip ccpai protoheaders: This command displays messages that are sent between the originating and terminating gateways. If no headers are being received by the terminating gateway, verify that the header-passing command is enabled on the originating gateway.

For Fax and dtmf related issues:

debug voice rtp session named-event

debug voice rtp session dtmf-relay


Feature Design of SIP Debug Output Filtering Support

Prior to the SIP Debug Output Filtering Support feature, debugging and troubleshooting on the VoIP gateway was made more challenging by the extensive amounts of raw data generated by debug output.

This feature allows the debug output for a SIP call to be filtered according to a variety of criteria. The SIP Debug Output Filtering Support feature provides a generic call filtering mechanism that does the following:

•Allows you to define a set of matching conditions for filtering calls.

•Identifies the desired calls based on the configured matching conditions inside VoIP gateways.

•Coordinates the filtering effort on traced calls between multiple modules inside VoIP gateways.

•Displays the debugging trace for calls that match the specified conditions.

SIP Debug Commands that Support Output Filtering

•debug ccsip all

•debug ccsip calls

•debug ccsip events

•debug ccsip messages

•debug ccsip preauth

•debug ccsip states

Configuring Call Filters

This task configures the conditions for filtering SIP calls.


1. enable

2. configure terminal

3. call filter match-list number voice

4. incoming calling-number string

5. incoming called-number string

6. incoming signaling {local | remote} ipv4 ip-address

7. incoming media {local | remote} ipv4 ip-address

8. incoming dialpeer tag

9. outgoing calling-number string

10. outgoing called-number string

11. outgoing signaling {local | remote} ipv4 ip-address

12. outgoing media {local | remote} ipv4 ip-address

13. outgoing dialpeer tag

14. end


call filter match-list 1 voice
incoming called-number XXXXXXXX

SIP Messages and Codec Negotiation- Quick Recap

1. In the 1xx, 2xx, 3xx, 4xx, 5xx,1xx messages, look at the “To” field to know who is sending the messages.
2. In the Invite, Re-Invite, Subscribe, BYE, ACK, and PRACK look at the from field to know who is sending the messages. If the message is a re-invite then you will see the tag= in the ‘To’ field of the SIP message. Re-invite normally happens during Fax, hold etc. There will be 2 re-invites one with IP= to close the previous communication and another invite will come with IP=x.x.x.x. Both re-invites will have same tag.
3. If one end supports g729r8/g729ar8 and other end supports g729br8 the media will fail. To solve this either change the service parameter on the CUCM “silence suppression (annex b)” to True or use voice-class codec command with codecs preferences listed or use the “voice-class sip g729 annexb-all”. If you do not want CUBE to get involved in codec negotiation then in the dial-peer use “codec transparent” or on inbound dial-peer use “voice-class sip pass-thru content sdp”. If you use “voice-class sip pass-thru content sdp”, this does not have any impact. The command “voice-class sip pass-thru content sdp” is applicable only for sip-sip connections. Also when you use “voice-class sip pass-thru content sdp” then the codec command under dial-peer have no impact.
4. If you have both SIP and SCCP phones ensure you do not use “voice-class sip pass-thru content sdp” because this will break the communication between SIP to SCCP phones and viceversa. If you are going to use “voice-class sip pass-thru content sdp” to resolve the video issue (to avoid video codec getting converted from value 97 to 119) between 2 sites, then instead of using “voice-class sip pass-thru content sdp” first change the “rtp payload-type cisco-codec-fax-ack 112” and change the “rtp payload-type cisco-codec-video-h264 97”.
5. When you use voice-class sip pass-thru content sdp in the dial-peer then only dtmf-relay rtp-nte is supported/works.
6. When you do “show voice call active brief”, the “pid:dial-peer number answer” is the inbound dial-peer and “pid:dial-peer number originate” is the outbound dial-peer.
7. PRACK is used to for sending the IP of the device to which the media has to be sent so that early media is supported. For this on the “SIP Re1xx Options” needs to be enabled on the SIP profile to “Send PRACK for all 1xx messages”. Cube by default has this enabled.

Vendor-Neutral CERT

CompTIA – Well known for Security+, Network+
ISC2 – Well known for CISSP. Other certs like SSCP, CCSP etc are not very familiar in the market.
EC-Council – They have a wide variety of know certs in ethical hacking and forensics like CEH, CHFI.
ISACA – Famous in audit and other certs like CISA and CISM whereas CGEIT and CRISC are not very well recognized

CUCM integration with CMS

1. For Adhoc conference https is used. For meetme and Rendezvous based conference SIP is used.
2. CUCM uses API to create conferences so only https is supported not http. There are 2 services on CMS, webadmin (https) and callbridge (SIP). Callbridge service is responsible for terminating sip signaling and merging the media.
3. Calls between CUCM and CMS is SIP (SIP TLS or SIP TCP).
4. on CUCM add the CMS as the conference bridge, enable the “override SIP Trunk destination as HTTP address” and enable “use HTTPS”. The hostname of the CMS configured on the CUCM should match the x.509 certificate name of the CMS. There are two certificates one for the webadmin (conference bridge configuration) and another for callbridge (SIP Trunk). CMS self signed certificate will not work for the integration with CUCM. Also the wild card certificate is not supported by CUCM. If you use wildcard certificate the SIP trunk will be shown as in full service but the conference will not work. The bug id is CSCur85534.
5. The CMS will still register even if the username/pwd provided in the configuration of the conference bridge in CUCM but creating the conferences will fail because CMS will check the credentials only when creating conferences.
There is a Bug ID CSCvb35512.
6. Normalization script on the CUCM SIP Trunk:
– vcs -interop on CUCM < 11.5 SU2
– cisco-meeting-server-interop on CUCM ≥11.5SU2
7. CMS does not do any certificate checking, but you need to enable “SIP media encryption”.


Use the following commands to see the details for CMS
cms>pki inspect cms.cer, cms.cer is the certificate name
To generate a selfsigned cert on the cms, use
cms>pki selfsigned webadmin
To see more option use
cms>pki ?

EXP-C_CUCM and EXP-C_EXP-E certificate Trust

Expressway C looks into the SAN fields only and will not look into the CN field. So it is recommended to have the FQDN same on both the fields.
Tomcat certificate is used for this trust purpose. When trusting the certificate EXP-C checks two things, one is the verify the CA who has issued the certificate in the EXP-C trust store and check the identity certificate of the CUCM. Meaning the CUCM CSR should be signed by CA and CA certificate should be uploaded to the EXP-C Trust store. If CA certificate is not seen then the connection is denied at the OS level but if the identity certificate is not trusted the connection is denied at application level. The certificate check of CA and the identity of CUCM is done only if the TLS verify mode is set to ON. This means only httos connection to CUCM and not TLS connection. In this case the connection is secured and encrypted but you do not check to which server this secured/encrypted connection is sent. Also note that EXP-C certificate should be uploaded to the callmanager trust store or the CA certificate should be uploaded.TLS Neighbor zone and TLS search rules are automatically created.
On the CUCM SIP Trunk security profile ensure to add the X.509 Subject Name. This subject name is the subject name of the EXP-C certificate. You need “VCS-interop” script chosen on the SIP Trunk setup in CUCM.

EXP-C_EXP-E certificate negotiation:

Both need to trust each other, this means mutual TLS. Mutual TLS will not happen if Certificate does not contain the Extended Key Usage ‘TLS Web Client Authentication’. Normally this option is present in CSR unless the CA signing the certificate strips it