Tag Archives: NSX

VMware NSX logical routing LAB

Following the subject of previous post, I will continue with NSX routing LAB topology used by me in my NSX learning. LAB topology diagram are inserted below. 

VMware NSX logical routing LAB diagram

(click on picture for hi resolution image)

A brief description for lab topology:

  • There are tree VXLAN logical switches: 5001, 5002 and 5003 corresponding to segments in a formal three-tier application (WEB/APP/DB). Each segment have a related subnet network: for LS5003 (WEB), for LS5002 (APP) and for LS5001 (DB).
  • Inter VXLAN routing are provided by a Distributed Logical Router (DLR) and an Edge Services Gateway (ESG). DLR act as a default gateway for DB and APP segments and ESG provide routing for WEB network segment. First IP from each subnet are reserved for default gateway.
  • Both DLR and ESG are configured in HA configuration mode (active-standby) – that is why these services are represented by two VMs each (vShield Edge for ESG and Control VM for DLR) – x per Edge ESXi host.
  • A group of two ESG appliances configured for Equal Cost Multi Path (ECMP) routing provides upstream communications. ECMP are enabled as well for DLR and ESG routers.
  • All routers are interconnected trough a transit logical switch 5004. Routers in this segment will use IP addresses from subnet: .5 for ESG, .254 and .253 for each of upstream routers. The DLR router use two IP addresses: first (.2) as a forwarding address and second (.3) as a control plane address (protocol address).
  • DLR, ESG and upstream routers exchange routing information via OSPF. All routers forms OSPF neighbor relationships.
  • ESG router are configured with both interfaces (internal and uplink) as part of the OSPF process (mapped to OSPF Area 0). DLR router have OSPF activated only on uplink interface and configured to redistribute in OSPF directly connected networks (APP and DB).
  • Upstream routers use OSPF on its internal interfaces and form BGP peering with external physical routers via its uplink interfaces.
  • External communications take place trough a dedicated dvPG (EDGE) with uplinks via physical NICs – DirectPath IO pass-through from physical host to EDGE nested ESXi.
  • Upstream ESG routers will redistribute BGP routes in OSPF and conversely OSPF to BGP.  

Presented below, are the configurations applied on physical routers (RTR1/2). VLAN 100 are used to interconnect with ESG upstream routers (EDGE/ subnet) and VLAN 102 to reach next hop on Mikrotik router (subnet For Internet access, a NAT overload function is configured on Mikrotik network appliance.

VMware NSX logical routing - physical router configuration

After all the configurations applied, we can check the routing protocol adjacencies and routing table information. For instance, here’s what DLR routers showing us:

OSPF adjacencies with ESG router (.5) and upstream routers (.253 and .254)

VMware NSX logical routing - DLR ospf adjacencies

Respectively, the routing table: 

VMware NSX logical routing - DLR routing table

We can observe two routes learned via OSPF: one to (WEB) learned from ESG routers (.5) and a default route as external redistribute route. All other routers are directly connected type. Btw, is the IP address of the DLR’s router management interface and are the automatically assigned network used for heartbeating between DLR router failover peers.

BGP peering can be checked on one of the upstream ESG router:

VMware NSX logical routing - upstream ESG BGP peering

and corresponding, all known networks: 

VMware NSX logical routing - upstream ESG known networks

It’s interesting to verify the routes installed in routing table on physical routers. Bellow, are shown the routes on RTR1 physical routers.

VMware NSX logical routing - physical router routing table

Finally, we can check the connectivity to some internet resources from one of the VM connected to internal logical switch (5002/APP): 

VMware NSX logical routing - traceroute test

My NSX Lab environment – short description

In this post, I will insert a diagram from my NSX LAB environment. A brief description follow thereafter.

My NSX Lab environment – short description

(click on picture for hi resolution image)

General LAB environment description:

  • All NSX lab components (ESXi servers with NSX loadable modules, NSX manager, Controller Cluster VMs, DLR control VM, etc.) are performed as virtual machines, all running on a single physical server. It is the same physical server used by our students (at DNT) in their lab exercises. Physical server have sufficient capacity to support all of my NSX lab scenarios (in my particular case I had access to 8x 2,9GHz x5570 CPU cores, 64GB RAM, 500GB storage space on 8xHDD 10k RAID10 LUN).
  • Physical server resources are controlled by LAB vCenter Server that define a particular vAPP container with minimum reserved resources for each student. My NSX LAB act in such a container. Each student have privileges to manage VMs only in their own container.
  • All LAB scenario’s VMs can be connected only to one permitted port group: dvPortGroup-Students. This port group have no uplinks (in other words isolated from physical networks), configured to carry all VLAN numbers and act in promiscuous mode (mandatory for nested ESXi). Students will use a remote access VPN connection to get into this network (via a VPN gateway build as a VM connected at the same time to production and isolated students network). An http proxy server are configured to enable access to Internet http/s resources from isolated student network. Lab vCenter Server, VPN and Proxy Server are all part of ADM-INFRA-VMW vAPP container with restricted access.

Note: For more information about DNT classroom VMware’s lab environment check one of the previous article series (Mediu de laborator pentru studenții claselor de VMware, part I, part II, part III).

NSX lab architecture brief description:

  • NSX lab will use IP addresses from subnet (all IP allocations are shown in diagram)
  • a dedicated vCenter sever is installed (further integrated with NSX manager)
  • five nested ESXi are installed and configured. These are grouped in three clusters: two computer clusters and one edge/mgmt cluster.
  • two distributed switches are used, one for ESXi in compute clusters and other for edge/mgmt cluster. A single transport zone are configured across all ESXi clusters.
  • several VXLAN switches are configured and interconnected via a Distributed Logical Router or NSX Edge. Some test VMs are connected to these VXLAN switches.
  • EDGE cluster’s nested ESXi hosts are additionally equipped with dual port physical NIC brought here via DirectPath IO from physical server. Physical ports are connected to external routers and switches (Cisco equipment from our CCNP lab kit). 

Image below show the inventory views for LAB vCenter Server (left) and NSX LAB vCenter Server (right):

My NSX Lab environment – short description - inventory views

My list for useful links for VMware NSX studying

Recently, I had the chance to get a new VMware certification: VMware Certified Professional 6 – Network Virtualization. During the study, I made a list of useful links and references – a list that I want to share in this post. Certainly, the subject is continuously evolving so the list will become outdated shortly, but even so, it can be a starting point. The list is as is for the end of 2015, no updates will be added. The list is presented with no particular order.

  1. VMware NSX for vSphere (NSX-V) Network Virtualization Design Guide (pdf, 93 pag.) (link)
  2. VCP Network Virtualization – Exam Blue Print (pdf, 25 pag.)
  3. VMware NSX Install. Configure, Manage [6.0] – student guide (pdf, 480 pag.)
  4. VMware NSX Install. Configure, Manage [6.0] – lab manual (pdf, 226 pag.)
  5. vmware.com – microsegmentation using NSX distributed firewall (pdf, 30 pag.)
  6. vmware.com – NSX for vSphere, getting started guide (pdf, 43 pag.)
  7. blog.bertello.org – NSX for Newbies – Part 1-10 (link)
  8. routetocloud.com – NSX Distributed Logical Router Deep Dive (link)
  9. routetocloud.com – NSX Distributed Firewall Deep Dive (link)
  10. HOL-SDC-1403   VMware NSX Introduction (link)
  11. HOL-SDC-1425   VMware NSX Advanced (link)
  12. featurewalkthrough.vmware.com  – NSX – (link)
  13. buildVirtual – VCP-NV Objectives Study Guide (link)
  14. YAVB – Rich Dowling  – VCP-NV (link)
  15. Ivan Pepelnjak – Overlay Virtual Networks in SDN (pdf, 278 pag.)
  16. HOL-PRT-1462   Palo Alto Networks – Virtualized Data Center Security (link)
  17. HOL-SDC-1415   IT Outcomes – Security Controls Native to Infrastructure (link)
  18. HOL-SDC-1420   OpenStack with VMware vSphere and NSX (link)
  19. HOL-SDC-1424   VMware NSX and the vRealize Suite (link)
  20. HOL-SDC-1412   IT Outcomes – Data Center Virtualization and Standardization (link)
  21. HOL-SDC-1319  VMware NSX … (link)
  22. packetmischief – DCI: THE NEED FOR STRETCHED LAYER 2 (link)
  23. packetmischief – Five Functional Facts about VXLAN (link)
  25. packetmischief – DCI SERIES: OVERLAY TRANSPORT VIRTUALIZATION (link) // offtop
  26. packetmischief – FIVE FUNCTIONAL FACTS ABOUT OTV (link) // offtop
  27. telecomoccasionally – NSX for vSphere: Understanding Transport Zone scoping (link)
  28. VMware NSX Use Case – Simplifying Disaster Recovery – Part 1 (link)
  29. VMware NSX Use Case – Simplifying Disaster Recovery – Part 2 (link)
  30. Considerations for Management Interface of Distributed Logical Router Control VM (2122060)
  31. blog.ipspace.net – Does uRPF Make Sense in Internet Service Provider Networks? (link)
  32. NSX 6.2 admin guide – Add a Logical (Distributed) Router (link)
  33. blogs.vmware.com – Getting Started with VMware NSX Part I – Building Virtual Networks (link)
  34. blogs.vmware.com – Getting Started with VMware NSX Part II – Building Virtual Networks (link)
  35. NSX Compendium- VMware NSX for vSphere (link)
  36. Сетевые оверлейные технологии для ЦОД. Часть 1 (link)
  37. Сетевые оверлейные технологии для ЦОД. Часть 2 (link)
  38. Сетевые оверлейные технологии для ЦОД. Часть 3 (link)
  39. bradhedlund.com- Going Over the Edge with your VMware NSX and Cisco Nexus (link)
  40. bradhedlund.com- The vSwitch ILLUSION and DMZ virtualization (link) // offtop
  41. bradhedlund.com- Distributed virtual and physical routing in VMware NSX for vSphere (link)
  42. bradhedlund.com- On choosing VMware NSX or Cisco AICI (link)
  43. bradhedlund.com- What is Network Virtualization? (link)
  44. bradhedlund.com- What is a Distributed Firewall? (link)
  45. bradhedlund.com- An introduction to Zero Trust virtualization-centric security (link)
  48. VMworld 2014: SEC1746 – NSX Distributed Firewall Deep Dive (link)
  49. telecomoccasionally – Distributed Firewall (DFW) in NSX for vSphere, and “Applied To:” (link)
  50. telecomoccasionally – NSX for vSphere: VXLAN Control Plane modes explained (link)
  51. networkinferno.net – Validating Distributed Firewall rulesets in NSX (link)
  52. networkinferno.net – Implementing a  Zero Trust Security Architecture (link)
  53. Manage and report on a Distributed Firewall using NSX Manager and ESXi CLI commands (link)
  55. blogs.vmware.com – VMware vCloud Hybrid Service Direct Connect Primer (link)
  56. habr – Сети для самых маленьких. Часть восьмая. BGP и IP SLA (link)
  57. VMware NSX, Convergence, and Reforming Operational Visibility for the SDDC (link)
  58. Tom Fojta's Blog  – vCloud Director with NSX: Edge Cluster (link)
  59. VMware.com – vCloud Architecture Toolkit – vCAT-SP (link)
  60. blogs.technet.com – Border Gateway Protocol (BGP) with Windows Server 2012 R2 (link)
  61. blogs.technet.com – Multi-tenant Site-to-Site (S2S) VPN GW  with Windows Server 2012 R2 (link)
  62. NSX Use Case Diagrams (link)
  63. VMware vRealize Automation with NSX (link)
  64. Architecture Design: vSphere with IPv6 (link)
  65. Network Virtualization (NSX) and vSphere Data Protection Interop (link)
  66. VMware NSX Installation Part 5 – Checking NSX Controller Status (link)
  67. routetocloud.com – Troubleshooting NSX-V Controller (link)
  68. routedocloud.com – NSX Load Balancing (link)
  69. routedocloud.com – NSX-V Edge NAT (link)
  70. blogs.vmware.com – Useful VXLAN commands in ESXCLI 5.1 (link)
  71. blogs.vmware.com – Understanding and troubleshooting NSX for vSphere 6.x DFW (link
  72. SR-IOV support status FAQ (2038739) (link)
  73. NSX vSphere troubleshooting (link)
  74. LAG vs. LBT for vSwitch Uplinks in vSphere  (link)
  75. vSphere Distributed Switch Part 14 – Configuring dvPortGroup General Settings (link)
  76. IPv6 in vSphere 6 (link)
  77. blog.ine.com – The Inside and Outside of NAT (link)
  78. NSX Useful numbers – VCP-NV Study (link)
  79. Deploying, Troubleshooting, and Monitoring VMware NSX Distributed Firewall (link)
  81. Proxy ARP & ICMP Redirect in vShield Edge NIC – Explained (link)
  82. How to install and configure VMware NSX 6.1.2 SSL VPN-Plus Step by Step (link)

Following the subject, next I will post some info about my NSX lab environment.