Multi-Networked Nested ESXi 7.0 Home Lab Setup with VyOS router

  VMware _vSphere

I decided to write this article since there is not much source of full detailed information about Nested ESXi installation in the Home Lab environment which also explains the network side of the configuration. In this article, while we install the Nested ESXi, also we will do the network routing of the hosts in different network subnets with the VyOS open-source router. I tried to explain all the steps as clearly and comprehensively as possible to avoid possible confusion, especially for those who consider themselves novice.

 

    • — ARTICLE INDEX —

  1. Home Lab Requirements
  2. Creating vSwitches & Port Groups on Physical ESXi Host
  3. Installing Nested ESXi Hosts
  4. Installing VyOS
  5. Configuring VyOS
  6. DHCP Scopes
  7. Troubleshooting
  8. TEST VM
  9. Installing vCenter
  10. Creating a new Datacenter and adding ESXi Hosts to the vCenter 7.0

 

Multi-Networked Nested ESXi 7.0 Home Lab Setup with VyOS router

 

As you can see in our Home Lab schema,there are three Nested ESXi hosts under one main ESXi host on the physical server. Each ESXi host is connected to a vSwitch and VyOS open-source router is used for subnet IP routing.


Since this article will not include Windows10, Server 2019, and AD, DHCP, DNS installation instructions, I assume that you have already installed ESXi 7.0 on the physical host and installed Windows 10 TestClient and DHCP&DNS server virtual machines on this host’s ESXi. However, Nested ESXi, VyOS and Vcenter installations and configurations will be described in detail in this article.

 

 

 

    • — Home Lab Requirements —

       


-ESXi 7.0 ISO        : https://my.vmware.com/group/vmware/evalcenter
-vCenter 7.0 ISO  : https://my.vmware.com/group/vmware/evalcenter
-VyOS ISO             : https://downloads.vyos.io/?dir=rolling/current/amd64
-Windows 10 ISO : https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise
-Server 2019 ISO  : https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2019
-A physical host with enough RAM, CPU and HDD resources for this Home-Lab

 

 

 

    • — Creating vSwitches & Port Groups on ESXi —

       


* Firstly, let’s take a look at the ESXi host default port group and vSwitch settings on the physical server.

 

Default Port Groups

Default Port Groups

vSwitch0

vSwitch0 detail


* Now, let’s add vSwitches that we will connect to ESXi hosts on different networks and Port Groups that we will assign to vSwitches we created..


-While adding vSwitches, don’t forget that we need to set the MTU value to 9000 to allow Jumbo frames and set “Promiscuous Mode, Mac Address Changes and Forged Transmits” values as Allow in security options.

 

vSwitch1

Default Port Groups

vSwitch2

vSwitch3


-While adding Port Groups, it is important to assign them to the correct vSwitches.

 

Port Group 172.25.1.0/24

Port Group 10.10.40.0/24

Port Group 10.10.10.0/24

 

 

 

    • — Installing Nested ESXi —


      * Now it’s time to install Nested ESXi servers.
      – See Gallery image titles for instructions

      Installing Nested ESXi
















      *Repeat the steps for NestedEsxi-2 and NestedEsxi-3 Installations and don’t forget to set the MTU value to 9000 to allow Jumbo frames and set “Promiscuous Mode, Mac Address Changes, and Forged Transmits” values as Allow in security options for the vSwitches of the Nested ESXi’s.

       

       

       

        • — Installing VyOS —


          * VyOS is a Linux-based network operating system that provides software-based network routing, firewall, and VPN functionality and installing VyOS is really easy .
          – See Gallery image titles for instructions

          Select create a new virtual machine



















           

           

           

            • — Configuring VyOS —


              * Ok, we have finished 4 networked VyOS installation. But now we need to configure the Vyos router so that it can provide communication between 4 different networks and provide internet access to all networks. Let’s start it by reviewing our network plan.

              Multi-Networked Nested ESXi 7.0 Home Lab Setup with VyOS router


              We will talk about how to configure VyOS right below, but you can refer to the resources below for detailed information.


              https://docs.vyos.io/en/latest/index.html
              https://support.vyos.io/en/kb
              https://forum.vyos.io/


              At first I want to give you a basic information about VyOS CLI. There are 2 different modes in VyOS console CLI: operationmode and configmode. You will see the ~ $ sign under Operationmode and the # sign under the Configmode. When you login, you will be under operationmode. Just type the config command and press Enter to switch to the configmode. To exit configmode, you need to type exit. The commands you can use under config mode and operation mode are different. However, to run the operationmode command under the configmode, you can type run at the beginning of the operationmode command, leave a space, type any operrationmode command, and press enter. Let’s begin to configue the VyOS ;

               

              After logged in, run the command below to see the status of the network interface cards.
              show interfaces

              show interfaces

              Use the conf command to enter the configmode, then you can use the clear command to clear the console screen.

              Use the show system hostname command to see the system host name. You can see that the host name of our system is “vyos”. You can change this name with the set system-hostname command if you want. I will not change this name for now, but in order to change the system domain name, I will assign my domain name “ad.systemyoda.com” to the system with the set system domain-name command. Then I set the time zone of the system with the set system time-zone ETC/GMT+8 command. You can enter the appropriate parameter for your time zone. You can also define an NTP server to the system with set system ntp server -IP address or hostname of NTP- command.

              Now I want to tell you the most important commands, commit and save . Regardless of what configuration changes you made, the rule / configuration changes will not be active in VyOS unless you commit it. So do not forget to enter commit command after making any configuration changes to take effect. But it is not over yet, if you do not enter the save command, the entire configuration will be lost when you restart VyOS.

              Get the information of network adapter cards with show interfaces ethernet command while under configmode.

              Verify the Mac addresses by comparing them with the Mac addresses of network adapter cards we assigned to VyOS VM.

              OK, we can start to assign IP addresses to the VyOS network interface cards with set interfaces ethernet command. Also, we can assign descriptions to each adapter with the description parameter.

              At this point, all networks can communicate with each other without writing any static rules, because all the networks are directly connected to the interfaces on the same router. You can make a basic ping test to verify the connections among the networks. But we still haven’t provided the internet access to the networks, so we need to write NAT rule for each network with set nat source rulecommand.

              Now, we need to write a static route that points our home modem gateway as next-hop to route network traffic such as internet traffic which is not destined for the networks defined in VyOS. We will use set protocols static routecommand for static route. Also, it’s a good idea to commit and save the configuration that we made so far.

              So far so good. Everything looks fine. Now all networks can communicate with each other and have internet access. But if you have a large network that needs the DHCP server, you should set up a DHCP server on each network. Or how about just installing one DHCP server and setting up DHCP Relay on VyoS so that all networks can use a single DHCP server. As you see in the Home-Lab Schema at the beginning of the article, I assume that you already have a DHCP server on the 172.25.1.0 network. We will use set service dhcp-relay command to relay dhcp. So let’s do it.


              We have completed almost everything. But it would be better if there was easier access to VyOS than VM console. So, we need to enable SSH on VyOS by set service ssh port 22 command.


              Let’s see the latest configuration information of the VyOS network adapter card again by show interfaces ethernet command.
              You can review specific configurations with commands like; show interfaces, show nat source, show protocols static, show service dhcp-relay, show service ssh, show system domain-name, show system host-name, show system ntp, show system time-zone. Or you can review all of your configurations setting just with show command. Keep in mind that; while you type you can use the Tab key for completion of the command or see the available commands.


              Finally, let’s look at the use of reboot and poweroff commands under operationmode.

 

 

 

    • — DHCP Scopes —


      * Of course, you can set manual IP addresses for the clients on the networks. Also, you can set up VyOS as a DHCP server but in our Home-Lab scenario, we already have set the dhcp-relay on VyOS and ve have a single AD/DHCP/DNS server that has an IP address 172.25.1.10. In scope options which will be distributed to the networks by DHCP; DNS address and as the default gateway the Router IP address has been defined for each network.
      In the DHCP Server screenshots below, you can see the DHCP scopes created for the networks and DHCP scope options and IP leases.

       

       

       

        • — Troubleshooting —


          * You can set up VyOS as a DNS forwarder with some useful parameters but in our Home-Lab scenario, we already have a single AD/DHCP/DNS server that has an IP address 172.25.1.10. If you encounter any DNS related issue, firstly check the VyOS configuration settings with “show” command on the VyOS console. It will also help to run the nslookup and tracert commands at the client operating system. In addition, assigning the home modem gateway IP address to the client operating system as DNS IP instead of DNS server IP ( bypassing the AD DNS Server) and checking for internet access can help you determine if the DNS server is the source of the problem.By the way, since 192.168.1.0 is also the IP range that the home modem’s DHCP distributes, consider possible IP conflicts when manually assigning IP addresses to hosts and clients from the 192.168.1.0 network IP range. Excluding a specific IP range from the 192.168.1.0 on the home modem and using this range for the home lab is the best solution.


          I would also like to mention a specific rare DNS server problem that I have encountered twice and that you might be experiencing; “IPv4 root hints missing in DNS Server Root Hints List” Since we didn’t assign IPv6 addresses to network adapters on the VyOS and we didn’t make any IPv6 NAT or Static Routes; if you encounter this issue, PCs and servers on your network will not be able to perform DNS resolution. If you are going to configure IPv6 on VyOS, you should configure similar rules for IPv6, such as those you configure for IPv4. If you are not going to use IPv6 and there are only IPv6 addresses in the DNS Server Root Hints List, you must manually replace those IPv6 addresses with the IPv4 addresses. If you don’t want to deal with one by one editing Root Hints IPs, you can assign home modem gateway IP as a secondary DNS to DNS server’s network card temporarily then you can click edit & resolve buttons on Root Hints List to get IPv4 addresses. Last note; unfortunately, after restarting the server, Root Hints IPv6 addresses can reappear without IPv4 addresses. In this case, you may need to repeat the above procedure. (The server I use is MS 2019 and I haven’t updated it yet, maybe the update may prevent me from experiencing this problem again)

          missing IPv4 root hints in DNS Root Hints List IPv6

           

           

           

            • — TEST VM —


              * If you don’t plan to install vCenter or you plan to install it later like me, you may need a multi-networked test virtual machine that can perform dhcp, network & internet access tests and access the NestedEsxi hosts’ administration web interface to manage NestedESXi hosts.
              – By the way, 4 network cards are just for testing purposes. You don’t need 4 separate network cards to access each network resource. After VyOS setup & configuration you can reach all networks just one network card connected one of the networks.

              Test VM Network Adapters





               

               

               

                • — Installing vCENTER —


                  * The main reason we want to install vCSA is to manage all ESXI hosts and virtual machines in different networks from a single point. In addition, you can manage the resource usage of multiple Host machines with vCenter. And remember, if your hardware resources are limited, vCenter can give you a little headache because it’s hungry for Ram and CPU. Anyway, you can refer to the link below for the system requirements of the vCenter.


                  -System Requirements for the vCenter Server Appliance :
                  https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-752FCA83-1A9B-499E-9C65-D5625351C0B5.html


                  * I will only cite DNS-related requirements from the above link.
                  -When you deploy the vCenter Server appliance, similar to any network server, you can assign a fixed IP address and an FQDN that is resolvable by a DNS server so that clients can reliably access the service
                  -When you deploy the vCenter Server appliance with a static IP address, you ensure that in case of system restart, the IP address of the appliance remains the same.
                  -Before you deploy the vCenter Server appliance with a static IP address, you must verify that this IP address has a valid internal domain name system (DNS) registration.
                  -When you deploy the vCenter Server appliance, the installation of the web server component that supports the vSphere Client fails if the installer cannot look up the fully qualified domain name (FQDN) for the appliance from its IP address. Reverse lookup is implemented using PTR records.
                  -If you plan to use an FQDN for the appliance system name, you must verify that the FQDN is resolvable by a DNS server, by adding forward and reverse DNS A records.
                  -You can use the nslookup command to verify that the DNS reverse lookup service returns an FQDN when queried with the IP address and to verify that the FQDN is resolvable.
                  -If you use DHCP instead of a static IP address for the vCenter Server appliance, verify that the appliance name is updated in the domain name service (DNS). If you can ping the appliance name, the name is updated in DNS.
                  -Ensure that the ESXi host management interface has a valid DNS resolution from the vCenter Server and all vSphere Client instances. Ensure that the vCenter Server has a valid DNS resolution from all ESXi hosts and vSphere Client.

                  Test VM Network Adapters




























                   

                   

                   

LEAVE A COMMENT