Install ESXi in a VMware Workstation virtual machine
When you don't have hardware that supports running ESXi and you want to setup this OS for testing or educational purposes then installing it in a VMware Workstation virtual machine allows you to run the OS on generic hardware. This article explains how to do this. It is based on VMware Workstation version 10 and ESXi version 5.5. Doing this is also know as running a nested hypervisor because it runs on another virtualization technology.
When you run ESXi 5.5 in a VMware Workstation 10 virtual machine you can run virtual machines in that virtualized ESXi-host if your CPU has virtualization support. (You might need to enable it in the BIOS) You will always be able to run 32 bit vm's in the virtual ESXi-host. But running 64 bit virtual machine requires the EPT feature in your Intel-processor (for AMD it's called RVI). Intel has created a web page to list processors and their features. You can find it here: ark.intel.com/products/virtualizationtechnology
The prerequisites for this setup are that you already have VMware Workstation running. If you do not have that already you can read how to do that here.
From VMware Workstation create a new virtual machine and select the Custom creation wizard. The reason is that there are a few settings you want to change compared to the default. If nothing is mentioned in the following steps just follow the defaults.
If you run VMware Workstation on Linux and want to run a nested ESXi-server then run Workstation as root or add RW-access to the /dev/vmnet0 folder (for bridged networking) or vmnet0/8 for host only or NAT-networking. Otherwise the nested ESXi-host will not be able to use promiscuous mode networking which is required. This is not an issue on when you run VMware Workstation on Windows.
Select the ESXi installer ISO in the wizard. This will detect ESXi and will configure some default settings for you, such as using 2 processors and 4 GB of RAM.
Next select the type of networking for your virtual ESXi-host. If you leave it at the default setting of NAT then you will be able to communicate from the ESXi-host with your LAN but you can't login to your ESXi-server from other computers in your network. Only the computer that you are using for VMware Workstation will be able to communicate with the ESXi-host because it's also on the same internal NAT-network. If you want to communicate with your ESXi-host from other computers in your network then select bridged networking.
When you have provided all the information in the wizard it will display a summary of all the settings. You can also at this time customize the hardware of the virtual machine. One of the things I suggest you do is to add a few extra virtual network adapters to your ESXi-host. This will make the setup more realistic and will allow you to use different switches for vMotion or for IP-storage. Of course all the traffic will be sent through your computers' single NIC but your setup will reflect VMware's best practices from the ESXi-host's point of view.
When the virtual machine is finished and powers on the ESXi-install process will start. You can now perform a standard ESXi-installation, which is very straightforward. Once the installation procedure is finished you can let the ESXi-host reboot and it will be ready to use.
Once it is booted you can look at the server's console what IP address has been assigned to it and you can then access the host from a browser. In the image below you see there is a download link to download the vSphere Client. With that client you can create virtual machines and build your vSphere environment.
When you start the vSphere client to login for the first time you will see a message explaining that all new features are available through the vSphere Web Client. That client is only available when you have vCenter running. You can install vCenter on Windows, but much easier will be to import the vCenter Virtual Appliance into your environment.
Install vCenter Server
One you have your server up and running you can start configuring networking and storage and create virtual machines. Something else to do is install vCenter. You probably do not what to run your vCenter server virtual machine inside the virtualized ESXi-server. (Depending on your hardware it wouldn't even work since vCenter runs in a 64-bit OS and you might not be able to run 64 bit nested VM's.) And why would you? You can import the vCenter appliance from VMware Workstation. If you go to the File-menu to open a new virtual machine and point to the vCenter Appliance OVA-file that you can download (also with your evaluation download) then it imports the appliance into VMware Workstation.
If you can actually run 64 bit virtual machines in your nested ESXi-server and you want to import the appliance with the vSphere client then this article explains how to do that.