How Reversed ARP is used for MAC learning with vSphere vMotion

When a virtual machine is migrated from one ESXi host to another the traffic for that vm has to be delivered to the destination host and no longer to the source host. How is the physical switch informed about the move? ESXi sends out a Reversed Address Resolution packet on the destination host that will trigger the physical switch to update its MAC-address table. In this article I will explain how this process works and why it is important. 

What happens on the physical switch?

Ethernet layer 2 switches keep track of which MAC address is on what port so traffic can be delivered via the right port to reach it's destination. So let's say a virtual machine is on an ESXi host that is connected to physical switch port 3. All traffic for that virtual machine goes out on port 3 to reach that vm. This situation before we migrate a virtual machine can be seen in the below image. 

 

 

When we migrate the vm to the destination host the physical switch will need to know that the virtual machine is now behind port 7, this is the port where the destination ESXi host is connected. This situation after the vMotion migration is shown in the next image.
 

The physical switch maintains a table called the MAC-address table (also known as CAM-table) that is used to deliver traffic to the right destination port. This is well explained in this article: How Switch learns the MAC addresses Explained

So how do we inform the physical switch about this change? There is no industry standard API-call or command to send instructions to update a MAC-address table entry to a layer 2 switch so VMware uses a clever trick to make the physical switch aware of the change. The ESXi-host sends out a Reverserd ARP packet that will trigger the physical switch to update it's port assignment information. 

Why using Reversed ARP is important

The way the MAC-learning process works is that when an Ethernet-frame is received the MAC-address where that frame came from is used to populate or update the MAC-address table. With this dynamic process the physical switch learns which addresses are behind each port. After a vm has been migrated to the destination host the physical switch would eventually learn the new address as soon as the vm starts sending traffic itself. But what if the virtual machine does not send traffic right after the vMotion process? It might take seconds or longer and in that case any traffic destined for that vm would still be delivered on the 'old' port. So this is where the VMKernel/ESXi steps in and sends out a RARP-packet to notify the switch of the virtual machine's new location.

This process is triggered by the Notify Switches feature that is enabled by default. On a standard switch this is configured at the switch level and for distributed virtual switches on the port group level.  

The Reversed ARP Ethernet-frame sent by the ESXi-host (and not by the vm) immediately after the vMotion migration has completed uses the virtual machine's MAC-address as the source. When the physical switch sees that frame it triggers an update to it's MAC-address table and therefor any traffic that will have that MAC-address as it's destination will be delivered to the 'new' switch port.

The way this process works is also described in this knowledge base article: IP to MAC mapping, GARP, RARP and Notify Switch. The same process is also used when a failover occurs for a vm because then we also must inform the physical switch that a virtual machine\s MAC-address is on a different switch port.

How to capture the Reversed ARP packets

I have recorded a video that shows this process in action from the vSphere-side. If you want to investigate this process yourself the instructions to do this can be found in this article below the video.

The see the traffic in action you can perform a packet capture on the destination ESXi-host. Make sure that you know on which uplink the virtual machine will be connected. To simplify the process when I recorded the video I made sure that the port group used was only using one active physical adapter. Also I used the --ethtype parameter for the pktcap-uw command to filter for RARP-packets. If there is a lot of traffic on the uplink you might otherwise be capturing too much other irrelevant traffic. The Ethernet protocol number (EtherType) for Reversed ARP is 0x8035. Also I am using the --dir 1 parameter to only capture outbound traffic.

On the destination ESXi host start the capture process with the following command:

pktcap-uw --uplink <vmnicX> --dir 1 --ethtype 0x8035 -o /tmp/<filename>

For example the command I used in the video was for vmnic5:

pktcap-uw --uplink vmnic5 --dir 1 --ethtype 0x8035 -o /tmp/rarp.pcap

Then migrate a virtual machine to the destination host. Packets will be captured and will be written into the output file. Download the file to your computer to analyze it with wireshark or investigate the packet capture file with the tcpdump-uw command directly on the ESXi-host.

tcpdump-uw -r /tmp/rarp.pcap

 

 

Tags