Out of 2 VMs and their 1 host, only 2 at a time are reachable from the LAN

I have a VMware vSphere hypervisor set up with two VMs. I need to be able to reach the host (for configuration, etc.) as well as both VM's from my client (which is also in the same subnet). I want to set up a PoC Hadoop cluster, and the client will need to connect to all of the VM's.

Yesterday, I was able to ping the host and one of the VM's, but not the other one. Pinging the reachable VM from the unreachable one worked, as well as pinging the host (if I remember correctly).

Today, I am able to ping both VM's, but not the host, and I haven't changed anything.

All three have static IPs configured, but the one not reachable doesn't seem to be part of the network (maybe ARP requests don't reach it or something). Checking the ARP table on my client just confirms that the IP of the unreachable host has invalid entry. I tried deleting the invalid entry, even flushing the cache completely, but to no avail.

I checked that there was no MAC-address conflict, and found the VM's and the host all have different MACs. The hostnames are different, too. And the assigned IPs, obviously.

I thought of port protection of the router, but it doesn't seem to make sense as there are at least two IPs on the same port. Also, I can't easily check whether that is activated, as I don't have access to the router.

The basic neetwork configuration is:

VM Host: 172.16.40.11   ------ Gateway: 172.16.40.1 ----- Client: 172.16.40.22
   VM 1: 172.16.40.191   -|
   VM 2: 172.16.40.193   -|

And a short command line session on the client showing the host being unavailable:

C:\Users\>ping 172.16.40.11

Pinging 172.16.40.11 with 32 bytes of data:
Reply from 172.16.40.21: Destination host unreachable
Reply from 172.16.40.21: Destination host unreachable
Reply from 172.16.40.21: Destination host unreachable

Ping statistics for 172.16.40.11:
    Packets: Sent = 3, Received = 3, Lost = 0 (0% los
Control-C
^C
C:\Users\>arp -a -v

Interface: 127.0.0.1 --- 0x1
  Internet Address      Physical Address      Type
  224.0.0.22                                  static
  239.255.255.250                             static

Interface: 0.0.0.0 --- 0xffffffff
  Internet Address      Physical Address      Type
  224.0.0.22            01-00-5e-00-00-16     static

Interface: 172.16.40.21 --- 0xc
  Internet Address      Physical Address      Type
  172.16.40.1           00-22-56-f7-62-c1     static
  172.16.40.2           00-22-90-0d-18-90     dynamic
  172.16.40.11          00-00-00-00-00-00     invalid
  172.16.40.191         00-0c-29-5e-4f-c1     dynamic
  172.16.40.193         00-0c-29-53-fd-25     dynamic
  172.16.40.255         ff-ff-ff-ff-ff-ff     static
  224.0.0.22            01-00-5e-00-00-16     static
  224.0.0.252           01-00-5e-00-00-fc     static
  239.255.255.250       01-00-5e-7f-ff-fa     static
  255.255.255.255       ff-ff-ff-ff-ff-ff     static

Interface: 0.0.0.0 --- 0xffffffff
  Internet Address      Physical Address      Type
  224.0.0.22            01-00-5e-00-00-16     static

Interface: 192.168.199.1 --- 0x15
  Internet Address      Physical Address      Type
  192.168.199.255       ff-ff-ff-ff-ff-ff     static
  224.0.0.22            01-00-5e-00-00-16     static
  224.0.0.252           01-00-5e-00-00-fc     static
  239.255.255.250       01-00-5e-7f-ff-fa     static

Interface: 192.168.109.1 --- 0x16
  Internet Address      Physical Address      Type
  192.168.109.255       ff-ff-ff-ff-ff-ff     static
  224.0.0.22            01-00-5e-00-00-16     static
  224.0.0.252           01-00-5e-00-00-fc     static
  239.255.255.250       01-00-5e-7f-ff-fa     static

I restarted the host and started the VMs again, and I am back with the situation I had yesterday:

C:\Users\>ping 172.16.40.11

Pinging 172.16.40.11 with 32 bytes of data:
Reply from 172.16.40.11: bytes=32 time<1ms TTL=64
Reply from 172.16.40.11: bytes=32 time<1ms TTL=64

Ping statistics for 172.16.40.11:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
C:\Users\>ping 172.16.40.191

Pinging 172.16.40.191 with 32 bytes of data:
Reply from 172.16.40.191: bytes=32 time<1ms TTL=64
Reply from 172.16.40.191: bytes=32 time<1ms TTL=64

Ping statistics for 172.16.40.191:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
C:\Users\>ping 172.16.40.193

Pinging 172.16.40.193 with 32 bytes of data:
Reply from 172.16.40.21: Destination host unreachable.
Reply from 172.16.40.21: Destination host unreachable.
Reply from 172.16.40.21: Destination host unreachable.

Ping statistics for 172.16.40.193:
    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),

Meanwhile on VM1, I am able to connect to the host and VM2, and the gateway:

[hadoop@hadoop1 ~]$ ping 172.16.40.1
PING 172.16.40.1 (172.16.40.1) 56(84) bytes of data.
64 bytes from 172.16.40.1: icmp_seq=1 ttl=255 time=0.593 ms
64 bytes from 172.16.40.1: icmp_seq=2 ttl=255 time=0.591 ms
^C
--- 172.16.40.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1307ms
rtt min/avg/max/mdev = 0.591/0.592/0.593/0.001 ms
[hadoop@hadoop1 ~]$ ping 172.16.40.11
PING 172.16.40.11 (172.16.40.11) 56(84) bytes of data.
64 bytes from 172.16.40.11: icmp_seq=1 ttl=64 time=1.02 ms
64 bytes from 172.16.40.11: icmp_seq=2 ttl=64 time=0.109 ms
^C
--- 172.16.40.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1103ms
rtt min/avg/max/mdev = 0.109/0.567/1.025/0.458 ms
[hadoop@hadoop1 ~]$ ping 172.16.40.191
PING 172.16.40.191 (172.16.40.191) 56(84) bytes of data.
64 bytes from 172.16.40.191: icmp_seq=1 ttl=64 time=0.033 ms
64 bytes from 172.16.40.191: icmp_seq=2 ttl=64 time=0.018 ms
^C
--- 172.16.40.191 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1194ms
rtt min/avg/max/mdev = 0.018/0.025/0.033/0.009 ms
[hadoop@hadoop1 ~]$ ping 172.16.40.193
PING 172.16.40.193 (172.16.40.193) 56(84) bytes of data.
64 bytes from 172.16.40.193: icmp_seq=1 ttl=64 time=1.98 ms
64 bytes from 172.16.40.193: icmp_seq=2 ttl=64 time=0.134 ms
^C
--- 172.16.40.193 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1587ms
rtt min/avg/max/mdev = 0.134/1.058/1.982/0.924 ms

But on VM2, I am only able to ping the host and VM1, not the gateway:

[hadoop@hadoop3 ~]$ ping 172.16.40.1
PING 172.16.40.1 (172.16.40.1) 56(84) bytes of data.
^C
--- 172.16.40.1 ping statistics ---
25 packets transmitted, 0 received, 100% packet loss, time 24652ms

[hadoop@hadoop3 ~]$ ping 172.16.40.11
PING 172.16.40.11 (172.16.40.11) 56(84) bytes of data.
64 bytes from 172.16.40.11: icmp_seq=1 ttl=64 time=1.17 ms
64 bytes from 172.16.40.11: icmp_seq=2 ttl=64 time=0.082 ms
^C
--- 172.16.40.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1272ms
rtt min/avg/max/mdev = 0.082/0.630/1.178/0.548 ms
[hadoop@hadoop3 ~]$ ping 172.16.40.191
PING 172.16.40.191 (172.16.40.191) 56(84) bytes of data.
64 bytes from 172.16.40.191: icmp_seq=1 ttl=64 time=0.187 ms
64 bytes from 172.16.40.191: icmp_seq=2 ttl=64 time=0.170 ms
^C
--- 172.16.40.191 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1181ms
rtt min/avg/max/mdev = 0.170/0.178/0.187/0.015 ms
[hadoop@hadoop3 ~]$ ping 172.16.40.193
PING 172.16.40.193 (172.16.40.193) 56(84) bytes of data.
64 bytes from 172.16.40.193: icmp_seq=1 ttl=64 time=0.021 ms
64 bytes from 172.16.40.193: icmp_seq=2 ttl=64 time=0.019 ms
^C
--- 172.16.40.193 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1076ms
rtt min/avg/max/mdev = 0.019/0.020/0.021/0.001 ms

And last but not least, the host can also ping both VMs and the gateway:

~ # ping 172.16.40.1
PING 172.16.40.1 (172.16.40.1): 56 data bytes
64 bytes from 172.16.40.1: icmp_seq=0 ttl=255 time=0.545 ms
64 bytes from 172.16.40.1: icmp_seq=1 ttl=255 time=3.338 ms

--- 172.16.40.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.545/1.942/3.338 ms
~ # ping 172.16.40.11
PING 172.16.40.11 (172.16.40.11): 56 data bytes
64 bytes from 172.16.40.11: icmp_seq=0 ttl=64 time=0.055 ms

--- 172.16.40.11 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.055/0.055/0.055 ms
~ # ping 172.16.40.191
PING 172.16.40.191 (172.16.40.191): 56 data bytes
64 bytes from 172.16.40.191: icmp_seq=0 ttl=64 time=0.131 ms
64 bytes from 172.16.40.191: icmp_seq=1 ttl=64 time=0.124 ms

--- 172.16.40.191 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.124/0.128/0.131 ms
~ # ping 172.16.40.193
PING 172.16.40.193 (172.16.40.193): 56 data bytes
64 bytes from 172.16.40.193: icmp_seq=0 ttl=64 time=0.205 ms
64 bytes from 172.16.40.193: icmp_seq=1 ttl=64 time=0.128 ms

--- 172.16.40.193 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.128/0.166/0.205 ms

A screenshot of the network configuration in vSphere (hadoop1 and hadoop3 are the two VMs):

What is going on here? How can I further diagnose what is happening?

I suspect that for some reason only two IP's are allowed to come from the network interface, but what could be causing this and how can I fix it?

I would check the port configuration of the connecting switch to see whether there is a MAC address limit--some companies will enforce to allow only a certain number of MAC addresses per port.