LVS Cluster Help


I'm trying to set up an LVS Cluster for a project and I don't have a lot of knowledge in these layers of networking. I've run into an issue where the real server rejects any packets from the director, but will allow packets to the same port from my client. I'm using suse linux on a kernel.

director script:

#set ip_forward OFF for lvs-dr director (1 on, 0 off)
#(there is no forwarding in the conventional sense for LVS-DR)
cat       /proc/sys/net/ipv4/ip_forward
echo "0" >/proc/sys/net/ipv4/ip_forward

#director is not gw for realservers: leave icmp redirects on
echo 'setting icmp redirects (1 on, 0 off) '
echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects
cat       /proc/sys/net/ipv4/conf/all/send_redirects
echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects
cat       /proc/sys/net/ipv4/conf/default/send_redirects
echo "1" >/proc/sys/net/ipv4/conf/eth1/send_redirects
cat       /proc/sys/net/ipv4/conf/eth1/send_redirects

#add ethernet device and routing for VIP
/sbin/ifconfig eth1:110 broadcast netmask
/sbin/route add -host dev eth1:110
#listing ifconfig info for VIP
/sbin/ifconfig eth1:110

#check VIP is reachable from self (director)
/bin/ping -c 1
#listing routing info for VIP
/bin/netstat -rn

#clear ipvsadm table
/sbin/ipvsadm -C
#installing LVS services with ipvsadm
#add telnet to VIP with round robin scheduling
/sbin/ipvsadm -A -t -s rr

#forward telnet to realserver using direct routing with weight 1
#/sbin/ipvsadm -a -t -r -g -w 1
/sbin/ipvsadm -a -t -r -g -w 1
#check realserver reachable from director
ping -c 1

#forward telnet to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t -r -g -w 1
#check realserver reachable from director
ping -c 1

#displaying ipvsadm settings

#not installing a default gw for LVS_TYPE vs-dr

real server script

#installing default gw for vs-dr
#/sbin/route add default gw
/sbin/route add default gw
#showing routing table
/bin/netstat -rn
#checking if DEFAULT_GW is reachable
ping -c 1

#set_realserver_ip_forwarding to OFF (1 on, 0 off).
echo "0" >/proc/sys/net/ipv4/ip_forward
cat       /proc/sys/net/ipv4/ip_forward

#looking for DIP
ping -c 1

#looking for VIP (will be on director)
ping -c 1


/sbin/ifconfig lo broadcast netmask 0xffffffff up
#ifconfig output
/sbin/ifconfig lo:110
#installing route for VIP on device lo:110
/sbin/route add -host dev lo:110
#listing routing info for VIP
/bin/netstat -rn

#hiding interface lo:0, will not arp
echo "1" >/proc/sys/net/ipv4/conf/all/hidden
cat       /proc/sys/net/ipv4/conf/all/hidden
echo "1" >/proc/sys/net/ipv4/conf/lo/hidden
cat       /proc/sys/net/ipv4/conf/lo/hidden


The above arp fix does not run because /hidden does not exist. I modified /etc/sysctl1.conf to add:

net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

and ran sysctl -p

When I run through the VIP my packets using tcpdump on a real server looks like:

sudo /usr/sbin/tcpdump -n -i any port 8080
tcpdump: WARNING: Promiscuous mode not supported on the "any" device
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
14:37:53.617753 IP > S 3169956976:316995697                                                                              6(0) win 65535 
14:37:53.617851 IP > R 0:0(0) ack 316995697                                                                              7 win 0

When I run directly to the real server the packets are pushed:

14:58:23.595729 IP > S 2691152412:2691152412(0) win 65535 
14:58:23.595794 IP > S 3988256836:3988256836(0) ack 2691152413 win 5840 
14:58:23.602995 IP > . ack 1 win 65535
14:58:23.600264 IP > P 1:483(482) ack 1 win 65535
14:58:23.600275 IP > . ack 483 win 6432
14:58:23.605755 IP > P 1:175(174) ack 483 win 6432
14:58:23.622787 IP > P 483:969(486) ack 175 win 65361
14:58:23.619738 IP > P 175:349(174) ack 969 win 7504
14:58:23.674299 IP > P 969:1473(504) ack 349 win 65187
14:58:23.671410 IP > P 349:524(175) ack 1473 win 8576
14:58:23.852775 IP > . ack 524 win 65012
If there is more information needed to help debug this issue please let me know.

asked on Server Fault Jul 14, 2009 by Ann • edited Jul 14, 2009 by Ann

1 Answer


Posted my question on the LVS mailing list and the first response asked:

on the realserver is the httpd listening on

Turns out I had set up the LVS cluster correctly, but JBoss was not set up to handle the VIP.

answered on Server Fault Jul 15, 2009 by Ann

User contributions licensed under CC BY-SA 3.0