Openstack instance lost internet access after ataching floating ip
Maybe someone has same problem
I have installed Openstack Victoria on two virtualmachines (1 controller node, 1 compute node) running ubuntu 20.04. Each node has two network interfaces, mgmt network and provider network. I have created private network and I have attached it to router. With this configuration I am able to access internet.
But, when I attach floating IP to my instance, it lost internet connectivity. I can access this instance from outside, but instance cannot access network gateway. I checked it with ip netns exec <qrouter-id> ping 8.8.8.8. It is working until I attach FIP.
I think that is a routing problem but I cannot find where? Do you guys have any ideas?
10.0.0.0/24 - mgmt network
10.0.2.0/24 - external (provider) network
Configuration of Linuxbridge:
root@compute1:/# grep -v "^#" /etc/neutron/plugins/ml2/linuxbridge_agent.ini | grep -v "^$"
[DEFAULT]
[agent]
extensions = qos
[linux_bridge]
physical_interface_mappings = provider:ens34
[network_log]
[securitygroup]
enable_security_group = true
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
[vxlan]
enable_vxlan = true
local_ip = 10.0.0.131
l2_population = true
Provider Network:
root@controller1:/# openstack subnet show provider
| Field | Value |
| allocation_pools | 10.0.2.50-10.0.2.150 |
| cidr | 10.0.2.0/24|
| created_at | 2021-02-22T16:17:20Z |
| description | |
| dns_nameservers | 8.8.8.8|
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 10.0.2.1|
| host_routes | |
| id | 7d07101a-4696-4ff8-88bc-fa4ffde1622f |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | provider |
| network_id | d65d17fe-9829-44d5-bf07-1abb70f9d523 |
| prefix_length | None |
| project_id | 957f142f850240b5801023369eace69a |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
Router:
root@controller1:/# openstack router show router1
| Field | Value |
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2021-02-22T16:17:51Z |
| description | |
| distributed | False |
| external_gateway_info | {"network_id": "d65d17fe-9829-44d5-bf07-1abb70f9d523", "external_fixed_ips": [{"subnet_id": "7d07101a-4696-4ff8-88bc-fa4ffde1622f", "ip_address": "10.0.2.51"}], "enable_snat": true} |
| flavor_id | None |
| ha | False |
| id | fa11f06e-906c-4ae9-8176-20fb74e1cacd |
| interfaces_info | [{"port_id": "67d37c5f-1250-45e7-a003-78493921b4d6", "ip_address": "172.16.1.1", "subnet_id": "b0762924-6c7a-453f-a9b8-788e15e5f0c0"}] |
| name | router1 |
| project_id | 957f142f850240b5801023369eace69a |
| revision_number | 4 |
| routes | |
| status | ACTIVE |
Network namespaces:
root@controller1:/# ip netns
qrouter-fa11f06e-906c-4ae9-8176-20fb74e1cacd (id: 3)
qdhcp-d65d17fe-9829-44d5-bf07-1abb70f9d523 (id: 0)
qdhcp-f6a245eb-001d-47b1-8af5-38178585fe87 (id: 6)
qdhcp-0fb79928-ae24-4d85-8c58-b1acb9c8c9d2 (id: 2)
qdhcp-0ab1f94c-1e06-485c-b024-548a927a5e36 (id: 1)
root@controller1:/# ip netns exec qrouter-fa11f06e-906c-4ae9-8176-20fb74e1cacd ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=128 time=11.7 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.679/11.679/11.679/0.000 ms
root@controller1:/# ip netns exec qrouter-fa11f06e-906c-4ae9-8176-20fb74e1cacd ip route
default via 10.0.2.1 dev qg-61a6ea6f-7e proto static
10.0.2.0/24 dev qg-61a6ea6f-7e proto kernel scope link src 10.0.2.51
172.16.1.0/24 dev qr-67d37c5f-12 proto kernel scope link src 172.16.1.1
So everything is working fine... And now i am attaching FIP
root@controller1:/# openstack floating ip list
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
| 8a3333a9-345d-4b2a-9d63-420f09e4c020 | 10.0.2.106| 172.16.1.236| edef7b03-25a9-43b4-9953-831539056ac3 | d65d17fe-9829-44d5-bf07-1abb70f9d523 | 957f142f850240b5801023369eace69a |
It is pingable from my local PC and i can access instance via SSH as well, but I cannot access internet from provider network:
root@controller1:/# ip netns exec qrouter-fa11f06e-906c-4ae9-8176-20fb74e1cacd ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2040ms
This is tcpdump from compute node:
root@compute1:/# tcpdump -i ens34 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens34, link-type EN10MB (Ethernet), capture size 262144 bytes
17:30:00.258697 IP 10.0.2.106 > 8.8.8.8: ICMP echo request, id 41872, seq 0, length 64
17:30:01.259844 IP 10.0.2.106 > 8.8.8.8: ICMP echo request, id 41872, seq 1, length 64
So packets are going through provider interface ens34. I think that is routing problem on compute node but I cannot find where it is.
EDIT
Interfaces in namespace
root@controller1:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: qr-67d37c5f-12@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether fa:16:3e:cb:0e:3a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.16.1.1/24 brd 172.16.1.255 scope global qr-67d37c5f-12
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fecb:e3a/64 scope link
valid_lft forever preferred_lft forever
3: qg-61a6ea6f-7e@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fa:16:3e:60:e9:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.0.2.51/24 brd 10.0.2.255 scope global qg-61a6ea6f-7e
valid_lft forever preferred_lft forever
inet 10.0.2.106/32 brd 10.0.2.106 scope global qg-61a6ea6f-7e
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe60:e9e4/64 scope link
valid_lft forever preferred_lft forever
Routing table in namespace
root@controller1:/# ip r
default via 10.0.2.1 dev qg-61a6ea6f-7e proto static
10.0.2.0/24 dev qg-61a6ea6f-7e proto kernel scope link src 10.0.2.51
172.16.1.0/24 dev qr-67d37c5f-12 proto kernel scope link src 172.16.1.1
iptables in namespace
root@controller1:/# iptables-save
# Generated by iptables-save v1.8.4 on Tue Feb 23 12:51:04 2021
*raw
:PREROUTING ACCEPT [105:8611]
:OUTPUT ACCEPT [65:6090]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
# Completed on Tue Feb 23 12:51:04 2021
# Generated by iptables-save v1.8.4 on Tue Feb 23 12:51:04 2021
*nat
:PREROUTING ACCEPT [8:1322]
:INPUT ACCEPT [18:1372]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 10.0.2.106/32 -j DNAT --to-destination 172.16.1.236
-A neutron-l3-agent-POSTROUTING ! -o qg-61a6ea6f-7e -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 10.0.2.106/32 -j DNAT --to-destination 172.16.1.236
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-float-snat -s 172.16.1.236/32 -j SNAT --to-source 10.0.2.106 --random-fully
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-61a6ea6f-7e -j SNAT --to-source 10.0.2.51 --random-fully
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 10.0.2.51 --random-fully
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed on Tue Feb 23 12:51:04 2021
# Generated by iptables-save v1.8.4 on Tue Feb 23 12:51:04 2021
*mangle
:PREROUTING ACCEPT [105:8611]
:INPUT ACCEPT [99:7701]
:FORWARD ACCEPT [2:102]
:OUTPUT ACCEPT [65:6090]
:POSTROUTING ACCEPT [67:6192]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A neutron-l3-agent-POSTROUTING -o qg-61a6ea6f-7e -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-mark
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-scope
-A neutron-l3-agent-PREROUTING -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-floatingip
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
-A neutron-l3-agent-float-snat -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-mark -i qg-61a6ea6f-7e -j MARK --set-xmark 0x2/0xffff
-A neutron-l3-agent-scope -i qr-67d37c5f-12 -j MARK --set-xmark 0x4000000/0xffff0000
-A neutron-l3-agent-scope -i qg-61a6ea6f-7e -j MARK --set-xmark 0x4000000/0xffff0000
COMMIT
# Completed on Tue Feb 23 12:51:04 2021
# Generated by iptables-save v1.8.4 on Tue Feb 23 12:51:04 2021
*filter
:INPUT ACCEPT [3:741]
:FORWARD ACCEPT [2:102]
:OUTPUT ACCEPT [65:6090]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-67d37c5f-12 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
COMMIT
# Completed on Tue Feb 23 12:51:04 2021
tcpdump from namespace
root@controller1:/# tcpdump -e -i any host 8.8.8.8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:56:31.517183 Out fa:16:3e:d9:60:22 (oui Unknown) ethertype IPv4 (0x0800), length 100: 10.0.2.131 > 8.8.8.8: ICMP echo request, id 58998, seq 14, length 64
13:56:31.524754 In 00:50:56:f5:43:f8 (oui Unknown) ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 10.0.2.131: ICMP echo reply, id 58998, seq 14, length 64
13:56:31.524803 Out fa:16:3e:e3:79:16 (oui Unknown) ethertype IPv4 (0x0800), length 100: 8.8.8.8 > 172.16.1.236: ICM P echo reply, id 34561, seq 14, length 64
13:56:32.518197 In fa:16:3e:6e:82:9e (oui Unknown) ethertype IPv4 (0x0800), length 100: 172.16.1.236 > 8.8.8.8: ICM P echo request, id 34561, seq 15, length 64
13:56:32.518237 Out fa:16:3e:d9:60:22 (oui Unknown) ethertype IPv4 (0x0800), length 100: 10.0.2.131 > 8.8.8.8: ICMP echo request, id 58998, seq 15, length 64
13:56:33.519420 In fa:16:3e:6e:82:9e (oui Unknown) ethertype IPv4 (0x0800), length 100: 172.16.1.236 > 8.8.8.8: ICM P echo request, id 34561, seq 16, length 64
13:56:33.519463 Out fa:16:3e:d9:60:22 (oui Unknown) ethertype IPv4 (0x0800), length 100: 10.0.2.131 > 8.8.8.8: ICMP echo request, id 58998, seq 16, length 64
13:56:34.520250 In fa:16:3e:6e:82:9e (oui Unknown) ethertype IPv4 (0x0800), length 100: 172.16.1.236 > 8.8.8.8: ICM P echo request, id 34561, seq 17, length 64
13:56:34.520291 Out fa:16:3e:d9:60:22 (oui Unknown) ethertype IPv4 (0x0800), length 100: 10.0.2.131 > 8.8.8.8: ICMP echo request, id 58998, seq 17, length 64
13:56:35.521179 In fa:16:3e:6e:82:9e (oui Unknown) ethertype IPv4 (0x0800), length 100: 172.16.1.236 > 8.8.8.8: ICM P echo request, id 34561, seq 18, length 64
13:56:35.521216 Out fa:16:3e:d9:60:22 (oui Unknown) ethertype IPv4 (0x0800), length 100: 10.0.2.131 > 8.8.8.8: ICMP echo request, id 58998, seq 18, length 64
13:56:36.522122 In fa:16:3e:6e:82:9e (oui Unknown) ethertype IPv4 (0x0800), length 100: 172.16.1.236 > 8.8.8.8: ICM P echo request, id 34561, seq 19, length 64
13:56:36.522158 Out fa:16:3e:d9:60:22 (oui Unknown) ethertype IPv4 (0x0800), length 100: 10.0.2.131 > 8.8.8.8: ICMP echo request, id 58998, seq 19, length 64
13:56:37.522683 In fa:16:3e:6e:82:9e (oui Unknown) ethertype IPv4 (0x0800), length 100: 172.16.1.236 > 8.8.8.8: ICM P echo request, id 34561, seq 20, length 64
twojold0 Answers
Nobody has answered this question yet.
User contributions licensed under CC BY-SA 3.0