CONNMARK restore not working for UDP based traffic ( exposed vi K8s Nodeport )

I have a K8s cluster hosting a UDP based application that is exposed via nodeport with externaltraffic policy local.

The return traffic needs to routed back to two separate LBs which are deployed in DNAT mode. The default gateway of the worker nodes is local router, so the return traffic needs to be policy routed based on a 'match' on specific nodeport.

CNI is calico without any network policy or enforcement and Kube-proxy is iptables based.
K8s: v1.20.4
iptables: iptables v1.8.4 (legacy)
uname -r :5.4.0-66-generic
Ubuntu 20.04.2 LTS

I CAN successfully do this for a TCP app, where I can set CONNMARKs for a nodeport and restore the CONNMARK to packet marks in the "PREROUTING MANGLE Chain and table". The restore takes place from traffic that is returned from "calic+" interfaces.

TCP service related rules that WORK with CONNMARK and policy routing:

root@worker1:~# iptables -t mangle -S PREROUTING
-P PREROUTING ACCEPT
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -i ens3 -p tcp -m tcp --dport 30080 -j CONNMARK --set-xmark 0xe/0xffffffff
-A PREROUTING -i calic+ -m connmark --mark 0xe -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -i ens3 -p tcp -m tcp --dport 30090 -j CONNMARK --set-xmark 0xc/0xffffffff
-A PREROUTING -i calic+ -m connmark --mark 0xc -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

The mark are picked by policy routing

32763:  from all fwmark 0xe lookup 100
32764:  from all fwmark 0xc lookup 12

You do have to modify the calico PREROUTING MANGLE Chain from "ACCEPT" to "RETURN" so that I can get my rules to take affect. Calico provides a knob for this.

However same set up does not work for UDP. The "restore" rule does not get any hits. I do see the conntrack events and the mark.

-A PREROUTING -i ens3 -p udp -m udp --dport 30095 -j CONNMARK --set-xmark 0xd/0xffffffff
-A PREROUTING -i calic+ -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

$ sudo conntrack -E | grep 10.10.24.136
    [NEW] udp      17 30 src=10.10.24.136 dst=10.10.2.244 sport=45628 dport=30095 [UNREPLIED] src=10.244.64.71 dst=10.10.24.136 sport=5005 dport=45628 mark=13
 [UPDATE] udp      17 30 src=10.10.24.136 dst=10.10.2.244 sport=45628 dport=30095 src=10.244.64.71 dst=10.10.24.136 sport=5005 dport=45628 mark=13
[DESTROY] udp      17 src=10.10.24.136 dst=10.10.2.244 sport=45628 dport=30095 src=10.244.64.71 dst=10.10.24.136 sport=5005 dport=45628 mark=13

root@worker1:~# iptables -t mangle -L PREROUTING --line-numbers -v
Chain PREROUTING (policy ACCEPT 475 packets, 79663 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    7880K 3462M cali-PREROUTING  all  --  any    any     anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
2        2    86 CONNMARK   udp  --  ens3   any     anywhere             anywhere             udp dpt:30095 CONNMARK set 0xd
3        0     0 CONNMARK   all  --  calic+ any     anywhere             anywhere             connmark match  0xd CONNMARK restore

Ofcourse UDP does not send any ACKs, however the UDP app i am testing with sends a packet back to the client which can be seen in TCPDUMPs on the server and client without any policy routing and normal nodeport traffic.

The client tcpdump with just simple nodeport.

21:22:45.394428 IP 10.10.24.136.38116 > 10.10.2.244.30095: UDP, length 15 ---> client request to nodeport
21:22:45.396230 IP 10.10.2.244.30095 > 10.10.24.136.38116: UDP, length 15 ---> pod return to client

Appreciate any help.