Backtrace on ARM has repeating entries
I experience a strange behaviour when creating a backtrace on an ARM platform running under Linux. Sometimes the backtrace output seems to be corrupted, depending on the code executed prior to the fault.
Here's my Crash.cpp code:
#include <cstdio>
#include <execinfo.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <ucontext.h>
const unsigned int CRASH_MAX_BACKTRACE_DEPTH = 30u;
void * crashData[CRASH_MAX_BACKTRACE_DEPTH] = { 0 };
void sigHandler(int signum, siginfo_t * siginfo, void * context)
{
int numFrames = backtrace(crashData, CRASH_MAX_BACKTRACE_DEPTH);
// restore faulting address
#if defined(__i386__)
crashData[2] = (void *)(((ucontext_t *)context)->uc_mcontext.gregs[REG_EIP]);
#elif defined(__arm__)
crashData[2] = (void *)(((ucontext_t *)context)->uc_mcontext.arm_pc);
#else
#error "Unsupported platform."
#endif
char ** symbols = backtrace_symbols(crashData, numFrames);
for (int i = 0; i < numFrames; i++)
{
printf("%d: %s\n", i, symbols[i]);
}
// resend the signal to the default handler in order to produce a core dump
(void) signal(signum, SIG_DFL);
(void) kill(syscall(__NR_gettid), signum);
}
void three()
{
char str[1];
printf("%s\n", "foo");
// produce SIGSEGV
int * p = 0;
*p = 1;
}
void two()
{
three();
}
void one()
{
two();
}
int main(int argc, char ** argv)
{
struct sigaction action;
sigemptyset(&action.sa_mask);
action.sa_sigaction = &sigHandler;
action.sa_flags = SA_SIGINFO; // we want the 3rd parameter of the handler to be the siginfo_t additional data
sigaction(SIGSEGV, &action, 0);
one();
return 0;
}
I cross-compiled it on a X86/Linux like that:
/path-to-cross-gcc/g++ -g3 -O0 Crash.cpp -o Crash -funwind-tables -rdynamic
When I run this, it gives me:
root@armbox:/# ./Crash
foo
0: ./Crash(_Z10sigHandleriP9siginfo_tPv+0x24) [0x8a3c]
1: /lib/libc.so.6(__default_rt_sa_restorer_v2+0) [0x76c56110]
2: ./Crash(_Z5threev+0x24) [0x8b1c]
3: ./Crash(_Z5threev+0x24) [0x8b1c]
4: ./Crash(_Z5threev+0x24) [0x8b1c]
5: ./Crash(_Z5threev+0x24) [0x8b1c]
6: ./Crash(_Z5threev+0x24) [0x8b1c]
7: ./Crash(_Z5threev+0x24) [0x8b1c]
8: ./Crash(_Z5threev+0x24) [0x8b1c]
9: ./Crash(_Z5threev+0x24) [0x8b1c]
10: ./Crash(_Z5threev+0x24) [0x8b1c]
11: ./Crash(_Z5threev+0x24) [0x8b1c]
12: ./Crash(_Z5threev+0x24) [0x8b1c]
13: ./Crash(_Z5threev+0x24) [0x8b1c]
14: ./Crash(_Z5threev+0x24) [0x8b1c]
15: ./Crash(_Z5threev+0x24) [0x8b1c]
16: ./Crash(_Z5threev+0x24) [0x8b1c]
17: ./Crash(_Z5threev+0x24) [0x8b1c]
18: ./Crash(_Z5threev+0x24) [0x8b1c]
19: ./Crash(_Z5threev+0x24) [0x8b1c]
20: ./Crash(_Z5threev+0x24) [0x8b1c]
21: ./Crash(_Z5threev+0x24) [0x8b1c]
22: ./Crash(_Z5threev+0x24) [0x8b1c]
23: ./Crash(_Z5threev+0x24) [0x8b1c]
24: ./Crash(_Z5threev+0x24) [0x8b1c]
25: ./Crash(_Z5threev+0x24) [0x8b1c]
26: ./Crash(_Z5threev+0x24) [0x8b1c]
27: ./Crash(_Z5threev+0x24) [0x8b1c]
28: ./Crash(_Z5threev+0x24) [0x8b1c]
29: ./Crash(_Z5threev+0x24) [0x8b1c]
Segmentation fault (core dumped)
The stack looks corrupted, from the 3rd frame on the faulting address is just repeated. However, if I make a core analysis, the stack seems to be fine:
root@armbox:/# gdb Crash Crash.core
GNU gdb (GDB) 7.4.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-oe-linux-gnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /jci/blackforest/Crash...done.
[New LWP 32356]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `./Crash'.
Program terminated with signal 11, Segmentation fault.
#0 0x00008b1c in three () at Crash.cpp:43
43 *p = 1;
(gdb) bt
#0 0x00008b1c in three () at Crash.cpp:43
#1 0x00008b40 in two () at Crash.cpp:48
#2 0x00008b50 in one () at Crash.cpp:53
#3 0x00008ba0 in main (argc=1, argv=0x7ef12b94) at Crash.cpp:64
Now here's the really interesting part. If I change the line char str[1]; in the faulting function three() to char str;, then the backtrace is flawless:
root@armbox:/# ./Crash
foo
0: ./Crash(_Z10sigHandleriP9siginfo_tPv+0x24) [0x8a3c]
1: /lib/libc.so.6(__default_rt_sa_restorer_v2+0) [0x76c3a110]
2: ./Crash(_Z5threev+0x24) [0x8b1c]
3: ./Crash(_Z3twov+0xc) [0x8b38]
4: ./Crash(_Z3onev+0xc) [0x8b48]
5: ./Crash(main+0x4c) [0x8b98]
6: /lib/libc.so.6(__libc_start_main+0x114) [0x76c23e1c]
Segmentation fault (core dumped)
The gdb core backtrace is the same. This is also true if I just remove the line printf("%s\n", "foo"); and let char str[1]; in place. For some reason the combination of the two is affecting the stack in a way that backtrace() fails to get a proper result. Interestingly enough this only happens on ARM/Linux, on X86/Linux it all works fine.
I know about the fact, that calling backtrace_symbols() and printf() in a signal handler is a very bad idea. However, this is just a showcase and I experienced the problem first within a software that uses backtrace_symbols_fd() to write the data into a file. So this should not be the issue here.
I'm using libc version 2.16. Any help would be greatly appreciated.
Edit: Here's the output of objdump -d Crash:
Crash: file format elf32-littlearm
Disassembly of section .init:
0000881c <_init>:
881c: e92d4008 push {r3, lr}
8820: eb00003e bl 8920 <call_weak_fn>
8824: e8bd8008 pop {r3, pc}
Disassembly of section .plt:
00008828 <.plt>:
8828: e52de004 push {lr} ; (str lr, [sp, #-4]!)
882c: e59fe004 ldr lr, [pc, #4] ; 8838 <_init+0x1c>
8830: e08fe00e add lr, pc, lr
8834: e5bef008 ldr pc, [lr, #8]!
8838: 000085a4 .word 0x000085a4
883c: e28fc600 add ip, pc, #0
8840: e28cca08 add ip, ip, #32768 ; 0x8000
8844: e5bcf5a4 ldr pc, [ip, #1444]! ; 0x5a4
8848: e28fc600 add ip, pc, #0
884c: e28cca08 add ip, ip, #32768 ; 0x8000
8850: e5bcf59c ldr pc, [ip, #1436]! ; 0x59c
8854: e28fc600 add ip, pc, #0
8858: e28cca08 add ip, ip, #32768 ; 0x8000
885c: e5bcf594 ldr pc, [ip, #1428]! ; 0x594
8860: e28fc600 add ip, pc, #0
8864: e28cca08 add ip, ip, #32768 ; 0x8000
8868: e5bcf58c ldr pc, [ip, #1420]! ; 0x58c
886c: e28fc600 add ip, pc, #0
8870: e28cca08 add ip, ip, #32768 ; 0x8000
8874: e5bcf584 ldr pc, [ip, #1412]! ; 0x584
8878: e28fc600 add ip, pc, #0
887c: e28cca08 add ip, ip, #32768 ; 0x8000
8880: e5bcf57c ldr pc, [ip, #1404]! ; 0x57c
8884: e28fc600 add ip, pc, #0
8888: e28cca08 add ip, ip, #32768 ; 0x8000
888c: e5bcf574 ldr pc, [ip, #1396]! ; 0x574
8890: e28fc600 add ip, pc, #0
8894: e28cca08 add ip, ip, #32768 ; 0x8000
8898: e5bcf56c ldr pc, [ip, #1388]! ; 0x56c
889c: e28fc600 add ip, pc, #0
88a0: e28cca08 add ip, ip, #32768 ; 0x8000
88a4: e5bcf564 ldr pc, [ip, #1380]! ; 0x564
88a8: e28fc600 add ip, pc, #0
88ac: e28cca08 add ip, ip, #32768 ; 0x8000
88b0: e5bcf55c ldr pc, [ip, #1372]! ; 0x55c
88b4: e28fc600 add ip, pc, #0
88b8: e28cca08 add ip, ip, #32768 ; 0x8000
88bc: e5bcf554 ldr pc, [ip, #1364]! ; 0x554
88c0: e28fc600 add ip, pc, #0
88c4: e28cca08 add ip, ip, #32768 ; 0x8000
88c8: e5bcf54c ldr pc, [ip, #1356]! ; 0x54c
88cc: e28fc600 add ip, pc, #0
88d0: e28cca08 add ip, ip, #32768 ; 0x8000
88d4: e5bcf544 ldr pc, [ip, #1348]! ; 0x544
88d8: e28fc600 add ip, pc, #0
88dc: e28cca08 add ip, ip, #32768 ; 0x8000
88e0: e5bcf53c ldr pc, [ip, #1340]! ; 0x53c
Disassembly of section .text:
000088e4 <_start>:
88e4: e3a0b000 mov fp, #0
88e8: e3a0e000 mov lr, #0
88ec: e49d1004 pop {r1} ; (ldr r1, [sp], #4)
88f0: e1a0200d mov r2, sp
88f4: e52d2004 push {r2} ; (str r2, [sp, #-4]!)
88f8: e52d0004 push {r0} ; (str r0, [sp, #-4]!)
88fc: e59fc010 ldr ip, [pc, #16] ; 8914 <_start+0x30>
8900: e52dc004 push {ip} ; (str ip, [sp, #-4]!)
8904: e59f000c ldr r0, [pc, #12] ; 8918 <_start+0x34>
8908: e59f300c ldr r3, [pc, #12] ; 891c <_start+0x38>
890c: ebffffd0 bl 8854 <_init+0x38>
8910: ebffffcc bl 8848 <_init+0x2c>
8914: 00008c20 .word 0x00008c20
8918: 00008b54 .word 0x00008b54
891c: 00008bbc .word 0x00008bbc
00008920 <call_weak_fn>:
8920: e59f3014 ldr r3, [pc, #20] ; 893c <call_weak_fn+0x1c>
8924: e59f2014 ldr r2, [pc, #20] ; 8940 <call_weak_fn+0x20>
8928: e08f3003 add r3, pc, r3
892c: e7932002 ldr r2, [r3, r2]
8930: e3520000 cmp r2, #0
8934: 012fff1e bxeq lr
8938: eaffffcb b 886c <_init+0x50>
893c: 000084ac .word 0x000084ac
8940: 00000044 .word 0x00000044
00008944 <deregister_tm_clones>:
8944: e92d4008 push {r3, lr}
8948: e59f0020 ldr r0, [pc, #32] ; 8970 <deregister_tm_clones+0x2c>
894c: e59f3020 ldr r3, [pc, #32] ; 8974 <deregister_tm_clones+0x30>
8950: e0603003 rsb r3, r0, r3
8954: e3530006 cmp r3, #6
8958: 98bd8008 popls {r3, pc}
895c: e59f3014 ldr r3, [pc, #20] ; 8978 <deregister_tm_clones+0x34>
8960: e3530000 cmp r3, #0
8964: 08bd8008 popeq {r3, pc}
8968: e12fff33 blx r3
896c: e8bd8008 pop {r3, pc}
8970: 00010e2c .word 0x00010e2c
8974: 00010e2f .word 0x00010e2f
8978: 00000000 .word 0x00000000
0000897c <register_tm_clones>:
897c: e59f002c ldr r0, [pc, #44] ; 89b0 <register_tm_clones+0x34>
8980: e59f102c ldr r1, [pc, #44] ; 89b4 <register_tm_clones+0x38>
8984: e92d4008 push {r3, lr}
8988: e0601001 rsb r1, r0, r1
898c: e1a01141 asr r1, r1, #2
8990: e0811fa1 add r1, r1, r1, lsr #31
8994: e1b010c1 asrs r1, r1, #1
8998: 08bd8008 popeq {r3, pc}
899c: e59f3014 ldr r3, [pc, #20] ; 89b8 <register_tm_clones+0x3c>
89a0: e3530000 cmp r3, #0
89a4: 08bd8008 popeq {r3, pc}
89a8: e12fff33 blx r3
89ac: e8bd8008 pop {r3, pc}
89b0: 00010e2c .word 0x00010e2c
89b4: 00010e2c .word 0x00010e2c
89b8: 00000000 .word 0x00000000
000089bc <__do_global_dtors_aux>:
89bc: e92d4010 push {r4, lr}
89c0: e59f4018 ldr r4, [pc, #24] ; 89e0 <__do_global_dtors_aux+0x24>
89c4: e5d43000 ldrb r3, [r4]
89c8: e3530000 cmp r3, #0
89cc: 18bd8010 popne {r4, pc}
89d0: ebffffdb bl 8944 <deregister_tm_clones>
89d4: e3a03001 mov r3, #1
89d8: e5c43000 strb r3, [r4]
89dc: e8bd8010 pop {r4, pc}
89e0: 00010e2c .word 0x00010e2c
000089e4 <frame_dummy>:
89e4: e59f0024 ldr r0, [pc, #36] ; 8a10 <frame_dummy+0x2c>
89e8: e92d4008 push {r3, lr}
89ec: e5903000 ldr r3, [r0]
89f0: e3530000 cmp r3, #0
89f4: 0a000003 beq 8a08 <frame_dummy+0x24>
89f8: e59f3014 ldr r3, [pc, #20] ; 8a14 <frame_dummy+0x30>
89fc: e3530000 cmp r3, #0
8a00: 0a000000 beq 8a08 <frame_dummy+0x24>
8a04: e12fff33 blx r3
8a08: e8bd4008 pop {r3, lr}
8a0c: eaffffda b 897c <register_tm_clones>
8a10: 00010cd8 .word 0x00010cd8
8a14: 00000000 .word 0x00000000
00008a18 <_Z10sigHandleriP9siginfo_tPv>:
8a18: e92d4800 push {fp, lr}
8a1c: e28db004 add fp, sp, #4
8a20: e24dd020 sub sp, sp, #32
8a24: e50b0018 str r0, [fp, #-24]
8a28: e50b101c str r1, [fp, #-28]
8a2c: e50b2020 str r2, [fp, #-32]
8a30: e59f00b8 ldr r0, [pc, #184] ; 8af0 <_Z10sigHandleriP9siginfo_tPv+0xd8>
8a34: e3a0101e mov r1, #30
8a38: ebffff94 bl 8890 <_init+0x74>
8a3c: e1a03000 mov r3, r0
8a40: e50b300c str r3, [fp, #-12]
8a44: e51b3020 ldr r3, [fp, #-32]
8a48: e593305c ldr r3, [r3, #92] ; 0x5c
8a4c: e1a02003 mov r2, r3
8a50: e59f3098 ldr r3, [pc, #152] ; 8af0 <_Z10sigHandleriP9siginfo_tPv+0xd8>
8a54: e5832008 str r2, [r3, #8]
8a58: e59f0090 ldr r0, [pc, #144] ; 8af0 <_Z10sigHandleriP9siginfo_tPv+0xd8>
8a5c: e51b100c ldr r1, [fp, #-12]
8a60: ebffff87 bl 8884 <_init+0x68>
8a64: e50b0010 str r0, [fp, #-16]
8a68: e3a03000 mov r3, #0
8a6c: e50b3008 str r3, [fp, #-8]
8a70: ea00000b b 8aa4 <_Z10sigHandleriP9siginfo_tPv+0x8c>
8a74: e51b3008 ldr r3, [fp, #-8]
8a78: e1a03103 lsl r3, r3, #2
8a7c: e51b2010 ldr r2, [fp, #-16]
8a80: e0823003 add r3, r2, r3
8a84: e5933000 ldr r3, [r3]
8a88: e59f0064 ldr r0, [pc, #100] ; 8af4 <_Z10sigHandleriP9siginfo_tPv+0xdc>
8a8c: e51b1008 ldr r1, [fp, #-8]
8a90: e1a02003 mov r2, r3
8a94: ebffff80 bl 889c <_init+0x80>
8a98: e51b3008 ldr r3, [fp, #-8]
8a9c: e2833001 add r3, r3, #1
8aa0: e50b3008 str r3, [fp, #-8]
8aa4: e51b2008 ldr r2, [fp, #-8]
8aa8: e51b300c ldr r3, [fp, #-12]
8aac: e1520003 cmp r2, r3
8ab0: a3a03000 movge r3, #0
8ab4: b3a03001 movlt r3, #1
8ab8: e20330ff and r3, r3, #255 ; 0xff
8abc: e3530000 cmp r3, #0
8ac0: 1affffeb bne 8a74 <_Z10sigHandleriP9siginfo_tPv+0x5c>
8ac4: e51b0018 ldr r0, [fp, #-24]
8ac8: e3a01000 mov r1, #0
8acc: ebffff63 bl 8860 <_init+0x44>
8ad0: e3a000e0 mov r0, #224 ; 0xe0
8ad4: ebffff7f bl 88d8 <_init+0xbc>
8ad8: e1a03000 mov r3, r0
8adc: e1a00003 mov r0, r3
8ae0: e51b1018 ldr r1, [fp, #-24]
8ae4: ebffff75 bl 88c0 <_init+0xa4>
8ae8: e24bd004 sub sp, fp, #4
8aec: e8bd8800 pop {fp, pc}
8af0: 00010e30 .word 0x00010e30
8af4: 00008c30 .word 0x00008c30
00008af8 <_Z5threev>:
8af8: e92d4800 push {fp, lr}
8afc: e28db004 add fp, sp, #4
8b00: e24dd008 sub sp, sp, #8
8b04: e59f0024 ldr r0, [pc, #36] ; 8b30 <_Z5threev+0x38>
8b08: ebffff69 bl 88b4 <_init+0x98>
8b0c: e3a03000 mov r3, #0
8b10: e50b3008 str r3, [fp, #-8]
8b14: e51b3008 ldr r3, [fp, #-8]
8b18: e3a02001 mov r2, #1
8b1c: e5832000 str r2, [r3]
8b20: ea000000 b 8b28 <_Z5threev+0x30>
8b24: ebffff53 bl 8878 <_init+0x5c>
8b28: e24bd004 sub sp, fp, #4
8b2c: e8bd8800 pop {fp, pc}
8b30: 00008c38 .word 0x00008c38
00008b34 <_Z3twov>:
8b34: e92d4800 push {fp, lr}
8b38: e28db004 add fp, sp, #4
8b3c: ebffffed bl 8af8 <_Z5threev>
8b40: e8bd8800 pop {fp, pc}
00008b44 <_Z3onev>:
8b44: e92d4800 push {fp, lr}
8b48: e28db004 add fp, sp, #4
8b4c: ebfffff8 bl 8b34 <_Z3twov>
8b50: e8bd8800 pop {fp, pc}
00008b54 <main>:
8b54: e92d4800 push {fp, lr}
8b58: e28db004 add fp, sp, #4
8b5c: e24dd098 sub sp, sp, #152 ; 0x98
8b60: e50b0098 str r0, [fp, #-152] ; 0x98
8b64: e50b109c str r1, [fp, #-156] ; 0x9c
8b68: e24b3090 sub r3, fp, #144 ; 0x90
8b6c: e2833004 add r3, r3, #4
8b70: e1a00003 mov r0, r3
8b74: ebffff30 bl 883c <_init+0x20>
8b78: e59f3038 ldr r3, [pc, #56] ; 8bb8 <main+0x64>
8b7c: e50b3090 str r3, [fp, #-144] ; 0x90
8b80: e3a03004 mov r3, #4
8b84: e50b300c str r3, [fp, #-12]
8b88: e24b3090 sub r3, fp, #144 ; 0x90
8b8c: e3a0000b mov r0, #11
8b90: e1a01003 mov r1, r3
8b94: e3a02000 mov r2, #0
8b98: ebffff42 bl 88a8 <_init+0x8c>
8b9c: ebffffe8 bl 8b44 <_Z3onev>
8ba0: e3a03000 mov r3, #0
8ba4: ea000000 b 8bac <main+0x58>
8ba8: ebffff32 bl 8878 <_init+0x5c>
8bac: e1a00003 mov r0, r3
8bb0: e24bd004 sub sp, fp, #4
8bb4: e8bd8800 pop {fp, pc}
8bb8: 00008a18 .word 0x00008a18
00008bbc <__libc_csu_init>:
8bbc: e92d45f8 push {r3, r4, r5, r6, r7, r8, sl, lr}
8bc0: e1a07000 mov r7, r0
8bc4: e59f604c ldr r6, [pc, #76] ; 8c18 <__libc_csu_init+0x5c>
8bc8: e1a08001 mov r8, r1
8bcc: e59f5048 ldr r5, [pc, #72] ; 8c1c <__libc_csu_init+0x60>
8bd0: e1a0a002 mov sl, r2
8bd4: e08f6006 add r6, pc, r6
8bd8: ebffff0f bl 881c <_init>
8bdc: e08f5005 add r5, pc, r5
8be0: e0656006 rsb r6, r5, r6
8be4: e1b06146 asrs r6, r6, #2
8be8: 08bd85f8 popeq {r3, r4, r5, r6, r7, r8, sl, pc}
8bec: e2455004 sub r5, r5, #4
8bf0: e3a04000 mov r4, #0
8bf4: e2844001 add r4, r4, #1
8bf8: e5b53004 ldr r3, [r5, #4]!
8bfc: e1a00007 mov r0, r7
8c00: e1a01008 mov r1, r8
8c04: e1a0200a mov r2, sl
8c08: e12fff33 blx r3
8c0c: e1540006 cmp r4, r6
8c10: 1afffff7 bne 8bf4 <__libc_csu_init+0x38>
8c14: e8bd85f8 pop {r3, r4, r5, r6, r7, r8, sl, pc}
8c18: 000080f8 .word 0x000080f8
8c1c: 000080ec .word 0x000080ec
00008c20 <__libc_csu_fini>:
8c20: e12fff1e bx lr
Disassembly of section .fini:
00008c24 <_fini>:
8c24: e92d4008 push {r3, lr}
8c28: e8bd8008 pop {r3, pc}
bselu1 Answer
Looks like this may have been reported and fixed here. I don't see a workaround other than recompiling libstdc++ with the recent patch. I observe the same behaviour with gcc 4.6.1 and libstdc++ 3.4.16.
answered on Stack Overflow Sep 30, 2014 by 
Dave McMordie
Dave McMordieUser contributions licensed under CC BY-SA 3.0