Segmentation fault using pthread and arm-linux-gnueabihf-gcc
I have been trying all day to fix a segmentation fault that only occurs on one device. But from the beginning...
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include "pthread.h"
void* start_rtn(void* param);
int main(int argc, char **argv)
{
printf("Hallo World\n");
pthread_t tid;
int err = pthread_create(&tid, NULL, start_rtn, NULL);
pthread_join(tid, NULL);
printf("Exit\n");
}
void* start_rtn(void* param)
{
printf("Thread test\n");
}
And here are the relevant parts of my CMakeList.txt:
SET(CMAKE_SYSTEM_PROCESSOR arm)
SET(CMAKE_C_COMPILER /usr/bin/arm-linux-gnueabihf-gcc)
file(GLOB SOURCES "main.c")
add_executable(test ${SOURCES})
target_link_libraries(test -pthread)
This test application works on all my ARM devices except one. The one ARM device immediately delivers a segmentation fault. When I run gdb the following output appears:
warning: core file may not match specified executable file.
[New LWP 10825]
warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `./test'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000 in ?? ()
But... If I compile the same application on my Raspberry Pi, it is working even on the special device.
If I remove the call to pthread_create it is working no matter on which device it was compiled.
So there seems to be a problem with my cross compiler, but this is only noticeable on the specific device and only if pthread methods are used.
I am helpless and grateful for every hint
This is ldd of the none working executable:
libpthread.so.0 => /lib/libpthread.so.0 (0xb6f2e000)
libc.so.6 => /lib/libc.so.6 (0xb6e02000)
ld-linux-armhf.so.3 => /lib/ld-linux-armhf.so.3 (0xb6dda000)
/lib/ld-linux-armhf.so.3 (0xb6f4d000)
Version information:
./testapp:
libpthread.so.0 (GLIBC_2.4) => /lib/libpthread.so.0
ld-linux-armhf.so.3 (GLIBC_2.4) => /lib/ld-linux-armhf.so.3
libc.so.6 (GLIBC_2.4) => /lib/libc.so.6
/lib/libpthread.so.0:
ld-linux.so.3 (GLIBC_2.4) => /lib/ld-linux-armhf.so.3
ld-linux.so.3 (GLIBC_PRIVATE) => /lib/ld-linux-armhf.so.3
libc.so.6 (GLIBC_PRIVATE) => /lib/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib/libc.so.6
/lib/libc.so.6:
ld-linux.so.3 (GLIBC_2.4) => /lib/ld-linux-armhf.so.3
ld-linux.so.3 (GLIBC_PRIVATE) => /lib/ld-linux-armhf.so.3
And this is ldd output from the working executable (compiled by raspberry pi)
libpthread.so.0 => /lib/libpthread.so.0 (0xb6ee1000)
libc.so.6 => /lib/libc.so.6 (0xb6db5000)
/lib/ld-linux-armhf.so.3 (0xb6f00000)
Version information:
./testapp_working:
libc.so.6 (GLIBC_2.4) => /lib/libc.so.6
libpthread.so.0 (GLIBC_2.4) => /lib/libpthread.so.0
/lib/libpthread.so.0:
ld-linux.so.3 (GLIBC_2.4) => /lib/ld-linux-armhf.so.3
ld-linux.so.3 (GLIBC_PRIVATE) => /lib/ld-linux-armhf.so.3
libc.so.6 (GLIBC_PRIVATE) => /lib/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib/libc.so.6
/lib/libc.so.6:
ld-linux.so.3 (GLIBC_2.4) => /lib/ld-linux-armhf.so.3
ld-linux.so.3 (GLIBC_PRIVATE) => /lib/ld-linux-armhf.so.3
P.S.: This thread seems to be very equal. I have already made sure that all gcc parameters of my environment and the Raspberry-Pi are identical.
Edit - Compiler command requested by Erlkoenig:
/usr/bin/arm-linux-gnueabihf-gcc -marm -march=armv6 -mfpu=vfp -mtp=soft -pthread -g2 -gdwarf-2 -DDEBUG -D LINUX -D _GNU_SOURCE -D _DEBUG -std=gnu11 -o CMakeFiles/testapp.dir/main.c.o -c main.c
/usr/bin/arm-linux-gnueabihf-gcc -marm -march=armv6 -mfpu=vfp -mtp=soft -pthread -g2 -gdwarf-2 -DDEBUG CMakeFiles/testapp.dir/main.c.o -o ARM/Debug/testapp -pthread
Edit - Additional gdb output requested by Erlkoenig:
Core was generated by `./testapp'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000 in ?? ()
(gdb) disas
No function contains program counter for selected frame.
(gdb) bt
#0 0x00000000 in ?? ()
#1 0xb6ecb5b0 in __pthread_initialize_minimal () from /lib/libpthread.so.0
#2 0xb6ec9ba0 in ?? () from /lib/libpthread.so.0
Cannot access memory at address 0x0
#3 0xb6ec9ba0 in ?? () from /lib/libpthread.so.0
Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info reg
r0 0xb6f08504 3069216004
r1 0xb6ee41e0 3069067744
r2 0x800fff 8392703
r3 0x0 0
r4 0xb6f08504 3069216004
r5 0xb6ee2000 3069059072
r6 0xb6d99050 3067711568
r7 0xaf 175
r8 0x0 0
r9 0xb6ee4198 3069067672
r10 0xb6ecb2e0 3068965600
r11 0x0 0
r12 0xb6e979e0 3068754400
sp 0xbe9bec30 0xbe9bec30
lr 0xb6ecb5b0 -1226000976
pc 0x0 0x0
cpsr 0x60000010 1610612752
(gdb) info thread
Id Target Id Frame
* 1 LWP 24127 0x00000000 in ?? ()
(gdb)
Edit - exakt CPU info requested by Erlkoenig:
processor : 0
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 1594.16
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part : 0xc09
CPU revision : 0
Hardware : Altera SOCFPGA
Edit - objdump output requested by Erlkoenig:
test: file format elf32-littlearm
Disassembly of section .init:
000004f4 <_init>:
4f4: e92d4008 push {r3, lr}
4f8: eb000035 bl 5d4 <call_weak_fn>
4fc: e8bd8008 pop {r3, pc}
Disassembly of section .plt:
00000500 <.plt>:
500: e52de004 push {lr} ; (str lr, [sp, #-4]!)
504: e59fe004 ldr lr, [pc, #4] ; 510 <.plt+0x10>
508: e08fe00e add lr, pc, lr
50c: e5bef008 ldr pc, [lr, #8]!
510: 00010a9c .word 0x00010a9c
00000514 <__cxa_finalize@plt>:
514: e28fc600 add ip, pc, #0, 12
518: e28cca10 add ip, ip, #16, 20 ; 0x10000
51c: e5bcfa9c ldr pc, [ip, #2716]! ; 0xa9c
00000520 <printf@plt>:
520: e28fc600 add ip, pc, #0, 12
524: e28cca10 add ip, ip, #16, 20 ; 0x10000
528: e5bcfa94 ldr pc, [ip, #2708]! ; 0xa94
0000052c <pthread_self@plt>:
52c: e28fc600 add ip, pc, #0, 12
530: e28cca10 add ip, ip, #16, 20 ; 0x10000
534: e5bcfa8c ldr pc, [ip, #2700]! ; 0xa8c
00000538 <__stack_chk_fail@plt>:
538: e28fc600 add ip, pc, #0, 12
53c: e28cca10 add ip, ip, #16, 20 ; 0x10000
540: e5bcfa84 ldr pc, [ip, #2692]! ; 0xa84
00000544 <pthread_create@plt>:
544: e28fc600 add ip, pc, #0, 12
548: e28cca10 add ip, ip, #16, 20 ; 0x10000
54c: e5bcfa7c ldr pc, [ip, #2684]! ; 0xa7c
00000550 <puts@plt>:
550: e28fc600 add ip, pc, #0, 12
554: e28cca10 add ip, ip, #16, 20 ; 0x10000
558: e5bcfa74 ldr pc, [ip, #2676]! ; 0xa74
0000055c <__libc_start_main@plt>:
55c: e28fc600 add ip, pc, #0, 12
560: e28cca10 add ip, ip, #16, 20 ; 0x10000
564: e5bcfa6c ldr pc, [ip, #2668]! ; 0xa6c
00000568 <__gmon_start__@plt>:
568: e28fc600 add ip, pc, #0, 12
56c: e28cca10 add ip, ip, #16, 20 ; 0x10000
570: e5bcfa64 ldr pc, [ip, #2660]! ; 0xa64
00000574 <pthread_join@plt>:
574: e28fc600 add ip, pc, #0, 12
578: e28cca10 add ip, ip, #16, 20 ; 0x10000
57c: e5bcfa5c ldr pc, [ip, #2652]! ; 0xa5c
00000580 <abort@plt>:
580: e28fc600 add ip, pc, #0, 12
584: e28cca10 add ip, ip, #16, 20 ; 0x10000
588: e5bcfa54 ldr pc, [ip, #2644]! ; 0xa54
Disassembly of section .text:
0000058c <_start>:
58c: f04f 0b00 mov.w fp, #0
590: f04f 0e00 mov.w lr, #0
594: bc02 pop {r1}
596: 466a mov r2, sp
598: b404 push {r2}
59a: b401 push {r0}
59c: f8df a024 ldr.w sl, [pc, #36] ; 5c4 <_start+0x38>
5a0: a308 add r3, pc, #32 ; (adr r3, 5c4 <_start+0x38>)
5a2: 449a add sl, r3
5a4: f8df c020 ldr.w ip, [pc, #32] ; 5c8 <_start+0x3c>
5a8: f85a c00c ldr.w ip, [sl, ip]
5ac: f84d cd04 str.w ip, [sp, #-4]!
5b0: 4b06 ldr r3, [pc, #24] ; (5cc <_start+0x40>)
5b2: f85a 3003 ldr.w r3, [sl, r3]
5b6: 4806 ldr r0, [pc, #24] ; (5d0 <_start+0x44>)
5b8: f85a 0000 ldr.w r0, [sl, r0]
5bc: f7ff efce blx 55c <__libc_start_main@plt>
5c0: f7ff efde blx 580 <abort@plt>
5c4: 000109e8 .word 0x000109e8
5c8: 00000034 .word 0x00000034
5cc: 00000048 .word 0x00000048
5d0: 0000004c .word 0x0000004c
000005d4 <call_weak_fn>:
5d4: e59f3014 ldr r3, [pc, #20] ; 5f0 <call_weak_fn+0x1c>
5d8: e59f2014 ldr r2, [pc, #20] ; 5f4 <call_weak_fn+0x20>
5dc: e08f3003 add r3, pc, r3
5e0: e7932002 ldr r2, [r3, r2]
5e4: e3520000 cmp r2, #0
5e8: 012fff1e bxeq lr
5ec: eaffffdd b 568 <__gmon_start__@plt>
5f0: 000109c8 .word 0x000109c8
5f4: 00000044 .word 0x00000044
000005f8 <deregister_tm_clones>:
5f8: 4806 ldr r0, [pc, #24] ; (614 <deregister_tm_clones+0x1c>)
5fa: 4b07 ldr r3, [pc, #28] ; (618 <deregister_tm_clones+0x20>)
5fc: 4478 add r0, pc
5fe: 4a07 ldr r2, [pc, #28] ; (61c <deregister_tm_clones+0x24>)
600: 447b add r3, pc
602: 4283 cmp r3, r0
604: 447a add r2, pc
606: d003 beq.n 610 <deregister_tm_clones+0x18>
608: 4b05 ldr r3, [pc, #20] ; (620 <deregister_tm_clones+0x28>)
60a: 58d3 ldr r3, [r2, r3]
60c: b103 cbz r3, 610 <deregister_tm_clones+0x18>
60e: 4718 bx r3
610: 4770 bx lr
612: bf00 nop
614: 00010a08 .word 0x00010a08
618: 00010a04 .word 0x00010a04
61c: 000109a4 .word 0x000109a4
620: 0000003c .word 0x0000003c
00000624 <register_tm_clones>:
624: 4808 ldr r0, [pc, #32] ; (648 <register_tm_clones+0x24>)
626: 4909 ldr r1, [pc, #36] ; (64c <register_tm_clones+0x28>)
628: 4478 add r0, pc
62a: 4a09 ldr r2, [pc, #36] ; (650 <register_tm_clones+0x2c>)
62c: 4479 add r1, pc
62e: 1a09 subs r1, r1, r0
630: 447a add r2, pc
632: 0fcb lsrs r3, r1, #31
634: eb03 01a1 add.w r1, r3, r1, asr #2
638: 1049 asrs r1, r1, #1
63a: d003 beq.n 644 <register_tm_clones+0x20>
63c: 4b05 ldr r3, [pc, #20] ; (654 <register_tm_clones+0x30>)
63e: 58d3 ldr r3, [r2, r3]
640: b103 cbz r3, 644 <register_tm_clones+0x20>
642: 4718 bx r3
644: 4770 bx lr
646: bf00 nop
648: 000109dc .word 0x000109dc
64c: 000109d8 .word 0x000109d8
650: 00010978 .word 0x00010978
654: 00000050 .word 0x00000050
00000658 <__do_global_dtors_aux>:
658: b508 push {r3, lr}
65a: 4b0a ldr r3, [pc, #40] ; (684 <__do_global_dtors_aux+0x2c>)
65c: 4a0a ldr r2, [pc, #40] ; (688 <__do_global_dtors_aux+0x30>)
65e: 447b add r3, pc
660: 447a add r2, pc
662: 781b ldrb r3, [r3, #0]
664: b96b cbnz r3, 682 <__do_global_dtors_aux+0x2a>
666: 4b09 ldr r3, [pc, #36] ; (68c <__do_global_dtors_aux+0x34>)
668: 58d3 ldr r3, [r2, r3]
66a: b123 cbz r3, 676 <__do_global_dtors_aux+0x1e>
66c: 4b08 ldr r3, [pc, #32] ; (690 <__do_global_dtors_aux+0x38>)
66e: 447b add r3, pc
670: 6818 ldr r0, [r3, #0]
672: f7ff ef50 blx 514 <__cxa_finalize@plt>
676: f7ff ffbf bl 5f8 <deregister_tm_clones>
67a: 4b06 ldr r3, [pc, #24] ; (694 <__do_global_dtors_aux+0x3c>)
67c: 2201 movs r2, #1
67e: 447b add r3, pc
680: 701a strb r2, [r3, #0]
682: bd08 pop {r3, pc}
684: 000109a6 .word 0x000109a6
688: 00010948 .word 0x00010948
68c: 00000038 .word 0x00000038
690: 00010992 .word 0x00010992
694: 00010986 .word 0x00010986
00000698 <frame_dummy>:
698: e7c4 b.n 624 <register_tm_clones>
69a: bf00 nop
0000069c <main>:
69c: e92d4800 push {fp, lr}
6a0: e28db004 add fp, sp, #4
6a4: e24dd018 sub sp, sp, #24
6a8: e50b0018 str r0, [fp, #-24] ; 0xffffffe8
6ac: e50b101c str r1, [fp, #-28] ; 0xffffffe4
6b0: e59f20b0 ldr r2, [pc, #176] ; 768 <main+0xcc>
6b4: e08f2002 add r2, pc, r2
6b8: e59f30ac ldr r3, [pc, #172] ; 76c <main+0xd0>
6bc: e7923003 ldr r3, [r2, r3]
6c0: e5933000 ldr r3, [r3]
6c4: e50b3008 str r3, [fp, #-8]
6c8: e3a03000 mov r3, #0
6cc: e59f309c ldr r3, [pc, #156] ; 770 <main+0xd4>
6d0: e08f3003 add r3, pc, r3
6d4: e1a00003 mov r0, r3
6d8: ebffff9c bl 550 <puts@plt>
6dc: ebffff92 bl 52c <pthread_self@plt>
6e0: e50b0010 str r0, [fp, #-16]
6e4: e51b1010 ldr r1, [fp, #-16]
6e8: e59f3084 ldr r3, [pc, #132] ; 774 <main+0xd8>
6ec: e08f3003 add r3, pc, r3
6f0: e1a00003 mov r0, r3
6f4: ebffff89 bl 520 <printf@plt>
6f8: e24b0014 sub r0, fp, #20
6fc: e3a03000 mov r3, #0
700: e59f2070 ldr r2, [pc, #112] ; 778 <main+0xdc>
704: e08f2002 add r2, pc, r2
708: e3a01000 mov r1, #0
70c: ebffff8c bl 544 <pthread_create@plt>
710: e50b000c str r0, [fp, #-12]
714: e51b3014 ldr r3, [fp, #-20] ; 0xffffffec
718: e3a01000 mov r1, #0
71c: e1a00003 mov r0, r3
720: ebffff93 bl 574 <pthread_join@plt>
724: e59f3050 ldr r3, [pc, #80] ; 77c <main+0xe0>
728: e08f3003 add r3, pc, r3
72c: e1a00003 mov r0, r3
730: ebffff86 bl 550 <puts@plt>
734: e3a03000 mov r3, #0
738: e59f1040 ldr r1, [pc, #64] ; 780 <main+0xe4>
73c: e08f1001 add r1, pc, r1
740: e59f2024 ldr r2, [pc, #36] ; 76c <main+0xd0>
744: e7912002 ldr r2, [r1, r2]
748: e5921000 ldr r1, [r2]
74c: e51b2008 ldr r2, [fp, #-8]
750: e0321001 eors r1, r2, r1
754: 0a000000 beq 75c <main+0xc0>
758: ebffff76 bl 538 <__stack_chk_fail@plt>
75c: e1a00003 mov r0, r3
760: e24bd004 sub sp, fp, #4
764: e8bd8800 pop {fp, pc}
768: 000108f0 .word 0x000108f0
76c: 00000040 .word 0x00000040
770: 00000130 .word 0x00000130
774: 0000011c .word 0x0000011c
778: 00000078 .word 0x00000078
77c: 000000e8 .word 0x000000e8
780: 00010868 .word 0x00010868
00000784 <start_rtn>:
784: e92d4800 push {fp, lr}
788: e28db004 add fp, sp, #4
78c: e24dd008 sub sp, sp, #8
790: e50b0008 str r0, [fp, #-8]
794: e59f3018 ldr r3, [pc, #24] ; 7b4 <start_rtn+0x30>
798: e08f3003 add r3, pc, r3
79c: e1a00003 mov r0, r3
7a0: ebffff6a bl 550 <puts@plt>
7a4: e1a00000 nop ; (mov r0, r0)
7a8: e1a00003 mov r0, r3
7ac: e24bd004 sub sp, fp, #4
7b0: e8bd8800 pop {fp, pc}
7b4: 00000080 .word 0x00000080
000007b8 <__libc_csu_init>:
7b8: e92d 43f8 stmdb sp!, {r3, r4, r5, r6, r7, r8, r9, lr}
7bc: 4607 mov r7, r0
7be: 4e0c ldr r6, [pc, #48] ; (7f0 <__libc_csu_init+0x38>)
7c0: 4688 mov r8, r1
7c2: 4d0c ldr r5, [pc, #48] ; (7f4 <__libc_csu_init+0x3c>)
7c4: 4691 mov r9, r2
7c6: 447e add r6, pc
7c8: f7ff ee94 blx 4f4 <_init>
7cc: 447d add r5, pc
7ce: 1b76 subs r6, r6, r5
7d0: 10b6 asrs r6, r6, #2
7d2: d00a beq.n 7ea <__libc_csu_init+0x32>
7d4: 3d04 subs r5, #4
7d6: 2400 movs r4, #0
7d8: f855 3f04 ldr.w r3, [r5, #4]!
7dc: 3401 adds r4, #1
7de: 464a mov r2, r9
7e0: 4641 mov r1, r8
7e2: 4638 mov r0, r7
7e4: 4798 blx r3
7e6: 42a6 cmp r6, r4
7e8: d1f6 bne.n 7d8 <__libc_csu_init+0x20>
7ea: e8bd 83f8 ldmia.w sp!, {r3, r4, r5, r6, r7, r8, r9, pc}
7ee: bf00 nop
7f0: 000106d6 .word 0x000106d6
7f4: 000106cc .word 0x000106cc
000007f8 <__libc_csu_fini>:
7f8: 4770 bx lr
7fa: bf00 nop
Disassembly of section .fini:
000007fc <_fini>:
7fc: e92d4008 push {r3, lr}
800: e8bd8008 pop {r3, pc}
Edit - Here you can download both executables
Thomas2 Answers
Now that I have spent two days on this problem, a solution seems to have been found.
On a RaspberryPi (on which the application starts successfully) there was always SIGABRT (signal 6) with "stack smashing detected". But these were unexplainable. So I decided to disable them (temporarily) via -fno-stack-protector.
The result: The application also starts on the special device and there are no crashes yet.
Is it possible that there is a bug of the cross compiler "gcc-arm-linux-gnueabihf" in combination with stack smashing? The problem is reproducible with gcc version 7.5.0 and version 9.3.0.
To be honest, the story makes me very doubtful about my build setup.
This is the new ldd output:
ldd ./testapp
libpthread.so.0 => /lib/libpthread.so.0 (0xb6f6f000)
libc.so.6 => /lib/libc.so.6 (0xb6e43000)
/lib/ld-linux-armhf.so.3 (0xb6f8e000)
Many thanks to all supporters
start_rtn is not returning anything - it should return something, even if it's NULL.
You may find that on different platforms you'll have different patterns of uninitialised memory and registers. On most platforms you're probably lucky.
That's not to say there could be an issue with loading the pthread lib as well - are you able to load the program with gdb and at least step in to the first few lines?
Also - is there the option of installing compilers on the target and performing a direct compile with that? I know someone who set up a build server on a RPI when he failed to get a cross-compiler to work ;)
Den-JasonUser contributions licensed under CC BY-SA 3.0