memset large memory region allocated by shm_open/ftruncate/mmap crashes with bus error

0

I have a demo program which uses shm_open/ftruncate/mmap to allocate memory:

#include <unistd.h>
#include <sys/stat.h>
#include <sys/file.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>

void shm_alloc(const char* path, size_t size) {
    int shmfd = shm_open(path, O_CREAT | O_EXCL | O_TRUNC | O_RDWR, 0666);
    if (shmfd == -1) {
        perror("shm_open");
        exit(1);
    }
    printf("shm_open success, path: %s, fd: %d\n", path, shmfd);

    int ret = ftruncate(shmfd, size);
    if (ret != 0) {
        perror("ftruncate");
        exit(1);
    }
    printf("ftruncate success, fd: %d, size: %#lx\n", shmfd, size);

    void* addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, shmfd, 0);
    if (addr == MAP_FAILED) {
        perror("mmap");
        exit(1);
    }
    printf("mmap success, fd: %d, size: %#lx, addr: %p\n", shmfd, size, addr);

    close(shmfd);
    memset(addr, 0x00, size); // <===== Crashes here
    printf("memset success, size: %#lx, addr: %p\n", size, addr);
}

int main() {
    shm_alloc("/a", 0x100000000); // 4GB
    shm_alloc("/b", 0x100000000); // 4GB, crashes at the `memset` in this call
    return 0;
}

The program creates two 4GB shared memory regions and memset them. Run the program it outputs the following and then crashes:

shm_open success, path: /a, fd: 3
ftruncate success, fd: 3, size: 0x100000000
mmap success, fd: 3, size: 0x100000000, addr: 0x7f4c15625000
memset success, size: 0x100000000, addr: 0x7f4c15625000
shm_open success, path: /b, fd: 3
ftruncate success, fd: 3, size: 0x100000000
mmap success, fd: 3, size: 0x100000000, addr: 0x7f4b15625000
[1]    1250849 bus error (core dumped)  ./a.out

As you see, the first memset succeeds, and the second crashes, I gdbed the program, and it shows the second region mmaped at 0x7f4b15625000 does not have enough space (4GB) (execute p (char*)addr + 0xFFFFFFFF in gdb produces Cannot access memory error), but it do have space smaller than 4GB (might be 2GB or 3GB, I didn't test for the accurate size).

I am running on Debian 9 and the physical memory is 16GB without a swap space, so it seems not to be a no-enough-memory problem, anyone has any idea? Shouldn't I trust the ftruncate/mmap calls?


EDIT: Running df -h shows the size of /dev/shm is 7.8G, so it might be the root cause. However, why ftruncate succeeds even there is no enough space in /dev/shm? And why when I ls -alh /dev/shm, it shows two files "a" and "b" both with size 4.0GB?

Output of free -h:

# before running the program:
              total        used        free      shared  buff/cache   available
Mem:            15G        701M        9.0G        161M        5.7G         14G
Swap:            0B          0B          0B

# after running the program:
              total        used        free      shared  buff/cache   available
Mem:            15G        702M        1.4G        7.8G         13G        6.6G
Swap:            0B          0B          0B

Output of ulimit -a:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-u: processes                       65535
-n: file descriptors                1024768
-l: locked-in-memory size (kbytes)  64
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 63053
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited
c
linux
shared-memory
mmap
asked on Stack Overflow Jun 2, 2020 by Kelvin Hu • edited Jun 2, 2020 by Kelvin Hu

1 Answer

0

I have reproduced this issue with a 16GB Ubuntu VM which has also 8 GB by default for /dev/shm.

After increasing /dev/shm to 9GB it works:

$ ./tm
shm_open success, path: /a, fd: 3
ftruncate success, fd: 3, size: 0x100000000
mmap success, fd: 3, size: 0x100000000, addr: 0x7ff849486000
memset success, size: 0x100000000, addr: 0x7ff849486000
shm_open success, path: /b, fd: 3
ftruncate success, fd: 3, size: 0x100000000
mmap success, fd: 3, size: 0x100000000, addr: 0x7ff749486000
memset success, size: 0x100000000, addr: 0x7ff749486000
$ df /dev/shm -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           9.0G  8.1G  1.0G  89% /dev/shm
$ 

I cannot answer questions about ftruncate and file size in /dev/shm but it works.

answered on Stack Overflow Jun 2, 2020 by pifor

User contributions licensed under CC BY-SA 3.0