How to eliminate stack overflow error even after allocating a large stack size?

Unhandled exception at 0x00AA9379 in A.exe: 0xC00000FD: Stack overflow (parameters: 0x00000000, 0x00802000).

I am facing a stack overflow error. I'm using VS15 as my IDE. I tried to allocate more memory for the stack. For that I used Project >> Properties >> Linker >> System >> Stack allocation and allocated 4GB for the stack. But the error continues to stop at chkstk.asm at this line

99 sub eax, _PAGESIZE_ ; decrease by PAGESIZE.

But the problem isn't solved. How do I know in advance that how much stack size would I need? I've used dynamic memory allocation for all of the large variables. But cannot solve the problem. Here is a verifiable example...

Here is my code:

#include <stdio.h>

void main(void)
{
    FILE    *fp1;
    char    datfile[132];
    int    nod[1024 * 1024];
    int    Enod[8 * 1024 * 1024];
    double    nodS[1024 * 1024], nodF[1024 * 1024];
}

The default stack size on Windows using MS compilers is 1 MiB. You put on the stack several arrays that hold millions of integers and doubles.

You said you increased stack size to 4GB. In that case, the following becomes relevant:

The reserved memory size represents the total stack allocation in virtual memory. As such, the reserved size is limited to the virtual address range. The initially committed pages do not utilize physical memory until they are referenced; however, they do remove pages from the system total commit limit, which is the size of the page file plus the size of the physical memory. The system commits additional pages from the reserved stack memory as they are needed, until either the stack reaches the reserved size minus one page (which is used as a guard page to prevent stack overflow) or the system is so low on memory that the operation fails. (emphasis mine)

In addition, Intel's notes list

Given these definitions, the following lists the limits on 32-bit and 64-bit variants of Windows:

32-bit

• Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size)

...

64-bit

• Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size)

...

Note that the limit on static and stack data is the same in both 32-bit and 64-bit variants. This is due to the format of the Windows Portable Executable (PE) file type, which is used to describe EXEs and DLLs as laid out by the linker. It has 32-bit fields for image section offsets and lengths and was not extended for 64-bit variants of Windows. As on 32-bit Windows, static data and stack share the same first 2GB of address space. (emphasis mine)

Finally, look closely at the instruction in the beginning of your post:

But the error continues to stop at chkstk.asm at this line

99       sub     eax, _PAGESIZE_         ; decrease by PAGESIZE

To look more closely at this, I wrote a small program which would generate the same effect as yours (it only "worked" with a 32-bit build—I am not sure what needs to be done to cause a 64-bit executable to crash):

#include <stdio.h>
#include <stdlib.h>
#define MYSIZE (8 * 1024 * 1024)

int main(void) {
    int x[MYSIZE];
    x[MYSIZE - 1] = rand();
    printf("%d\n", x[MYSIZE - 1]);
    return 0;
}

I set the stack size in the linker options to 4294967296 and then ran the program under the debugger. It crashed with a stack overflow, and broke at the same instruction you observed. Scrolling up in the stack checking code, I noted the following comments:

; Handle allocation size that results in wraparound.
; Wraparound will result in StackOverflow exception.

As far as I can figure out, the routine is trying to move the top-of-stack down by PAGESIZE at a time to reserve the applicable stack size.

So, trying to set the stack size to 4 GB seems to be the root cause of your immediate problem. You can try setting it to 1 GB which might solve that problem. Indeed, I changed the stack size for the program above to 1073741823 (= 1024 * 1024 * 1024 - 1), and I did not get a stack overflow. I think the fact that link does not at the very least warn about an invalid stack size value is a bug.

Indeed, looking at the hexdump of an executable built with /STACK:1000000000 and comparing it to one built with /STACK:4294967296 highlights the problem:

00000150: 0000 0000 0300 4081 00ca 9a3b 0010 0000  ......@....;....
00000150: 0000 0000 0300 4081 0000 0000 0010 0000  ......@.........

Note that 0x3b9aca00 is 1,000,000,000 in hex. Looking at the header format, those refer to the long SizeOfStackReserve; entry. That is, when you set stack size to 4 GB (actually, anything above 0xfffffffc), that has the effect of setting it to zero.

While setting a stack size to larger but still unsupported sizes does result in a positive value being set in the executable header, e.g.:

cl main.c /link /STACK:0xdeadbead
xxd main.exe |more
...
00000150: 0000 0000 0300 4081 b0be adde 0010 0000  ......@.........
...

the resulting executable cannot be run:

C:\...> main
Not enough storage is available to process this command.

However even though setting the stack to a smaller but still large size may enable your program to run, relying on a huge stack is not necessarily a good idea. The immediate alternative is to allocate those arrays on the heap using malloc and remembering to free them.

That is, instead of

int    Enod[8 * 1024 * 1024];

you need declare int *Enod and then allocate memory for the array using

Enod = malloc(8 * sizeof(*Enod) * 1024 * 1024);
/* remember to check that Enod is not NULL */

Answers to this question discuss why the stack is much more constrained.

In addition, your code would benefit from replacing arbitrary looking numbers with meaningful mnemonics using defines.