Stack overflow error in assembly

0

I am trying to write code that takes input from the user and displays the factorial, in assembly x64. Every time I run the code it gives me Unhandled exception at 0x00007FF64B910B11 in finalProjectAssembly.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x00000019F1603FF8)., but I do not understand how the stack has overflowed. What am I missing?

INCLUDELIB libcmt.lib
INCLUDELIB legacy_stdio_definitions.lib

EXTERN printf:PROC
EXTERN scanf:PROC

.DATA
prompt  BYTE    "Enter a number: ", 0
inFmt   BYTE     "%d", 0
prompt3 BYTE    "The factorial is: %d",10,0
cont    QWORD   ?
disp    BYTE    "You Entered %d", 10,0
num     QWORD   ?
num2    REAL8   1.5

.CODE
main PROC C
sub     rsp, 24
lea     rcx, prompt          
call    printf
lea     rdx, num             
lea     rcx, inFmt           
call    scanf
push    num

mov     rdx,num              
lea     rcx, disp            
call    printf
call    factorial
mov     rdx, rax
lea     rcx, prompt3
call    printf

lea     rcx, prompt2
call    printf
lea     rdx, cont
lea     rcx, inFmt
call    scanf

add     rsp,24             
mov     rax,0
ret
main ENDP

factorial PROC
push    rbp
mov     rbp,rsp
mov     rax, [rbp + 16]
cmp     rax, 1
jle     quit
dec     rax 
push    rax
call    factorial
mov     rbx, [rbp+16]
imul    rbx

quit:
mov     rsp, rbp
pop     rbp
ret
factorial ENDP
assembly
64-bit
asked on Stack Overflow May 10, 2018 by cheezt

1 Answer

0

If you call foreign functions (printf,scanf) you must obey their calling conventions. The appropriate Microsoft x64 software convention determines that you have to reserve at leat space for four qwords (=32 bytes) in the stack before calling the function. The function can freely use that "shadow space". Furthermore, ESP have to be aligned to an address divisible by 16.

Change

sub     rsp, 24

to

sub     rsp, 32
and     spl, -16        ; Align to 16

You don't need the add rsp,24 at the end of the main function. The correct prolog (push rbp; mov rsp, rbp) and the correct epilog (leave) will be inserted automatically by the assembler ML64.

The shadow space has to be straight in front of the function's return address pushed by the call instruction. Move the push num close to call factorial and don't forget to clean up the stack:

push    num
call    factorial
add     rsp, 8

Don't forget to clean up the stack:

push    rax
call    factorial
add     rsp, 8

I didn't check the format specifier %d as recommended by David Wohlferd. The program basically works with the changes above.

answered on Stack Overflow May 10, 2018 by rkhb

User contributions licensed under CC BY-SA 3.0