ALIGN not align the section in link script

I try to set the payload section align to 0x1000 with MEMORY:

MEMORY {
  flash (rxa)  : ORIGIN = 0x40000000, LENGTH = 16M
  ram   (rwxa) : ORIGIN = 0x80000000, LENGTH = 64M
}

SECTIONS {
  . = ORIGIN(flash);
  _start = .;
  . = ALIGN(0x1000);
  .text : {
      *(.text .text.*)
  }

  . = ALIGN(0x1000);
  .payload : {
      *(.payload)
  }

  _end = .;
}

And compile with following commands:

riscv64-unknown-elf-gcc -march=rv64g -mabi=lp64 -nostdlib -nostartfiles -I... -T linker.ld load.S -o test

But I found the section in the elf has no aligment to 0x1000:

$ riscv64-unknown-elf-readelf -S test

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000040000000  00001000
       00000000000000c4  0000000000000000  AX       0     0     16
  [ 2] .payload          PROGBITS         00000000400000d0  000010d0
       00000000000004c8  0000000000000000  WA       0     0     16

Why the ALIGN not works here ?

Update: I found if I set . = ORIGIN(flash); -> . = 0x40000000, and comment out the corresponding MEMORY code, it will work:

$ riscv64-unknown-elf-readelf -S test

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000040000000  00001000
       00000000000000c4  0000000000000000  AX       0     0     16
  [ 2] .payload          PROGBITS         0000000040001000  00002000
       0000000000000448  0000000000000000  WA       0     0     16