MIPS Branch Datapath

I am programming a single cycle MIPS CPU and I am confused how the branch address is computed. Given this MIPS datapath diagram

Why is it that after the sign extend and shift left by two, the address is added by PC+4?

My main question is when I do the

 SignExtImm(instruction.imm) << 2 + (PC + 4)

I would be off by one instruction. Looking at my MIPS assembly Fibonacci test case that I wrote:

[0x004000a0]    0x00112020  add $4, $0, $17        <-- # loop label here
[0x004000a4]    0x0c100002  jal 0x00400008 [fib] 
[0x004000a8]    0x00000000  nop                  
[0x004000ac]    0x00029821  addu $19, $0, $2     
[0x004000b0]    0x22310001  addi $17, $17, 1     
[0x004000b4]    0x1632fffb  bne $17, $18, -20 [loop-0x004000b4];
[0x004000b8]    0x00000000  nop                             

Say I am at 0x004000b4 and the branch is taken

The instruction is 0x1632fffb

So imm val := 0xFFFB and PC+4 := 0x004000B8

sign extend imm val ==> 0xFFFFFFFB

shift left by 2 ==> 0xFFFFFFEC

then 0xFFFFFFEC + 0x004000B8 ==> 0x1004000A4 ==> 0x004000A4

but my loop starts at 0x004000A0

If I follow that MIPS datapath diagram, I am unable to calculate the correct branch address.