Same HTTP request passes with CURL_CLI but fails with LIBCURL

Am using libcurl to communicate with Amazon S3.

GET calls are success whereas PUT (for uploading files) calls are failing with 403.

PUT call with same headers when ran through CURL-CLI are success.

I have disabled certificate checking. [[ curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0) ]]

Using CURLOPT_DEBUGFUNCTION option as specified in https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html, am able to capture the following log.

=> Send header, 0000000578 bytes (0x00000242)
0000: PUT /M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_0603202
0040: 0065517_fota_dlready HTTP/1.1
005f: Host: fota.test.nvtl.s3.amazonaws.com
0086: Accept: */*
0093: Authorization : AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ
00d3: /20200603/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-co
0113: ntent-sha256;x-amz-date, Signature=61613a8357e45ac6c14067856f1e1
0153: 56449adcba20d3b8d295ed1c67a126dda0c
0178: x-amz-content-sha256:e2b646fdb491ec4be82661a9ea86ce3b08fe5fabfda
01b8: 2b9a9f8c3d9a783135837
01cf: x-amz-date:20200603T065517Z
01ec: content-length: 3394
0202: content-type: application/octet-stream
022a: Expect: 100-continue
0240: 
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....P
<= Recv header, 0000000024 bytes (0x00000018)
0000: HTTP/1.1 403 Forbidden
<= Recv header, 0000000036 bytes (0x00000024)
0000: x-amz-request-id: BE05183350F1A797
<= Recv header, 0000000090 bytes (0x0000005a)
0000: x-amz-id-2: b6rrLoMPXn0Umvv8YzFiVL8CX27oszT0mQqWnjxaBtD49DoqsI6C
0040: NedmUoOebVAf3R96Q7c59tg=
<= Recv header, 0000000031 bytes (0x0000001f)
0000: Content-Type: application/xml
<= Recv header, 0000000028 bytes (0x0000001c)
0000: Transfer-Encoding: chunked
<= Recv header, 0000000037 bytes (0x00000025)
0000: Date: Wed, 03 Jun 2020 06:55:19 GMT
<= Recv header, 0000000019 bytes (0x00000013)
0000: Connection: close
<= Recv header, 0000000018 bytes (0x00000012)
0000: Server: AmazonS3
<= Recv header, 0000000002 bytes (0x00000002)
0000: 
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....0
<= Recv data, 0000000254 bytes (0x000000fe)
0000: f3
0004: <?xml version="1.0" encoding="UTF-8"?>.<Error><Code>AccessDenied
0044: </Code><Message>Access Denied</Message><RequestId>BE05183350F1A7
0084: 97</RequestId><HostId>b6rrLoMPXn0Umvv8YzFiVL8CX27oszT0mQqWnjxaBt
00c4: D49DoqsI6CNedmUoOebVAf3R96Q7c59tg=</HostId></Error>
00f9: 0
00fc: 
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....0
== Info: TLSv1.2 (OUT), TLS alert, Client hello (1):
=> Send SSL data, 0000000002 bytes (0x00000002)
0000: ..
== Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 0000000512 bytes (0x00000200)
0000: .........;r..f..g.]..50.=.+.xh..M..Ia.....0.,.(.$.............k.
0040: j.i.h.9.8.7.6.........2...*.&.......=.5.../.+.'.#.............g.
0080: @.?.>.3.2.1.0.........E.D.C.B.1.-.).%.......<./...A.............
00c0: ............3...$."...fota.test.nvtl.s3.amazonaws.com...........
0100: ................................ ...............................
0140: ......3t.........http/1.1.......................................
0180: ................................................................
01c0: ................................................................
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....W
== Info: TLSv1.2 (IN), TLS handshake, Server hello (2):
<= Recv SSL data, 0000000087 bytes (0x00000057)
0000: ...S...Z..J....v...6.F........i3..j:2. ..ptS.oT..h...8..J_..:o..
0040: )..a...................
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....T
== Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
<= Recv SSL data, 0000002900 bytes (0x00000b54)
0000: ...P..M...0...0...........-........r..8..0...*.H........0d1.0...
0040: U....US1.0...U....DigiCert Inc1.0...U....www.digicert.com1#0!..U
0080: ....DigiCert Baltimore CA-2 G20...191109000000Z..210312120000Z0l
00c0: 1.0...U....US1.0...U....Washington1.0...U....Seattle1.0...U....A
0100: mazon.com, Inc.1.0...U....*.s3.amazonaws.com0.."0...*.H.........
0140: ....0.........[...)X.+\.x....<1j.....p.y.c..hx..c=.@..e......t,.
0180: g.^c3......z...}~>.2.eC.;...Y.;h..e....A..y....$#.K.#.&...2O..fD
01c0: ..7k..1.R..3....+~...{}.XT.gp4.l.0$.V.)Z.o.6..?..^Oz,%s....~...~
0200: ........z.c..P...~7G..-....!6K.....\....}....F.7..H.T.z..#p;..F
0240: #Jr...)L..xa..........0..|0...U.#..0......(thFg.p%t..E[.}\D0...U
0280: ........&.d...\.......'...0/..U...(0&..*.s3.amazonaws.com..s3.am
02c0: azonaws.com0...U...........0...U.%..0...+.........+.......0....U
0300: ...z0x0:.8.6.4http://crl3.digicert.com/DigiCertBaltimoreCA-2G2.c
0340: rl0:.8.6.4http://crl4.digicert.com/DigiCertBaltimoreCA-2G2.crl0L
0380: ..U. .E0C07..`.H...l..0*0(..+.........https://www.digicert.com/C
03c0: PS0...g.....0y..+........m0k0$..+.....0...http://ocsp.digicert.c
0400: om0C..+.....0..7http://cacerts.digicert.com/DigiCertBaltimoreCA-
0440: 2G2.crt0...U.......0.0..}..+.....y......m...i.g.v.......X......g
0480: p.<5.......w.........nMv;0.....G0E.!....)Y.!i...a.......4.w....w
04c0: .e..y. 3..cv.....R]....A.......a_.......v.D.e......@....(.......
0500: 1.?.3........nMv;(.....G0E.!...B).+.S..[..^..mD..&2.c8.*....7. B
0540: ..#.....f.s..B.....#mA&.u..v.C..u.......q...#...{G8W...R....d6..
0580: .....nMv;5.....F0D. ..._A/^..Y..V.....oJ.#.k.8...fDj. A..^..o...
05c0: .......09O.....dEgX.K.0...*.H.................x".V...6....2@....
0600: ..*.6q........CS.....-...=..ep...V......M.C......@.$.y.k.F?.v.&(
0640: .<N....a.o......i../..Q8K+^.;k86.F.....rW..._..>...3.q.Tc...l..
0680: $...fx0.;...)[.......f_....0T*....g.....<u{........8/6....."...J
06c0: .38...@ %.K.nt.5M...'m5Pk..wH.0...Z....-.c..g0..c0..K...........
0700: ...&..;'....0...*.H........0Z1.0...U....IE1.0...U....Baltimore1.
0740: 0...U....CyberTrust1"0 ..U....Baltimore CyberTrust Root0...15120
0780: 8120507Z..250510120000Z0d1.0...U....US1.0...U....DigiCert Inc1.0
07c0: ...U....www.digicert.com1#0!..U....DigiCert Baltimore CA-2 G20..
0800: "0...*.H.............0..............s....\.u.\ps...z...#....?.!.
0840: .M..-...1..k..].Q..^r..f.....A..&....i.S..O>..:C+.Y.....Y@rZg...
0880: U..+............%.E......t....eP...E../..6......R....6q+.......[
08c0: [9...................Z../r....g....J. ...r....OB.M'BM.....u.6`.&
0900: T...7....)Y ..%........................0...0...U.........(thFg.p
0940: %t..E[.}\D0...U.#..0.....Y0.GX....T6.{:..M.0...U.......0.......0
0980: ...U...........04..+........(0&0$..+.....0...http://ocsp.digicer
09c0: t.com0:..U...3010/.-.+.)http://crl3.digicert.com/Omniroot2025.cr
0a00: l0=..U. .60402..U. .0*0(..+.........https://www.digicert.com/CPS
0a40: 0...*.H............./.7f....U.)..P.....(.t..i;D0=..I.h6..0....IB
0a80: cFR.i..I...W.....u....3.b.CT.c...S.........ex.3...>.p....x.3...,
0ac0: X..@.mA...../...pk.:.{....K..o..)....T.T.... .i,./?P...W...s..$.
0b00: ...E....).f...l.O.+.LG....A....FwB......%K.PW...N.?.%.A...bmo...
0b40: ......).......T.KI..
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....M
== Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
<= Recv SSL data, 0000000333 bytes (0x0000014d)
0000: ...I...A.>...,R..3R.ry.../WTDH..b...lcB`.RB<dPQJ..2..$M.`6.A....
0040: ..2e.C.......$W+.8.g.].....A[q...,G.......u;.....q....8..:......
0080: G ....F.-.o.b54...P-.o..:|H..!......5.....O.7..d..El7.;....\....
00c0: t......'.v&...#..Y*.y..yoN....3 >...?...vH...i.J....K.....ar.:..
0100: ..Ll.............(Z..mx.;.....BQ...7.b.[...|..z.;..d....*. ..]M/
0140: ./.........=.
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
<= Recv SSL data, 0000000004 bytes (0x00000004)
0000: ....
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....F
== Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
=> Send SSL data, 0000000070 bytes (0x00000046)
0000: ...BA.wm.Jg'... .......,y8%B.....1..........}X.5.~....oC...T.].6
0040: .Q\..8
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (OUT), TLS change cipher, Client hello (1):
=> Send SSL data, 0000000001 bytes (0x00000001)
0000: .
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....@
== Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
=> Send SSL data, 0000000016 bytes (0x00000010)
0000: ......1.e.].....
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.2 (IN), TLS change cipher, Client hello (1):
<= Recv SSL data, 0000000001 bytes (0x00000001)
0000: .
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....@
== Info: TLSv1.2 (IN), TLS handshake, Finished (20):
<= Recv SSL data, 0000000016 bytes (0x00000010)
0000: ....t.....%. 1..
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: ....P

When i ran the same from CLI it is success.

# curl --insecure -v -X PUT https://fota.test.nvtl.s3.amazonaws.c
om/M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_06032020065517_fota_dlrea
dy -H 'Authorization: AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ/20200603/
us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, S
ignature=61613a8357e45ac6c14067856f1e156449adcba20d3b8d295ed1c67a126dda0c' -H x-
amz-content-sha256:e2b646fdb491ec4be82661a9ea86ce3b08fe5fabfda2b9a9f8c3d9a783135
837 -H x-amz-date:20200603T065517Z -H 'content-length: 3394' -H 'content-type: a
pplication/octet-stream' -T /opt/nvtl/data/8c51f2240f9fc1e7d2329a24210e30c9_
200_06032020065517_fota_dlready
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
> PUT /M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_06032020065517_fota_dlready HTTP/1.1
> Host: fota.test.nvtl.s3.amazonaws.com
> User-Agent: curl/7.52.1
> Accept: */*
> Authorization: AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ/20200603/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=61613a8357e45ac6c14067856f1e156449adcba20d3b8d295ed1c67a126dda0c
> x-amz-content-sha256:e2b646fdb491ec4be82661a9ea86ce3b08fe5fabfda2b9a9f8c3d9a783135837
> x-amz-date:20200603T065517Z
> content-length: 3394
> content-type: application/octet-stream
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< x-amz-id-2: xnaNU1mmmuynkWQQDA0OJsgKKAYBOCRbx6lpTkj54HGEicXWOMtnd3qb4ZbY6vmogULiq7vOFcA=
< x-amz-request-id: BAE5F92ED54058EC
< Date: Wed, 03 Jun 2020 06:57:53 GMT
< ETag: "595f51bb7d2cc4c5c3f30b3bc3d350c3"
< Content-Length: 0
< Server: AmazonS3
< 

Could any give hints what might actually be wrong here.

Is there any other method/way where we can have more verbose ouput regards the error.

Note: HEADERS that are passed to both the requests (libcurl and curl_cli) are same. Certificate check is disabled in both.

Response after adding user-agent.

0000: .....
=> Send header, 0000000609 bytes (0x00000261)
0000: PUT /M2000/activity/8c51f2240f9fc1e7d2329a24210e30c9_200_0603202
0040: 0142553_fota_dlready HTTP/1.1
005f: Host: fota.test.nvtl.s3.amazonaws.com
0086: User-Agent: libcurl-agent/1.0
00a5: Accept: */*
00b2: Authorization : AWS4-HMAC-SHA256 Credential=AKIAJ2ZI2YKOFDBS4UMQ
00f2: /20200603/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-co
0132: ntent-sha256;x-amz-date, Signature=9a72f22ce185e48c023a8d3314004
0172: ac2a76b91ccd56b208d769e27bafe7b1446
0197: x-amz-content-sha256:83a7ba5f8f0c829537965116840aad257f6b71e7899
01d7: bafc4c93542b681c9454a
01ee: x-amz-date:20200603T142554Z
020b: content-length: 3394
0221: content-type: application/octet-stream
0249: Expect: 100-continue
025f: 
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....P
<= Recv header, 0000000024 bytes (0x00000018)
0000: HTTP/1.1 403 Forbidden
<= Recv header, 0000000036 bytes (0x00000024)
0000: x-amz-request-id: AD607FF3FA529447
<= Recv header, 0000000090 bytes (0x0000005a)
0000: x-amz-id-2: w0ZHrVJ9R8uslrB6kf9KLLRtshjksK9cLxeovH53GXL4uRoK17U6
0040: MJaWuhiGhZMKdIphBWkY+mE=
<= Recv header, 0000000031 bytes (0x0000001f)
0000: Content-Type: application/xml
<= Recv header, 0000000028 bytes (0x0000001c)
0000: Transfer-Encoding: chunked
<= Recv header, 0000000037 bytes (0x00000025)
0000: Date: Wed, 03 Jun 2020 14:25:57 GMT
<= Recv header, 0000000019 bytes (0x00000013)
0000: Connection: close
<= Recv header, 0000000018 bytes (0x00000012)
0000: Server: AmazonS3
<= Recv header, 0000000002 bytes (0x00000002)
0000: 
<= Recv SSL data, 0000000005 bytes (0x00000005)
0000: ....0
<= Recv data, 0000000254 bytes (0x000000fe)
0000: f3
0004: <?xml version="1.0" encoding="UTF-8"?>.<Error><Code>AccessDenied
0044: </Code><Message>Access Denied</Message><RequestId>AD607FF3FA5294
0084: 47</RequestId><HostId>w0ZHrVJ9R8uslrB6kf9KLLRtshjksK9cLxeovH53GX
00c4: L4uRoK17U6MJaWuhiGhZMKdIphBWkY+mE=</HostId></Error>

UPDATED

Figured out the issue.

Issue seems to be with spacing. "Authorization : AWS4-XXX" gives 403 error whereas "Authorization: AWS4-XXX" is success.

Thanks, Trinadh

best guess: fota.test.nvtl.s3.amazonaws.com blocks requests without any User-Agent. several websites do this, for example Wikipedia.org does the same thing (not sure why),

curl the cli program has a default user-agent, it looks like User-Agent: curl/7.52.1, but libcurl doesn't have any default user-agent, you can set one with the CURLOPT_USERAGENT option.