Why can't I remove the transfer-encoding header in a node proxy?

2

I have a Node http-proxy server doing some response body rewriting that basically does this:

  1. Client GET localhost:8000/api/items
  2. Node Proxy send localhost:8000 -> to example.com/api
  3. Server responds with json [{ id: 1234, url: http://example.com/api/items/1234 }]
  4. Node proxy rewrites json to [{ id: 1234, url: http://localhost:8000/api/items/1234 }]
  5. Node proxy calculates the new content-length header, sets it, and returns the response to the client

This was working fine until the backend server enabled compression. So now, by default, responses were being gzipped. I worked around this by setting this in my proxy:

req.headers['accept-encoding'] = 'deflate';

So after that, responses weren't being gzipped, I could parse them and rewrite the body as necessary before. However, this stopped working with IE. I think the problem is that the response still has a transfer-encoding=chunked header, so IE expects a chunked response. Because that transfer-encoding header is present, there is no content-length header, even though I'm explicitly setting it (those two headers being mutually exclusive). I've tried everything I can think of to remove the transfer-encoding header and get the content-length header in instead, but nothing is working. I've tried all of these:

// In the context of my middleware response.writeHead function
res.setHeader('transfer-encoding', null);
res.setHeader('transfer-encoding', '');
res.removeHeader('transfer-encoding');
res.setHeader('content-length', modifiedBuffer.length); // this line alone worked before
res.originalWriteHead.call(res, statusCode, { 'Content-Length', modifiedBuffer.length });

// In the context of my middleware response.write function res.write(data, encoding)
// Here, encoding parameter is undefined
// According to docs, encoding defaults to utf8, could be 'chunked'
res.oldWrite.call(res, modifiedBuffer, 'utf8');
res.oldWrite.call(res, modifiedBuffer, '');
res.oldWrite.call(res, modifiedBuffer, null);
// tried all three previous the same for res.end

Basically, no matter what I do, the response is not chunked, but has the transfer-encoding header set, and not the content-length. Firefox, safari, chrome all seem to handle this fine, but IE fails with the error XMLHttpRequest: Network Error 0x800c0007, No data is available for the requested resource.. This is (from what I can tell) because it's waiting for chunks (because of the transfer-encoding header), but gets the end of the response, and doesn't have a content-length to read it.

Does anyone know how I can solve this? Am I doing something wrong in trying to remove the transfer-encoding header in favor of content-length?

node.js
middleware
http-proxy
asked on Stack Overflow Oct 27, 2014 by xdumaine

1 Answer

0

I figured this out myself:

I was effectively using two middleware components (my own, described in the question), and express.js compression. My middleware was decompressing the response, but compression ensures that the response is always written with transfer-encoding=chunked and content-length is removed.

Removing the express.js compression module solved this for me.

answered on Stack Overflow Oct 27, 2014 by xdumaine

User contributions licensed under CC BY-SA 3.0