WCF streamed response failing on large data

I'm currently experiencing an issue using WCF streaming to return a large set of serialized objects over the internet while hosted through IIS. I'm not sure if it is a network issue or a WCF issue, but the symptoms are hard for me to justify as network issues, as I'll describe below.

The objects are a variety of different objects sent over the one stream, most of which are small but there are also jpegs in there which range from 10k->500k per object. The different cases that I can't really reconcile are:

1. If we run it over our internal network it works perfectly, even if we increase the data transferred 10-fold higher than the real case so we're sending gigabytes of data.
2. If we run it on the servers hosting the solution we get an error (detailed below) after a few minutes and a few hundred megabytes transferred.
3. If we run it on the servers but remove the photos (Sending only the other data) it succeeds and sends roughly 40mb over a longer time period than the failure takes (The longer time is because it takes longer to process the other data, so that's expected).
4. We have tested simply downloading a 1GB file off the webserver (Significantly more than any of the tests other than the ones we tried on our local network) and it succeeded multiple times without issue.

The error we receive client-side is (Note: We went nowhere near the 3 hour timeout we have set):

System.Reflection.TargetInvocationException: An exception occurred during the operation, making the result invalid. Check InnerException for exception details. ---> 
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '03:00:00'. ---> 
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> 
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

And server-side we see:

System.ServiceModel.CommunicationException,
System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 The remote host closed the connection. The error code is 0x80070016.

The WCF binding setup is (With a little anonymizing):

  <system.serviceModel>
    <serviceHostingEnvironment>
      <baseAddressPrefixFilters>
        <add prefix="http://anonymous"/>
      </baseAddressPrefixFilters>
    </serviceHostingEnvironment>
    <bindings>
      <basicHttpBinding>
        <binding name="StreamedHttpBindingConfig" transferMode="StreamedResponse" maxReceivedMessageSize="2147483647" maxBufferSize="8388608" sendTimeout="01:00:00">
          <readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647"
            maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" maxDepth="2147483647" />
          <security mode="None" />
        </binding>
      </basicHttpBinding>
    </bindings>
    <services>
      <service behaviorConfiguration="wsHttpBindingBehaviour" name="AnonymousName">
        <endpoint address="" binding="basicHttpBinding" bindingConfiguration="StreamedHttpBindingConfig" name="wsHttpBinaryBindingEndpoint" contract="Service.IAnonymous" />
        <endpoint address="mex" binding="mexHttpBinding" name="mexHttpBindingEndpoint" contract="IMetadataExchange" />
        <host>
          <baseAddresses>
            <add baseAddress="http://anonymous/Service" />
          </baseAddresses>
        </host>
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="wsHttpBindingBehaviour">
          <serviceThrottling maxConcurrentCalls="160" maxConcurrentSessions="100" maxConcurrentInstances="100" />
          <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
          <serviceMetadata httpGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>

And the service contract is (Method/param names changed to anonymize, but types the same):

System.ServiceModel.Channels.Message GetData(System.ServiceModel.Channels.Message request);

Which looks like a network issue to me, but I can't imagine why it would succeed on the longer time-period with smaller data if it was intermittent network issues and I can't see why downloading a large file would've been unaffected if this was the case. Does anyone know a potential cause for this, or any idea what the next step of diagnosing it is?

CommunicationException is genericized and does not reveal the underlying exception. In the future, when reporting errors, I'd recommend turning on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the configuration behavior) on the server in order to send the exception information back to the client, and then report the nested exception you get in Detail.

For example:

<behaviors> 
            <serviceBehaviors> 
                <behavior 
                    <serviceMetadata httpGetEnabled="true" /> 
                    <serviceDebug includeExceptionDetailInFaults="true" /> 
                </behavior> 
            </serviceBehaviors> 
        </behaviors> 

It's hard to tell from what is going wrong at all. Here is some more things I suggest, which I also suggested in another post:

• Enable tracing on the service side and the client side, generate tracing logs, and analyze with SvcTraceViewer. To do this, follow the instructions at this MSDN article on using the service trace viewer.

• Turn on debug exceptions. This is done by turning in includeExceptionDetailInFaults, which you can do by following the instructions here.

• Use Fiddler to monitor the wire traffic on both the client side and the service side.

Generally, once you do this, you should plenty of more info on what's going funky at the service side and can diagnose the issue pretty quickly. Try it, and please report back! :)