jquery data table, passing large json string

I have a jQuery data table and use ajax call to populate it. Data in a C# datatable is converted to JSOn string in a web method API and the result is passed to data table. It works fine when I have a couple of thousand rows of data but if I have something like 100,000 rows I get out of memory error. I looked around for this issue and tried to implement some of the suggestions but it didn't work. I don't believe I have circular reference because (I guess) in that case even the one with smaller data would fail. Here is what I have (so far), where in response to a submit button click, data table is populated:

var tblFacCert = $("#tblFacCert").DataTable({
    jQueryUI: true,
    data: [],
    dom: 'lfrtip',
    processing: true,
    stateSave: true,
    ....
});

$("#btnSubmit").on("click", function (event) {
    var facCertUrl = "../services/easg.asmx/GetComplianceReportData";
    var facCertParams = "{ 'startDate': '" + $("#tbStartDate").val() + "', 'certID': '" + $('#ddlCertificate').val() + "'}";
    populteTable(facCertUrl, facCertParams, tblFacCert);
})

function populteTable(ws_url, parameters, table) {
    $.ajax({
        type: "POST",
        dataType: "json",
        contentType: "application/json; charset=utf-8",
        url: ws_url,
        cache: false,
        data: parameters,
    }).done(function (result) {
        table.clear().draw();
        table.processing = true;
        jResult = JSON.parse(result.d);
        table.rows.add(jResult).draw();
    }).fail(function (jqXHR, textStatus, errorThrown) {debugger
        alert(textStatus + ' - ' + errorThrown); <--- Here I can see the following error thrown
    });
}
// Error: 
Exception of type 'System.OutOfMemoryException' was thrown.
StackTrace: 
at System.Text.StringBuilder.ToString() 
at System.Web.Script.Serialization.JavaScriptSerializer.Serialize(Object obj, 
SerializationFormat serializationFormat) 
at System.Web.Script.Services.RestHandler.InvokeMethod(HttpContext context, WebServiceMethodData methodData, IDictionary`2 rawParams) 
at System.Web.Script.Services.RestHandler.ExecuteWebServiceCall(HttpContext context, WebServiceMethodData methodData)
ExceptionType:System.OutOfMemoryException


[WebMethod]
[ScriptMethod(UseHttpGet = false, ResponseFormat = ResponseFormat.Json)]

public string GetComplianceReportData(string startDate, string certID)
{
    DateTime dtStartDate = Convert.ToDateTime(startDate);
    int iCertItemID = int.Parse(certID);
    DataTable dt = FacCompliance.GetFacCompliances(dtStartDate, iCertItemID);
    string JSONresult;
    JSONresult = JsonConvert.SerializeObject(dt);

    /* I tried this, as was suggested in a post, no luck */

    //StringBuilder sb = new StringBuilder();
    //StringWriter sw = new StringWriter(sb);

    //using (JsonWriter writer = new JsonTextWriter(sw))
    //{
    //    var serializer = new JsonSerializer();
    //    serializer.Serialize(writer, dt);
    //}

    return JSONresult;
}

another suggestion, web.config change:

<configuration>
    <appSettings>
        ....
        <add key="aspnet:MaxJsonDeserializerMembers" value="500000"/>
    </appSettings>

<system.web.extensions>
    <scripting>
        <webServices>
            <jsonSerialization maxJsonLength="2147483647"/>
        </webServices>
    </scripting>
</system.web.extensions>

I also tried adding "serevrSide: true" to data tale initialization but get this error if I do so:

"Unhandled exception at line 36, column 442 in https://cdn.datatables.net/1.10.16/js/jquery.dataTables.min.js
0x800a138f - JavaScript runtime error: Unable to set property 'data' of undefined or null reference"