Memory access violation summing large CSV file into array

1

My code reads a comma delimited file (CSV) and average numbers into an array. It works for number_x_nodes = 1000, however when run for number_x_nodes = 1000 (ie. the whole csv file) it gives a 0xc0000005 exit code. The error occurs in Clion but not Codeblocks.

The CSV file is as follows. The entire file contains 28,325,381 lines and is 1.8GB

CoordinateX,CoordinateY,CoordinateZ,Pressure,Temperature,VelocityX,VelocityY,VelocityZ,
0,0,0.0904,33.5797,300,-0.00146382,0.000389435,-0.00147085,
0,0.0003,0.0904,33.5795,300,0.126682,-0.000382509,0.00330599,
0,0.0006,0.0904,33.5793,300,0.250278,-0.00151828,0.0100881,
0,0.0009,0.0904,33.5788,300,0.365407,-0.00287706,0.0184123,
...
after 100 lines, CoordinateZ changes to the next value
after 72 CoordinateZ iterations, CoordinateX changes to the next value
after 3930 CoordinateX iterations, file ends.

The code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main()
{
    /* Open CSV file */
    int BUFFER_SIZE = 128;
    char buffer[BUFFER_SIZE];
    FILE *pFile = fopen("39900.csv","r");
    char *pCell;

    /* Set node counts */
    int number_x_nodes = 3930; //if number_x_nodes = 1000, the code runs as expected
    int number_y_nodes = 100;
    int number_z_nodes = 72;

    /* Setup Array */
    double property_array[number_y_nodes][6]; //property_array[j][n]
    double number_xz_plane_nodes = number_x_nodes*number_z_nodes;
    int n = 0; //property index
    int i = 0; //x-coordinate index
    int j = 0; //y-coordinate index

    /* Initialize entries of array */
    rewind(pFile);
    fgets(buffer,BUFFER_SIZE,pFile);
    for(j=0; j<number_y_nodes; j++) {
        fgets(buffer,BUFFER_SIZE,pFile);
        pCell = strtok(buffer, ",");
        pCell = strtok(NULL, ",");
        property_array[j][0] = strtod(pCell, &pCell);
        pCell = strtok(NULL, ",");
        for (n=1; n<6; n++) {
            pCell = strtok(NULL, ",");
            property_array[j][n] = strtod(pCell, &pCell)/number_xz_plane_nodes;
        }
    }

    /* Average values into the array */
    for (i=0; i<number_xz_plane_nodes; i++) {
        for (j=0; j<number_y_nodes; j++) {
            fgets(buffer, BUFFER_SIZE, pFile);
            pCell = strtok(buffer, ",");
            pCell = strtok(NULL, ",");
            pCell = strtok(NULL, ",");
            for (n=1; n<6; n++) {
                pCell = strtok(NULL, ",");
                property_array[j][n] += strtod(pCell, &pCell)/number_xz_plane_nodes; //program crashes on this line
            }
        }
    }

    /* Print array */
    printf("Printing array...\n");
    for(j=0; j<number_y_nodes; j++){
        printf("y_%i = %.7lf --> ",j, property_array[j][0]);
        printf("vel_x = %.30lf\n", property_array[j][3]);
    }

    /* Close file */
    fclose(pFile);

    return(0);
}

Result for int number_x_nodes = 1000

Prints array as expected.

Result for int number_x_nodes = 3930

Process finished with exit code -1073741819 (0xC0000005)

Values at crash

buffer = {char [128] of random characters}

number_xy_plane_nodes = 282960

pCell = NULL

n = 1

i = 73799

j = 0

Editor's Note

The error that lead to this question has been found. Please refer to my answer below.

c
asked on Stack Overflow Jul 27, 2019 by Riley • edited Aug 6, 2019 by Riley

1 Answer

1

I have found the error. The CSV file contains a break 7,380,002 lines in. This is the first thing I checked for when I received the error. I must have checked incorrectly the first time. I would like to thank everyone who commented for their efforts.

...
0.65536,0.0686213,0.133,30.3213,300,1.59667,-0.00783055,-0.0092234,
0.65536,0.0705461,0.133,30.1529,300,1.59745,-0.00759862,-0.00799852,
0.65536,0.0725,0.133,29.9784,300,1.59777,-0.00753752,-0.00714456,

0,0,0.0904,33.5797,300,-0.00146382,0.000389435,-0.00147085,
0,0.0003,0.0904,33.5795,300,0.126682,-0.000382509,0.00330599,
0,0.0006,0.0904,33.5793,300,0.250278,-0.00151828,0.0100881,
...
answered on Stack Overflow Jul 28, 2019 by Riley

User contributions licensed under CC BY-SA 3.0