Reading large matrix in CSV format (file) in C++

0

I am trying to read a large matrix stored in CSV format in C++. The same code works well in reading matrix size up to 500*500. But for larger data size, the code throws the following error:

Unhandled exception at 0x00007FF612D42068 in z650test3.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000C4532A3000).

I am attaching the code below. Could someone please help me figure out the issue? I believe the issue is coming from handling the memory for large scale matrix size. I really appreciate your ideas and help.

Also, I can work with MAT file as well, I would appreciate if anyone can advise me on ways to read from MAT file as well. Thanks

#include <stdio.h>
#include <iostream>
#include <fstream>
#include <vector>
#include <string>
#include<cmath>
#include <math.h>
#include <sstream> //istringstream
#include <fstream> // ifstream

using namespace std;

#define N 500



vector<vector<float>> parse2DCsvFile(string inputFileName) {

    vector<vector<float> > data;
    ifstream inputFile(inputFileName);
    int l = 0;

    while (inputFile) {
        l++;
        string s;
        if (!getline(inputFile, s)) break;
        if (s[0] != '#') {
            istringstream ss(s);
            vector<float> record;

            while (ss) {
                string line;
                if (!getline(ss, line, ','))
                    break;
                try {
                    record.push_back(stof(line));
                }
                catch (const std::invalid_argument e) {
                    cout << "NaN found in file " << inputFileName << " line " << l
                        << endl;
                    e.what();
                }
            }

            data.push_back(record);
        }
    }

    if (!inputFile.eof()) {
        cerr << "Could not read file " << inputFileName << "\n";
        //__throw_invalid_argument("File not found.");
    }

    return data;
}


int main()
{

    float CSVdata[N][N];

    vector<vector<float>> data1 = parse2DCsvFile("data_from_CSV.csv");
    for (int i = 0; i < N; i++)
    {
        for (int j = 0; j < N; j++)
        {
            CSVdata[i][j] = data1[i][j];
        }
        cout << endl;
    }

    system("Pause");
    return 0;
}
c++
csv
large-data
mat
asked on Stack Overflow Jun 13, 2020 by Asif

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0