Pycharm error code Process finished with exit code -1073741819 (0xC0000005)

0

I am trying to put multiple relatively large CSVs into a Df -- I am getting the error Process finished with exit code -1073741819 (0xC0000005) instead of it printing out my new DF. It makes it through the forloop ok but can't go any further.

I thought it was a memory error at first so I increased the memory heap in pycharm to 6gb. after that, I realized I was running 32bit python3 so I switched to 64bit python. I had a few issues in my pip environment so I switched to a conda environment. This is where I am at. Any advice?

The full terminal output looks like this:

C:\Users\Jack\Desktop\safegraph\2020-03-01-weekly-patterns.csv.gz 10%|█ | 1/10 [00:49<07:23, 49.25s/it]C:\Users\Jack\Desktop\safegraph\2020-03-08-weekly-patterns.csv.gz C:\Users\Jack\Desktop\safegraph\2020-03-15-weekly-patterns.csv.gz 30%|███ | 3/10 [02:21<05:31, 47.38s/it]C:\Users\Jack\Desktop\safegraph\2020-03-22-weekly-patterns.csv.gz 40%|████ | 4/10 [03:01<04:31, 45.20s/it]C:\Users\Jack\Desktop\safegraph\2020-03-29-weekly-patterns.csv.gz 50%|█████ | 5/10 [03:41<03:38, 43.73s/it]C:\Users\Jack\Desktop\safegraph\2020-04-05-weekly-patterns-corrected.csv.gz C:\Users\Jack\Desktop\safegraph\2020-04-12-weekly-patterns.csv.gz 70%|███████ | 7/10 [05:09<02:11, 43.80s/it]C:\Users\Jack\Desktop\safegraph\2020-04-19-weekly-patterns.csv.gz 80%|████████ | 8/10 [05:57<01:29, 44.95s/it]C:\Users\Jack\Desktop\safegraph\2020-04-26-weekly-patterns.csv.gz 90%|█████████ | 9/10 [06:47<00:46, 46.74s/it]C:\Users\Jack\Desktop\safegraph\2020-05-03-weekly-patterns.csv.gz

Process finished with exit code -1073741819 (0xC0000005)

The code looks like this:

import pandas as pd
import numpy as np
import sqlite3
import os
import psutil
from sqlalchemy import create_engine
from tqdm import tqdm
import warnings
import glob



### Add path to your unzipped data here -- OR -- add the file to this directory and change the f_path var to your files name +  extension
f_path = r"C:\Users\Jack\Desktop\safegraph"
all_files = glob.glob(os.path.join(f_path, "*.csv.gz"))
print(all_files)
svmem = psutil.virtual_memory()
print (svmem.available)
li = []
for filename in tqdm(all_files):
    print(filename)
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)
print("concatenating...")
frame = pd.concat(li, axis=0, ignore_index=True)
## Preview your dataset
print(frame.shape)
print(frame.head())

pandas
pycharm
out-of-memory
bigdata
asked on Stack Overflow May 19, 2020 by Jack Lindsay

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0