dask concat fails for unequal sized dataframes

3

I am experiencing a strange behavior when performing a concatenation of two dask dataframes (lazy objects) that have different number of columns/rows. The dataframes are read from hdf5 files using:

df1 = dd.read_hdf( f1, 'hf', mode='r' )

the final concat is:

concat_df = dask.concat( list_of_data_frames_list, axis=0 )

The operation completes but a subsequent persist or compute of the concatenated dataframe causes Python to exit with:

Process finished with exit code -1073741819 (0xC0000005)

if i do the same thing using two identical dataframes, concat works fine. it also works fine if the dataframes are read from csv files and not hdf5!.

i am using Python 3.6.6, Pandas 0.23.3 and dask 0.18.2

any thoughts?

python-3.x
concat
dask
asked on Stack Overflow Aug 20, 2018 by Kostas Markakis • edited Aug 22, 2018 by Kostas Markakis

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0