When joining two pandas dataframes with MultiIndex that are not "in order", my pandas installation makes python crash with error code 0xC00000FD. It took me a while to find my bug, and when I found it I got even more confused. Why did this happen, and how should I get better at spotting it?
Consider the following code:
import pandas as pd
df = pd.DataFrame.from_records([
("foo", 1, .1),
("foo", 2, .2),
("bar", 1, .3),
("bar", 2, .4)
], columns=["Level1", "Level2", "Value"])
df2 = df.set_index(["Level1","Level2"])
df3 = df.set_index(["Level2","Level1"])
combination = pd.merge(left=df2, right=df2, left_index=True, right_index=True)
print("ok")
combination2 = pd.merge(left=df2, right=df3, left_index=True, right_index=True)
print("fail")
It gives the following output:
Level1 Level2 Value
0 foo 1 0.1
1 foo 2 0.2
Value
Level1 Level2
foo 1 0.1
2 0.2
Value
Level2 Level1
1 foo 0.1
2 foo 0.2
ok
Process finished with exit code -1073741571 (0xC00000FD)
The problem arose when I had a table, made some pivots/melts, and wanted to go back to the original multiindex and join my results. At first I did not understand the error message. From this page I learned that the error code is a stack overflow. From this PR to pandas I learned that the recursion is somehow related to calculating group indices. This information made me able to hunt down my bug and also create a minimal reproducing example.
Now to my question:
EDIT The error as stated occurs in the latsest python and pandas to date, pandas 0.24.2 and python 3.7.3
User contributions licensed under CC BY-SA 3.0