PyODBC Cursor.fetchall() causes python to crash (segfault)

I am using Python 2.7 on Windows XP.

I have a simple python script on a schedule that uses pyodbc to grab data from an AR database which has worked perfectly until today. I get a segfault once the cursor reaches a particular row. I have similar code in C++ which has no problem retrieving the results, so I figure this is an issue with pyodbc. Either way, I'd like to "catch" this error. I've tried to use the subprocess module, but it doesn't seem to work since once the script hits a segfault it just hangs on the "python.exe has encountered a problem and needs to close." message. I guess I could set some arbitrary time frame for it to complete in and, if it doesn't, force close the process, but that seems kind of lame.

I have reported the issue here as well - http://code.google.com/p/pyodbc/issues/detail?id=278

@paulsm4 - I have answered your questions below, thanks!

Q: You're on Windows/XP (32-bit, I imagine), Python 2.7, and BMC Remedy AR. Correct?

A: Yes, it fails on Win XP 32 bit and Win Server 2008 R2 64 bit.

Q: Is there any chance you (or perhaps your client, if they purchased Remedy AR) can open a support call with BMC?

A: Probably not...

Q: Can you isolate which column causes the segfault? "What's different" when the segfault occurs?

A: Just this particular row...but I have now isolated the issue with your suggestions below. I used a loop to fetch each field until a segfault occurred.

cursor.columns(table="mytable")
result = cursor.fetchall()
columns = [x[3] for x in result]
for x in columns:
    print x
    cursor.execute("""select "{0}"
                      from "mytable"
                      where id = 'abc123'""".format(x))
    cursor.fetchall()

Once I identified the column that causes the segfault I tried a query for all columns EXCEPT that one and sure enough it worked no problem.

The column's data type was CHAR(1024). I used C++ to grab the data and noticed that the column for that row had the most characters in it out of any other row...1023! Thinking that maybe there is a buffer in the C code for PyODBC that is getting written to beyond its boundaries.

2) Enable ODBC tracing: http://support.microsoft.com/kb/274551

3) Post back the results (including the log trace of the failure)

Ok, I have created a pastebin with the results of the ODBC trace - http://pastebin.com/6gt95rB8. To protect the innocent, I have masked some string values.

Looks like it may have been due to data truncation.

Does this give us enough info as to how to fix the issue? I'm thinking it's a bug within PyODBC since using the C ODBC API directly works fine.

Update

So I compiled PyODBC for debugging and I got an interesting message -

Run-Time Check Failure #2 - Stack around the variable 'tempBuffer' was corrupted.

While I don't currently understand it, the call stack is as follows -

pyodbc.pyd!GetDataString(Cursor * cur=0x00e47100, int iCol=0)  Line 410 + 0xf bytes C++
pyodbc.pyd!GetData(Cursor * cur=0x00e47100, int iCol=0)  Line 697 + 0xd bytes   C++
pyodbc.pyd!Cursor_fetch(Cursor * cur=0x00e47100)  Line 1032 + 0xd bytes C++
pyodbc.pyd!Cursor_fetchlist(Cursor * cur=0x00e47100, int max=-1)  Line 1063 + 0x9 bytes C++
pyodbc.pyd!Cursor_fetchall(_object * self=0x00e47100, _object * args=0x00000000)  Line 1142 + 0xb bytes C++

Resolved!

Problem was solved by ensuring that the buffer had enough space.

In getdata.cpp on line 330

char tempBuffer[1024];

Was changed to

char tempBuffer[1025];

Compiled and replaced the old pyodbc.pyd file in site-packages and we're all good!

Thanks for your help!

Q: You're on Windows/XP (32-bit, I imagine), Python 2.7, and BMC Remedy AR. Correct?

Q: Is there any chance you (or perhaps your client, if they purchased Remedy AR) can open a support call with BMC?

Q: Can you isolate which column causes the segfault? "What's different" when the segfault occurs?

Please do the following:

1) Try different "select a,b,c" statements using Python/ODBC to see if you can reproduce the problem (independent of your program) and isolate a specific column (or, ideally, a specific column and row!)

2) Enable ODBC tracing: http://support.microsoft.com/kb/274551

3) Post back the results (including the log trace of the failure)

4) If that doesn't work - and if you can't get BMC Technical Support involved - then Plan B might be to debug at the ODBC library level:

• How to debug C extensions for Python on Windows

  Q: What C/C++ compiler would work best for you?