PyQt5: too many signals fired in short timespan causes a stack overflow
1. Background information
I'm currently building a custom file explorer in Python 3.6 with PyQt5. The following figure shows how it looks like:
I use a multithreaded approach to keep the GUI responsive at all times. The side-threads take on a lot of work, but they cannot directly touch the GUI (add or remove widgets, refresh widgets, ...). Code in such a side-thread can emit signals to slots from the main-thread. That's what I typically do when interaction with the GUI is needed.
2. The problem
In my application is a specific function - let's call it someFunc() - that causes a stack overflow error when it runs. I've noticed that it only causes the error when a lot of files are opened in the custom explorer. This function runs from a side-thread and emits lots of signals to the main-thread. I believe the error is related to too many signal emissions. But we'll get into that later.
The error manifests itself in the following way:
The GUI freezes.
Windows 10 shows a popupwindow telling me that something went wrong, and asking me if I want to debug. I click "Debug"
Visual Studio opens. I can see the following error message:
Unhandled exception at 0x00007FF8D18E614A (ntdll.dll) in python.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x0000006F3B403FF0). occurred
The error message is sometimes a little different, like:Unhandled exception at 0x000000005ECE0688 (python36.dll) in python.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000EC38003DA0).The console from which I started my Python application doesn't show any error message.
3. The error reproduced in a minimal example
The full file explorer application is way to big to post here. So I have done my best to design a minimalistic example that reproduces the error. Please keep in mind that this example is not doing anything useful.
To run the sample code, please copy-past the code to a .py file and run it from your console. Make sure you are working in Python3.6 with PyQt5.
import sys
import threading
from PyQt5.QtWidgets import *
from PyQt5.QtCore import *
from PyQt5.QtGui import *
class File(QObject):
refresh_signal = pyqtSignal()
def __init__(self, name):
super().__init__()
self.__name = name
self.btn = QPushButton(name)
self.btn.setFixedHeight(30)
self.btn.setFixedWidth(50)
def refresh(self):
assert threading.current_thread() is threading.main_thread()
print("File '" + str(self.__name) + "' received the 'refresh()' signal in " + str(threading.current_thread()))
self.btn.style().unpolish(self.btn)
self.btn.style().polish(self.btn)
self.btn.update()
global app
app.processEvents()
# Do some heavy work
k = 0
for i in range(10000):
for j in range(100):
k += (i + j)
print("File '" + str(self.__name) + "' finished the 'refresh()' signal in " + str(threading.current_thread()))
def emit_signal(self):
assert threading.current_thread() is not threading.main_thread()
print("File '" + str(self.__name) + "' emits the 'refresh()' signal from " + str(threading.current_thread()))
self.refresh_signal.emit()
class Dir(QObject):
def __init__(self, name):
super().__init__()
self.__name = name
self.__fileList = []
for i in range(40):
file = File("file_" + str(i))
file.refresh_signal.connect(file.refresh)
self.__fileList.append(file)
def display(self, lyt):
assert threading.current_thread() is threading.main_thread()
i = 0
for file in self.__fileList:
if i < 10:
lyt[0].addWidget(file.btn)
elif i < 20:
lyt[1].addWidget(file.btn)
elif i < 30:
lyt[2].addWidget(file.btn)
else:
lyt[3].addWidget(file.btn)
i += 1
def loop_over_files(self):
assert threading.current_thread() is not threading.main_thread()
for file in self.__fileList:
file.emit_signal()
class MyApplication(QMainWindow):
my_signal = pyqtSignal(object)
def __init__(self):
super().__init__()
# -------------------------------- #
# Window setup #
# -------------------------------- #
# 1. Define the geometry of the main window
# ------------------------------------------
self.setGeometry(100, 100, 800, 800)
self.setWindowTitle("Pyqt signals tester")
# 2. Create frame and layout
# ---------------------------
self.__frm = QFrame(self)
self.__frm.setStyleSheet("QWidget { background-color: #efefef }")
self.__lyt = QVBoxLayout()
self.__frm.setLayout(self.__lyt)
self.setCentralWidget(self.__frm)
# 3. Insert a button
# -------------------
self.__btn = QPushButton("fire signals")
self.__btn.setFixedWidth(100)
self.__btn.setFixedHeight(50)
self.__btn.clicked.connect(self.__btn_clicked)
self.__lyt.addWidget(self.__btn)
# 4. Creat frame for the file objects
# ------------------------------------
self.__fileLyt = []
self.__fileLyt.append(QHBoxLayout()) # row 0
self.__fileLyt.append(QHBoxLayout()) # row 1
self.__fileLyt.append(QHBoxLayout()) # row 2
self.__fileLyt.append(QHBoxLayout()) # row 3
self.__lyt.addLayout(self.__fileLyt[0])
self.__lyt.addLayout(self.__fileLyt[1])
self.__lyt.addLayout(self.__fileLyt[2])
self.__lyt.addLayout(self.__fileLyt[3])
# -------------------------------- #
# Test setup #
# -------------------------------- #
self.__rootDir = Dir("rootDir")
self.__rootDir.display(self.__fileLyt)
self.__threadSpawner = ThreadSpawner()
self.my_signal.connect(self.__threadSpawner.workerObject.doWork)
self.show()
def __btn_clicked(self):
self.my_signal.emit(self.__rootDir)
class ThreadSpawner(QObject):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# 1. Create 'workerThread' and 'workerObject' objects
# ----------------------------------------------------
self.workerThread = QThread()
self.workerObject = WorkerObject()
self.workerObject.moveToThread(self.workerThread)
# 2. Connect all relevant signals
# --------------------------------
self.workerThread.finished.connect(self.workerObject.deleteLater)
self.workerThread.finished.connect(lambda: print("workerThread finished."))
# 3. Start the thread
# --------------------
self.workerThread.start()
def __del__(self):
self.workerThread.quit()
self.workerThread.wait()
class WorkerObject(QObject):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@pyqtSlot(object)
def doWork(self, dir):
dir.loop_over_files()
if __name__ == '__main__':
global app
app = QApplication(sys.argv)
QApplication.setStyle(QStyleFactory.create('Fusion'))
myApplication = MyApplication()
sys.exit(app.exec_())
So, what does this code do? It creates a window with a "fire" button on top that starts the test. Below that button is a matrix of buttons, each of them representing a File()-object.
When you click the "fire" button, it calls the loop_over_files() function in the one and only Dir()-object. This function loops over all the File() objects and calls their emit_signal() function - one after the other very quickly. All of this happens in the workerThread.
As the drawing above shows, the emit_signal() function emits a signal (duh..) which crosses the bridge from the workerThread to the mainThread. The slot is the refresh() function, doing some GUI refreshing.
I've also added some "heavy work" to the refresh function, to trigger the error more easily.
3.1 Generated output
Click the "fire" button, and you'll see some output like this:
File 'file_0' emits the 'refresh()' signal from <_DummyThread(Dummy-1, started daemon 12580)>
File 'file_1' emits the 'refresh()' signal from <_DummyThread(Dummy-1, started daemon 12580)>
File 'file_2' emits the 'refresh()' signal from <_DummyThread(Dummy-1, started daemon 12580)>
File 'file_3' emits the 'refresh()' signal from <_DummyThread(Dummy-1, started daemon 12580)>
...
File 'file_0' received the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
File 'file_1' received the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
File 'file_2' received the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
File 'file_3' received the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
...
File 'file_0' finished the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
File 'file_1' finished the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
File 'file_2' finished the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
File 'file_3' finished the 'refresh()' signal in <_MainThread(MainThread, started 1324)>
...
The output is generally mixed up because of the multithreaded nature.
3.2 Trigger the error
I get the stack overflow error only if I push the limits by adding much more File()-objects. Remember, each of the File()-objects emits a signal. I believe the error is due to an abundance of signals getting fired in a short timespan.
In the Dir class, change the for-loop range from 40 to 160 (so you multiply the amount of File() objects):
for i in range(160):
file = File("file_" + str(i))
file.refresh_signal.connect(file.refresh)
self.__fileList.append(file)
In the same class, adapt the display() function to display the File()-objects properly:
def display(self, lyt):
assert threading.current_thread() is threading.main_thread()
i = 0
for file in self.__fileList:
if i < 40:
lyt[0].addWidget(file.btn)
elif i < 80:
lyt[1].addWidget(file.btn)
elif i < 120:
lyt[2].addWidget(file.btn)
else:
lyt[3].addWidget(file.btn)
i += 1
Run the script. You should get a window like this:
Click the "fire" button and behold, there is your stack overflow error! It looks like PyQt5 has a problem dealing with 160 signals emitted in a short timespan.
3.3 Error workaround
I found a way to "work around" the error. But it's an ugly solution. Consider the emit_signal() function in the File class. Add the codeline time.sleep(0.010) at the end of the function:
def emit_signal(self):
assert threading.current_thread() is not threading.main_thread()
print("File '" + str(self.__name) + "' emits the 'refresh()' signal from " + str(threading.current_thread()))
self.refresh_signal.emit()
time.sleep(0.010)
Now rerun the script, and behold: there is no more stack overflow error! I believe this tiny delay helps Python to overcome the massive influx of signals somehow. In other words: the influx of signals is spread out just a tiny bit more in time...
But I don't like the solution. It doesn't solve the issue at the "problem root". It is like a tissue for the bleeding - nothing more. I want something more stable.
4. Solution
Please help me to find a solution, cleaner than the time-delay shown above.
EDIT:
I've noticed that the actual problem is app.processEvents(). When I remove that codeline from the refresh() function, it works - no more stack overflow! For some weird reason, PyQt5 struggles with a stack overflow if you call the processEvents() function too often. However, I cannot remove it from the application.That would bring down the GUI responsiveness.
I found the function hasPendingEvents(), which could tell me if I need to call app.processEvents() or not. Unfortunately, the Qt docs tell me that this function is not thread safe. It must be called from the main-thread (that's okay) but no other threads must be running in the entire application (that's not okay). You can find the docs here: http://doc.qt.io/qt-5/qcoreapplication-obsolete.html#hasPendingEvents
K.Mulier0 Answers
Nobody has answered this question yet.
User contributions licensed under CC BY-SA 3.0