I have a script that is part of an automated test suite. It runs very slowly on Windows but not on Linux and I have found out why. The process that we are testing ('frank') creates a child process (so a grandchild). The python code won't return until that grandchild process also ends (on Windows - doesn't do this on Linux). The grandchild process will kill itself off after 5 seconds if there is no parent (it hangs around in case another process talks to it)
I've found I can stop the communicate function from hanging in this way if I don't capture stdout. But I need stdout. I read somewhere that the communicate function is waiting for all pipes to be closed. I know that the stdout handle is duplicated for the grandchild but I can't change the code I'm testing.
I've been searching for a solution. I tried some creation flags (still in the code) but that didn't help.
This is the cut down test -
import os
import sys
import threading
import subprocess
def read_from_pipe(process):
last_stdout = process.communicate()[0]
print (last_stdout)
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
# start process
command = 'frank my arguments'
cwd = "C:\\dev\\ui_test\\frank_test\\workspace\\report183"
p = subprocess.Popen(command,
stdout=subprocess.PIPE,
cwd=cwd)
# run thread to read from output
t = threading.Thread(target=read_from_pipe, args=[p])
t.start()
t.join(30)
print('finished')
Any ideas?
Thanks.
Peter.
After tips from @eryksun and a lot of Googling, I have this rather complicated lot of code! At one point, I considered cheating and doing os.system and redirecting to a temp file but then I realised that our test code allows for a command timing out. os.system would just block forever if the child process doesn't die.
import os
import sys
import threading
import subprocess
import time
if os.name == 'nt':
import msvcrt
import ctypes
# See https://stackoverflow.com/questions/55160319/python-subprocess-waiting-for-grandchild-on-windows-with-stdout-set for details on Windows code
# Based on https://github.com/it2school/Projects/blob/master/2017/Python/party4kids-2/CarGame/src/pygame/tests/test_utils/async_sub.py
from ctypes.wintypes import DWORD
if sys.version_info >= (3,):
null_byte = '\x00'.encode('ascii')
else:
null_byte = '\x00'
def ReadFile(handle, desired_bytes, ol = None):
c_read = DWORD()
buffer = ctypes.create_string_buffer(desired_bytes+1)
success = ctypes.windll.kernel32.ReadFile(handle, buffer, desired_bytes, ctypes.byref(c_read), ol)
buffer[c_read.value] = null_byte
return ctypes.windll.kernel32.GetLastError(), buffer.value
def PeekNamedPipe(handle):
c_avail = DWORD()
c_message = DWORD()
success = ctypes.windll.kernel32.PeekNamedPipe(handle, None, 0, None, ctypes.byref(c_avail), ctypes.byref(c_message))
return "", c_avail.value, c_message.value
def read_available(handle):
buffer, bytesToRead, result = PeekNamedPipe(handle)
if bytesToRead:
hr, data = ReadFile(handle, bytesToRead, None)
return data
return b''
def read_from_pipe(process):
if os.name == 'posix':
last_stdout = process.communicate()[0]
else:
handle = msvcrt.get_osfhandle(process.stdout.fileno())
last_stdout = b''
while process.poll() is None:
last_stdout += read_available(handle)
time.sleep(0.1)
last_stdout += read_available(handle)
print (last_stdout)
# start process
command = 'frank my arguments'
cwd = "C:\\dev\\ui_test\\frank_test\\workspace\\report183"
p = subprocess.Popen(command,
stdout=subprocess.PIPE,
cwd=cwd)
# run thread to read from output
t = threading.Thread(target=read_from_pipe, args=[p])
t.start()
t.join(30)
print('finished')
User contributions licensed under CC BY-SA 3.0