python subprocess waiting for grandchild on Windows with stdout set

0

I have a script that is part of an automated test suite. It runs very slowly on Windows but not on Linux and I have found out why. The process that we are testing ('frank') creates a child process (so a grandchild). The python code won't return until that grandchild process also ends (on Windows - doesn't do this on Linux). The grandchild process will kill itself off after 5 seconds if there is no parent (it hangs around in case another process talks to it)

I've found I can stop the communicate function from hanging in this way if I don't capture stdout. But I need stdout. I read somewhere that the communicate function is waiting for all pipes to be closed. I know that the stdout handle is duplicated for the grandchild but I can't change the code I'm testing.

I've been searching for a solution. I tried some creation flags (still in the code) but that didn't help.

This is the cut down test -

import os
import sys
import threading
import subprocess


def read_from_pipe(process):
    last_stdout = process.communicate()[0]
    print (last_stdout)

CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008

# start process
command = 'frank my arguments'

cwd = "C:\\dev\\ui_test\\frank_test\\workspace\\report183"

p = subprocess.Popen(command,
                     stdout=subprocess.PIPE,
                     cwd=cwd)

# run thread to read from output
t = threading.Thread(target=read_from_pipe, args=[p])
t.start()
t.join(30)
print('finished')

Any ideas?

Thanks.

Peter.

python
windows
subprocess
pipe
stdout
asked on Stack Overflow Mar 14, 2019 by Peter S

1 Answer

0

After tips from @eryksun and a lot of Googling, I have this rather complicated lot of code! At one point, I considered cheating and doing os.system and redirecting to a temp file but then I realised that our test code allows for a command timing out. os.system would just block forever if the child process doesn't die.

import os
import sys
import threading
import subprocess
import time
if os.name == 'nt':
    import msvcrt
    import ctypes

# See https://stackoverflow.com/questions/55160319/python-subprocess-waiting-for-grandchild-on-windows-with-stdout-set for details on Windows code
# Based on https://github.com/it2school/Projects/blob/master/2017/Python/party4kids-2/CarGame/src/pygame/tests/test_utils/async_sub.py

from ctypes.wintypes import DWORD
if sys.version_info >= (3,):
    null_byte = '\x00'.encode('ascii')
else:
    null_byte = '\x00'

def ReadFile(handle, desired_bytes, ol = None):
    c_read = DWORD()
    buffer = ctypes.create_string_buffer(desired_bytes+1)
    success = ctypes.windll.kernel32.ReadFile(handle, buffer, desired_bytes, ctypes.byref(c_read), ol)
    buffer[c_read.value] = null_byte
    return ctypes.windll.kernel32.GetLastError(), buffer.value


def PeekNamedPipe(handle):
    c_avail = DWORD()
    c_message = DWORD()
    success = ctypes.windll.kernel32.PeekNamedPipe(handle, None, 0, None, ctypes.byref(c_avail), ctypes.byref(c_message))
    return "", c_avail.value, c_message.value


def read_available(handle):
    buffer, bytesToRead, result = PeekNamedPipe(handle)
    if bytesToRead:
        hr, data = ReadFile(handle, bytesToRead, None)
        return data
    return b''

def read_from_pipe(process):
    if os.name == 'posix':
        last_stdout = process.communicate()[0]
    else:
        handle = msvcrt.get_osfhandle(process.stdout.fileno())
        last_stdout = b''
        while process.poll() is None:
            last_stdout += read_available(handle)
            time.sleep(0.1)
        last_stdout += read_available(handle)
    print (last_stdout)

# start process
command = 'frank my arguments'

cwd = "C:\\dev\\ui_test\\frank_test\\workspace\\report183"

p = subprocess.Popen(command,
                     stdout=subprocess.PIPE,
                     cwd=cwd)

# run thread to read from output
t = threading.Thread(target=read_from_pipe, args=[p])
t.start()
t.join(30)
print('finished')
answered on Stack Overflow Mar 15, 2019 by Peter S

User contributions licensed under CC BY-SA 3.0