Popen subprocess process stop reading after a specific reply

1

I'm using pygdbmi and I think I've hit a bug there. In a nutshell: it seems that non-blocking subprocess Pipe is interacting in a funny way with GDB.

My problem is NOT to read from a subprocess started with popen, that's working. The problem seems that the subprocess (GDB in this case) is printing something to its stdout, I can read it, but after a very specific GDB command, something breaks and then I can't read anymore. This doesn't seem a matter of how many GDB commands are sent (and thus how much data is exchanged through the pipe).

I'm using pygdbmi to talk to an embedded board through JLink JLinkGDBServerCL. This is working 99.999 % of the time. Pygdbmi uses subprocess.Popen(stdin=subprocess.PIPE) to spawn a GDB process. I then use

response = gdbmi.write('-target-select remote localhost:2331', timeout_sec=5)

To connect to a running instance of JLinkGDBServerCL and interact with my board. I can download code, set breakpoints, interrupt, start again, the works. I can even send -data-list-register-values x to a board running a Cortex-M3 core. When I try to run the same command on a board with an Arm Cortex-M33 board, I don't get any reply.

I've enabled GDB logging with

-gdb-set trace-commands on
-gdb-set logging on

and if I check on my gdb.txt file, I get the expected reply:

^done,register-values=[{number="0",value="0x0"},{number="1",value="0x0"},{number="2",value="0x0"},{number="3",value="0x80730"},{number="4",value="0x40000100"},{number="5",value="0x101"},{number="6",value="0x3ff01"},{number="7",value="0x0"},{number="8",value="0xffffffff"},{number="9",value="0x40001430"},{number="10",value="0xffffffff"},{number="11",value="0xffffffff"},{number="12",value="0xffffffff"},{number="13",value="0x20001e58"},{number="14",value="0x20001e69"},{number="15",value="0xeffffffe"},{number="25",value="0x41000003"},{number="91",value="0x20001e58"},{number="92",value="0x0"},{number="93",value="0x1"},{number="94",value="0x0"},{number="95",value="0x0"},{number="96",value="0x0"},{number="97",value="0x0"},{number="98",value="0x0"},{number="99",value="0x0"},{number="100",value="0x0"},{number="101",value="0x0"},{number="102",value="0x0"},{number="103",value="0x0"},{number="104",value="0x0"},{number="105",value="0x0"},{number="106",value="0x0"},{number="107",value="0x0"},{number="108",value="0x0"},{number="109",value="0x0"},{number="110",value="0x0"},{number="111",value="0x0"},{number="112",value="0x0"},{number="113",value="0x0"},{number="114",value="0x0"},{number="115",value="0x0"},{number="116",value="0x0"},{number="117",value="0x0"},{number="118",value="0x0"},{number="119",value="0x0"},{number="120",value="0x0"},{number="121",value="0x0"},{number="122",value="0x0"},{number="123",value="0x0"},{number="124",value="0x0"},{number="125",value="0x0"},{number="126",value="0x0"},{number="127",value="0x0"},{number="128",value="0x0"},{number="129",value="0x0"},{number="130",value="0x0"},{number="131",value="0x0"},{number="132",value="0x0"},{number="133",value="0x0"},{number="134",value="0x0"},{number="135",value="0x0"},{number="136",value="0x0"},{number="137",value="0x0"},{number="138",value="0x0"},{number="139",value="0x0"},{number="140",value="0x0"},{number="141",value="0x0"},{number="142",value="0x0"},{number="143",value="0x0"},{number="144",value="0x0"},{number="145",value="0x0"},{number="146",value="0x20001e58"},{number="147",value="0x0"},{number="148",value="0x0"},{number="149",value="0x0"},{number="150",value="0x0"},{number="151",value="0x0"},{number="152",value="0xfffffffc"},{number="153",value="0x0"},{number="154",value="0x0"},{number="155",value="0x0"},{number="156",value="0x0"},{number="157",value="0x1"},{number="158",value="0x0"},{number="159",value="0x0"},{number="160",value="0x0"},{number="161",value="0x0"}]

This means pygdbmi is sending the command to the child GDB process, but it's not being able to read. Pygdbmi is using this gist to make readline() non-blocking:

def make_non_blocking(file_obj: io.IOBase):
    """make file object non-blocking
    Windows doesn't have the fcntl module, but someone on
    stack overflow supplied this code as an answer, and it works
    http://stackoverflow.com/a/34504971/2893090"""

    if USING_WINDOWS:
        LPDWORD = POINTER(DWORD)
        PIPE_NOWAIT = wintypes.DWORD(0x00000001)

        SetNamedPipeHandleState = windll.kernel32.SetNamedPipeHandleState
        SetNamedPipeHandleState.argtypes = [HANDLE, LPDWORD, LPDWORD, LPDWORD]
        SetNamedPipeHandleState.restype = BOOL

        h = msvcrt.get_osfhandle(file_obj.fileno())

        res = windll.kernel32.SetNamedPipeHandleState(h, byref(PIPE_NOWAIT), None, None)
        if res == 0:
            raise ValueError(WinError())

    else:
        # Set the file status flag (F_SETFL) on the pipes to be non-blocking
        # so we can attempt to read from a pipe with no new data without locking
        # the program up
        fcntl.fcntl(file_obj, fcntl.F_SETFL, os.O_NONBLOCK)

And it's reading with:

        while True:
            responses_list = []
            try:
                self.stdout.flush()
                raw_output = self.stdout.readline().replace(b"\r", b"\n")
                responses_list = self._get_responses_list(raw_output, "stdout")
            except IOError as e:
                pass

Because readline() is non-blocking, it throws many exceptions, but ends up reading GDB's output. That is, until -data-list-register-values x. The funny thing is that if I use -data-list-register-values (omitting the format specifier), I can read GDB's error message complaining about the missing argument:

        response = gdbmi.write('-data-list-register-values', timeout_sec=10)
        pprint(response)

[{'message': 'error',
  'payload': {'msg': '-data-list-register-values: Usage: '
                     '-data-list-register-values [--skip-unavailable] <format> '
                     '[<regnum1>...<regnumN>]'},
  'stream': 'stdout',
  'token': None,
  'type': 'result'}]

At the very bottom of the GDB log, I see

~"Exception condition detected on fd 0\n"
~"error detected on stdin\n"

I'm not sure if this is a red-herring or not.

Any suggestions on how to debug why readline() is not actually reading the output from GDB?

python
subprocess
gdb
pipe
popen
asked on Stack Overflow Mar 23, 2021 by Leonardo

1 Answer

0

Turns out that the problem is with JLinkGDBServerCL.

I was originally spawning the subprocess with:

command = ['C:\\Program Files (x86)\\SEGGER\\JLink_V694d\\JLinkGDBServerCL.exe',
'-select', 'USB', '-if', 'SWD', '-device', 'RSL15', '-endian', 'little', '-speed', 
'1000', '-port', '2331', '-vd', '-ir', '-localhostonly', '1', '-noreset', '-singlerun',
'-strict', '-timeout 0', '-nogui']

self.gdbServer = subprocess.Popen(command,
    shell=False,
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE,
    stderr=subprocess.PIPE)

Turns out that I have to call popen with:

self.gdbServer = subprocess.Popen(command,
    shell=False,
    stdout=subprocess.DEVNULL,
    stdin=subprocess.PIPE,
    stderr=subprocess.PIPE)

I have no idea why, considering that I'm never reading from self.gdbServer.stdout. I've also tried to use make_non_blocking:

from pygdbmi.IoManager import make_non_blocking

self.gdbServer = subprocess.Popen(command,
    shell=False,
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE,
    stderr=subprocess.PIPE)
make_non_blocking(self.gdbServer.stdout)
answered on Stack Overflow Mar 29, 2021 by Leonardo

User contributions licensed under CC BY-SA 3.0