Basic HTTPServer hanging at "socket.readinto : return self._sock.recv_into(b)"

2

I have a very basic implementation of the http.server.HTTPServer i'm spinning up in Python 3.6. I'm using it to authenticate OAuth2 against the google Analytics API. As is standard, i've created a simple handle to receive the OAuth2 grant key and am using the serve_forever() method. However, the thread used to create the serve forever is hanging, and it seems to be stuck deep in the Python 3.6 implementation.

Interestingly, I don't think my code matters at all in this case. Look at the call stack that is being reported by listeners I setup for debugging:

service_action started
service_action complete
service_action started
service_action complete
service_action started
service_action complete
service_action started
service_action complete
"socketserver.BaseServer.serve_forever : self._handle_request_noblock()"
"socketserver.BaseServer._handle_request_noblock : self.get_request()"
"socketserver.BaseServer._handle_noblock_request : self.process_request()"
"socketserver.BaseServer.process_request : self.finish_request()"
"socketserver.BaseServer.finish_request : self.RequestHandlerClass(request, client_address, self) #  request == <socket.socket fd=1828, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8080), raddr=('127.0.0.1', 57504)>,  client_address == ('127.0.0.1', 57504),  self == <MDGCoreLib.Utilities.HttpServer.Server.Server object at 0x0000026304E3FC88>"
"socketserver.BaseRequestHandler.__init__ : set self.client_address"
"socketserver.BaseRequestHandler.__init__ : set self.server"
"socketserver.BaseRequestHandler.__init__ : setup()"
"socketserver.StreamRequestHandler.setup : self.connection = self.request"
"socketserver.BaseRequestHandler.__init__ : self.handle()"
"server.BaseHttpRequestHandler.handle  : self.close_connection = True"
"server.BaseHTTPServer.handle_one_request : self.raw_requestline = self.rfile.readline(65537)"
"socket.readinto : self._checkClosed()"
"socket.readinto : return self._sock.recv_into(b)  #b == <memory at 0x0000026304E38C48>"
"server.BaseHTTPHandler.handle_one_request : if len(self.raw_requestline) > 65536"
"server.BaseHTTPHandler.handle_one_request : if not self.raw_requestline"
"server.BaseHTTPHandler.handle_one_request : if not self.parse_request()"
"server.BaseHTTPHandler.handle_one_request : if not hassattr(self, mname)"
#My handler runs
*"OAuthGrantRequestHandler : self.send_response"
127.0.0.1 - - [27/Aug/2018 14:35:16] "GET /?(REDACTED) HTTP/1.1" 200 -
Request handler completed*
#My handler finishes

Parent thread : joining Thread 2
"server.BaseHTTPHandler.handle_one_request : flushing wfile to socket finishing request"
"socketserver.BaseRequestHandler.__init__ : self.finish()"
"socketserver.StreamRequestHandler.finish : "
"socketserver.BaseServer.process_request : self.shutdown_request"
service_action started
service_action complete
"socketserver.BaseServer.serve_forever : self._handle_request_noblock()"
"socketserver.BaseServer._handle_request_noblock : self.get_request()"
"socketserver.BaseServer._handle_noblock_request : self.process_request()"
"socketserver.BaseServer.process_request : self.finish_request()"
"socketserver.BaseServer.finish_request : self.RequestHandlerClass(request, client_address, self) #  request == <socket.socket fd=1840, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8080), raddr=('127.0.0.1', 57505)>,  client_address == ('127.0.0.1', 57505),  self == <MDGCoreLib.Utilities.HttpServer.Server.Server object at 0x0000026304E3FC88>"
"socketserver.BaseRequestHandler.__init__ : set self.client_address"
"socketserver.BaseRequestHandler.__init__ : set self.server"
"socketserver.BaseRequestHandler.__init__ : setup()"
"socketserver.StreamRequestHandler.setup : self.connection = self.request"
"socketserver.BaseRequestHandler.__init__ : self.handle()"
"server.BaseHttpRequestHandler.handle  : self.close_connection = True"
"server.BaseHTTPServer.handle_one_request : self.raw_requestline = self.rfile.readline(65537)"
"socket.readinto : self._checkClosed()"
"socket.readinto : return self._sock.recv_into(b)  #b == <memory at 0x0000026304E38C48>"
"server.BaseHTTPHandler.handle_one_request : if len(self.raw_requestline) > 65536"
"server.BaseHTTPHandler.handle_one_request : if not self.raw_requestline"
"server.BaseHTTPHandler.handle_one_request : if not self.parse_request()"
"server.BaseHTTPHandler.handle_one_request : if not hassattr(self, mname)"
#My Handler Runs
"OAuthGrantRequestHandler : self.send_response"
127.0.0.1 - - [27/Aug/2018 14:35:18] "GET /favicon.ico HTTP/1.1" 200 -
Request handler completed
#My Handler Finishes

"server.BaseHTTPHandler.handle_one_request : flushing wfile to socket finishing request"
"socketserver.BaseRequestHandler.__init__ : self.finish()"
"socketserver.StreamRequestHandler.finish : "
"socketserver.BaseServer.process_request : self.shutdown_request"
service_action started
service_action complete
"socketserver.BaseServer.serve_forever : self._handle_request_noblock()"
"socketserver.BaseServer._handle_request_noblock : self.get_request()"
"socketserver.BaseServer._handle_noblock_request : self.process_request()"
"socketserver.BaseServer.process_request : self.finish_request()"
"socketserver.BaseServer.finish_request : self.RequestHandlerClass(request, client_address, self) #  request == <socket.socket fd=2008, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8080), raddr=('127.0.0.1', 57506)>,  client_address == ('127.0.0.1', 57506),  self == <MDGCoreLib.Utilities.HttpServer.Server.Server object at 0x0000026304E3FC88>"
"socketserver.BaseRequestHandler.__init__ : set self.client_address"
"socketserver.BaseRequestHandler.__init__ : set self.server"
"socketserver.BaseRequestHandler.__init__ : setup()"
"socketserver.StreamRequestHandler.setup : self.connection = self.request"
"socketserver.BaseRequestHandler.__init__ : self.handle()"
"server.BaseHttpRequestHandler.handle  : self.close_connection = True"
"server.BaseHTTPServer.handle_one_request : self.raw_requestline = self.rfile.readline(65537)"
"socket.readinto : self._checkClosed()"
"socket.readinto : return self._sock.recv_into(b)  #b == <memory at 0x0000026304E38C48>"
The program '[54321] Python @ tcp://localhost:55098/?legacyUnitTest' has exited with code -1 (0xffffffff).

The above is everything from the thread running the "serve_forever()" method, and you can see the parent thread is waiting to join where output reads Parent thread : joining Thread 2. My handler, get encountered and completed twice when the output is "OAuthGrantRequestHandler : self.send_response" and Request handler completed: but then is never called again before the internal python library hangs at socket.readinto : return self._sock.recv_into(b) while trying to process the last request.

Anyone had this Python hangup before, and is there anyway around it? I'm kind of at a loss as its hanging between server_forever() loops where my handler/code is encountered.

UPDATE 1 So comparing fiddler traces to the requests in the HTTPServer I see there are only two HTTPRequest, but the HTTPServer.serve_forever() attempts to read a third request which is where it hangs. I've also verified (you can see it in the output above) that the "shutdown_request()" method and underlying methods are being called to shutdown the socket. So why does the serve_forever loop continue to pass the check below and call self._handle_request_noblock() if it hasn't recieved a new request:

ready = selector.select(poll_interval)
                    if ready:
                        self._handle_request_noblock()
python
python-3.x
httpserver
asked on Stack Overflow Aug 27, 2018 by Jamie Marshall • edited Aug 27, 2018 by Jamie Marshall

1 Answer

0

Had pretty much the same issue, so after digging through socketserver.StreamRequestHandler implementation I found how to set timeout so that those erroneous read attempts would eventually unblock.

    class MyRequestHandler(BaseHTTPRequestHandler):
        timeout = 2  # This is what makes the difference (seconds)

        def do_GET(self):
            ...
answered on Stack Overflow Nov 7, 2018 by kikobyte

User contributions licensed under CC BY-SA 3.0