Apache22 on FreeBSD - Starts, does not respond to requests

I'm running Apache 2.2.17 with the peruser MPM on FreeBSD 8.2-RC1 on Amazon's EC2 (so it's XEN). It was installed from ports.

My problem is that, although Apache is running, listening for, and accepting connections, it doesn't actually respond to any or show them in the log at all.

If I telnet to the port it's listening on and type out an HTTP request:

GET / HTTP/1.1
Host: asdfasdf

And hit enter a couple of times, it just sits there... Nothing. No response requesting with a browser either. There doesn't appear to be anything helpful in the error log:

[Sun Jan 09 16:56:24 2011] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Sun Jan 09 16:56:25 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 09 16:56:25 2011] [notice] Digest: done
[Sun Jan 09 16:56:25 2011] [notice] Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 

The access log stays empty:

root:/var/log# wc httpd-access.log 
       0       0       0 httpd-access.log
root:/var/log#

I've tried with accf_http and accf_data both enabled and disabled, and with both the stock configuration and my customized config. I also tried uninstalling apache22-peruser-mpm and just installing straight apache22... Still no luck. I tried removing all of the LoadModule lines from httpd.conf and just re-enabled the ones that were necessary to parse the config. Ended up with only the following loaded:

root:/usr/local/etc/apache22# /usr/local/sbin/apachectl -M
Loaded Modules:
 core_module (static)
 mpm_peruser_module (static)
 http_module (static)
 so_module (static)
 authz_host_module (shared)
 log_config_module (shared)
 alias_module (shared)
Syntax OK
root:/usr/local/etc/apache22#

Same results.

Apache is definitely what's listening on port 80:

root:/usr/local/etc/apache22# sockstat -4 | grep httpd
root     httpd      43789 3  tcp4 6 *:80                  *:*
root     httpd      43789 4  tcp4   *:*                   *:*
root:/usr/local/etc/apache22#

And I know it's not a firewall issue as there is nothing running locally, and connecting from the local box to 127.0.0.1:80 results in the same issue.

Does anyone have any idea what's going on? Why it would be doing this? I've exhausted all of my debugging expertise. :/

Thanks for any suggestions!

EDIT: As per Phil Hollenback's suggestion, I ran a trace on the system calls. This is what I found through both ktrace and truss.

48443 httpd    CALL  select(0,0,0,0,0xbf7feab4)
48443 httpd    RET   select 0
48443 httpd    CALL  gettimeofday(0xbf7feae4,0)
48443 httpd    RET   gettimeofday 0
48443 httpd    CALL  wait4(0xffffffff,0xbf7fea98,WNOHANG|WUNTRACED,0)
48443 httpd    RET   wait4 -1 errno 10 No child processes
48443 httpd    CALL  select(0,0,0,0,0xbf7feab4)
48443 httpd    RET   select 0
48443 httpd    CALL  gettimeofday(0xbf7feae4,0)
48443 httpd    RET   gettimeofday 0
48443 httpd    CALL  wait4(0xffffffff,0xbf7fea98,WNOHANG|WUNTRACED,0)
48443 httpd    RET   wait4 -1 errno 10 No child processes
48443 httpd    CALL  select(0,0,0,0,0xbf7feab4)

Same thing repeated over and over. (Those are the last few lines.) What appears to be happening is it's running through to the select, hangs a moment, returns 0, then tries again. I don't know if that's normal behaviour, but it seems okay to me as it sounds like it's just polling the socket?

What I am kind of concerned about is the wait returning an error that the process doesn't exist. The output of truss (which I started the process with, getting the initial setup) shows a call to fork() that seems to return fine (returns a PID, no error).

In case it helps, the output of truss: http://nucleardog.com/apache-truss.txt (Updated for httpd -X.)

I did actually connect during that run - select never seems to return a new connection? As well, I turned LogLevel up to Debug in the Apache config and tried running again - it never logs any connection.

I think this is perhaps more of a FreeBSD issue than Apache, then, as it seems Apache never receives the request. :/

Any (more) ideas?

EDIT 2: This appears to be an issue with something in FreeBSD, given that Apache never seems to receive the connection request. I made a post at the FreeBSD forums (forums.freebsd.org/showthread.php?p=118612) to see if I can find any help there. Thanks folks for your time!

Try the following:

1. pstree | less to find the master apache process pid
2. strace -p <pid> -f to watch the system calls from the process form step 1.
3. connect to apache with telnet or a web browser.

You may need to install pstree and strace from freebsd ports. You should see scrolling output from strace as it logs the system calls (file opens, etc.) generated by the running apache process. This should give you some clue as to whether apache is actually running and just not producing output for some reason.

This page about debugging apache contains the above info as well as other troubleshooting tips you can use.

Also you mention your access log but not your error log, have you looked at that? There could be some additional diagnostic info there.

EDIT: Duh, you did say you looked at error log, sorry. Also yes, truss is the default freebsd call tracing tool, not strace. Using truss is a better answer since you don't have to compile it from ports like strace.