SQL Server timeout period elapses during pre-login handshake on remote connections

I'm having an odd issue where I get a timeout with the DB if there's no activity for about 30 mins. Then, after a second or so it starts working again. Here's the error in my .NET code:

System.Data.SqlClient.SqlException (0x80131904): Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=6118; handshake=0; ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out

Is there some feature of SQL Server where it goes to sleep after a certain period and then slowly wakes up again when probed?

All of the solutions I've found on the web are fixes that don't make sense given that it only happens after a consistent duration and then goes away after making another query.