API Explorer gives error - Failed to retrieve API configs with status: 500

I have been building a backend for my Android App using GAE and I am using Endpoints API. I have been able to deploy the backend successfully until now on the dev server. Today, out of the blue I am getting this exception while deploying the backend. I haven't changed the backend code at all but was exclusively working on the frontend. Following is the exception:

    "C:\Program Files\Java\jdk1.8.0_102\bin\java" -javaagent:C:\Users\Digvijay\.gradle\appengine-sdk\appengine-java-sdk-1.9.49\lib\agent\appengine-agent.jar -Xbootclasspath/p:C:\Users\Digvijay\.gradle\appengine-sdk\appengine-java-sdk-1.9.49\lib\override\appengine-dev-jdk-overrides.jar -Didea.launcher.port=7533 "-Didea.launcher.bin.path=C:\Program Files\Android\Android Studio\bin" -Dfile.encoding=windows-1252 -classpath "C:\Users\Digvijay\.gradle\appengine-sdk\appengine-java-sdk-1.9.49\lib\appengine-tools-api.jar;C:\Program Files\Android\Android Studio\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain com.google.appengine.tools.development.DevAppServerMain --address=localhost --port=8080 C:\Users\Digvijay\AndroidStudioProjects\LearnCity\backend\build\exploded-app
    Feb 09, 2017 9:51:16 PM java.util.prefs.WindowsPreferences <init>
    WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
    Feb 09, 2017 9:51:18 PM com.google.appengine.tools.development.SystemPropertiesManager setSystemProperties
    INFO: Overwriting system property key 'java.util.logging.config.file', value 'C:\Users\Digvijay\.gradle\appengine-sdk\appengine-java-sdk-1.9.49\config\sdk\logging.properties' with value 'WEB-INF/logging.properties' from 'C:\Users\Digvijay\AndroidStudioProjects\LearnCity\backend\build\exploded-app\WEB-INF\appengine-web.xml'
    Feb 09, 2017 9:51:18 PM com.google.apphosting.utils.jetty.JettyLogger info
    INFO: Logging to JettyLogger(null) via com.google.apphosting.utils.jetty.JettyLogger
    Feb 09, 2017 4:21:18 PM com.google.apphosting.utils.jetty.JettyLogger info
    INFO: jetty-6.1.x
    Feb 09, 2017 4:21:19 PM com.google.apphosting.utils.jetty.JettyLogger info
    INFO: Started SelectChannelConnector@localhost:8080
    Feb 09, 2017 4:21:19 PM com.google.appengine.tools.development.AbstractModule startup
    INFO: Module instance default is running at http://localhost:8080/
    Feb 09, 2017 4:21:19 PM com.google.appengine.tools.development.AbstractModule startup
    INFO: The admin console is running at http://localhost:8080/_ah/admin
    Feb 09, 2017 9:51:19 PM com.google.appengine.tools.development.DevAppServerImpl doStart
    INFO: Dev App Server is now running
    Feb 09, 2017 9:51:48 PM com.google.appengine.tools.development.LocalResourceFileServlet doGet
    WARNING: No file found for: /favicon.ico
    Feb 09, 2017 9:51:50 PM com.google.apphosting.utils.jetty.JettyLogger warn
    WARNING: Error for /_ah/spi/BackendService.getApiConfigs
    java.lang.StackOverflowError
        at java.lang.System.getProperty(System.java:714)
        at com.google.appengine.tools.development.agent.runtime.Runtime.checkRestricted(Runtime.java:62)
        at com.googlecode.objectify.impl.FieldProperty.<init>(FieldProperty.java:36)
        at com.googlecode.objectify.impl.translate.ClassPopulator.getDeclaredProperties(ClassPopulator.java:209)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:84)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at 
  com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEmbeddedClassTranslator(ClassTranslatorFactory.java:75)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:50)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)
        at com.googlecode.objectify.impl.translate.Translators.get(Translators.java:117)
        at com.googlecode.objectify.impl.translate.CreateContext.getTranslator(CreateContext.java:27)
        at com.googlecode.objectify.impl.translate.ClassPopulator.<init>(ClassPopulator.java:88)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.createEntityClassTranslator(ClassTranslatorFactory.java:66)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:49)
        at com.googlecode.objectify.impl.translate.ClassTranslatorFactory.create(ClassTranslatorFactory.java:36)
        at com.googlecode.objectify.impl.translate.Translators.create(Translators.java:138)

    Feb 09, 2017 9:51:50 PM com.google.apphosting.utils.jetty.JettyLogger warn
    WARNING: /_ah/api/explorer: java.io.IOException: Failed to retrieve API configs with status: 500
    Feb 09, 2017 9:52:07 PM com.google.apphosting.utils.jetty.JettyLogger warn
    WARNING: /_ah/spi/BackendService.getApiConfigs
    java.lang.NullPointerException
        at com.google.api.server.spi.SystemServiceServlet.execute(SystemServiceServlet.java:100)
        at com.google.api.server.spi.SystemServiceServlet.doPost(SystemServiceServlet.java:71)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
        at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:122)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
        at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
        at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:95)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:508)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

    Feb 09, 2017 9:52:07 PM com.google.apphosting.utils.jetty.JettyLogger warn
    WARNING: /_ah/api/discovery/v1/apis/myApi/v1/rest: java.io.IOException: Failed to retrieve API configs with status: 500

    Process finished with exit code -1

Does anyone have any clue in regards to the above regards?

I am posting the answer after shooting in the dark for answers for several hours now.

First clue: The exceptions trace show repeated call by Objectify and as @stickfigure has asked me to post the Entities I am trying to persist, his guess is right - the problem lies in the entities.

I don't know if this is intended or accidental but it is a caveat of Objectify that I am going to expose now. I am using Objectify 5.1.14(the latest as of right now) and this probably applies to older versions as well.

Apparently Objectify doesn't allow 2 entites with cross references. For example, I have two entities Entity1 and Entity2 as follows:

@Entity
public class Entity1 {

    @Id
    private long id;
    private Entity2 entity2;
}

@Entity
public class Entity2 {

    @Id
    private long id;
    private Entity1 entity1;
}

AND if you generate endpoints for them(yourself or the GAE plugin), the backend will compile just fine and shall be deployed. But any attempt to persist or accessing the API Explorer shall result in the above exception.

Solution: The obvious solution is to not having cross references. There is another solution however in case you don't care having the cross references NOT being persisted in the datastore - Use the @Ignore Objectify annotation as follows:

@Entity
    public class Entity1 {

    @Id
    private long id;
    @Ignore
    private Entity2 entity2;
}

@Entity
public class Entity2 {

    @Id
    private long id;
    @Ignore
    private Entity1 entity1;
}

NOTE: Just having 1 @Ignore for either entities is fine.

You can test the above scenario and will get exception as explained in case of cross references.

Lastly, I am not sure but based on searching, the problem seems to lie somewhere in generation of discovery docs.