We resolved this issue and wanted to share with the community (Knowledge Base)...
The Problem:
One of our ArcGIS for Server machines was in a stopped state and would not start up. The only valuable log message we could find in the C:\arcgisserver\logs<machinename>\server directory was:
"Failed to start the server machine. Premature end of file"
This article suggested some sort of corruption in the installation... which we followed, and compared to a working deployment. The specified file (config.xml) had no obvious issues.. it had content and appeared comparable to a working installation.
We found a few log file in the C:\Program Files\ArcGIS\Server\framework\etc\service\logs directory and specifically found the following in the service.log:
Nov 10, 2016 9:21:05 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
org.xml.sax.SAXParseException; systemId: file: C:/Program Files/ArcGIS/Server/framework/runtime/tomcat/conf/server.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
<STACK TRACE>
and sure enough... the server.xml file was empty (0 bytes)
THE FIX:
- Copied a server.xml file from an alternate location (a deployment of the same arcgis server version 10.3.1)
- Modified the port 6443 connector (bottom of file)
<Connector SSLEnabled="true" clientAuth="false" keyAlias="SelfSignedCertificate" keystoreFile="\\server\share\config-store\machines\servername\arcgis.keystore" keystorePass="<OBSCURED> maxHttpHeaderSize="65535" maxPostSize="10485760" maxThreads="150" port="6443" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" secure="true" sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1" sslProtocol="TLS" />
- Specifically updated the keyAlias, keystoreFile, and keystorePass
- The keystorePass was a little tricky... Fortunately we found that the value of password in the \\server\share\config-store\security\super\\super.json file was the same as the keystorepass on other installations... so we put that value into the keystorePass field in our 'hacked' server.xml file.
Note: when the alias, file, or password values were in-correct... the ArcGIS Server would actually startup but only listening on HTTP port 6080. The HTTPS port 6443 (this connector) was non-operational. This was also the case access HTTPS/443 through the web-adaptor as I assume the web-adaptor sent the back-end request to 6443 not to 6080.
I hope someone finds this useful if they run into a similar issue. We will be watching the site for stability, but a quick cursory test shows success and high confidence of this fix.