Dear Jonathan,
Throughout this thread, you asked Richelle the following:
"In the error message Could not connect to the ArcGIS Server on machine '[Internal machine address], is the '[Internal machine address]' your Portal machine name?"
In my case, i am getting the above message where my internal machine address is the portal machine which is strange!
Note that i got this error whenever trying to validate the federation between portal and server. I also got a java.net error connection timeout. Also note that all my highly available system was working fine using Microsoft NLBs until we migrated to F5 (same VIP names, no reverse proxy rules, valid SSL certificates..).
Here are the portal log:
/Msg>
<Msg time="2018-04-04T12:20:43,991" type="WARNING" code="207051" source="Portal Admin" process="2620" thread="13" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">Validation failed for federated server 'https://prdgisapp-nlb.domain.local:6443/arcgis'.</Msg>
<Msg time="2018-04-04T12:20:43,991" type="FINE" code="207051" source="Portal Admin" process="2620" thread="13" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">Validation failed for federated server 'https://prdgisapp-nlb.domain.local:6443/arcgis'. Validation steps. Step1: Error: java.net.ConnectException: Connection timed out: connect.
</Msg>
<Msg time="2018-04-04T12:21:28,619" type="INFO" code="216003" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Node prdgisweb1.domain.local is configured to be master.</Msg>
<Msg time="2018-04-04T12:21:28,619" type="INFO" code="216004" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Monitoring the standby nodes.</Msg>
<Msg time="2018-04-04T12:22:07,710" type="DEBUG" code="9999" source="Portal Admin" process="2620" thread="13" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">java.lang.RuntimeException: java.net.ConnectException: Connection timed out: connect
at com.esri.arcgis.portal.admin.core.util.HttpClient.get(HttpClient.java:152)
at com.esri.arcgis.portal.admin.core.client.ServerClient.getServerAdminInfo(ServerClient.java:260)
at com.esri.arcgis.portal.admin.core.federation.ServerFederationManager.a(ServerFederationManager.java:1625)
at com.esri.arcgis.portal.admin.core.federation.ServerFederationManager.a(ServerFederationManager.java:1558)
at com.esri.arcgis.portal.admin.core.federation.ServerFederationManager.validateAllServers(ServerFederationManager.java:1902)
at com.esri.arcgis.portal.admin.rest.federation.ServerFederationResource.getValidateAllServers(ServerFederationResource.java:111)
at sun.reflect.GeneratedMethodAccessor812.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:910)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:858)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:812)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.esri.commons.web.AppFilter.doFilter(AppFilter.java:265)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.esri.arcgis.portal.admin.rest.filters.AdminFilter.doFilter(AdminFilter.java:87)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at com.esri.arcgis.portal.util.TomcatValve.invoke(TomcatValve.java:43)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:442)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1083)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:640)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:524)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:403)
at com.esri.arcgis.portal.admin.core.util.HttpClient$b.connectSocket(HttpClient$b.java:479)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at com.esri.arcgis.portal.admin.core.util.HttpClient.get(HttpClient.java:136)
... 52 more
</Msg>
<Msg time="2018-04-04T12:22:07,710" type="WARNING" code="207051" source="Portal Admin" process="2620" thread="13" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">Validation failed for federated server 'https://prdgisapp-nlb.domain.local:6443/arcgis'.</Msg>
<Msg time="2018-04-04T12:22:07,710" type="FINE" code="207051" source="Portal Admin" process="2620" thread="13" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">Validation failed for federated server 'https://prdgisapp-nlb.domain.local:6443/arcgis'. Validation steps. Step1: Error: java.net.ConnectException: Connection timed out: connect.
</Msg>
<Msg time="2018-04-04T12:22:29,50" type="INFO" code="216003" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Node prdgisweb1.domain.local is configured to be master.</Msg>
<Msg time="2018-04-04T12:22:29,50" type="INFO" code="216004" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Monitoring the standby nodes.</Msg>
<Msg time="2018-04-04T12:23:29,701" type="INFO" code="216003" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Node prdgisweb1.domain.local is configured to be master.</Msg>
<Msg time="2018-04-04T12:23:29,701" type="INFO" code="216004" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Monitoring the standby nodes.</Msg>
<Msg time="2018-04-04T12:24:30,193" type="INFO" code="216003" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Node prdgisweb1.domain.local is configured to be master.</Msg>
<Msg time="2018-04-04T12:24:30,194" type="INFO" code="216004" source="Portal" process="1728" thread="1" methodName="" machine="PRDGISWEB1.domain.local" user="" elapsed="">HA: Monitoring the standby nodes.</Msg>