Hi Kenneth,
I've been experiencing the same issue at a few locations, and have gone through the same troubleshooting issues listed above with no success, however I've found a solution for my particular version of this problem which may be of assistance.
A user connect to the servers seems to work fine from ArcCatalog, but publisher and admin fail with this 504 error.
Firstly, all instances where I have seen this, the portal instance is registered to use Windows AD users, so IWA and SSO via the WebAdaptor.
What seems to be the problem is running the ArcMap/Catalog desktop application as a Windows AD User that is not a member of the portal identity store. If you start up ArcMap as an admin user not registered in portal (in the instances I've tried) you will run into a 504 problem.
It also seems that you cannot bypass the webadaptor and simply use the site administrator account when ArcMap/Catalog is running as a non-portal admin/publisher user, which is where it seems to be buggy.
To solve it we found that it's possible to do a run as of ArcMap/Catalog with an AD user that is either publisher or admin on the portal. After this the 504 error stopped occurring and both Portal users psa credentials and Portal AD credentials could be used to create an admin connection to AGS.
Hopfully this helps!
Dean