Sorry for the late response. If you already got around this, could you detail how?
If all of the servers are behind the firewall I would suggest trying authentication with Active Directory/LDAP at the Portal Tier. IWA may not be required if web tier authentication (e.g. outside the firewall) is not required, Portal Tier authentication may be quicker.
See:
Configure your portal to use Windows Active Directory
Use Integrated Windows Authentication with your portal—Portal for ArcGIS (10.5.x) | ArcGIS Enterpris...
You have a lot of overhead on a single server as well. May see a bit of a performance if you are using a single machine deployment and it is federated, I assume ArcGIS Server is also on this machine. I would consider adding a server.
If you are hosting Image Server, GeoEvent, and ArcGIS Server on the same box you will also see a performance hit. Better to segregate if possible.