We also had this issues where the Portal under Settings-->Servers-->was failing to connect to a federated server.
Portaladmin api was unable to unfedereate the server:
Failed to unfederate server ... Could not decrypt token. Token may not be valid.
In the ArcGIS Server admin api I was able to change the Security configuration to use internal Server authentication and removed the Server role of Hosted server. But Portal still thought that this ArcGIS Server was federated.
Then I remembered that in the Portal Sharing api federated servers where also listed.
https://portal.domain.net/portal/sharing/rest/portals/XXXXXXXXXXXX/servers
From here I was able to finally get the server unfederated with Portal.
Once everything was cleaned up I was able to Federate this ArcGIS Server instance and also register it as the Hosting Server.