After some investigation I am thinking now that it is crucial that the shared keys are different.
IF the shared keys are the same i think that server performs different validation:it acts as if token was generated by server itself, hence expects different token structure (can be seen after token decryption)
IF keys are different then debug logs of the Federated Server DO report an error, but then pass on to the section that causes this error:
java.io.IOException: com.esri.arcgis.discovery.admin.security.InvalidTokenException: com.esri.arcgis.discovery.admin.security.AGSSecurityException: com.esri.arcgis.discovery.admin.security.AGSSecurityException: Server machine 'https://portal.<HOST>.com/portal/sharing/rest/community/self' returned an error. 'User does not exist or is inaccessible.'
This could mean that the token is passed back by server internally to portal (where original encryption has happened, with the different key). Only then Portal decides if the token is valid or not.
For reason i cannot determine my tokens generated with AppID get rejected with
'User does not exist or is inaccessible.'
After token (not acces_token, but the one that my proxy uses to talk to server!) decryption I can see that it contains:
{"s":"9f22","t":"a","c":1528196962495,"d":"<Federated-Site-Id>","e":true,"g":"<App-ID>","h":"0123456789ABCDEF","l":0,"m":0}
Federated-Site-Id IS the ArcGIS Server site that is federated with my portal and that i try to reach, so it matches overall logic. This has no user info though, i suppose <AppID> plays its role?
After seeing the above error in Server Logs, I cannot see any meaningful error in Portal logs that would indicate a crash or so.
Am I missing some settings on the Application? I have shared it with everyone but did not help.
Regards,
Szymon