Thanks to Esri Technical Support for help recently with this same question!
It turns out there is a new system property "WebSocketContextURL" for this purpose, accessible via the ArcGIS for Server rest admin API.
Log in e.g. via http://locallost:6080/arcgis/rest/admin, use admin credentials, navigate to system -> properties -> update, and specify your reverse proxy URL in the JSON.
If the reverse proxy does not does not place /arcgis/ws/services at root, include the full path up to and including /arcgis, e.g. { "WebSocketContextURL" : "wss://exposed.server/prefix/arcgis" }.