RabbitMQ - ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN

[PeterBirksmith]

Branched from Diagnostic utility for checking the status of the RabbitMQ platform service​

Hi RJ

I have a similar issue with GEP even though I have reset GEP so there are now no configured services I am still getting the error below.

I've used your utility and all indications are that the services are available.

I'm running 10.3 Arcgis Server and GEP on Windows 2012

Error

Failed to create connection cxn-73279 com.rabbitmq.client.AuthenticationFailureException:

ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN.

I noticed that there is a notice on the RABBTMQ - http://www.rabbitmq.com/access-control.html#loopback-users

That this process no longer allows guest logon against localhost.

This is the only thing I can think of short of uninstalling and re-installing GEP knowing that this issue may still exist on completion.

Have you ever seen this error?

Cheers

Peter

[RJSunderman]

Hey Peter -

I'm not sure that RabbitMQ's restriction of only allowing guest to log-in via a loopback interface (i.e. localhost) is relevant here. ArcGIS Server is exposing RabbitMQ as a platform service. If we log-in to the ArcGIS Server Admin API we can obtain a user-id and password from the platform service and use those credentials to log-in to the RabbitMQ management console, for example. I've verified that I can do this from a remote machine.

For this issue, reinstalling the GeoEvent product is not going to do anything for us. The platform service is created by ArcGIS Server during its product installation. GeoEvent is using the platform service created by ArcGIS Server. If we end up having to re-install anything to fix RabbitMQ it will be Server, not GeoEvent.

Are you experiencing this issue on a single server, with localhost the only machine in the ArcGIS Server site's default cluster? To be clear, the RabbitMQ platform service must be available whether or not your site architecture consists of a single machine or multiple machines participating in a cluster -- I just want to be clear about your system setup.

- RJ

[PeterBirksmith]

Hi RJ,

Thanks for the response.

This is a single installation of ArcGis and GeoEvent 10.3 on a windows 2012 R2 Windows Sever.

I installed ArcGis Server 10.3 and applied patch ArcGIS-103-S-SCFSSP-Patch, then installed GeoEvent 10.3 and applied the patch GeoEvent-10.3-Patch1

Each service is started by a domain account with full admin rights.

I changed the server names to the fully qualified name of the host.  I've done this in ArcGis Server Manager under Site > Machines

Under Cluster there is only the default and I changed this also to be the FQDN.

In ArcGis GeoEvent Manager, under Site > Data Stores, I changed this also to be the FQDN.  The Name remained default.

I've lodged a support call but haven't heard back yet with any steps to take to resolve the issue. It's our prod box so I can't just uninstall the server where as I can uninstall and re-install geoevent as this component is just about to be implemented.

Again thanks for the assistance.

Cheers

Peter

[RJSunderman]

Thank you Peter. If you'll send me the technical support incident number I'll work to follow-up with my POC in Esri Support. Sorry, but I'm having trouble finding it in Esri Support's database...

I've been told by the Server product team that the ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN error message is returned when:

1. The guest account of Rabbit MQ is used to log-in.
2. A RabbitMQ client connection to the RabbitMQ service is attempted with a different version number.

The Server team has indicated that the first cannot be the case as we (Esri) delete the guest account and create a new account for log-in.

The second option, then, could occur when a client (GeoEvent or an external / non-Esri app) whose RabbitMQ version is different than the one used by the ArcGIS Server platform service attempts a connection.

For example, an ArcGIS Server 10.3.1 machine I accessed specifies that version 3.2.3 of RabbitMQ is being run (https://esri-server:6443/arcgis/admin/system/platformservices/?f=pjson). If a client leveraging a JAR file from RabbitMQ 3.2.0 attempted to connect, we'd receive the ACCESS_REFUSED error. I don't see how this can be the case, though, if GeoEvent is the only client attempting a connection.

Let's plan on using the technical support incident to collaborate further. I can post a SUM back here if / when we figure out what's going on. Please feel free to post back to this thread if you discover a work around or solution yourself.

Thanks -

RJ

[PeterBirksmith]

Hi RJ

Thanks again for the support and comments.  Here is the support call details.

Esri Australia Ticket Number  F8JA375119

I sent these details to Support

D:\Temp\CheckRabbitMQ>

{

  "platformservices": [

    {

"id": "5592604b-879d-4349-95e3-8ba65ffb82bc",

"type": "MESSAGE_BUS",

"provider": "RabbitMQ",

"info": {

"port": 27271,

"platformTopics": ["arcgis-admin-events"],

"ssl": true,

"password": "xxxxxxxxxxxxxxx",

"user": "xxxxxxxxxxxxxxx",

"version": "3.2.3"

      }

    },

    {

"id": "d8969064-636b-4236-a74a-c9e07cc41dfb",

"type": "MESSAGE_BUS",

"provider": "RabbitMQ",

"info": {

"port": 27271,

"platformTopics": ["arcgis-admin-events"],

"ssl": true,

"password": "xxxxxxxxxxxxxxxxx",

"user": "xxxxxxxxxxxxx",

       "version": "3.2.3"

      }

    }

  ],

  "status": "success"

}

Cheers

Peter

[PeterBirksmith]

Hi RJ,

Support and myself have come to the same conclusion that this may be the OS.  It's not replicated in Windows 2008 but is in 2012.

We'll have to see what Esri Server group respond with.

Cheers

Peter

[RJSunderman]

Thanks Peter for the follow-up.  It looks like somehow there are two "RabbitMQ" messaging service / platform services configured, each with its own GUID / id. I don't know how this would happen; maybe there is an OS factor to consider in issue replication.

I think I would expect some sort of failure if GeoEvent went searching for a platform service of type "MESSAGE_BUS" and found more than one candidate. I don't know that I would have expected the ACCESS_REFUSED log-in failure ... but in any case you're tracking to the root cause of the issue.

Good luck -

RJ

[PeterBirksmith]

Hi RJ

Thanks for the feedback.  I've had feedback from Esri support that the issue relates to Win 2012 R2 using the patches for 10.3

I've been advised to upgrade to 10.3.1

So will do this once I've completed my Birst training and let you know how I go.

Cheers

Peter

[PeterBirksmith]

Hi RJ,

The upgrade to 10.3.1 has resolved the issue with RabbitMQ errors in Win 2012 R2

Thanks to the hard working ESRI support team for getting this issue resolved.

Cheers

Peter

[MarcGraham1]

Hi GeoEvent Team​ RJ Sunderman​ Adam Mollenkopf​, I have experienced this same bug after installing this patch ArcGIS for Server Publishing Patch  on ArcGIS for Server 10.4.1 on Windows Server 2012 R2.  I would recommend that nobody using geoevent installs this patch, as it seems to mess up geoevent.  This isse went away after an uninstall reinstall of server and geoevent.

Regards,

Marc