If you turn on the logging for the connection:
35704 - Diagnose ArcSDE connection and performance issues using SDEINTERCEPT
you might get an idea of how much those domains affect the connection speed.
Also, I agree with Blake. Keep it clean.
As I guy who just stepped into a position of overseeing a rather large legacy system, I spend an awful lot of time just figuring out if something in the system is still in use, be it in the database, mxds, services on the servers, etc...
When you leave detritus laying around, it's just more stuff to trip over.