Well, the normal way to do it is to create a GIS Server connection via ArcCatalog.
Then in ArcMap you can navigate to that connection and add any service just like a normal layer / dataset.
But as you are posting this in "Developers", I guess you want some sort of programmatic solution.