Jason,
All of our map services are public. One of our users created a simple ASP.NET page that will send 15 queries spread accross 10 map services and the geometry service. The results are displayed in a table. If the 15 query results match the expected result the then the text in the table is green. If the results do not match then the corresponding table row is red.
The user created this ASP.NET page to test our map services that they are using in an application. They posted it to a public web server so we could also take advantage of it as a diagnostic / monitoring tool. It is simple but it is very helpful. Send me a private message if you want more info.