Hi there, the print runs on your ArcGiS Server as you have indicated.
This is most likely where the time out is occuring. The default setting for custom GP tools is 600 sec, which is 10 minutes. I would check these settings in the first instance.
Are you running the service as synchronus or asynchronus? It may pay to change the service type to async if your are experiencing timeout issues, this will then stop the browser blocking the UI. Which could be another reason you are receiving timeouts.
As to why your print is taking a long time, I would check the logs to see if there are any errors being generated from the custom print service. The other questions to ask are :
How much information are you trying to print, eg how many layers, at what scale are you trying to print, what is the size of the print layout.
Does the ArcGIS Server have access to all of the required layers that are trying to be printed?
Again, I would be looking at the logs in the first instance to see what information is there as that should show you any issues or errors being generated off your custom service.
Cheers