Hello, I have a data pipeline using Apache Airflow that kicks off our ingestion process on a geoprocessing server. For the most part things work well, however when the gp server gets bogged down with too many jobs in a submitted state, they no longer move to 'executing' when a previous job completes. They are asynchronous jobs so my question is two-fold:
1) What mechanism triggers the job state to move from submitted to executing?
2) How are the number of asynchronous jobs being executing simultaneously determined?
Right now I have 8 different jobs that are submitted with 0 executing, 0 waiting, 0 failed. They have been submitted for over 12 hours. I have to go in and turn off gp server, then turn it back on again in order to get things flowing again. Why is this?