6 Replies Latest reply on Jul 20, 2011 6:57 PM by graeme

    ArcGIS Server 10 services not starting

    donbrown44
      We have a virtual machine with Postgres & ArcSDE 10 SP1 installed. Another virtual machine with ArcGIS Server 10 with SP1 installed on Linux.
      When I restart the server, sometimes not all services start back up.  I can manually start these services by clicking on them in arcgismanager.  It is not the same services each time that don't start up. When services don't start, I get the following log file errors for each service:

      Configuration myMapService.MapServer can not be started.
      Server Object instance creation failed on machine myServer. Error: 0x80080005 (Server execution failed )
      Container failed to create on machine myServer. Error: 0x80080005 (Server execution failed )


      Maybe ArcGIS Server 10 just does not work on Linux??
        • Re: ArcGIS Server 10 services not starting
          randall_williams-esristaff
          ArcGIS Server runs fine on Linux. The issues you're reporting sound to me like resource and capacity issues.

          I have a few questions:

          1. How many services are you running?
          2. Can you describe the specs of the machine you're running AGS on?
          3. Are you using optimized or classic map services?
          4. How complex are these services?
          5. How many SOC instances do you have dedicated to these services?
          6. Are these services operating in high isolation or low isolation?
          7. Pooled or non-pooled?

          When started, ArcGIS Server will attempt to start each service individually until no further memory can be allocated to an ArcSOC.exe process. At that point, you'll start to see errors like those you're reporting. I'd be curious to know how many arcsoc.exe processes are running, and to know the amount of RAM they are consuming. I've never seen a single SOC host, no matter the amount of memory there is on the machine, instantiate more than ~120 SOC instances.
          • Re: ArcGIS Server 10 services not starting
            donbrown44
            1. How many services are you running?
            25 services spread throughout 5 �??folders�??.

            2. Can you describe the specs of the machine you're running AGS on?
            Here is the output of �??cpuinfo�?? (which is repeated for 4 total CPU�??s):
            processor       : 0
            vendor_id       : GenuineIntel
            cpu family      : 6
            model           : 15
            model name      : Intel(R) Xeon(R) CPU           X5355  @ 2.66GHz
            stepping        : 7
            cpu MHz         : 2660.002
            cache size      : 4096 KB
            physical id     : 0
            siblings        : 1
            core id         : 0
            cpu cores       : 1
            fpu             : yes
            fpu_exception   : yes
            cpuid level     : 10
            wp              : yes
            flags           : fpu tsc msr pae cx8 apic cmov pat clflush acpi
            bogomips        : 6652.77
            clflush size    : 64
            cache_alignment : 64
            address sizes   : 36 bits physical, 48 bits virtual
            power management:

            And the output of �??meminfo�??:
            MemTotal:      8388608 kB
            MemFree:       3611528 kB
            Buffers:        207904 kB
            Cached:        1725348 kB
            SwapCached:          0 kB
            Active:        2905736 kB
            Inactive:       632276 kB
            HighTotal:           0 kB
            HighFree:            0 kB
            LowTotal:      8388608 kB
            LowFree:       3611528 kB
            SwapTotal:     4192956 kB
            SwapFree:      4192956 kB
            Dirty:             100 kB
            Writeback:           0 kB
            AnonPages:     1604744 kB
            Mapped:         131600 kB
            Slab:          1002608 kB
            PageTables:      20100 kB
            NFS_Unstable:        0 kB
            Bounce:              0 kB
            CommitLimit:   8387260 kB
            Committed_AS:  5606376 kB
            VmallocTotal: 34359738367 kB
            VmallocUsed:      4220 kB
            VmallocChunk: 34359734147 kB

            3. Are you using optimized or classic map services?
            About half are .msd and half are .mxd

            4. How complex are these services?
            Most are simple.  One is a feature access.  One or two MXD�??s reference a lot of data and have unique symbolization.

            5. How many SOC instances do you have dedicated to these services?
            The SOM and SOC are both installed on the above virtual machine.

            6. Are these services operating in high isolation or low isolation?
            All are high isolation

            7. Pooled or non-pooled?
            All are pooled.

            You mentioned �?? �??I'd be curious to know how many arcsoc.exe processes are running, and to know the amount of RAM they are consuming.�?�
            I�??m not sure how to check this.  I would be happy to if you could explain how.  Thanks for your help.  I�??d love to hear any recommendations of better settings/configurations.
            • Re: ArcGIS Server 10 services not starting
              randall_williams-esristaff
              Thanks! By this:

              5. How many SOC instances do you have dedicated to these services?
              The SOM and SOC are both installed on the above virtual machine.

              I looking to know how many ArcSOC.exe processes are defined for each service - by default, it's set to minimum 1, maximum 2.

              Would it be possible to switch some of these services to low isolation? That may be your bottleneck, as in high isolation each instance runs in its own process.

              see:
              Isolation

              http://help.arcgis.com/en/arcgisserver/10.0/help/arcgis_server_java_help/index.html#/Tuning_and_configuring_services/009200000063000000/
              • Re: ArcGIS Server 10 services not starting
                astauffer-esristaff
                You might want to look at increasing the startup timeout in the service .cfg files. The problem could be that starting the 25 services takes too much time so some of the services actually timeout before they get a chance to instantiate.

                Cheers,
                Andrew
                • Re: ArcGIS Server 10 services not starting
                  donbrown44
                  I had the default: min 1, max 2

                  I switched all services to low isolation.  I think that fixed the startup problem.  Thanks for the help!
                  • Re: ArcGIS Server 10 services not starting
                    graeme
                    I'm getting a similar error when trying to start my 15th Geoprocessing Service:

                    Configuration OEMS/<GPServiceName>.GPServer can not be started.
                    Configuration OEMS/<GPServiceName>.GPServerJobScheduler can not be started.
                    Server Object instance creation failed on machine <MachineName>.
                      Error: 0x80080005 (Server execution failed
                    )


                    If I Stop one of the first 14 services then I can Start the one that I previously could not start.

                    This is in a Test/Dev environment using ArcGIS 10.0 SP2 on Windows XP SP3 so I think will "evaporate" as an issue when we move to a Production server.  However, I'm keen to hear whether the limit I'm hitting sounds like it is to be expected by answering Randall's questions from above.

                    1. How many services are you running? 14 - 15th won't start
                    2. Can you describe the specs of the machine you're running AGS on? i7 CPU 870 @ 2.93 GHz, 3.42 GB RAM
                    3. Are you using optimized or classic map services? Geoprocessing Services published using A Map
                    4. How complex are these services? Based on Python scripts that open several MXDs, apply some logic using ArcPy and then save a couple of maps and sometimes a report into a single PDF
                    5. How many SOC instances do you have dedicated to these services? Have not specified this but with 14 GP Services there are 43 x ArcSOC.exe running
                    6. Are these services operating in high isolation or low isolation? High Isolation - was only option I had to choose from
                    7. Pooled or non-pooled? Pooled

                    Any comments on whether I am hitting an inevitable limit are welcome.

                    - Graeme