Data store connection issue when hard drive full

[ahagopian_coj]

Hello.  Our data store server filled up the hard drive with all the backups.  We have deleted most of the backups to free up space but now we are having issues with our hosted feature layers.  We can't create new records in anything. We can't edit any hosted feature layers.  Workforce isn't working since we can't update any work orders or create new ones.  I have rebooted everything.  I re-ran the ArcGIS Server configuration wizard to no avail.  Does anyone know how to fix this? We are running version 10.9.1 of enterprise in a federated environment.

I have ticket into ESRI but the tech keeps asking me about my ODBC 17 driver version.  It's Portal, no driver is used.  Losing faith in ESRI support so wanted to see if my fellow GIS peers had any ideas.

Thanks!

[MarceloMarques]

ArcGIS Data Store utility reference—ArcGIS Enterprise | Documentation for ArcGIS Enterprise

Change the read-only threshold for the relational data store

By default, the relational data store is placed in read-only mode when free disk space drops to 1,024 MB. In the following example, there are other apps writing to the machine, so the administrator configures the relational data store to be placed in read-only mode sooner: when free disk space on the machine drops to 4,000 MB:

changedbproperties --store relational --disk-threshold-readonly 4000

You are changing the following database properties:
         ...disk space threshold to change relational data store into READONLY mode to 4000MB)

Changing database configurations could cause the database to restart. Please do not interrupt the process once it has started.

Do you want to continue (Yes or No)?Yes

 describedatastore command will show if the database is in read-only mode.

changedatastoremode

This utility is used with relational and spatiotemporal big data stores.

The changedatastoremode utility allows you to place a relational data store in read-only mode while you perform maintenance on the data store. For example, if you need to perform a maintenance task that will cause the data store to restart, such as changing the backup location from one drive to another drive or changing database properties, you can place the relational data store in read-only mode so no users are in the process of publishing or editing data when the data store restarts.

The changedatastoremode utility is also used to place the relational data store back in a read-write mode after you finish maintenance or after you add enough disk space to the primary data store machine to allow the data store to function properly in read-write mode.

You must run the changedatastoremode utility on the primary relational data store machine, not the standby machine.

Note:

If ArcGIS Data Store places your relational data store in read-only mode due to insufficient disk space, automatic backups are also disabled to avoid filling the disk further. Therefore, you must reset your automatic backup schedule using the updatebackupschedule utility after you place the relational data store back in read-write mode.

When your spatiotemporal big data store gets close to running out of disk space, ArcGIS Data Store places it in read-only mode to avoid data corruption. You can use the changedatastoremode utility to return a spatiotemporal big data store to read-write mode after you have added sufficient disk space to the spatiotemporal big data store machines.

Syntax

changedatastoremode {readonly | readwrite} [--prompt {yes | no}] --store {relational | spatiotemporal}

Examples

In this example, the relational data store is placed back in read-write mode, meaning clients can resume such activities as publishing hosted feature layers to ArcGIS Enterprise, editing data through a hosted feature layer, or adding CSV files to a map viewer.

changedatastoremode readwrite --store relational --prompt no

To place a spatiotemporal big data store in read-write mode, specify spatiotemporal with the --store option as shown in the following example:

changedatastoremode readwrite --prompt no --store spatiotemporal

[ahagopian_coj]

Thank you so much for your reply!!!  So the datastore put itself into read only mode when the hard drive had no space left?  How do I get it out of read only mode?  Restarting the data store server didn't work.

[MarceloMarques · Accepted solution]

changedatastoremode readwrite --store relational --prompt no

describedatastore

Available ArcGIS Data Stores
==============================================================
Relational Data Store
Tile Cache Data Store
Spatiotemporal Big Data Store
Graph Store
Object Store

General Information of ArcGIS Data Store on machine.domain.com
==============================================================
ArcGIS Data Store release....11.2.0.1234
Staging location.............C:\arcgisdatastore\staging
Log location.................C:\arcgisdatastore\logs
Free disk space..............174.00GB
Threshold for READONLY mode..2048MB

Information for relational data store ds_sthiu0_5T
==============================================================
Backup location...........\\nwshare\dsbackups
Is backup folder shared...true
Backup schedule...........{"schedule-starttime":"00:00:00","schedule-frequency":"Every 7 DAYS"}
Days backup retained......31
Data store status.........Started
SSL enabled...............true
Last failover.............20150130190334005
Member machines...........MACHINE1.DOMAIN.COM, MACHINE4.DOMAIN.COM
Maximum connections.......150
Owning system URL.........https://gisserver_webadaptor.domain.com/server
Portal for ArcGIS URL.....https://portal_webadaptor.domain.com/portal
Number of connections.....8 connection(s) to managed database
Data Store mode.....................READWRITE
Is Point-in-time recovery enabled...No
Query optimizer enabled.............Yes

Information for tile cache data store ds_wztxj7um
==============================================================
Data location...................C:\arcgisdatastore\nosqldata
Data store status...............Started
Backup location.................\\mybackups\scenebackup
Backup schedule.................{schedule-frequency":"Every 1 DAYS"}
Is backup folder shared.........true
Mode............................cluster
Member tile cache machines......MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM
Registered tile cache machines..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM
Owning system URL...............https://gisserver_webadaptor.domain.com/server
Portal for ArcGIS URL...........https://portal_webadaptor.domain.com/portal

Information for spatiotemporal data store ds_qpko99Cl
==============================================================
Backup location.....................\\nwshare2\sbds_bu
Max rebalance off time..............60 minutes
Automatic rebalance ................On
Machines in spatiotemporal cluster..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM
Current coordinator in cluster......MACHINE1.DOMAIN.COM
Registered spatiotemporal machines..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM
Owning system URL...................https://gisserver_webadaptor.domain.com/server
Portal for ArcGIS URL...............https://portal_webadaptor.domain.com/portal
Data Store mode.....................READWRITE

Information for graph store x2b7s0n
==============================================================
Deployment mode..................primaryStandby
Access endpoint..................MACHINE1:9829
Backup location..................\\nwshare3\graphbu
Registered graph store machines..MACHINE1.DOMAIN.COM
Owning system URL................https://gisserver_webadaptor.domain.com/server
Portal for ArcGIS URL............https://portal_webadaptor.domain.com/portal

Information for object store w1tiknx
==============================================================
Deployment mode...................singleInstance
Access endpoint...................MACHINE1:9900
Backup location...................\\nwshare4\myfiles\ods_bu
Registered object store machines..MACHINE1.DOMAIN.COM
Owning system URL.................https://gisserver.domain.com:6443/server/admin
Portal for ArcGIS URL.............https://portal_webadaptor.domain.com/portal

[ahagopian_coj]

Thank you!  That worked perfectly!  You are a life saver!  Or at least my database saver!

[MarceloMarques]

You are welcome!