@MkbRMIT I think the /etc/security/limits.conf file is only really useful for setting the limits for the install process and if you are not using the Systemd start up script as Systemd seems to impose its own limits.
If you are using the arcgisserver.service systemd start up copied from the <install path>/framework/etc/scripts directory then you need to update it.
I am using the systemd startup script so I have edited /etc/systemd/system.conf to set DefaultTasksMax=infinity
and /etc/systemd/system/arcgisserver.service to set LimitNOFILE=1048576
in /etc/security/limits.conf I have set
<user that runs the software> soft nofile 1048576
<user that runs the software>hard nofile 1048576
<user that runs the software>soft nproc 25059
<user that runs the software>hard nproc 25059