Sub groups don't exist in ArcGIS Online.
Assign user types and roles to match the privileges required for the user (Viewer, Editor). Don't use this to control editing specific maps/layers.
Then create groups to manage access. e.g. for a project you could create:
- Group1: Project X (Management)
- 5 users with editor privileges
- Group2: Project X
- 100 users with viewer privileges, as well as the 5 mgmt users
Create Hosted Feature Layer Views with editing enabled (or disabled), add them to separate maps, then share to the appropriate group.
https://doc.arcgis.com/en/arcgis-online/share-maps/groups.htm