Happy to help you with this.
There's not quite enough information here for me to determine whether this will work for your needs, but this downloadable toolset for solving large Network Analyst problems may do the trick: https://github.com/Esri/large-network-analysis-tools If you just need the travel time and/or distance between origins and the closest destination (and not the route geometry), you can use the Solve Large OD Cost Matrix tool, and will go waaaaayyyy faster than what you're doing now. It does a bunch of pre-processing to improve performance, chunks up the problem and solves it in parallel across all the cores of your machine, and also uses the arcpy.nax solver objects under the hood, which are faster than the network analysis layers you can use in a model.
Also, here is a video from a DevSummit 2022 presentation about how to solve large network analysis problems: https://mediaspace.esri.com/media/t/1_hzppnv8p