Hi Sean,
Currently the Connect Origins to Destinations analysis tool does not support specifying origins that connect to multiple destinations. If you have multiple destinations, the origins can only have one feature. So to achieve your workflow, you need to run the tool multiple times such that you have only one origin in each iteration and then combine the results from the iterations.
I have logged an enhancement request to update the Connect Origins to Destinations tool to support multiple origins with multiple destinations.
On a related note, to visualize the results, would you prefer a straight line connecting each origins to their destinations even though you are using network travel distance? Or in your workflow, you prefer to see the connections using the true shapes derived from the streets?