If you have a big server with space for many VMs then put one component per VM -- one for Portal, one for Server, one for Datastore, one for your flavor of database server. I ended up with Portal and Web Adaptors and IIS on one, which works fine.
If it were up to me alone, I would put them on Linux and use PostgreSQL as the database but that's me (and I will retire soon). We are a mixed shop here so Esri software runs on Windows. (Our concurrent license manager is on CentOS.)
I don't feel there would be an advantage of using PostgreSQL, but I have more experience with it. Our IT group runs SQL Server and they manage it for us, so we use that.
One of the reasons for one component per VM is snapshot backups. Whenever you call in to Esri support and they hit a broken component that they can't fix, they will say "can you drop back to yesterday's snapshot?" and if the SQL Server is on the same machine, the answer will be "NO, we will lose work".
Another reason is that you will have some license limit on the CPUs and if you have (like us) a 4 CPU limit, you get 4 per machine = 16 vs 4 total for everything.
Interesting about federation. I don't remember that being an issue. Esri support has always been good enough over the phone for us. (Though occasionally they can't help at all.) I can't imagine what it would cost to bring someone in here physically.
Upgrades have always been a pain. I am getting ready to go from 10.9.1 to 11 now. Every time we upgrade, something breaks. Last time, EVERY Web App Builder application stopped working (and no, Esri could not figure it out.) Only the Developer Edition apps survived so that's all we use now.