I did some more testing with 2.5. I was actually unable to publish any layers using 2.5 to our Portal or to our AGOL account. This included trying to overwrite the layers. I must have published all our layers prior to me updating.
I downgraded to 2.4 and it appears to be working fine again.
So far, 2.5 will be a no go for us.