Hi, Thanks for reporting this issue. Once you Sign-in, all the AGOL data sources should not remain red and not connected. We'll address this issue in the product.
Meanwhile, there is another way you can access the AGOL layer after you open back the Excel project. In 'ArcGIS for Excel', While Signing In, select the checkbox - “Keep me signed in” option, Add your AGOL layer, save your Excel sheet, close the Excel, re-open the Excel, and I think now the AGOL layer will display without any issues.
Thanks,
Gaurav
ArcGIS for Excel Team