Thanks for the suggestion. I actually need to use it in the Web AppBuilder, so that non-GIS users can use the app to extract addresses.
As of now, I've deleted all the unnecessary fields and made the layer scale-depended. This improved the speed display a little bit (which was actually the original problem since my layer has 600000 features).