Given the density of your data set, make sure you are not projecting on the fly. On-the-fly projection is handy, but it is a performance killer when dealing with dense data sets.
Contour lines pose a challenge for spatial indexes because a single contour line tends to span very large distances so the efficacy of using spatial indexes is reduced. I commonly take dense data sets like contour lines and break them up using a grid, e.g., USGS 1:24,000 topographic map index grid. Splitting the contour lines allows for better spatial index efficacy and quicker draw times when zoomed into small areas. For analysis, some of how contours are selected might need to be modified, but results are not impacted once the selections are made correctly.