The empirical semivariogram is created by averaging the squared differences between pairs of points that are approximately the same distance apart. Even a few outliers can heavily influence this average (particularly because you're squaring the difference).
Outliers are almost always problematic for kriging, but they're particularly bad when the outliers are scattered randomly throughout the study region (rather than being clustered together). This is because randomly scattered outliers will affect the empirical semivariances at small distances (because they might be right next to low values), but if the outliers are clustered, the squared difference between two outliers might still be small, allowing for accurate semivariogram estimation at small distances (which is the most important part of the semivariogram because closer neighbors get the highest weights).