You aren't working with projected data.
The cell size and circle radius units suggest this.
I would use projected data for the inputs.
As for the mask, for now, drop the mask to see if it works without the mask.
Simply use Select by Attributes to select the points that fall within the convex hull and save those as a new (projected) point file.
As for the output... make it a *.tif file avoid an esri grid... simply add tif to the end of the file name once you have chosen your saving location.
I think it is a combination of not using projected data for the inputs and perhaps the mask