Random Rounding Error??

2988
1
04-18-2014 11:39 AM
Labels (1)
AnthonyGalante
New Contributor
I have a field that is double with no defined precision or scale and populated with real numbers.
I create a new field, same format, and use field calc to subtract each number by .01
I create another field, same format, use field calc to subtract new number from original.
I get results, seemingly randomly, that range from .01 to .009xx.
WHY?
ver 10.1 build 3143
0 Kudos
1 Reply
ChrisBuscaglia
Esri Contributor
I have a field that is double with no defined precision or scale and populated with real numbers.
I create a new field, same format, and use field calc to subtract each number by .01
I create another field, same format, use field calc to subtract new number from original.
I get results, seemingly randomly, that range from .01 to .009xx.
WHY?
ver 10.1 build 3143


Hello,

You may want to post this question in the GeoDatabase forum:  http://forums.arcgis.com/forums/32-Geodatabase-amp-ArcSDE

Thanks,

Chris - Solutions
0 Kudos