Latest Contributions by MichaelPark

This error is usually thrown when the HDFS namenode is accessible to the Enterprise machines, but the datanodes are not. The namenode allows us to list files in a directory, but the actual content of the files are stored on the datanodes. I don't know exactly how Oracle BDA exposes the HDFS API, but it likely behaves the same. I would check to make sure that all of the machines in your BDA are accessible to the Enterprise machine(s). Make sure they aren't blocked by a firewall and that the advertised host names/IP addresses can be resolved by the Enterprise machines. ... View more

I see. There is currently not a way to directly run the Hive queries and return the results to Desktop using those GP tools.  ... View more

The GIS Tools for Hadoop were written mainly to support Hive queries. Please see the wiki  (Hive Spatial · Esri/spatial-framework-for-hadoop Wiki · GitHub) for more information including the list of supported ST_Geometry operators. ... View more

Do the Hadoop Esri GP Tools work on Polygon-in-polygon scenarios? Yes, the GIS tools for Hadoop supports a number of spatial relationships including polygon-intersect-polygon. See the full list here​ How efficient are they when counting, for example, buildings in a state or county? A spatial join in Hive will run relatively efficiently when you are joining a large data set with a small data set.  If you are summarizing 50 million polygons into ~50 polygons, you will be OK.  If you are summarizing that same 50 million into ~3200 polygons, you'll start to see some real slow downs. 2. In the presentations by Eric Hoel / Dave Parker in 2015 Esri Dev Summit at one stage (21:40 min), it was mentioned that Copy to HDFS tools should be avoided for loading Tera Bytes of data from SDE DB to HDFS. In that case, how do I load large spatial data (50 mil buildings ` 0.5 TB) from SDE GDB in to Hadoop? There are a number of open source tools for moving large amounts of data from a relational database to Hadoop.  Take a look at Apache Sqoop​, for example.  ... View more

Either option would be good. Sounds good.  We will probably just add both options anyway. Have you ever used the spatial framework for hadoop with ORC files? I get an error about vector not instantiated when I use ORC data. I have not.  Can you create an issue in the above github repository with some information about which Hive/Hadoop versions you are using and the full error message?  If you don't have an account there, you can just reply with that information here and I'll create an issue.   ... View more

This has been answered on Github. Copy to HDFS not working · Issue #16 · Esri/gis-tools-for-hadoop · GitHub ... View more

If you're not set on using Windows for your cluster, Linux is the way to go.  We have used both Ambari (Hortonworks) and CDH (Cloudera) to set up our clusters.  The end result for each is more or less the same.  Are you able to share some insight into what you're trying to accomplish with a Hadoop cluster?  ... View more