Accessing ArcGIS Online REST Services through a proxy with Python

6896
6
09-04-2014 07:14 AM
TrilliumLevine1
Occasional Contributor


Hello All,

 

I'm trying to run a script to access an ArcGIS Online REST web service and am behind an orginizational firewall, so I need to go over an authenticated proxy. My understanding is that urllib2 is supposed to pull the proxy settings from my system automatically so I shouldn't need to add any code dealing with that, but I keep getting a socket timeout error (60010) right after the script attempts to establish a connection -- I think this has to do with the proxy, but maybe I'm wrong.  Has anyone else run into this problem and found a sollution?  I tried importing the'socket' module and setting the timout value to 'NONE', as well as setting the expiration to a larger value, but the error keeps appearing after 1 minute.  I've attached the part of my code dealing with the connection to the service -- does anyone out there have any ideas?  Any feedback would be highly appreciated!

 

 

socket.PNG

 

import json, urllib, urllib2 import sys, os import arcpy import shutil  class GetToken(object):     # find feature service attachments, send to directory     def urlopen(self, url, data=None):         # open url, send response         referer = "http://www.arcgis.com/arcgis/rest"         req = urllib2.Request(url)         req.add_header('Referer', referer)         if data:             response = urllib2.urlopen(req, data)         else:             response = urllib2.urlopen(req)         return response     def gentoken(self, username, password,         referer = 'www.arcgis.com', expiration=60):         # gets token from referrer         query_dict = {'username': username,             'password': password,             'expiration': str(expiration),             'client': 'referer',             'referer': referer,             'f': 'json'}         query_string = urllib.urlencode(query_dict)         token_url = "https://www.arcgis.com/sharing/rest/generateToken"         token_response = urllib.urlopen(token_url, query_string)         token = json.loads(token_response.read())         if "token" not in token:             print token['messages']             exit()         else:             return token['token']
0 Kudos
6 Replies
JasonScheirer
Occasional Contributor III

No, you need to configure your proxies in your code. See this thread on Stack Overflow.

0 Kudos
TrilliumLevine1
Occasional Contributor

Thanks for your response, Jason.  I've tried all the variations mentioned on the link you sent, as well as others.  Right now the top of my script looks like below, and I also set my environment variables appropriately for http_proxy and https_proxy.  Right now I'm getting an error on the splituser def inside of urllib telling me that a string or buffer is expected.


import sys, os
import arcpy
import shutil
import re


# set proxies
proxy = urllib2.ProxyHandler({
    'http': '192.168.104.103',
    'https': '192.168.104.103'
})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
class GetToken(object):
    # find feature service attachments, send to directory


    def urlopen(self, url, data=None):
        # open url, send response
        referer = "http://www.arcgis.com/arcgis/rest"
        req = urllib2.Request(url)
        req.add_header('Referer', referer)
        if data:
            response = urllib2.urlopen(req, data)
        else:
            response = urllib2.urlopen(req)
        return response


    def gentoken(self, username, password,
        referer = 'www.arcgis.com', expiration=60):
        # gets token from referrer
        query_dict = {'username': username,
            'password': password,
            'expiration': str(expiration),
            'client': 'referer',
            'referer': referer,
            'f': 'json'}
        query_string = urllib.urlencode(query_dict)
        token_url = "https://www.arcgis.com/sharing/rest/generateToken"
        token_response = urllib.urlopen(token_url, query_string)
        token = json.loads(token_response.read())
        if "token" not in token:
            print token['messages']
            exit()
        else:
            return token['token']

0 Kudos
JasonScheirer
Occasional Contributor III

Can you post the traceback?

0 Kudos
TrilliumLevine1
Occasional Contributor

yep, here's the traceback:

Traceback   

    <module> C:\Users\tle\Desktop\Scripts\dl_extract2.py 161 

    main C:\Users\tle\Desktop\Scripts\dl_extract2.py 157 

    __init__ C:\Users\tle\Desktop\Scripts\dl_extract2.py 53 

    gentoken C:\Users\tle\Desktop\Scripts\dl_extract2.py 40 

    urlopen C:\Python26\ArcGIS10.0\lib\urllib.py 88 

    open C:\Python26\ArcGIS10.0\lib\urllib.py 207 

    open_https C:\Python26\ArcGIS10.0\lib\urllib.py 439 

    endheaders C:\Python26\ArcGIS10.0\lib\httplib.py 904 

    _send_output C:\Python26\ArcGIS10.0\lib\httplib.py 776 

    send C:\Python26\ArcGIS10.0\lib\httplib.py 735 

    connect C:\Python26\ArcGIS10.0\lib\httplib.py 1112 

    wrap_socket C:\Python26\ArcGIS10.0\lib\ssl.py 350 

    __init__ C:\Python26\ArcGIS10.0\lib\ssl.py 118 

    do_handshake C:\Python26\ArcGIS10.0\lib\ssl.py 293 

IOError: [Errno socket error] [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol   

0 Kudos
TrilliumLevine1
Occasional Contributor

This actually turned out to be a pretty easy fix.  All that were needed in my case were the system variables:

http_proxy: http//:user:password@server:port

https_proxy: http//:user:password@server:port

This link explains why the http protocol is used for both.

The following code was not needed -- in fact, the script threw a 404 error when it was included:

proxy = urllib2.ProxyHandler({ 

    'http': '192.168.104.103'

    'https': '192.168.104.103' 

}) 

opener = urllib2.build_opener(proxy) 

urllib2.install_opener(opener) 

BertKraan1
Occasional Contributor III

One more discovery:

Turns out that (in my situation, ymmv) the authentication is key. So setting the username in the proxy"definition" is mandatory. Either in the environment setting or in proxies = urllib2.ProxyHandler I needed to add http://username:password@abc.com:1234

hth, Bert