Make Yahoo! Web Service REST calls with Python

Python provides a number of modules for performing HTTP requests. This HOWTO describes how to perform GET and POST requests using the urllib and urllib2 modules from the Python standard library.

• Simple GET requests
• Simple POST requests
• Authenticated requests
• Using httplib2
• Further reading

Simple GET requests

The simplest way of retrieving data from a URL uses the urllib.urlopen function:

import urllib

url = 'http://developer.yahoo.com/'
u = urllib.urlopen(url)
# u is a file-like object
data = u.read()

This can be condensed in to a single line:

data = urllib.urlopen(url).read()

This is fine for quickly retrieving some data, but does not give you the ability to easily distinguish between normal page retrievals and 404 or 500 errors. The urllib2 module offers a similar function that throws an exception should one of these HTTP errors be encountered:

import urllib2

try:
	data = urllib2.urlopen(url).read()
except urllib2.HTTPError, e:
	print "HTTP error: %d" % e.code
except urllib2.URLError, e:
	print "Network error: %s" % e.reason.args[1]

An HTTPError is thrown if the server returns a status code other than 200 (OK) or an HTTP redirect. Redirects are followed automatically.

Simple POST requests

Some APIs require you to make POST requests. The urlopen method provided by both urllib and urllib2 supports POST requests through an optional second argument:

import urllib

url = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'
appid = 'YahooDemo'

context = '''
Italian sculptors and painters of the renaissance favored
the Virgin Mary for inspiration
'''
query = 'madonna'

params = urllib.urlencode({
	'appid': appid,
	'context': context,
	'query': query
})

data = urllib.urlopen(url, params).read()

Authenticated requests

The del.icio.us API requires you to make authenticated requests, passing your del.icio.us username and password using HTTP authentication. urllib2 provides a mechanism for doing this.

import urllib2

delicious_user = 'Your del.icio.us username'
delicious_pass = 'Your del.icio.us password'

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(
	None, 'https://api.del.icio.us/', delicious_user, delicious_pass
)
auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
xml = urllib2.urlopen('https://api.del.icio.us/v1/posts/recent').read()

Using httplib2

If the urllib2 code for making authenticated requests strikes you as over-complicated, you may be interested in Joe Gregorio's httplib2 library. It is not included in the Python standard library so you will need to download and install it separately.

All requests are passed through an instance of the Http class. You can optionally enable an HTTP cache by passing a directory to that constructor; this is not necessary for the Yahoo! APIs as they do not directly support this kind of caching.

A simple GET request looks like this:

import httplib2
http = httplib2.Http()
response, content = http.request('http://developer.yahoo.com/')

At this point, response is a dictionary-like object containing the headers from the server response, while content is the body of the response. You can check for error codes using response.status.

POST requests are similar to urllib and urllib2, but require you to explicitly set the Content-Type header of your request to application/x-www-form-urlencoded.

import httplib2
http = httplib2.Http()

url = 'http://search.yahooapis.com/ContentAnalysisService/V1/termExtraction'
appid = 'YahooDemo'

context = '''
Italian sculptors and painters of the renaissance favored the Virgin Mary for inspiration
'''
query = 'madonna'

import urllib
params = urllib.urlencode({
	'appid': appid,
	'context': context,
	'query': query
})

response, content = http.request(url, 'POST', params,
	headers={'Content-type': 'application/x-www-form-urlencoded'}
)

Finally, an authenticated request can be set up using the add_credentials method:

import httplib2
http = httplib2.Http()

delicious_user = 'Your del.icio.us username'
delicious_pass = 'Your del.icio.us password'

http.add_credentials(delicious_user, delicious_pass)

response, content = http.request('https://api.del.icio.us/v1/posts/recent')

Further reading

Related information on the web

• urllib and urllib2 documentation
• httplib documentation
• httplib2 documentation
• Dive Into Python Chapter 11: HTTP Web Services