Web Object Grabber in Python

I had a goal to write a simple Python functions that takes either a url or html document and an XPath query and returns the queried element. This was surprisingly difficult.

After researching some through some blogs I decided to go with lxml. I also installed beautiful soup which supports the parsing of html that is pretty far from structured. I needed to use something that is as system compatible and os independent as possible. I was testing on Ubuntu 8.10.

sudo apt-get install python-lxml
sudo apt-get install python-beautifulsoup

I'll go step by step through the lines of code I wrote give comments

from lxml.html import parse
root = parse('http://www.hotels.com').getroot()
or
root = parse('hotels_test.html').getroot()

The above code first imports the parse function from the appropriate package. The parse command takes either a url or a file name as a parameter. One note about taking in a URL is that websites do not produce the same html as your browser, hence the second argument. I saved the html in a file and parsed that instead. The command .getroot() is called on a document object and returns the HTMElement at the root of the document (<>).

x = root.xpath('/html/body/div[2]/div')

The above command is the magical statement I wanted to return to me the node in the document which corresponds to that XPath statement. HOWEVER, lxml does not support the position elements ([#]). This is a big monkey wrench in my plans and it requires a significant work around. The naive method is to replace all the elements containing ' [#]' with the descendant operator ('//'). The does grow the size of the result set.

I am currently thinking of solution which elegantly fit in my methodology. I would love suggestions!! Below are some websites and resources I used and recorded.

• http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/
• http://codespeak.net/lxml/xpathxslt.html
• http://codespeak.net/lxml/lxmlhtml.html
• http://codespeak.net/lxml/dev/api/index.html [API]
• http://www.somebits.com/weblog/tech/python/xpath.html
• http://www.boddie.org.uk/python/HTML.html
  

Update

It turns out that in order to properly specify which in a number of elements an xpath is specifying you need to put parenthesis around the xpath string.

Instead of:

x = root.xpath('/html/body/div[2]/div[3]/li/ul')

we have:

x = root.xpath('((/html/body/div)[2]/div)[3]/li/ul')

This seems to work fine.

Update 2:

I was grabbing XPath's via Firebug... but it seems like either Firefox or Firebug change the the underlying DOM for some reason.  Now I need to make sure I either get the transformed HTML or an XPath query which built from the source html.