I was recently asked to write a script that scrapes an ASP.NET page at work. It was a paginated list of people, and each person was linked to a page with more details about them. Naturally, the HTML code was also a horrible mess.

The usual first step in such is to try if the XML functionality in your language of choice can make something workable out from the code. Since that will only work with well-formed XHTML, it was not an option here. The next thing is regular expressions, but they are such a huge pain to write and maintain for something like parsing specific data out from HTML.

Luckily, there’s a better way to do it in Python, using a library called BeautifulSoup. It’s definitely the best tool for this job I’ve seen.

No more regular expressions

I have used my fair share of regular expressions to get data out from HTML. Often it’s not a simple task to write a regex to get some data from HTML, and sometimes you’ll need tens of them to get all the data you want.

With BeautifulSoup, it’s really simple. You probably know what tag the data in is, and what’s near it. You can look up elements by attributes, parents, find text nodes based on what they contain etc.

For example, if the page has a person’s name and a phone number listed in a table:

<!-- this is somewhere in some malformed HTML -->
<td>Name:</td><td>Dirty Harry</td>
<td>Phone:</td><td>123 345 6000</td>

so to get the data out:

from BeautifulSoup import BeautifulSoup
 
# Let's assume the_html contains the html code for the page
s = BeautifulSoup(the_html)
 
# Look up element which contains "Name:", and get the next node's string contents
name = s.find(text="Name:").next.string
 
# Do same for "Phone:"
phone = s.find(text="Phone:").next.string

Doing the above with regex could’ve been tricky, for example if there was a random amount of space between the tds, or maybe the td’s weren’t organized like that etc. etc.

That was just one small example, BeautifulSoup has much more functionality to offer, so you should definitely check out the BeautifulSoup homepage for downloads and documentation!

I’m going to compare it to Zend Framework, as it’s the framework I’m most familiar with.

Reduce redundancy

Django has many features to reduce the amount of redundant code you need to write.

• Generic Views
• Forms
• Modelforms
• Automatic admin panel

Generic views

This was initially slightly confusing to me, but then I figured it out. Django’s “generic views” are basically kind of like functions – they perform some task based on the parameters you pass to them.

Instead of always writing an action in a controller for all your requests, you can create a generic view for common tasks, which could for example be a list of something in your database.

Combined with Django’s way of defining URLs separately for all actions, you can do some things in your app even without writing anything else than a url pattern.

Forms

Django’s newforms library makes it really easy to do forms. Just define a form class using the simple methods provided and you can have a nice form with validation ready in no time.

Comparing this to ZF, it’s really a lot less of typing. In ZF, the syntax for defining the form is longer, but not by much, but the main factor is that in it you’ll need to define the validation by hand. Of course, if you don’t like Django’s validation, it’s easy to define custom validation for it as well… it just has sensible defaults, which in my opinion is a Good Thing ™

Modelforms

Django’s modelforms basically let you create forms based on your models by just defining a form class and adding a Meta class parameter that tells which model class the form is for. You won’t need to define any fields or anything, and you’ll get a ready-to-use form for creating and editing model data! Simply add a view that will display the form and call form.save() when done.

Often you will need to spend time to make forms to edit your DB stuff and with modelforms it’s almost no lines of code. Even relationships between objects work – you get a select box for one-to-many relations etc.

Of course, if needed, modelforms can be extended to perform their job a bit differently too. It will require a bit of going through Django’s code, though, as at least I wasn’t able to find any examples. Definitely doable and not too difficult if you know what you’re doing.

Automatic admin panel

This is one very very useful feature. Many many times I’ve written similar admin panel code for different projects – that stuff isn’t usually very reusable. Django all that away and gives you a great automagically generated admin panel which you can even customize to a good degree to make it even better. Need to create a database driven website? Just this should be a reason good enough you should definitely at least give Django a try – not to mention all the other things it has.

Unicode support

Python, unlike PHP, has good unicode support out of the box. No weird mbstring stuff or such, and Django’s svn release is fully unicode aware. Will make your life a whole lot easier if you need to work with multiple languages.

PS: Regarding Zend Framework

While Zend Framework has a different approach to things – it is meant to be loosely coupled etc. with most components being usable without needing the others, and Django has parts that are much more tightly coupled (like modelforms with the ORM) – it could still learn a thing or two from Django.