Building a Bucketlist Site with Django

Half a year ago, I got this crazy idea to build a site where people could log and record all the things they wanted to accomplish before they died. But more than just simple list-making, I wanted to make it easy for people to tell stories about their goals, and to add images and video. I wanted to let people “follow” other people’s lists, to receive email when their friends accomplished their goals, to start discussions about getting the most out of life. I wanted it to be a place where people could get inspired by the goals of others, and to easily make copies of those goals in their own bucketlists.

The result is bucketlist.org.

I had a pre-existing love affair with the Python-based Django framework – there was never a question of what platform to build on. But no matter how good the platform, the devil’s in the details.

Data Modeling

While the basic concept was pretty simple, some of the implementation details became challenges. First, there was abstract stuff, like how to do the data modeling and logic for items copied from one list into another. If we wanted to have views like “most copied items,” it was important that people not be able to edit the copied items (if they did, the copy would no longer be a copy). But if copied items weren’t editable, no one would want to use the feature. Should a copied item have a foreign-key relationship to the original, or be a free-standing record? And what about copies of copies? Would we track them in the aggregate by traversing a relationship chain, or by counting references to a single original? Wrangling data can be a brain buzz. After trying several approaches, I decided it would be most performant to make copies into free-standing records, but with a “copied_from” relationship field. Users would be allowed to edit the details of the item, but not the title. That protected the integrity of the relationship but still allowed enough freedom to keep the feature useful.

User Registration

Handling user registration and profiles is something nearly every site needs, and the magic pairing of django-registration and django-profiles covers all the bases, keying off Django’s built-in base User object with a lot of implementation flexibility. But the documentation for them is written for rocket scientists, and customization can be a chore if you don’t have a CS background. In fact, the “Missing Manual” I wrote for django-registration a year ago has been one of the most popular items on Birdhouse since it went up.

During the Bucketlist soft-launch period, I began to realize how different user expectations can be from the assumptions you make during build. For example, a lot of people now expect to be able to use email addresses as usernames when signing up for a site. But I’m using the username as part of the user’s bucketlist URL, and personal email addresses change. No go on that. Ironically, Django 1.2 started supporting email addresses as usernames, so I had to figure out a way to disable the new feature.

I’m no fan of anonymity on the internet. One of the few things Facebook (which I generally dislike) really gets right is that it strongly encourages users to use their real names. So my signup form initially required First and Last names, and bucketlist pages were clearly labeled as such. But within days of the soft launch, I had several users trying to find ways to work around the last name requirement. Who am I to judge? Backed off and made last name optional, and found conditional ways to represent the user’s name depending on how much info they were willing to give.

Also had to figure out things like how to modify the base User object when fields in the linked Profile object is changed.

Rich Text Editor

Because they’re difficult to do securely, rich text editors in form fields are most commonly found on the back-end of content management systems, not on public facing pages. Since a user can bypass the options provided by the rich text editor simply by turning off JavaScript, input has to be carefully inspected and post-processed on the back end as well.

To get around this problem, my first versions of the site used Markdown to allow fancy formatting, but I was never happy with that approach. Developers may think Markdown is “user friendly,” but trust me — it isn’t. And because it requires a unique syntax, you always need a helper guide to go with it, which is just not user friendly. I really wanted a proper rich text editor, but needed it to be secure.

Took a lot of trial and error, but I documented the solution I finally came up with: Allowing Secure User Input With Django. I’m happy with the results.

Handling Media

From the start, I wanted to make it easy to for users to add images and video to their stories. But I didn’t want to get into the business of storing huge amounts of media – people already have accounts at Flickr and YouTube and Vimeo, and there’s a wealth of freely embeddable content at those sites ready to be re-used (Bucketlist users are encouraged to add media created by other people if they don’t have their own). At the same time, I wasn’t willing to allow users to paste embed code from those sites – that’s a security nightmare waiting to happen (even WordPress only lets site administrators paste embed code – not normal authors or end users).

My first stab at a solution was to build some kind of shortcode solution, where users could type something like:

[youtube 83kx78y]

and have that string auto-replaced with proper embed code. There’s already a django-shortcodes project out there, and I figured I’d build on that. But after spending a few evenings working with APIs of various providers and realizing how many more I had to create, decided that was a fool’s errand (I even submitted patches to the python-flickr API interface tool along the way).

Then I learned about the awesome oembed system. Oembed is a standard that lets you hand a media ID to a supporting provider, and they pass back all of the metadata for that piece of media. So a site can implement a tool that parses user-submitted text for URLs of supporting sites (like Flickr or Vimeo), extracts the media ID, and uses the returned metadata in json format to reconstruct a known-safe embed code block. Thus, users need only paste media URLs into the rich text field and the rest happens automagically.

Started off with the various forks of django-oembed, with mixed results. The codebase was really large (for what it did), and the results were buggy. URL fragments would be left in the text, and some providers simply failed to resolve (that part turned out to be a problem with the ooembed endpoint, which I was able to fix by switching to embed.ly. After trying and failing to fix django-oembed, switched to jquery-oembed, which did a cleaner job with half the lines of code. I’d rather not rely on Javascript for this functionality, but will wait until someone writes a leaner, cleaner oembed implementation for Django.

Twitter API

From the start, I had planned to let users Tweet their new goals directly from the site. Found a nice Python library for working with the Twitter API, but posting content without sending passwords around meant learning all about the OAuth dance – a little tarantella that involves passing request tokens and auth keys back and forth. In retrospect, it doesn’t look too nasty, but it took several evenings to get it just right. It’ll be easier next time :)

I didn’t implement Facebook’s posting tools into this version of the site – in part because of all the politics and privacy concerns swirling around them right now, and in part because I understand they’re moving everything to OAuth2. Will wait for the dust to settle on that a bit.

Of course posting to Twitter meant I needed shortened versions of Bucketlist URLs, so I got to play with the bit.ly API too. That part was a cakewalk – no tarantella required.

And of course I needed a character counter in the Tweet field. Found a nice JQuery plugin for that.

Design

I’m more of a developer than a designer. I know good design when I see it, and even teach a lecture to visiting journalists on Web Design Principles. But when it comes to conceptualizing original designs, I kind of freeze up. Also, I wanted to keep on trucking with the 960 Grid system I use elsewhere — 960gs makes it trivial to manipulate columns in complex layouts. Unfortunately, there are few freely available designs based on 960gs – you’re still on your own to come up with look and feel. What I ended up with isn’t ideal, but it’s not bad either. Rather than designing the whole thing in advance, I just started filling out the grid as I went along – it evolved through chipping and plucking away over time. If the site does well, I may hire a real designer at some point.

Comments

Getting the Comments system working the way I wanted them to was fun too. Django’s native commenting framework works well out of the box, but when implemented per the docs, always presents a name and email field to the user. But I only want to take comments from authenticated users, making those fields redundant and annoying. Ultimately did find a way to get this working, and decided to contribute documentation to save future users the same hassles. The ticket was accepted, but the patch was never committed (got lost in the ramp-up to Django 1.2, I think). A common problem with open source projects — to make it worth the while for the public to submit patches, those patches need to be handled in a timely way.

Deployment, DVCS

Along the way, also learned pip and the virtualenvwrapper, using requirements.txt for tracking dependencies rather than doing it all manually. And switched from svn to git for version control. Lots of learning steps through all that, including a fun version incompatibility between dev and production that yielded awesomely dadaistic error messages like:

fatal: fetch-pack: unable to fork off sideband demultiplexer

But nothing I couldn’t work through. Overall, really happy to have made the changes. Deployment onto new development machines, and onto the server, is almost trivial now.

Birdhouse runs on cPanel and cPanel doesn’t yet support Django natively – admins have to jump through quite a few hoops to get Django sites running. I’ve filed an official feature request for cPanel to support Django like it does Ruby on Rails, and it’s getting a fair number of votes, but it needs more. If you’re a cPanel admin, please add your voice! For now, my Django-on-cPanel process looks something like this.

Avatars

Thumbnails for profile avatars are provided through two mechanisms: User’s choice of Gravatar or an uploaded image. If the user does not upload an avatar, and if their registered email address matches an account at gravatar.com, their Gravatar is automatically presented via the django-gravatar template tag. If they do upload an avatar, we verify and resize it with the sorl-thumbnails library… which unfortunately appears to have been recently abandoned. Still, sorl isn’t going to break anytime soon, and I’m not seeing anything out there that looks like a great replacement, so went with it. sorl-thumbnail depends on the ubiquitous Python Imaging Library. Which is fine, except that I got dragged through the trenches trying to get PiL working on Snow Leopard. Little obstacles every step of the way add up.

Bits and Pieces

The dynamic list reordering (see the bottom of your own list) came from some Ajaxy bits I assembled from various sources when building django-todo last year – that code ported over pretty clean.

The slippy slidey two-pane views that let you toggle/slide between incomplete and completed items came from a JQuery plugin by Gaya Design (original concept here).

All of the various RSS views were assembled pretty much according to the docs on Django’s syndication framework.

The “Featured Items” (best items rotating randomly in the banner) are selected (by me) with a custom admin action which was trivial to write. The resulting queryset is randomized with the little-known “?” syntax on order_by in a global context_processor:

def featured_items(request):
    return {'featured': Item.objects.filter(featured=True).order_by('?')[:4]}

I’ve heard that using “?” isn’t very performant with large querysets, but this query will never return a large recordset, so it should be fine.

Tagging of course comes via the awesome django-tagging package.

Overall, the project took six months of intermittent nights and weekends. And despite the many hassles, it was a fantastic learning experience, and a gas to build. Now that summer is pretty much here, it’s harder to stay indoors and hack. Time to get out and actually do some of the things on my bucketlist :)