Displaying Django User Messages with Angular.js

Django’s Messages framework is an elegant workhorse, and I’ve never built a Django site that didn’t use it for displaying success/failure/info messages to users after certain actions are taken (like logging in successfully or adding an item to a cart).

But wouldn’t it be cool if you could use that functionality client-side, delivering user messages to be processed as JSON data rather than statically outputting messages to generated HTML? On a recent project, I needed to do this because Varnish caching doesn’t let you mark page fragments as non-cacheable, so statically generated messages were not an option. But there are all sorts of reasons you might want to handle Django Messages client-side.


Here’s how to accomplish the job in a really lightweight way, without the need for a full-blown REST API app like Django Rest Framework or Tastypie, and with Angular.js (which is, IMO, the best of the current crop of JavaScript application frameworks).
Continue reading “Displaying Django User Messages with Angular.js”

django-allauth: Retrieve First/Last Names from FB, Twitter, Google

Of the several libraries/packages available for setting up social network logins for Django projects, I currently find django-allauth the most complete, with the best docs and the most active development. Doesn’t hurt that the lead dev on the project is super friendly and responsive on StackOverflow!

But not everything about it is intuitive. After wiring up Twitter, Facebook and Google as login providers, I found that first and last names were not being retrieved from the remote services when an account was successfully created. I also, frustratingly, could find only the most oblique references online to how to accomplish this.

There are a couple of ways to go about it – you can either receive and handle the allauth.account.signals.user_signed_up signal that allauth emits on success, or set up allauth.socialaccount.adapter.DefaultSocialAccountAdapter, which is also unfortunately barely documented.

I decided to go the signals route. The key to making this work is in intercepting the sociallogin parameter your signal handler will receive when an account is successfully created. I then installed a breakpoint with import pdb; pdb.set_trace() to inspect the contents of sociallogin. Once I had access to those goodies, I was able to post-populate the corresponding User objects in the database.

This sample code grabs First/Last names from Twitter, Facebook or Google; season to taste:

# When account is created via social, fire django-allauth signal to populate Django User record.
from allauth.account.signals import user_signed_up
from django.dispatch import receiver

def user_signed_up_(request, user, sociallogin=None, **kwargs):
    When a social account is created successfully and this signal is received,
    django-allauth passes in the sociallogin param, giving access to metadata on the remote account, e.g.:

    sociallogin.account.provider  # e.g. 'twitter' 

    See the socialaccount_socialaccount table for more in the 'extra_data' field.

    if sociallogin:
        # Extract first / last names from social nets and store on User record
        if sociallogin.account.provider == 'twitter':
            name = sociallogin.account.extra_data['name']
            user.first_name = name.split()[0]
            user.last_name = name.split()[1]

        if sociallogin.account.provider == 'facebook':
            user.first_name = sociallogin.account.extra_data['first_name']
            user.last_name = sociallogin.account.extra_data['last_name']

        if sociallogin.account.provider == 'google':
            user.first_name = sociallogin.account.extra_data['given_name']
            user.last_name = sociallogin.account.extra_data['family_name']


Center for Investigative Reporting

cir A year and a half ago, I left the Berkeley J-School to experience life in a high-energy web development shop with central campus. I learned a ton in that short time – the Agile process, Angular.js, building sites as Single Page Applications, strict separation between back-end and front-end systems, rigorous code review processes, and much more. And I had the opportunity to work with a crew of Java, Ruby, and Javascript rock stars, from whom I’ve learned so much.

Since my career to date had been as a web tech generalist (i.e. one person wearing all the hats), I found the experience incredibly illuminating. And yet… the project and I had some “creative differences” which ultimately resulted in me leaving the department at the end of May.

I’ve spent the past month working on personal and freelance projects, studying, and job hunting. I longed to work with journalists again, and really missed working with Django, which still feels like the most natural and effective way to build highly customized data-driven web sites I’ve ever encountered. At the same time, I wanted to make sure that my work had some kind of higher purpose – I wanted to be part of something with social and political impact.

Unfortunately, I wasn’t able to find anything on campus that really fit my requirements, and finally made the tough decision to start looking off-campus.

Today, I’m thrilled to say that I believe I’ve found the perfect fit, as a full-time Django developer at the Center for Investigative Reporting in Berkeley.

At The Center for Investigative Reporting (CIR), we believe journalism that moves citizens to action is an essential pillar of democracy. Since 1977, CIR has relentlessly pursued and revealed injustices that otherwise would remain hidden from the public eye. Today, we’re upholding this legacy and looking forward, working at the forefront of journalistic innovation to produce important stories that make a difference and engage you, our audience, across the aisle, coast to coast and worldwide.

CIR recently merged with the Bay Citizen and California Watch, two excellent journalism organizations that have had myriad overlapping projects with the J-School over the years. In fact, walking around the CIR offices today, I’m meeting former J-School students and instructors I haven’t seen in years – kind of a homecoming!

I’ll be enthusiastically starting work in mid-July. Yes, it’s tough to say goodbye to the University, but it really is an ideal evolutionary step for me right now.


Building mod_wsgi with EasyApache for WHM/cPanel

Note: These instructions are for root owners of WHM/cPanel systems, not end users.

If you want to run Django sites on a cPanel server, you’ll probably want to use the mod_wsgi Apache module. There are plenty of instructions out there on compiling mod_wsgi, but if you create it outside of the cPanel system, mod_wsgi.so will vanish each time you run easy_apache to upgrade your apache and php.

The key is to install this mod_wsgi for cPanel module. But before you go there, you’re going to want a more recent version of Python installed, since RedHat and CentOS still ship with Python 2.4, which will be deprecated by Django soon. However, you can’t overwrite the system-provided Python because yum and Mailman depend on it.

Download Python 2.7 (or whatever the latest is) into /usr/local/src. It’s critical that you build Python with shared libraries enabled, since mod_wsgi will be wanting to use them. So unpack the Python archive and cd into it, then:

./configure --enable-shared
make install

You’ll get a new build of python in /usr/local/bin, without disrupting the native version in /usr/bin. Any user wanting python2.7 to be their default can add this to their .bash_profile:


You’ll also get new libpython shared objects in /usr/local/lib. When you go to build mod_wsgi, easy_apache will need to look for python libs in that location. I found that copying the libs into standard library locations such as /lib and /usr/lib as suggested here didn’t do the trick. What did work was to add a system configuration file pointing to the new libs. Do this:

cd /etc/ld.so.conf.d
echo "/usr/local/lib/" > python27.conf

Now you’re ready to build mod_wsgi through easy_apache. Download custom_opt_mod-mod_wsgi.tar.gz from this ticket at google code and run:

tar -C /var/cpanel/easy/apache/custom_opt_mods -xzf custom_opt_mod-mod_wsgi.tar.gz

That unpacks the module into the right location so that easy_apache will find it and present it as a build option. Run easy_apache as usual (either via script or through WHM) and select the mod_wsgi option. When complete, you’ll find mod_wsgi.so along with all your other modules in /usr/local/apache/modules. The best part is, this will now become part of the default easy_apache build process, so Django sites won’t break when you rebuild apache+php in the future.

Many thanks to challgren for creating the module and to Graham Dumpleton for all of his mod_wsgi evangelism and support.

Migrating from Django-Tagging to Taggit

When Bucketlist launched a year ago and I needed a good app to let users create a taxonomy for their life goals, django-tagging was the main contender, and that’s what we went with.

Django-tagging worked pretty well overall, but had one critical bug: Because it only had a tag “name” field but no slug field, users could enter tags with slashes in them. Accessing lists of those tags would then generate a 500 error – a bad user experience, unclean, and I was getting tired of seeing the error reports. Unfortunately, django-tagging hasn’t been been updated in quite a while – starting to look like abandon-ware.

At Djangocon 2010, buzz was that Alex Gaynor’s django-taggit was picking up the slack and becoming the go-to tagging library for Django. Unfortunately, Taggit provides no migration strategy to move your existing tag base over. I held off on migration hoping one would appear, then finally decided this week to try it myself. Thought I’d document the process for others in the same boat.
Continue reading “Migrating from Django-Tagging to Taggit”

Shorter URLs with Base62 in Django

Update, 4/2017: See this StackOverflow answer for a different (and probably shorter) approach to this problem.

URL shorteners have become a hot commodity in the age of Twitter, where every byte counts. Shorteners have their uses, but they can also be potentially dangerous, since they mask the true destination of a link from users until it’s too late (shorteners are a malware installer’s wet dream). In addition, they work almost as a second layer of DNS on top of the internet, and a fragile one at that – if a shortening company goes out of business, all the links they handle could potentially break.

On bucketlist.org, a Django site that lets users catalog life goals, I’ve been using numerical IDs in URLs. As the number of items stored started to rise, I watched my URLs getting longer. Thinking optimistically about a hypothetical future with tens of millions of records to serve, and inspired by the URL structure at the Django-powered photo-sharing site Instagr.am, decided to do some trimming now, while the site’s still young. Rather than rely on a shortening service, decided to switch to a native Base 62 URL schema, with goal page URIs consisting of characters from this set:

BASE62 = "abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"

rather than just the digits 0-9. The compression is significant. Car license plates use just seven characters and no lower-case letters (base 36), and are able to represent tens of millions of cars without exhausting the character space. With base 62, the namespace is far larger. Here are some sample encodings – watch as the number of characters saved increases as the length of the encoded number rises:

Numeric Base 62
1 b
22 w
333 fx
4444 bjG
55555 o2d
666666 cN0G
7777777 6Dwb
88888888 gaYdK
999999999 bfFTGp
1234567890 bv8h5u

I was able to find several Django-based URL shortening apps, but I didn’t want redirection – I wanted native Base62 URLs. Fortunately, it wasn’t hard to roll up a system from scratch. Started by finding a python function to do the basic encoding – this one did the trick. I saved that in a utils.py in my app’s directory.

Of course we need a new field to store the hashed strings in – I created a 5-character varchar called “urlhash” … but there’s a catch – we’ll come back to this.

The best place to call the function is from the Item model’s save() method. Any time an Item is saved, we grab the record ID, encode it, and store the return value in urlhash. By putting it on the save() method, we know we’ll never end up with an empty urlhash field if the item gets stored in an unpredictable way (site users can either create new items, or copy items from other people’s lists into their own, for example, and there may be other ways in the future — we don’t want to have to remember to call the baseconvert() function from everywhere when a single place will do — keep it DRY!)).

Generating hashes

So in models.py:

from bucket.utils import BASE10, BASE62, baseconvert


def save(self):

    # Do a bunch of stuff not relevant here...

    # Initial save so the record gets an ID returned from the db
    super(Item, self).save()

    if not self.urlhash:
        self.urlhash = baseconvert(str(self.id),BASE10,BASE62)

Now create a new record in the usual way and verify that it always gets an accompanying urlhash stored. We also need to back-fill all the existing records. Easy enough via python manage.py shell:

from bucket.models import Item
from bucket.utils import BASE10, BASE62, baseconvert

items = Item.objects.all()
for i in items:
    print i.id
    i.urlhash = baseconvert(str(i.id),BASE10,BASE62)
    print i.urlhash

Examine your database to make sure all fields have been populated.

About that MySQL snag

About that “snag” I mentioned earlier: The hashes will have been stored with mixed-case letters (and numbers), and they’re guaranteed to be unique if the IDs you generated them from were. But if you have two records in your table with urlhashes ‘U3b’ and ‘U3B’, and you do a Django query like :

urlhash = 'U3b'
item = Item.objects.get(urlhash__exact=urlhash)

Django complains that it finds two records rather than one. That’s because the default collation for MySQL tables is case-insensitive, even when specifying case-sensitive queries with Django! This issue is described in the Django documentation and there’s nothing Django can do about it – you need to change the collation of the urlhash column to utf8_bin. You can do this easily with a good database GUI, or with a query similar to this:

ALTER TABLE `db_name`.`db_table_name` CHANGE COLUMN `urlhash` `urlhash` VARCHAR(5) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '' AFTER `id`;

or, if you’re creating the column fresh on an existing table:

ALTER TABLE `bucket_item` ADD `urlhash` VARCHAR( 5 ) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL AFTER `id` , ADD INDEX ( `urlhash` )

Season to taste. It’s important to get that index in there for performance reasons, since this will be your primary lookup field from now on.

Tweak URL patterns and views

Since the goal is to keep URLs as short as possible, you have two options. You could put a one-character preface on the URL to prevent it from matching other word-like URL strings, like:


but I wanted the shortest URLs possible, with no preface, just:


Since I have lots of other word-like URLs, and can’t know in advance how many characters the url hashes will be, I simply moved the regex to the very last position in urls.py – this becomes the last pattern matched before handing over to 404.

url(r'^(?P<urlhash>\w+)/$', 'bucket.views.item_view', name="item_view"),

Unfortunately, I quickly discovered that this removed the site’s ability to use Flat Pages, which rely on the same fall-through mechanism, so I switched to the “/i/B3j” technique instead.

url(r'^i/(?P<urlhash>\w+)/$', 'bucket.views.item_view', name="item_view"),

Now we need to tweak the view that handles the item details a bit, to query for the urlhash rather than the record ID:

from django.shortcuts import get_object_or_404

def item_view(request,urlhash):        
    item = get_object_or_404(Item,urlhash=urlhash)

It’s important to use get_object_or_404 here rather than objects.get(). That way we can still return 404 if someone types in a word-like URL string that the regex in urls.py can’t catch due to its open-endedness. Note also that we didn’t specify urlhash__exact=urlhash — case-sensitive lookups are the default in Django queries, and there’s no need to specify the default.

If you’ve been using something like {% url item_view item.id %} in your templates, you’ll obviously need to change all instances of that to {% url item_view item.urlhash %} (you may have to make similar changes in your view code if you’ve been using reverses with HttpResponseRedirect).

Handling the old URLs

Of course we still want to handle all of those old incoming links to the numeric URLs. We just need a variant of the original ID-matching pattern:

url(r'^(?P\d+)/$', 'bucket.views.item_view_redirect', name="item_view_numeric"),

which points to a simple view item_view_redirect that does the redirection:

def item_view_redirect(request,item_id):
    Handle old numeric URLs by redirecting to new hashed versions
    item = get_object_or_404(Item,id=item_id)
    return HttpResponseRedirect(reverse('item_view',args=[item.urlhash]))

Bingo – all newly created items get the new, permanently shortened URLs, and all old incoming links are handled transparently.

Loose Notes from Djangocon 2010

It’s been inspiring to watch the growth of the Django developer community, and the increasing traction the platform is getting from high-profile sites. NASA, The Onion, Washington Post, Mozilla, PBS, and many other prominent organizations are discovering the power of deploying on a pure Python framework, rather than on an opinionated CMS written in PHP that gets in your way as much as it helps. I was lucky to attend the first Djangocon at Google headquarters a couple of years ago, and lucky again to be able to attend the conference in Portland, OR this year.

Three solid days of panels on topics ran the gamut from low-level detail-oriented sessions like tips on working with forms to high-level recommendations from experts on things like scaling to high-traffic situations, automating the deployment process, and what could be done better. As with any conference, 3/4 of the value is in the panels, and the other 1/4 is in the networking – meeting and talking with people working with the same toolchains, exchanging tips and helping one another. I learned a ton this year. There were surprises, too – from everyone getting their own pony in their shwag bag  to the visit from Oregon congressman David Wu, to the realization that I wasn’t the most junior developer in the room, to the discovery that you could get full to the point of bursting at a vegan restaurant.

About that pony: It all started during a discussion on what features should go into the next version of Django, when someone said “I want a pony!” The feature under discussion was delivered, and the person got their pony. That led to the creation of playful sites like djangopony.com and My Little Django. Hilarious at the time, but honestly, I think the meme has played itself out, and may have just jumped the shark with everyone getting their own pony this year. I love the pony, and I love my new Pinky Pie, but I’m ready for the meme to go away now.

While most sessions were highly technical, one of the highlights was the keynote presentation by Eric Florenzano of the Djangodose podcast, “Why Django Sucks (And How We Can Fix It).” video | slides . The talk generated some controversy, but that’s healthy and good. The talk was refreshing for its honesty and forthcoming with actual solution proposals on most points. Django appeals to enterprise in part because it takes a conservative approach toward change, but the atmosphere of the platform must remain on its toes to stay competitive and forward-thinking.

Newly launched: whydjango.com – to become a collection of case studies explaining why Django is a good fit for organizations and enterprises. I plan to submit case studies for the Graduate School of Journalism and the Knight Digital Media Center soon.

Took copious notes at most of the sessions, but have only edited them lightly – apologies for typos and incomplete sentences. And sorry this is so long! (I didn’t have time to make it shorter). Downloadable slides from many of the talks are available here. And of course I only attended half of the sessions by definition. Full list of sessions here. Want to watch the whole thing? Videos of the sessions are already up!

Continue reading “Loose Notes from Djangocon 2010”

Building a Bucketlist Site with Django

Half a year ago, I got this crazy idea to build a site where people could log and record all the things they wanted to accomplish before they died. But more than just simple list-making, I wanted to make it easy for people to tell stories about their goals, and to add images and video. I wanted to let people “follow” other people’s lists, to receive email when their friends accomplished their goals, to start discussions about getting the most out of life. I wanted it to be a place where people could get inspired by the goals of others, and to easily make copies of those goals in their own bucketlists.

The result is bucketlist.org.

I had a pre-existing love affair with the Python-based Django framework – there was never a question of what platform to build on. But no matter how good the platform, the devil’s in the details.
Continue reading “Building a Bucketlist Site with Django”

Allowing Secure User Input with Django

Building a site that needs to accept formatted user input? There’s no way you’re going to let random users input any old HTML – you’d open the door to all kinds of cross-site-scripting attacks and other nastiness. Nor can you just filter out the tags you consider dangerous – that road is fraught with peril. The only solution is to white-list a small subset of tags and unceremoniously drop the rest.

There are two layers to the problem – how to support formatted text on the front-end, and how to process submitted text on the back-end.

For the front-end, some developers are drawn to the Markdown syntax – a supposedly user-friendly wiki-like syntax that can be re-rendered as safe HTML. But while Markdown may look friendly to developers, it doesn’t to normal users – trust me on this. Even for tech-savvy users, Markdown requires that you place syntax instructions on your site (inelegant). A better solution is to use a rich text editor for the web, like TinyMCE or WYMEditor.

Ever notice that you often see rich text editors in content management systems run by trusted users, but seldom on public-facing web pages? That’s because it’s tricky to do securely, and without giving users enough rope to hang themselves formatting-wise.

With a bit of configuration though, you can deploy public-facing rich textareas securely, allowing only the input of tags you specify. But you can’t stop there – all the user has to do is disable Javascript in the browser to bypass your rich text editor. You must process submitted text on the back-end with the same set of rules in your view logic.

Continue reading “Allowing Secure User Input with Django”