Django: Testing for Missing Migrations

When adding or altering model schemas in Django, developers typically generate and commit accompanying migration files. But, counterintuitively, Django wants to track all model changes in migration files, even if they don’t result in database schema changes. Those can be easy to miss.

Regardless the reason, you never want your repo to be in a state where migrations are detected as needed, but accompanying migration files aren’t committed.

Writing a pytest test that can be triggered on github, Circle, Travis, Jenkins, or whatever you use turns out to be trivial, but I couldn’t find documentation or examples on the interwebs, so am posting this here for posterity:

import pytest

from django.core.management import call_command

@pytest.mark.django_db
def test_for_missing_migrations():
    """ If no migrations are detected as needed, `result`
    will be `None`. In all other cases, the call will fail,
    alerting your team that someone is trying to make a
    change that requires a migration and that migration is
    absent.
    """

    result = call_command("makemigrations", check=True, dry_run=True)
    assert not result

It really is that simple! If no migrations are needed, result will be None. Any other return value means someone should run ./manage.py makemigrations to see what’s missing, and commit the results.

New features in django-todo

If you haven’t checked out django-todo for a while, the project has been super-active lately! In the past few months it’s gained support for file attachments, batch-task-import via CSV, and a fully integrated email tracker. Now at version 2.4.6 on pipy, or check out the live demo site.

I started this project as a small demo more than ten years ago, and it’s evolved into a piece of staple software in the Django ecosystem. Proud of what it’s become!

Announcing django-todo 2.0

django-todo is a pluggable, multi-user, multi-group, multi-list todo and ticketing system – a reusable app designed to be dropped into any existing Django project. Users can create tasks for themselves or for others, or create ”assigned tasks” that will be filed into a specific list (public tickets).

That was the original project description, and it hasn’t changed in 10+ years.

When I first created django-todo, it was a simple “let’s learn Django” project I gave to myself. I open sourced it, it’s been relatively successful, and the project has received numerous contributions over the years (grateful!). When I heard that it wasn’t compatible with Django 2.0, I looked back on that old code and realized it was time for a major refactor/upgrade. I’ve been working on the update for the past couple of months (evenings only).

Virtually every module and template has been refactored, much more in line with current best practices. The update started small, but by the end, I had made 75 commits and written the first suite of working tests (finally!). And I adopted Bootstrap as the default layout engine. And finally got around to creating a live demo site for the project.

django-todo 2.0 requires Django 2.0 and Python 3.x – no apologies. Unfortunately, this is a backwards-incompatible update (you’ll need to migrate old data manually, if you have any).

Hope it’s useful to a few teams or individuals out there. Contributions still very much welcome.

Testing Reusable Django Apps with Pytest

Django project dependencies are normally installed with a simple

pip install myapp

But if you want to be a code contributor to a dependency such as a reusable Django app (or if you are writing one), the usual pattern is to:

  1. git clone that app to another directory, outside of your project
  2. From within your project dir, use
    pip install -e /path/to/the/clone

This creates a foo.pth link and/or a foo.egg-link in your project’s virtualenv, rather than copying the code into the project itself, so you can work on the separate repo and have changes reflected in your project in real time.

That’s all well and good, but if you use pytest and need to run tests in the dependency, just running pytest from your project dir won’t find tests that live in the dependency. So can’t you just run pytest from within the dependency’s directory? Not quite – since it’s a Django app, it’s going to assume access to Django itself, and likely to other apps installed in the Django project, such as users. It would be redundant and cumbersome to install Django in the dependency’s directory, and that still wouldn’t solve the problem of access to users etc.

I can’t seem to find mention anywhere of a “best-practice” solution to this dilemma, but after much futzing around and feeling like I must be missing something obvious I tried what should have been plain from the outset:

  1. From the project dir, pip uninstall myapp
  2. From the project dir, create a symlink:
    ln -s /path/to/the/clone myapp
  3. Add myapp to your .gitignore

Now the separate reusable app appears to Django and to pytest like it’s present in the project, while maintaining physical separation between the project and the dependency. Tests run, and there’s no confusion. Just be sure to .gitignore the symlink.

There’s a gotcha to look out for here: Typically, a reusable app called “myapp” will live in a parent folder that’s also called “myapp”. If you make your symlink to the top-level folder, the imports inside it will not work. You’ll want to create your symlink to the inner folder, e.g.:

ln -s /Users/yourname/dev/myapp/myapp .

Unless I’m mistaken about something, a simple symlink (rather than pip install -e appears to be the easiest way to organize the relationship.

Django template tag to display latest git commit, date and tag in template

I found a few references to bits and pieces of this in various places, but this snippet ties all three bits of information together in a single string, and accounts for calling the path to the git repo from anywhere (which tends to work “magically” on localhost but breaks on servers).

If you don’t use git tags, season to taste.

@register.simple_tag
def git_ver():
    '''
    Retrieve and return the latest git commit hash ID and tag as a dict.
    '''

    git_dir = os.path.dirname(settings.BASE_DIR)

    try:
        # Date and hash ID
        head = subprocess.Popen(
            "git -C {dir} log -1 --pretty=format:\"%h on %cd\" --date=short".format(dir=git_dir),
            shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        version = head.stdout.readline().strip().decode('utf-8')

        # Latest tag
        head = subprocess.Popen(
            "git -C {dir} describe --tags $(git -C {dir} rev-list --tags --max-count=1)".format(dir=git_dir),
            shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        latest_tag = head.stdout.readline().strip().decode('utf-8')

        git_string = "{v}, {t}".format(v=version, t=latest_tag)
    except:
        git_string = u'unknown'

    return git_string

Then, in your template, you can simply:

{% if user.is_superuser %}{% git_ver %}{% endif %}

django-todo 1.6 released

I’m pleased to announce the release of django-todo 1.6, now with “Anyone” ticket support and py2/3 guarantee via Travis. Now available on pypi!

Django-Todo is a multi-user, multi-group todo/ticketing system for Django projects.

View 403, 404, 500 with media in Django DEBUG mode

When working with Django in DEBUG mode, it can be tough to see your 403, 404, and 500 views, since they raise visible stack traces instead of the UX the end user will see. But if you turn DEBUG off, runserver’s local media serving is disabled because it’s designed to work only with DEBUG = True. The solution is scattered throughout the Django docs, and I couldn’t find it compiled into one compact code block anywhere – just reference the handling functions directly from the end of your urls.py:

if settings.DEBUG:
    from django.views.defaults import server_error, page_not_found, permission_denied
    urlpatterns += [
        url(r'^500/$', server_error),
        url(r'^403/$', permission_denied, kwargs={'exception': Exception("Permission Denied")}),
        url(r'^404/$', page_not_found, kwargs={'exception': Exception("Page not Found")}),
    ]

And voila, your 403.html, 404.html, and 500.html templates will be displayed in full glory for developers.

Sane Password Strength Validation for Django with zxcvbn

While many admins and blog posts tell users that length is by far the most important factor in creating strong passwords/passphrases, the majority of password input fields are giving them a set of hide-bound rules: Eight characters, at least one upper- and one lowercase letter, some digits and punctuation marks, etc.

Even though it includes dictionary words, a passphrase like:

Sgt. Pepper's Mr. Kite

is far stronger than:

js72(.Tb8

(there’s a world of difference between 22 characters and 9, from a cracking perspective). But many password input fields would reject the first one. No wonder users are confused by the process of creating strong passwords!
Continue reading “Sane Password Strength Validation for Django with zxcvbn”

Django Unit Tests Against Unmanaged Databases

A Django project I’m working on defines two databases in its config: The standard/default internal db as well as a remote legacy read-only database belonging to my organization. Models for the read-only db were generated by inspectdb, and naturally have managed = False in their Meta class, which prevents Django from attempting any form of migration on them.

Unfortunately, that also prevents the Django test runner from trying to create a schema mirror of it during test runs. But what if you want to stub out some sample data from the read-only database into a fixture that can be loaded and accessed during unit tests? You’ll need to do the following:

  • Tell Django to create the second test database locally rather than on the remote host
  • Disable any routers you have that route queries for certain models through the remote db
  • Tell Django to override the Managed = False attribute in the Meta class during the test run

Putting that all together turned out to be a bit tricky, but it’s not bad once you understand how and why you need to take these steps. Because you’ll need to override a few settings during test runs only, it makes sense to create a separate test_settings.py to keep everything together:

from project.local_settings import *
from django.test.runner import DiscoverRunner


class UnManagedModelTestRunner(DiscoverRunner):
    '''
    Test runner that automatically makes all unmanaged models in your Django
    project managed for the duration of the test run.
    Many thanks to the Caktus Group: http://bit.ly/1N8TcHW
    '''

    def setup_test_environment(self, *args, **kwargs):
        from django.db.models.loading import get_models
        self.unmanaged_models = [m for m in get_models() if not m._meta.managed]
        for m in self.unmanaged_models:
            m._meta.managed = True
        super(UnManagedModelTestRunner, self).setup_test_environment(*args, **kwargs)

    def teardown_test_environment(self, *args, **kwargs):
        super(UnManagedModelTestRunner, self).teardown_test_environment(*args, **kwargs)
        # reset unmanaged models
        for m in self.unmanaged_models:
            m._meta.managed = False

# Since we can't create a test db on the read-only host, and we
# want our test dbs created with postgres rather than the default, override
# some of the global db settings, only to be in effect when "test" is present
# in the command line arguments:

if 'test' in sys.argv or 'test_coverage' in sys.argv:  # Covers regular testing and django-coverage

    DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
    DATABASES['default']['HOST'] = '127.0.0.1'
    DATABASES['default']['USER'] = 'username'
    DATABASES['default']['PASSWORD'] = 'secret'

    DATABASES['tmi']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
    DATABASES['tmi']['HOST'] = '127.0.0.1'
    DATABASES['tmi']['USER'] = 'username'
    DATABASES['tmi']['PASSWORD'] = 'secret'


# The custom routers we're using to route certain ORM queries
# to the remote host conflict with our overridden db settings.
# Set DATABASE_ROUTERS to an empty list to return to the defaults
# during the test run.

DATABASE_ROUTERS = []

# Set Django's test runner to the custom class defined above
TEST_RUNNER = 'project.test_settings.UnManagedModelTestRunner'

With that in place, you can now run your tests with:

./manage.py test --settings=project.test_settings

… leaving settings untouched during normal site operations. You can now serialize some data from your read-only host and load it as a fixture in your tests:

class DirappTests(TestCase):

    # Load test data into both dbs:
    fixtures = ['auth_group.json', 'sample_people.json']

    ...

    def test_stub_data(self):
        # Guarantees that our sample data is being loaded in the test suite
        person = Foo.objects.get(id=7000533)
        self.assertEqual(person.first_name, "Quillen")