Technology for ponies

Aug 01, 2016

Django 1.10 removed migrate list option

Django 1.10 removed the very helpful --list (or -l) option from the manage migrate command. This option was deprecated since 1.8. But don't worry, you can have the same output using:

./manage.py showmigrations

You can also see in which order your migrations will be applied using the --plan option:

./manage.py showmigrations --plan

Jun 12, 2015

Convert a Foreign Key to Many to Many using Django 1.7 buit-in migrations

Django 1.7 comes with a built-in migration system for your data model, replacing the need of a third party module, usually South. This system is pretty good at doing a lot of stuff automatically, but there are some situations where you need to get your hands dirty, especially when there is data migration involved.

This guide will show you, step by step, how to convert a Foreign Key field into a Many to Many, while preserving existing relations.

Initial set up

Let's say you are starting a pizzeria and want your first pizzas to be very simple, they'll have only one topping:

from django.db import models

class Topping(models.Model):
    name = models.CharField(max_length=20)

class Pizza(models.Model):
    name = models.CharField(max_length=20)
    topping = models.ForeignKey(Topping)

Create the initial migration file (assuming our app is called pizza) and run it:

$ ./manage.py makemigrations pizza
$ ./manage.py migrate

Migrations for 'pizza':
    - Create model Pizza
    - Create model Topping
    - Add field topping to pizza

And let's create a few pizzas to test our example:

$ ./manage.py shell
from pizza.models import Topping, Pizza
for topping_name, pizza_name in [('mozzarella','Margherita'), ('mushrooms', 'Capricciosa'), ('artichoke', 'Vegetariana'), ('pineapple', 'Hawaii')]:
    topping = Topping.objects.create(name=topping_name)
    pizza = Pizza.objects.create(name=pizza_name, topping=topping)

Here are our data:

sqlite> select * from pizza_topping;

sqlite> select * from pizza_pizza;

Convert to Many to Many

After making a few pizzas you soon realize only one topping per pizza is not tasty enough. Let's tweak our model and add a field which have many toppings:

class Pizza(models.Model):
    name = models.CharField(max_length=20)
    topping = models.ForeignKey(Topping, related_name='old_topping')
    toppings = models.ManyToManyField(Topping)

We must add a related_name attribute on one of the 2 fields to have the reverse accessor (from Topping to Pizza) to work. We now run an automatic migration to create the intermediary table that will hold the new relation:

$ ./manage.py makemigrations pizza

Migrations for 'pizza':
    - Add field toppings to pizza
    - Alter field topping on pizza

Great. If we run migrate we'll now have an empty table pizza_pizza_toppings that is able to store many toppings per pizza. But we want to keep toppings we already have on our pizzas. So let's create a data migration:

$ ./manage.py makemigrations pizza --empty

Migrations for 'pizza':

Edit the newly created file pizza/migrations/0003_auto_xxx.py. We'll create a function that for each pizza, copy data from the field topping to the new field toppings:

from django.db import models, migrations

def forward(apps, schema_editor):
    Pizza = apps.get_model("pizza", "Pizza")
    for pizza in Pizza.objects.all():

class Migration(migrations.Migration):

    dependencies = [
        ('pizza', '0002_auto_20150612_0903'),

    operations = [

After running migrate, we can see toppings relationships had been copied to the new table:

sqlite> select * from pizza_pizza_toppings;

Last step is to clean up our model and remove the old topping field:

class Pizza(models.Model):
    name = models.CharField(max_length=20)
    toppings = models.ManyToManyField(Topping)
$ ./manage.py makemigrations pizza

Migrations for 'pizza':
    - Remove field topping from pizza

Voila! You can now start making delicious pizza!

Mar 18, 2015

Python JSON modules benchmark

We are currently working on a project that loads and dumps very big JSON objects. We recently had some slowness issues and though giving a try to some C compiled JSON modules could give a boost to our API.

After some googling we found out a few exsting modules and decided to setup a benchmark to decide which one to use. The selected modules are:

We wrote a little code snippet to automate our benchmark. It is using the timeit standard module.

from timeit import timeit

modules = ['json', 'cjson', 'jsonlib', 'simplejson', 'ujson', 'yajl']

for module in modules:
        loads = "loads"
        dumps = "dumps"
        if module == "cjson":
                loads = "decode"
                dumps = "encode"

        print "[%s]" % module
        load_time = timeit("obj = json.%s(data)" % loads, setup="import %s as json; data = open('file.json').read()" % module, number=NUMBER)
        dump_time = timeit("json.%s(obj)" % dumps, setup="import %s as json; obj = json.%s(open('file.json').read())" % (module, loads), number=NUMBER)

        print "Load time: ", load_time
        print "Dump time: ", dump_time


  • The file.json is 15Mb.
  • cjson is not a drop-in replacement of the standard json module. We have to patch the loads and dumps functions.
  • Loading and dumping are done 20 times for a better accuracy

And the winners are:

Load time:  12.7440698147
Dump time:  6.37574481964
Load time:  6.24644708633
Dump time:  11.6356570721
Load time:  13.1087989807
Dump time:  15.4686369896
Load time:  4.67061305046
Dump time:  6.65470814705
Load time:  5.84970593452
Dump time:  5.00060105324
Load time:  6.7509560585
Dump time:  16.4374480247

simplejson for loading and ujson for dumping. Choose one or the other depending if your program is more loading or dumping data.

Next → Page 1 of 4