Ponytech

Technology for ponies

mars 18, 2015

Python JSON modules benchmark

We are currently working on a project that loads and dumps very big JSON objects. We recently had some slowness issues and though giving a try to some C compiled JSON modules could give a boost to our API.

After some googling we found out a few exsting modules and decided to setup a benchmark to decide which one to use. The selected modules are:

We wrote a little code snippet to automate our benchmark. It is using the timeit standard module.

from timeit import timeit

modules = ['json', 'cjson', 'jsonlib', 'simplejson', 'ujson', 'yajl']
NUMBER = 20

for module in modules:
        loads = "loads"
        dumps = "dumps"
        if module == "cjson":
                loads = "decode"
                dumps = "encode"

        print "[%s]" % module
        load_time = timeit("obj = json.%s(data)" % loads, setup="import %s as json; data = open('file.json').read()" % module, number=NUMBER)
        dump_time = timeit("json.%s(obj)" % dumps, setup="import %s as json; obj = json.%s(open('file.json').read())" % (module, loads), number=NUMBER)

        print "Load time: ", load_time
        print "Dump time: ", dump_time

Notes:

  • The file.json is 15Mb.
  • cjson is not a drop-in replacement of the standard json module. We have to patch the loads and dumps functions.
  • Loading and dumping are done 20 times for a better accuracy

And the winners are:

[json]
Load time:  12.7440698147
Dump time:  6.37574481964
[cjson]
Load time:  6.24644708633
Dump time:  11.6356570721
[jsonlib]
Load time:  13.1087989807
Dump time:  15.4686369896
[simplejson]
Load time:  4.67061305046
Dump time:  6.65470814705
[ujson]
Load time:  5.84970593452
Dump time:  5.00060105324
[yajl]
Load time:  6.7509560585
Dump time:  16.4374480247

simplejson for loading and ujson for dumping. Choose one or the other depending if your program is more loading or dumping data.

mars 25, 2014

Use HTTP basic authentification to login into Django

Let's imagine you have a view somewhere on your website you want to password protect using your usual django login but you are too lazy to design a form to input your credentials. This makes sense if the view is for your own use and you don't need to have a fancy login page. In such a case the easiest and fastest way to proceed is to use the standard HTTP basic authentification to let your browser asks for your credentials.

It is then easy to get the user / password back in your view and to authenticate yourself into django. Here is the code snippet to do this :

from django.http import HttpResponse
from django.contrib.auth import authenticate
import base64

def my_view(request):
      if 'HTTP_AUTHORIZATION' in request.META:
              auth = request.META['HTTP_AUTHORIZATION'].split()
              if len(auth) == 2:
                      if auth[0].lower() == "basic":
                              username, password = base64.b64decode(auth[1]).split(':', 1)
                              user = authenticate(username=username, password=password)
                              if user is not None and user.is_staff:
                                      # handle your view here
                                      return render_to_response('my_template.html')

      # otherwise ask for authentification
      response = HttpResponse("")
      response.status_code = 401
      response['WWW-Authenticate'] = 'Basic realm="restricted area"'
      return response

If you need to protect more than one view you should wrap this code in a view decorator.

Please not that using HTTP basic authentification your username and password are sent base64 encoded but as it can be easily decoded you should have your website served over https to keep your crendentials secured.

déc. 17, 2013

Admin interface with foreign key on a very long list

If your application has a model with a ForeignKey on an other model which gets a lot of entries in the database, the auto-generated admin interface can become a nightmare to load. Django will render your ForeignKey using a select drop-down and with thousands of entries loading time and browser memory are in bad shape.

Solution is rather simple, add your ForeignKey in the raw_id_fields parameter in the Django admin. Lets consider this example :

models.py

class MyModel(models.Model):
  user = models.ForeignKey(User)

admin.py

from models.py import MyModel

class MyModelAdmin(admin.ModelAdmin):
    raw_id_fields = ("user",)

admin.site.register(MyModel, MyModelAdmin)
Next → Page 1 of 3