Tag: model

Django powered AJAX Chat – Part 2

I’m sorry I left my last Django powered AJAX Chat – Part 1 post so abandoned, we have been dealing with an ever increasing amount of study stuff at the uni and I didn’t have much time to keep on writing about our chat application. I have been working hard to improve some of the functionality of the application, both on the Django side and the jQuery side. There are still many rough edges, but this time I’ve got some more along the lines of plug&play. So, just for those impatient ones who like to see to believe, this is a screenshot of it in action:

Screenshot of the Django chat application using the jQuery Javascript library
Screenshot of the Django chat application using the jQuery Javascript library, you can appreciate the use of special formatting for users who are joining the room, normal formatting for user messages and... yes... smileys!

Pretty neat, huh? There is a lot of ground to cover with this one, so maybe I’ll split this post in two… we will see.

The Idea

Have a pure Django + jQuery chat application working on our site. The users will be presented with a GUI like the one above, listing the messages in the chat room, and will be allowed to submit messages though a text box. A Javascript process on the user’s browser will be quering the server every N seconds for new messages and the server will respond in JSON format, returning a list of the new messages (if any) along with their authors, a timestamp (optional) and an id (very important). If there are new messages, they will be formatted and correctly appended to the chat box. Different types of messages should be supported, among them are: user joins, user leaves, system notifications, special messages, and of course normal user messages. Extras: Try to make it feel as natural as possible, using soft scrolling when new messages arrive, capturing the press of the Enter key when typing a message to automatically send the written text, clear the input box and allow the user to keep on typing… and smilies!

Prerequisites

Before we start, some prerequisites: Django must have the auth and contenttype apps installed, if you are using the Django Admin they are all set up and you can just keep on reading. Rule of thumb: if the admin works, chat will too.

Where Can I Download This and Other Announcements

I though I might as well upload this as a Google Code project (gotta love version control), so you can checkout the project at: http://code.google.com/p/django-jchat/, or download the zip file in the downloads tab. I still strongly recommend that you read my rambles about the code below, for you can gain an overall insight of the code and what I wanted to achieve with each line of code.

Also consider that this is a post I started writing in May… but got hold back until now due to time constraints… so the pasted code could be… a little bit outdated.

The Models

To begin let’s revise the models, as you might recall from the last post we had only two models, a Room model and a Message model. (I didn’t work on a Session model to display the list of users because we already had something like that, so if you want to add one yourself, be my guest, and make sure to drop me a mail on how you modified/improved it :D.)

These models basically behave like your average IRC chat server, you have several chat rooms which users can join and message in. I wanted to make sure my chat app was as portable as possible, so I first worked on a home made solution to allow my Room models to reference different kinds of objects, needless to say that is was sloppy. Unsatisfied with my solution I started looking around for better ones. It didn’t take me long until I stumbled upon Django Admin Log model, which had the capacity of referencing each of my site’s models. Looking into the code I discovered the magic of contenttype’s GenericType and generic.GenericForeignKey classes. They are a work of art! They allow me to establish relationships between my models and any other model with just three extra fields in my models and almost no programming! This is how the Room model looks now:

from django.db import models
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType, ContentTypeManager
from django.contrib.contenttypes import generic
from datetime import datetime

class RoomManager(models.Manager):
    '''Custom model manager for rooms, this is used for "table-level" operations.
    All methods defined here can be invoked through the Room.objects class.
    @see: http://docs.djangoproject.com/en/1.0/topics/db/managers/#topics-db-managers
    Also see GenericTypes from the contenttypes django app!
    @see: http://docs.djangoproject.com/en/1.0/ref/contrib/contenttypes/''' 

    def create(self, object):
        '''Creates a new chat room and registers it to the calling object'''
        r = self.model(content_object=object)
        r.save()
        return r

    def get_for_object(self, object):
        '''Try to get a room related to the object passed.'''
        return self.get(content_type=ContentType.objects.get_for_model(object), object_id=object.pk)

    def get_or_create(self, object):
        '''Save us from the hassle of validating the return value of get_for_object and create a room if none exists'''
        try:
            return self.get_for_object(object)
        except Room.DoesNotExist:
            return self.create(object)

class Room(models.Model):
    '''Representation of a generic chat room'''
    content_type = models.ForeignKey(ContentType) # to what kind of object is this related
    object_id = models.PositiveIntegerField() # to which instace of the aforementioned object is this related
    content_object = generic.GenericForeignKey('content_type','object_id') # use both up, USE THIS WHEN INSTANCING THE MODEL
    created = models.DateTimeField(default=datetime.now())
    comment = models.TextField(blank=True, null=True)
    objects = RoomManager() # custom manager

    def __add_message(self, type, sender, message=None):
        '''Generic function for adding a message to the chat room'''
        m = Message(room=self, type=type, author=sender, message=message)
        m.save()
        return m

    def say(self, sender, message):
        '''Say something in to the chat room'''
        return self.__add_message('m', sender, message)
    def join(self, user):
        '''A user has joined'''
        return self.__add_message('j', user)

    def leave(self, user):
        '''A user has leaved'''
        return self.__add_message('l', user)

    def messages(self, after_pk=None, after_date=None):
        '''List messages, after the given id or date'''
        m = Message.objects.filter(room=self)
        if after_pk:
            m = m.filter(pk__gt=after_pk)
        if after_date:
            m = m.filter(timestamp__gte=after_date)
        return m.order_by('pk')

    def last_message_id(self):
        '''Return last message sent to room'''
        m = Message.objects.filter(room=self).order_by('-pk')
        if m:
            return m[0].id
        else:
            return 0

    def __unicode__(self):
        return 'Chat for %s %d' % (self.content_type, self.object_id)
from django.db import models
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType, ContentTypeManager
from django.contrib.contenttypes import generic
from datetime import datetime
class RoomManager(models.Manager):
”’Custom model manager for rooms, this is used for “table-level” operations.
All methods defined here can be invoked through the Room.objects class.
Also see GenericTypes from the contenttypes django app!
def create(self, object):
”’Creates a new chat room and registers it to the calling object”’
r = self.model(content_object=object)
r.save()
return r
def get_for_object(self, object):
”’Try to get a room related to the object passed.”’
return self.get(content_type=ContentType.objects.get_for_model(object), object_id=object.pk)
def get_or_create(self, object):
”’Save us from the hassle of validating the return value of get_for_object and create a room if none exists”’
try:
return self.get_for_object(object)
except Room.DoesNotExist:
return self.create(object)
class Room(models.Model):
”’Representation of a generic chat room”’
content_type = models.ForeignKey(ContentType) # to what kind of object is this related
object_id = models.PositiveIntegerField() # to which instace of the aforementioned object is this related
content_object = generic.GenericForeignKey(‘content_type’,’object_id’) # use both up, USE THIS WHEN INSTANCING THE MODEL
created = models.DateTimeField(default=datetime.now())
comment = models.TextField(blank=True, null=True)
objects = RoomManager() # custom manager
def __add_message(self, type, sender, message=None):
”’Generic function for adding a message to the chat room”’
m = Message(room=self, type=type, author=sender, message=message)
m.save()
return m
def say(self, sender, message):
”’Say something in to the chat room”’
return self.__add_message(‘m’, sender, message)
def join(self, user):
”’A user has joined”’
return self.__add_message(‘j’, user)
def leave(self, user):
”’A user has leaved”’
return self.__add_message(‘l’, user)
def messages(self, after_pk=None, after_date=None):
”’List messages, after the given id or date”’
m = Message.objects.filter(room=self)
if after_pk:
m = m.filter(pk__gt=after_pk)
if after_date:
m = m.filter(timestamp__gte=after_date)
return m.order_by(‘pk’)
def last_message_id(self):
”’Return last message sent to room”’
m = Message.objects.filter(room=self).order_by(‘-pk’)
if m:
return m[0].id
else:
return 0
def __unicode__(self):

return ‘Chat for %s %d’ % (self.content_type, self.object_id)

I hope the commentaries are enough for most of you, I don’t want to bore you down with details, and the comments make the code (in my humble opinion) pretty much self explanatory. In any case, just use the comments below to share your inquiries on the code. Don’t despair, I will elaborate on the methods defined at RoomManager later on when we talk about the views.

In any case, the Message model was pretty much unchanged, for the sake of completness I’ll post it here too:

MESSAGE_TYPE_CHOICES = (
    ('s','system'),
    ('a','action'),
    ('m', 'message'),
    ('j','join'),
    ('l','leave'),
    ('n','notification')
)

class Message(models.Model):
    '''A message that belongs to a chat room'''
    room = models.ForeignKey(Room)
    type = models.CharField(max_length=1, choices=MESSAGE_TYPE_CHOICES)
    #sender = models.CharField(max_length=50, blank=True)
    author = models.ForeignKey(User, related_name='author', blank=True, null=True)
    message = models.CharField(max_length=255, blank=True, null=True)
    timestamp = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        '''Each message type has a special representation, return that representation.
        This will also be translator AKA i18l friendly.'''
        if self.type == 's':
            return u'SYSTEM: %s' % self.message
        if self.type == 'n':
            return u'NOTIFICATION: %s' % self.message
        elif self.type == 'j':
            return 'JOIN: %s' % self.author
        elif self.type == 'l':
            return 'LEAVE: %s' % self.author
        elif self.type == 'a':
            return 'ACTION: %s > %s' % (self.author, self.message)
        return self.message

Again it should be somewhat self explanatory ;). If in doubt, just ask.

Views

Phase one complete, we have our pretty nifty models done, now we need to take care of some basic views (and their respective urls) to interact with the chat app. The view itseft does nothing special, it recieves messages sent by users, or returns the list of messages after a certain message, recieves log in and logout events, among other things. I did develop a quick test view (marked in dark orange), and we’ll use it to explain how to create a chat room for your site. But first, the views:

# -*- encoding: UTF-8 -*-
'''
Chat application views, some are tests... some are not
@author: Federico Cáceres <fede.caceres@gmail.com>
'''
from datetime import datetime

from django.http import HttpResponse, Http404
from django.shortcuts import render_to_response
from django.contrib.auth.decorators import login_required
from django.template import RequestContext
from django.contrib.auth.models import User 

from chat.models import Room, Message

@login_required
def send(request):
    '''
    Expects the following POST parameters:
    chat_room_id
    message
    '''
    p = request.POST
    r = Room.objects.get(id=int(p['chat_room_id']))
    r.say(request.user, p['message'])
    return HttpResponse('')

@login_required
def sync(request):
    '''Return last message id

    EXPECTS the following POST parameters:
    id
    '''
    if request.method != 'POST':
        raise Http404
    post = request.POST

    if not post.get('id', None):
        raise Http404

    r = Room.objects.get(id=post['id'])

    lmid = r.last_message_id()    

    return HttpResponse(jsonify({'last_message_id':lmid}))

@login_required
def receive(request):
    '''
    Returned serialized data

    EXPECTS the following POST parameters:
    id
    offset

    This could be useful:
    @see: http://www.djangosnippets.org/snippets/622/
    '''
    if request.method != 'POST':
        raise Http404
    post = request.POST

    if not post.get('id', None) or not post.get('offset', None):
        raise Http404

    try:
        room_id = int(post['id'])
    except:
        raise Http404

    try:
        offset = int(post['offset'])
    except:
        offset = 0

    r = Room.objects.get(id=room_id)

    m = r.messages(offset)

    return HttpResponse(jsonify(m, ['id','author','message','type']))

@login_required
def join(request):
    '''
    Expects the following POST parameters:
    chat_room_id
    message
    '''
    p = request.POST
    r = Room.objects.get(id=int(p['chat_room_id']))
    r.join(request.user)
    return HttpResponse('')

@login_required
def leave(request):
    '''
    Expects the following POST parameters:
    chat_room_id
    message
    '''
    p = request.POST
    r = Room.objects.get(id=int(p['chat_room_id']))
    r.leave(request.user)
    return HttpResponse('')

@login_required
def test(request):
    '''Test the chat application'''

    u = User.objects.get(id=1) # always attach to first user id
    r = Room.objects.get_or_create(u)

    return render_to_response('chat/chat.html', {'js': ['/media/js/mg/chat.js'], 'chat_id':r.pk}, context_instance=RequestContext(request))

def jsonify(object, fields=None, to_dict=False):
    '''Funcion utilitaria para convertir un query set a formato JSON'''
    try:
        import json
    except ImportError:
        import django.utils.simplejson as json

    out = []

    if type(object) not in [dict,list,tuple] :
        for i in object:
            tmp = {}
            if fields:
                for field in fields:
                    tmp[field] = unicode(i.__getattribute__(field))
            else:
                for attr, value in i.__dict__.iteritems():
                    tmp[attr] = value
            out.append(tmp)
    else:
        out = object

    if to_dict:
        return out
    else:
        return json.dumps(out)

(sorry for the cheap template of the blog 😦 )

In any case, the code is quite documented, but I can brief you on the inner workings of each view:

  • send: this view recieves the messages sent by users and associates them to the corresponding Room model
  • sync: called when the interface loads, basically it requests the id of the last message sent to a chat room, this is used so that the user when joining a chat room will only request messages sent AFTER his arrival
  • receive: called by the client’s javacript code, returns a list of messages sent. It uses a variable sent though POST called “offset”, basically it is the id of the last message received by the client (or the id received by the sync view) so that the view only returns messages after it
  • join: called when a user joins a chat room
  • leave: called when a user leaves a chat room
  • test: silly test view, basically allows you to test the app by itself without having to associate it with other apps (more on this on the next parragaph)

In a nutshell this test view gets the Room instance that is related to the user whose id = 1 (in my case this is always the admin user, or the first super user created by the django.contrib.auth app, as it is the first user created when running the syncdb manage script for the first time on a project with the django.contrib.auth app installed.) I am doing this because in my case I know that in the worst scenario, at least I will have a user model to which I can bind this chat room to, so that I can save it and send messages to it…

Based on that test view you can learn how to bind a chat Room to anything you’d like. For example: Imagine that you have a complex site, where you have different user groups. you can enter the page of any particular user group and you would like to display a chat room that is exclusive for that group whose page you are joing. So, if you have a UserGroup model, you could write a view like this to get or create the Room instace associated to your UserGroup model:

from django.shortcuts import render_to_response
from yourapp.chat.models import Room
from models import UserGroup

def usergroup_index(request, group_id):
    group = UserGroup.models.get(id=group_id)
    room = Room.objects.get_or_create(group)
    return render_to_response("your_template.html", {'group':group, 'chat_id':room.id})

Just a quick and dirty example, but basically what that get_or_create manager function does is try to get the Room instance that is associated with the object instance you are passing it as a parameter. It is quite magical thanks again to the GenericType class. This allows you, like I mentioned before, to “attach” or “bind” a chat room to any kind of model.

In any case, it is imperative that your render_to_response call (or whichever method you use) sends the room id to the template (I’ll show you how I use it in the next section) or else nothing will happen!

To wrap this up, this is the urls.py file:

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    url(r'^request/$', 'forum.views.forum'),

    url(r'^$', 'chat.views.test'),
    url(r'^send/$', 'chat.views.send'),
    url(r'^receive/$', 'chat.views.receive'),
    url(r'^sync/$', 'chat.views.sync'),

    url(r'^join/$', 'chat.views.join'),
    url(r'^leave/$', 'chat.views.leave'),
)

With that done, let’s see how a basic template is set up, and the final piece of the puzzle, the Javascript… in front of it all :D.

Template

From now on, you can just rely on the basic copy-paste method, for you should have no need to touch the html (unless you want to modify the looks of it, in which case you should be modifying the css rules). Fortunately this is very easy, just paste the following somewhere in your code:

<div id="chat"></div>

<script type="text/javascript">
$(window).ready(function(){
	init_chat({{ chat_id }}, "chat");
})
</script>

What we see here is a blank <div> block with the chat id. The chat javscript files will place all html elements here for you, so you don’t need to worry. Then we have a short call to a javascript function, using jQuery’s event binding we tell it to “call init_chat” once the window is done loading (html is ready). That “chat_id” variable in the template is the id of the room that your view got from looking up (or creating) a chat Room, and it is necessary so that the javascript managing your chatroom initializes correctly and knows which is the Room it should query.

Of course, you should also include the jquery.js file, you can download it from jquery.com or directly link at a version left available online by the nice people of Google:

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3/jquery.min.js"></script>

You shuld place THAT and the next javascript file in your html so that everything works.

Javascript (jQuery)

This is the script done in Javascript using the jQuery Library… documentation to come soon!

var chat_room_id = undefined;
var last_received = 0;

/**
 * Initialize chat:
 * - Set the room id
 * - Generate the html elements (chat box, forms & inputs, etc)
 * - Sync with server
 * @param chat_room_id the id of the chatroom
 * @param html_el_id the id of the html element where the chat html should be placed
 * @return
 */
function init_chat(chat_id, html_el_id) {
	chat_room_id = chat_id;
	layout_and_bind(html_el_id);
	sync_messages();
}

/**
 * Asks the server which was the last message sent to the room, and stores it's id.
 * This is used so that when joining the user does not request the full list of
 * messages, just the ones sent after he logged in.
 * @return
 */
function sync_messages() {
    $.ajax({
        type: 'POST',
        data: {id:window.chat_room_id},
        url:'/chat/sync/',
		dataType: 'json',
		success: function (json) {
        	last_received = json.last_message_id;
		}
    });

	setTimeout("get_messages()", 2000);
}

/**
 * Generate the Chat box's HTML and bind the ajax events
 * @param target_div_id the id of the html element where the chat will be placed
 */
function layout_and_bind(html_el_id) {
		// layout stuff
		var html = '<div id="chat-messages-container">'+
		'<div id="chat-messages"> </div>'+
		'<div id="chat-last"> </div>'+
		'</div>'+
		'<form id="chat-form">'+
		'<input name="message" type="text" class="message" />'+
		'<input type="submit" value="Say"/>'+
		'</form>';

		$("#"+html_el_id).append(html);

		// event stuff
    	$("#chat-form").submit( function () {
            var $inputs = $(this).children('input');
            var values = {};

            $inputs.each(function(i,el) {
            	values[el.name] = $(el).val();
            });
			values['chat_room_id'] = window.chat_room_id;

        	$.ajax({
                data: values,
                dataType: 'json',
                type: 'post',
                url: '/chat/send/'
            });
            $('#chat-form .message').val('');
            return false;
	});
};

/**
 * Gets the list of messages from the server and appends the messages to the chatbox
 */
function get_messages() {
    $.ajax({
        type: 'POST',
        data: {id:window.chat_room_id, offset: window.last_received},
        url:'/chat/receive/',
		dataType: 'json',
		success: function (json) {
			var scroll = false;

			// first check if we are at the bottom of the div, if we are, we shall scroll once the content is added
			var $containter = $("#chat-messages-container");
			if ($containter.scrollTop() == $containter.attr("scrollHeight") - $containter.height())
				scroll = true;

			// add messages
			$.each(json, function(i,m){
				if (m.type == 's')
					$('#chat-messages').append('<div class="system">' + replace_emoticons(m.message) + '</div>');
				else if (m.type == 'm')
					$('#chat-messages').append('<div class="message"><div class="author">'+m.author+'</div>'+replace_emoticons(m.message) + '</div>');
				else if (m.type == 'j')
					$('#chat-messages').append('<div class="join">'+m.author+' has joined</div>');
				else if (m.type == 'l')
					$('#chat-messages').append('<div class="leave">'+m.author+' has left</div>');

				last_received = m.id;
			})

			// scroll to bottom
			if (scroll)
				$("#chat-messages-container").animate({ scrollTop: $("#chat-messages-container").attr("scrollHeight") }, 500);
		}
    });

    // wait for next
    setTimeout("get_messages()", 2000);
}

/**
 * Tells the chat app that we are joining
 */
function chat_join() {
	$.ajax({
		async: false,
        type: 'POST',
        data: {chat_room_id:window.chat_room_id},
        url:'/chat/join/',
    });
}

/**
 * Tells the chat app that we are leaving
 */
function chat_leave() {
	$.ajax({
		async: false,
        type: 'POST',
        data: {chat_room_id:window.chat_room_id},
        url:'/chat/leave/',
    });
}

// attach join and leave events
$(window).load(function(){chat_join()});
$(window).unload(function(){chat_leave()});

// emoticons
var emoticons = {
	'>:D' : 'emoticon_evilgrin.png',
	':D' : 'emoticon_grin.png',
	'=D' : 'emoticon_happy.png',
	':\\)' : 'emoticon_smile.png',
	':O' : 'emoticon_surprised.png',
	':P' : 'emoticon_tongue.png',
	':\\(' : 'emoticon_unhappy.png',
	':3' : 'emoticon_waii.png',
	';\\)' : 'emoticon_wink.png',
	'\\(ball\\)' : 'sport_soccer.png'
}

/**
 * Regular expression maddness!!!
 * Replace the above strings for their img counterpart
 */
function replace_emoticons(text) {
	$.each(emoticons, function(char, img) {
		re = new RegExp(char,'g');
		// replace the following at will
		text = text.replace(re, '<img src="/media/img/silk/'+img+'" />');
	});
	return text;
}

CSS

This is the default css I used:

/* CHAT */

#chat-messages-container {
	overflow: auto; /* used for scrolling */
	width: 100%;
	height: 100px;
	border: 1px solid #777;
	-moz-border-radius: 5px;
}

#chat-messages {
	padding: 5px;
}

#chat-messages .author {
	font-weight: bold;
	margin-right: 0.5em;
	display: inline;
}

#chat-messages .message {
	margin-bottom: 4px;
}

#chat-messages .system {
	font-weight: bold;
	color: #642;
}

#chat-messages .join {
	color: #696;
	background: url(img/silk/door_in.png) top left no-repeat;
	padding-left: 20px;
	margin: 4px 0;
}

#chat-messages .leave {
	color: #966;
	background: url(img/silk/door_out.png) top left no-repeat;
	padding-left: 20px;
	margin: 4px 0;
}

#chat .message {
	width: 80%;
}
The whole mess together

Phew, that was excruciating! Hopefully you got a grasp on how the pieces fit together! But if everything was done correctly, it will work like a charm!

Let me share with you a little extra for reading all the way to the bottom, a little video of me… chatting with my self!

Hope you really enjoyed this, it was an absolute pleasure to work on this app on the marvellous Django framework. As I mentioned on my previous post, there are other ways to do this, which are much more optimized and appropiate, and we also left behind several security and functional issues besides (sending messages to a Room where you have not joined for instance)… but even though of all that, this still works quite well, and is a nice addition to your site. Sure, it may also kill your database if the number of visitors is high enough :P!

Remember you can checkout the whole code from http://code.google.com/p/django-jchat/ it should work right out of the box on any given django project.

And again, if you have questions, just post a comment below and I’ll try to help you out.

Until next time!

Advertisements

Find relation between two models

Today one of my teammates and I were working on a complex SQL query which was beyond Django’s ORM normal API, so we had to use the extra method to build the query. After running with a couple of issues with PostgreSQL’s GROUP BY we decided to opt quickly for subquieries, and in no time we had our query done. The problem was though, that we had to do this operation on several models that where similar to each other, the only thing that changed was the WHERE clause depending on the models involved.

So I set off so find a way to automatically discover which was the relation between two models. In short time I discovered this: MyModel._meta.get_all_related_objects(). This returns a list of RelatedObject instances that tell us about which models point to MyModel  though a foreign key, cool! So, let us suppose we have a Car model and a Rental model. The Rental model has a ForeingKey to Car, to know how they are related I built something like this:

for r in target_model._meta.get_all_related_objects():
if r.model == ranking_model:
relation = r
for r in Car._meta.get_all_related_objects():
    if r.model == Rental:
        relation = r
        break

now in relation I have a RelatedObject (I cannot seem to find a lot of information about this on the net) instace, which has very useful fields like:

  • parent_model: the model that is being refered in this relation (in this case Car)
  • model: the Model (in this case Rental) that points to the parent model (in this case Car)
  • field: the field in model that references to parent_model

So now I know that if I want to build a custom SQL in Django and I really want to make it generic as possible, I can use this RelatedObject class along with other interesting model attributes like: Model._meta.db_table (getting the name that the model has on your database) and such to make my code 100% portable 😀 (or at least try to).

Just for the fun of it, this is the resulting code (not that it will make much sense to you… and sorry for the lousy formatting):

return target_model.objects.extra(select={"sum":"SELECT SUM(points) "
                                                "FROM %s "
                                                "WHERE date_part('year', reg_date) = %d "
                                                "AND %s.%s = %s.%s" % (ranking_model._meta.db_table, year, # year
                                                                       ranking_model._meta.db_table, # ranking table name
                                                                       relation.field.column, # column on ranking that points to user/team table
                                                                       target_model._meta.db_table, # user/team table
                                                                       target_model._meta.pk.column)}, # id column on user/team table
                                  order_by=['sum'])[:10]

Use with wisdom!

Now, if there is an easier way to do this, please tell me, I’m eager to learn!

Django powered AJAX Chat – Part 1

In the neverending adventure of developing our game website we have stumbled upon the “live chat” rock. I’ve seen some interesting things here using a pure django implementation for chat and there using django and comet for chat. I browsed the first one and I didn’t like it too much, and the comet implementation is clearly out of the picture for now. We have to finish this project in 4 to 6 weeks and we cannot take the risk of messing around with yet another technology (although I will keep this django+comet link bookmarked for future use).

So I decided to make one quick and dirty chat app myself. I had a very clear idea of how I wanted to implement the chat, it is really simple after all:

  1. Client loads HTML + Javascript
  2. Client queries the Server every X seconds for new messages
  3. Server responds messages serialized in JSON format
  4. Client attaches new messages to the message box

This does fail regarding performance, so I did a little bit of reading regarding the matter of creating html based chat applications. It turns out there’s this Comet model to allow persistent or long lived HTTP requests. The idea behind that is simple, for out chat application for instance, instead of quering the server every X seconds just make a request and wait… wait until the server prints SOMETHING into the request. This does save us of making many queries but falls short when implemeting over a normal web server, say Apache. This happens because for every open connection Apache assigns either a thread or a child process to the request. If you know a little about OSs or how the Apache HTTP server works, you would have figured out by now that this implementation can effectivelly occupy all of Apache’s threads discarding any other requests by other clients. FAIL!

To save us from that, diverse solutions exists. Long story short: You can use the Twisted Networking Engine along with Apache, having Apache serve the normal HTTP requests while all other requests (such as our nice chat application) requests are handled by Twisted, which is optimized for this scenarios. Sounds great, doesn’t it? Still, we won’t go this way in the making of this project, not now.

Instead, let’s just make a plain simple server-killer 100% Django chat application! We’ll be needing Django and jQuery.

Heed

I have already posted a follow-up for this post here: https://pythonhaven.wordpress.com/2009/07/13/django-powered-ajax-chat-–-part-2/. It covers the final implementation I used, explaining the application in-depth, and also points you to the google code site where you can get the source code wrapped up in a nice example project. The motivation behind this project is written down in the post you are reading now (part 1), but the implementation is covered in the linked post (part 2).

The models

The philosophy that I have for this application is:

  • Extreme low coupling with the rest of the apps
  • Internationalization capable

Bare with me as I struggle to respect those two aspects.

So first of all we create a chat room model. This will represent a virtual space where messages can be sent and where people can read other people’s messages, the model is as follows:

class Room(models.Model):
    '''Representation of a generic chat room'''
    belongs_to_type = models.CharField(max_length=100, blank=True, null=True)
    belongs_to_id = models.IntegerField(blank=True, null=True)
    created = models.DateTimeField()
    comment = models.TextField(blank=True, null=True)
    objects = RoomManager() # custom manager

    class Meta():
        unique_together = (('belongs_to_type', 'belongs_to_id'))

    def say(self, type, sender, message):
        '''Say something in le chat'''
        m = Message(self, type, sender, message)
        m.save()
        return m

    def messages(self, after=None):
        m = Message.objects.filter(room=self)
        if after:
            m = m.filter(pk__gt=after)
        return m

    def __unicode__(self):
        return 'Chat for %s %d' % (self.belongs_to_type, self.belongs_to_id)

As you have notices this model has two strange, very strange, fields: belongs_to_type and belongs_to_id. I took this off django’s admin login feature. Basically when creating a chat room, you can create a weak reference to ANYTHING you want. For instance let’s say you have project screens and you want each project to have a chatroom of it’s own, you just have to create a room with belongs_to_type = “project” and belongs_to_id = <the id of your project here>. This will then behave like a reverse ForeignKey, instead of having the project point to the chat room, the chat room will point to the project, but, the chat room can also point to… documents, profiles, groups, etc. The possibilities are endless.

Besides that I also defined a custom model manager in the line 06 , I added a couple of utility functions for the creation of rooms and the quick retrieval of rooms:

class RoomManager(models.Manager):
    '''Custom model manager for rooms, this is used for "table-level" operations'''
    def create(self, parent_type, parent_id):
        '''Creates a new chat room and registers it to the calling object'''
        # the first none is for the ID
        r = self.model(None, parent_type, parent_id, datetime.now())
        r.save()
        return r

    def get_room(self, parent_type, parent_id):
        '''Get a room through its parent.'''
        return self.get(belongs_to_type=parent_type, belongs_to_id=parent_id)

I also defined a say function to send a message to the chat room, and a messages function to retrieve the messages of the room.

On to the message, this is the model:

class Message(models.Model):
    '''A message that belongs to a chat room'''
    room = models.ForeignKey(Room)
    type = models.CharField(max_length=1, choices=MESSAGE_TYPE_CHOICES)
    sender = models.CharField(max_length=50, blank=True)
    message = models.CharField(max_length=255)
    timestamp = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        '''Each message type has a special representation, return that representation.
        This will also be translator AKA i18l friendly.'''
        if self.type in ['s','m','n']:
            return u'*** %s' % self.message
        elif self.type == 'j':
            return '*** %s has joined...' % self.sender
        elif self.type == 'l':
            return '*** %s has left...' % self.sender
        elif self.type == 'a':
            return '*** %s %s' % (self.sender, self.message)
        return ''

Simple huh? The message points to the room to which it belongs and then contains a variety of vital information. I was specially interested in the type field, I wanted to have several message types with different representations on the client side, these are the first ones that I thought of:

MESSAGE_TYPE_CHOICES = (
    ('s','system'),
    ('a','action'),
    ('m', 'message'),
    ('j','join'),
    ('l','leave'),
    ('n','notification')
)

So, those are the message types I was interested in having and as you can see the Message object has different representations depending on the message type and they are Internationalization friendly, great!. I am still not sure though on whether the sender field should be a simple CharField or a ForeingKey to django.contrib.auth.models.Users…

Well, that’s all for now, next time we’ll be seeing some jQuery in action to get this thing rolling.

Until next time!

Part 2

Remember, part 2 is here: https://pythonhaven.wordpress.com/2009/07/13/django-powered-ajax-chat-–-part-2/

Fixtures, default data for your models

So, we started working on our project (finally) and although some of the requirements of the project will change in the course of the following days, we are working on the more general aspects of the site. Among these general “apps” is the help app. Basically it has a couple of static pages and then a hierarchy of categories (folders) and topics presented in a similar way to Google’s Gmail Help Page.

This help app is a variation of the usual Polls or Books examples you get in Django’s Writing you first Django App Tutorial, and is a very good exercise to get up to speed with the development in Django. As the app started to take shape we began to wonder if Django provided a mechanism to load default data into the models…

Fortunately it does! You can read all about loading initial data into models it in the docs, but if you want to continue reading about our experiences and choices, read on!

In a nutshell: You can choose between two methods, fixtures or SQL. Fixtures are database agnostic representations of database rows in either XML, JSON, YAML, or Python objects notation. While SQL, you know, consists of SQL insert statements to populate your data back. Both of these can be stored in files and can later be loaded with the magical manage.py script. Obviously we chose fixtures because of their loose coupling with database layer and using XML for maximum compatibility!

So, to generate a fixture containing the current data in the database of the models of an app you only need to run the following manage.py command in the root folder of your Django project:

python manage.py dumpdata <app_name> --format <format>

Where app name is the name of the Django app whose models and format can either be xml, json or yaml. Is you run that command you’ll get the resulting fixture onscreen, that is of no use for us, we need to save it on a file! No worries, you just need to use the output redirection symbols of your console to push that data to a file. So lets say you want to create a fixture for your catalog app…

Do this under GNU/Linux (I assume it is the same under MacOS):

python manage.py dumpdata catalog --format xml > catalog/fixtures/catalog_data.xml

Do this under Windows:

python manage.py dumpdata catalog --format xml > catalog\fixtures\catalog_data.xml

You may have noticed I redirected the output to a fixtures directory inside the app, this is because the Django documentation recommends that for data dumping (using this path is crucial for the last step in this post!)

So, how do you load the data? Simple, use mange.py loaddata!

manage.py loaddata catalog/fixtures/catalog_data.xml

And you are done.

But the whole idea of using fixtures was to provide a simple way to transport data across the team of developers (we are going to be working with SVN, remember?). So reading the doc a little bit more we find that these fixtures can be automatically loaded when performing a syncdb! You only need to save the fixture file with a special name:  initial_data.xml (xml in our case, for you it can be xml, json or yaml) in your app’s fixtures directory.

Now every time you perform a syncdb, the models will be created on the database and the data from the fixture will be automagically loaded, isn’t that great? It is!

Hope this helps!

Until next time.