Comments on: Django snippet — Template tag: split list to n sublists

By: Kjell Magne Fauske

Kjell Magne Fauske — Mon, 06 Oct 2008 08:41:57 +0000

In an earlier comment I said that the change reduced the number queries from over a hundred to eight. That was not completely true. I did a few denomalization tricks that also reduced the number of queries a lot.

The reason that you don’t see any changes may be that you are already passing an evaluated queryset to the templatetag. If you only use two columns the extra overhead will not be that large either. In my case I split it into four columns.

The debug toolbar is indeed useful. It can be a bit slow for complex pages, but it is a great tool for spotting bottlenecks.

By: Herself

Herself — Fri, 03 Oct 2008 19:04:21 +0000

Oh wait, I take that back. The debug toolbar does sort of have one of the things I was looking for; it was under templates, which I hadn’t looked at yet. (The request context info toggle is what I’m looking at, though it still doesn’t give me exactly what I want, but it’s a start. *g*)

By: Herself

Herself — Fri, 03 Oct 2008 16:52:14 +0000

Thanks for the pointer to Rob’s debugging toolbar! (Very nice, though it still doesn’t have the one thing I really need, of course. *g*)

However, I’m not seeing any change in the number of queries performed (one for each try) on the pages I’m using the template tag as written versus the change you’ve suggested. (Though it makes more sense to me know why you’re suggesting that now, thanks. *g*)

Also, I’ve changed the variables used so list isn’t a variable name. Didn’t think about that when I started working on this. *g*

By: Kjell Magne Fauske

Kjell Magne Fauske — Thu, 02 Oct 2008 13:09:56 +0000

The reason for the high number of queries is due to the fact that Querysets are lazy. They are only evaluated when needed. So when you slice your list with code like len(list[i::cols]) you hit the database multiple times. For instance, determining the length of the queryset requires one query. The results are then thrown away and a new query is necessary for the next iteration. Likewise, extracting a slice requires a new query.

With the code “tmplist=tuple(list)” all items in the list are retrieved from the database with a single query and stored in a list. So writing len(tmplist) now does not require a database query. Neither do slicing. The downside is that all items are stored in memory, but that is usually not a big problem.

I hope that the above explanation makes things clearer. If you are interested in looking at the SQL-queries I recommend trying the “Django Debug Toolbar” (http://rob.cogit8.org/blog/2008/Sep/19/introducing-django-debug-toolbar/) That’s how I found out about the “problem” with your template tag. A page that used your tag required more than hundred queries. After I did the tuple/list trick the number of queries was reduced to eight.

PS. I used tuple instead of list to avoid confusion with your variable named list.

By: Herself

Herself — Thu, 02 Oct 2008 06:32:56 +0000

Glad you found it useful, Kjell. :) I’m trying really hard to document things that took me a while to figure out or find answers to while working on my current projects, ‘cuz goodness knows I’d be lost sometimes without other kind souls who have done the same thing. *vbg*

Now on to your suggested change (and this is my lack of knowledge of python showing, as I’m still learning it), how does adding the “tmplist=tuple(list)” bit help reduce the number of queries?

(I must admit to a great deal of frustration with how the queries and results from django are structured, as they just don’t make sense to me yet. Largely because I’m not really comfortable with the ins and outs of OOP, I think. I only just discovered the connection.queries trick to see the raw sql last week, and am still looking for a way that will display the results of the queries and their object names, because that would help me immensely in my debugging. Ah, the joys of a new language/framework. *g*)

By: Kjell Magne Fauske

Kjell Magne Fauske — Tue, 30 Sep 2008 06:34:20 +0000

Thanks for sharing. The templatetag is really useful.

The templatetag has some performance issues when splitting querysets though. A large number of SQL queries are executed. To reduce the number of queries you could evaluate the queryset prior to splitting. I posted one approach on the django-snippets page.

Thanks again for sharing.