Aggregated blog posts about Django, updated every hour

September 03, 2010

Python Decorators for Djangonauts

From Stuart Marsh on September 03, 2010 09:05 PM

If you’ve used Django for any amount of time then you’ve probably come across decorators.  They’re the things with the @ signs over the top of your view.

The first one you’re likely to see is the @login_required decorator, which requires a user to be logged in before the view will run.  If they are not, django will redirect to the login page.

If you’ve ever wondered what they do and how they work, I’m going to show you how to create your own.

What is a decorator?

A decorator is a function that wraps your decorated function (or view, as they’re functions too) in yet another function, allowing you to run code both before and after the main function.

What?

A simple example

Allow me to explain.  Let’s say we have a view, but we only want it to render if the HTTP_REFERER is www.example.com.  If it’s not, then we want it to redirect to the home page.

So we have our view:

def my_view(request):
    ....
    return render_to_response('myview.html')

We’re going to call the decorator refer_check. The decorator looks like this.

from urlparse import urlparse
from django.http import HttpResponseRedirect

def refer_check(func):
    def decorate(request):
        referer = request.META.get('HTTP_REFERER', '')

        if referer == 'http://www.example.com':
            return func(request)
        else:
            return HttpResponseRedirect('/')
    return decorate

So how does this work? Well, our view (my_view) gets passed to the refer_check function – the func parameter is the view function.

We then create a new function, called decorate, which will wrap around our view function. That’s why it takes the request parameter, just like the view.

Now we get the HTTP_REFERER and check it. If it’s what we’re looking for, we run the view – the line where it says return func(request). If it’s the wrong referer value, we return and http redirect to the root of the domain.

Once all that has finished, we return the decorate function from the refer_check function.

To finish, we put the decorator above the view.

@refer_check
def my_view(request):
    ....
    return render_to_response('myview.html')

Another example

In the above example, we did something before running the view. But you can also do things after the view has run.

Here’s a decorator that lets you time in seconds how long your function takes to run.

from datetime import datetime

def time_it(func):
    def decorator():
        a = datetime.now()
        func()
        b = datetime.now()
        print 'function: %s took %d seconds to run' % (func.__name__, (b-a).seconds)
    return decorator

@time_it
def myfunc():
    for i in range(1,100000):
        print i

We have a function that prints the numbers 1 to 100,000. This is passed into our decorator which notes the time before the function is run. We then run the function using func(), and we note the time again. Then we print out a message with the function name and the number of seconds it took to complete.

Conclusion

As you can see from these examples, the decorators aren’t absolutely necessary. You could code both these examples by putting the code in the view/function itself. But it’s very useful if you like reusing your code, as you can apply your decorator to any function you like.


Caktus Consulting Group Seeks Two Python/Django Web Developers

From Caktus Consulting Group on September 03, 2010 04:13 PM

I’m delighted to announce that Caktus is looking for two Python and/or Django web developers to join our team on a contract or part-time basis, with the potential for full-time work in the future.

Caktus builds custom web applications for local and remote clients using a variety of open-source technologies. We are a small team based in the Chapel Hill/Carrboro area of North Carolina (currently residing in Carrboro Creative Coworking). We believe in face-to-face contact, both with clients and amongst ourselves, and employ agile development techniques that emphasize teamwork and collaboration. We encourage you to meet the team and learn more about what we do.

We’re looking for two experienced Python and/or Django web developers who enjoy working on a team and are excited to work on new projects. We have a preference for local candidates, but will consider all submissions. Your work will involve creating and integrating Django apps, working on existing Django projects, deployment, and database work.

You will be working in Linux (Debian-flavor) production environments with Apache and WSGI. Python and relational database experience is required. Django experience is a (big) plus. HTML/CSS and JavaScript experience are also a must, and jQuery is a plus.

If you’re interested in one of these positions, please send us your resume, some sample Python code that you wrote, and links to any open-source projects you’ve contributed to. We’re looking forward to meeting you!

September 02, 2010

5 Command Line Tools for Shell Dwellers

From Stuart Marsh on September 02, 2010 09:25 PM

If you’re anything like me, then you’ll spend a large part of your time working at the command line. I often find it easier to use some things in the shell then as a GUI. Take Subversion for example. While the rest of my team use TortoiseSVN, I’m constantly trying to evangelise the command line version, as I think it makes more sense.

Perhaps that’s just me….or maybe you too.

Tools you can Use

Anyway, here’s some nice command line tools that do useful things.  Enjoy.

1. Todo List

Todo.txt is a command line to do list tool.  It runs on MacOSX, Linux and Cygwin for Windows.  You can quickly and easily add items to the list, assign each task to a project, tag a task, prioritise and de-prioritise.  Prioritised tasks are color coded.  You can also search for keywords, projects or tags.

There’s a screencast you can check out on their site.

2. Twitter

Twyt is a python api interface for Twitter, but it also comes with a command line tool, so you can post tweets, follow people and all the rest straight from the command line.  You’ll need python installed to run it, but you can get python for all platforms so it’s not a problem.

3. Image Magick

Image Magick is a tool for converting and editing images.  It does a huge array of stuff, including format conversion, resizing and special effects.  It’s available in Mac, Linux and Windows.

4. Facebook

FBCMD is a command line interface for Facebook.  It has a large list of commands, and you can do pretty much what ever you can do on Facebook in the CLI.

5. Google

Google CL is the daddy of the command line tools, allowing you access to Blogger, Calendar, Contacts, Docs, Picasa and YouTube.  Upload all your photos to Picasa at once, post videos to YouTube in one line, export your contacts, it can do it all.  This is another tool you’ll need to install python for.

Given the amount of data we give over to Google, this one has to be the most useful.

Hope you found some of these useful.  At least if we go back to the dark ages and lose our GUI interfaces, we’ll still be able post inane babble to twitter…


September 01, 2010

Google Webmaster Tools

From Mark van Lent on September 01, 2010 07:49 PM

Google's Webmaster Tools provide the modern webmaster/developer with some nice tools to improve a website and the way the site is indexed. In this article I'll focus on the crawler related tools. Specifically, how they helped me when I migrated from Plone to Django.

Crawler access

First up, crawler access (in the Site configuration menu). This item offers three tools: you can generate a robots.txt file, test a robots.txt file and request to remove a URL from the Google search results.

For those unfamiliar with a robots.txt file: it is a way to give instructions to a robot, like the ones used by search engines to index your website. With the tools in the crawler access section you can easily create such a file to e.g. restrict access to your internal search pages. Besides creating the robots.txt file, you can also test it by specifying a URL and the tool will tell you whether it is allowed to access the page or not. (Note: this is not a security mechanism. Robots can ignore the instructions.)

In my case I only used the testing tool since I already had a robots.txt file and just needed to adjust it.

Crawl errors

The next thing I'd like to discuss are the crawl errors (in the Diagnostics menu). Among other things, it shows you which pages Google expected to be on your site but could not find. This was useful since there were a number of pages that were not available anymore after the migration. (Especially pages that I forgot were accessible.) For instance:

  • /author/markvl which is now just /about
  • /front-page (the default page of the Plone site root) is now just /

Sure, I could have found those myself (and I probably should have), but it's very easy to miss URLs that are referenced somewhere. This tool efficiently lists those missing pages, so I could add some rewrite rules in my web server configuration to solve the problems. (Note that the results can change, so you might want to check them regularly.)

By the way, it's also a nice way to catch mistakes in the content: in once case I had a typo in the link from one blog entry to another. It popped up in the crawl errors and I was able to correct this.

Crawl stats

The crawl stats (also in the Diagnostics menu) display the Googlebot activity of the last 90 days. While there's only that much you can do to control the way Google crawls your site, it was funny to see how some changes made an impact.

Crawl stats (composition of two snapshots I made of the graphs)

The first change, which is visible in all three graphs, is the migration to Django on May 31. Google clearly started crawling more pages since the end of May. As a logical result more bandwidth is consumed. However, less time is spent downloading a page.

The second change is most clear in the middle graph (kilobytes downloaded per day). I moved to a different server and enabled gzip compression on June 25. The move was planned but I might have forgotten about the compression if I hadn't seen this graph.

Even more…

This article focussed only on a few of the available tools. There's much more to discover, like an overview of the queries that returned pages from your site or the ability to upload a sitemap (and being able to view how many of the pages in the sitemap have been indexed).

I really think that using the webmaster tools is a valuable addition to a webmaster's toolkit.

Justin.tv Android app – first broadcast test

From Stuart Marsh on September 01, 2010 07:10 PM

I’ve just installed the new Justin.tv Android app on my HTC Hero with Android 2.1 running.  It allows you to broadcast live streaming video from your phone, either over Wifi or 3G.

For this first test I used Wifi, so I can’t vouch for the speed or quality of a 3G connection.

First Impressions

The interface is pretty slick and easy to use.  It gives 3 buttons on the right hand side.  The top one is Chat, the bottom is Share, and the middle one is a big red button to start your broadcast.

Pressing the menu button brings up the account settings where you can either login to an existing account, or create a new one from within the app.

Results

The first few broadcasts I did, I found that it cut off the last 10 seconds of the recording.  I’m assuming this is to do with the lag, and that when you press the stop button on the app, it also stops sending data.  Which means if there is a 10 second lag, it will not send the last 10 seconds of your feed.

The audio was about half a second out of sync with the video, and the video quality was OK.  But overall it was a good experience, and considering I’m broadcasting very quickly from a mobile device, it’s still quite impressive.

The Video

This is what I recorded in the end.

Watch live video from Beardy Geek TV on Justin.tv


I’ve just installed the new Justin.tv Android app on my HTC Hero with Android 2.1 running.  It allows you to broadcast live streaming video from your phone, either over Wifi or 3G. For this first test I used Wifi, so I can’t vouch for the speed or quality of a 3G connection. First Impressions The [...]

Person X is...

From Andy McKay on September 01, 2010 11:09 AM

Many years ago when I was at ActiveState we had a new manager, I got one of those every month or so it wasn't a big deal. One of the things he said was - he would never use the excuse that a decision will be made because person X said so and they know best. It was 9 years ago, so I can't remember the exact wording.

What he was saying is that all decisions can be questioned, things need to be explained and we shouldn't be just blindly following some people. He was also saying that this is bit of elitism, putting other people down. This came back to me a few times as a manager when I made decisions and was questioned. But it really hit me home in slightly different circumstances on the Django mailing lists - and I will point out, this is a rare exception.

I've seen at a couple of times people say something like: "our code was written by X, so i doubt there's a bug there", "this was designed by X who has quite a reputation...".

Each time, it's felt a bit like the "person X knows best" card has been pulled out and people can't argue with it and it grates a little. As the questioner it would put me down.

There are many people on the Django mailing list whom by following their comments and decisions I've come to respect a lot (Russell, Jakob, Karen, Graham etc). The problem is here that people come into the situation phrasing their issue incorrectly and it comes across dis-respectfully. They haven't realized how smart and (generally) right these people are.

They assume that something is broken and how can we all be so stupid for not seeing it or fixing it. It can be tough not to pull out the card in this case and say "X is right because they are the god of Y".

If you are about to write that in the mailing list, maybe there's another way of phrasing it that will keep Django one of the friendliest and nicest communities out there.

August 31, 2010

JMeter testing from the command line

From Stuart Marsh on August 31, 2010 09:56 PM

I recently did a video post about website stress testing using JMeter.  Moving on from this, I’m going to show you how you can run JMeter tests from the command line, without using the GUI.

Why use the command line?

The main reason to use the command line is the most obvious: you don’t have a gui available.  If you want to run stress tests from a server, you’ll need to do it from the shell.

So why would you want to run the tests from a server?  Well one issue I discovered while running some tests on a media server recently, is that testing from your home machine means at some point during the test, you’re going to run out of bandwidth.

Of course you can just let JMeter loose with lots of threads and not record the responses, but it’s useful to know the response results of your pages or media, which means a download of the page or media you’re testing.

Using JMeter without a GUI

The JMeter binary allows you to run in GUI-less mode, using the -n option.

  1. Create your test in the JMeter UI as usual.
  2. Upload the JMX file to the server.  This contains all the configurations for your test that you just setup.
  3. Run the jmeter command:
    jmeter -n -t mytest.jmx -l log.jtl
  4. The test will run, and save the output in the log.jtl file.
  5. Download this file, and open it inside the JMeter UI.

If you need to change elements of the test, you can edit the JMX file directly.  It’s stored in an xml type format, so it’s fairly easy to see what you need to change.


Generic relationships in Django

From Marcin Mierzejewski on August 31, 2010 07:19 PM

"Last week I’ve completed an interesting task within our project: building internal link-shortening system based on persisted objects, not on constant URLs. The main requirement was to get as loosely-coupled design as possible. In the perfect world the “shortenable” classes should know absolutely nothing about link-shortening mechanism. How to get such a flexibility in Django? I’ve used the ContentTypes framework with generic relations and it works pretty well!" by Marcin Swierczynski

Another nashvegas Update

From Patrick Altman on August 31, 2010 03:59 AM

So something that has been bugging me since the recent release of nashvegas was the lack of handling errors in migration scripts.

There were several issues impeding me being able to handle errant scripts gracefully, by which I mean reporting the error via a sys.exit() call.

I was sub-shelling to manage.py dbshell where I would PIPE the contents of the script to stdin. Turns out that dbshell executes the database client process via os.execvp which as the Python document states:

These functions all execute a new program, replacing the current process; they do not return.

So, obviously, I was not going to get a return code from the underlying call, that I might check and take evasive action.

Because I was getting to the database client indirectly through dbshell, I was unable to control things like setting flags or variables to control the behavior of the database client. So, even if I were to somehow get around the first problem of os.execvp being used, I still would not be able to set the necessary flags needed to get the client to behave how I wanted it to.

Therefore, I decided that I'd just copy the argument building from each of the django.db.backends.*.client modules and execute the client commands directly. This would also give me the ability to fine tune the command line parameters as I needed.

I decided to focus on postgresql because it's my database of choice these days. I learned tonight that if I pass the command line parameter --set ON_ERROR_STOP=TRUE, that the execution of sql commands stops on error and exits the client with a return code of 3.

So, now, with nashvegas 0.2a1.dev3 and your database set to postgresql, you can rely on errors in migration scripts to abort, given you the ability to fix without recording the migration in the Migration model.

I haven't attempted to figure out how all the other clients work, though they are supported with the same options as found in django.db.backends. So if by default they exit with error status codes then they'll work like mentioned above. However, if they are require additional command line arguments like postgresql did to exhibit similar behavior, then I ask that someone submit a patch and I'll gladly incorporate it.

August 30, 2010

Organising your Django code

From Stuart Marsh on August 30, 2010 07:47 PM

There are several options when it comes to organising your code in a Django project.  There isn’t a ‘correct’ way, and it will depend on the size of the project, if it is a team development, or purely the preference of the developer.

Templates

Django gives you the ability to put your templates anywhere, either in single or multiple locations.  You can add a path to your TEMPLATE_DIRS setting, and Django will look at each path in turn to try and find your template to render.

Option 1

Have a single template directory, which contains all the templates for all your project apps.  The benefit here is that everything is in one place.  If your project is split up into a lot of apps, and there is crossover between apps, you may struggle to find where you put a template further down the line.

Another advantage is if you have a dedicated content team who just work on templates, and you either don’t want to bother them with python code, or you just don’t want them to go near it, you can keep it separate.  I’ve used this method, and the templates were kept in a separate source repository to the code.

The disadvantage is that if you don’t have a good system for arranging the directories within your templates directory, you’ll still struggle to find things.  So have a good strategy for file naming conventions if you go this route.

Option 2

Have a template directory in each of your installed apps.  If your apps are self contained, this is a good option, as it will be easy to organise.  Email templates in your email app, user profile templates in your user profile app.

The only problems you get here is when you have an app whose functionality spans more than one app, or more than one app accesses the same template.  You then have to decided where best to put it.

Media

Media really has to be in a single location so that it can be served by whichever webserver you choose.  But you still have the option of having the media stored somewhere within the project, or external to the project.  Again, when working within a team that has graphic designers, it can be beneficial to keep media outside the project.  It also makes sense if you want to re-use the media in another project.

Template Tags

Your template tags need to be inside a directory named templatetags, which has to reside inside an app.  If you have multiple apps, with multiple template tag directories, it can be an issue finding the correct app where the template tag lives.

If you want to put all your template tags in one place, and use them across your project, you can create an app specifically for your template tags.  Create your app, place a templatetags directory in it, with an __init__.py file, and remember to add the app to the INSTALLED_APPS setting.

Do you have any tips when it comes to organising your code?  Anything here you agree or don’t agree with? Have your say and leave a comment below.


How I solved the comment spam for my Django site

From Mark van Lent on August 30, 2010 06:33 PM

After this website migrated from Plone to Django, the comment spammers found my site more interesting. Instead of five spam comments a year, I suddenly got the same amount per week. Although those comments were never published (more on that later), it did annoy me. By no longer displaying the comment form below the blog entries, the problem of the spam seems to be solved. While this wasn't my goal, it is a nice side effect.

It appears that the initial comment system (I'm using Django's comment framework) was more interesting for comment spammers than the implementation of comments on my old Plone site was. I think this is because there is a URL field on the form now. And if the URL is present, the name of the commenter becomes a link to that URL.

Since I saw that coming, I am using a similar setup as in Practical Django Projects: for blog entries older than 60 days I'm assuming comments are most likely spam and they need to be moderated before they become visible. For the other comments, I run them past Akismet. This seems to work fine for now. (However, since I want to prevent false positives to go unnoticed for too long, an email is sent for every comment posted on the site. That is why comment spam is still a little annoying.)

Initially my comment form was displayed at the bottom of every blog entry. I didn't like this, so I decided to only display a “post comment” link. If you've got JavaScript enabled, the form is inserted in the page dynamically after clicking on the link. Without JavaScript the link acts as an ordinary link and the page is reloaded with the form appended to the end. (I'm using an ordinary GET parameter so in my view function I can detect whether the comment form should be shown or not.)

Apparently this has the nice side effect that the comment spammers cannot find the comment form anymore. That is, I haven't received any spam since this change. I'm curious how long this will last, but for now I'm happy.

Disclaimer: I don't want to claim this is a guaranteed way of preventing comment spam. I'm just reporting my observations here…

A different kind of URL shortener

From Paul Bissex on August 30, 2010 12:08 AM

Today I'm launching my first Google App Engine site. While I built it largely to play with GAE, it is also useful in its own right (I like to think so anyway). It does two different things:

Link shortening without redirection. Put in a godawful long Amazon link and get back a shorter Amazon link. Works with eBay and a few others too. I welcome recipes for other sites. (For the programmers in the audience, which is most of you -- yes, the processing is via regular expressions.)

It does some basic checks to confirm that the shortened URL returns the same page as the original one.

Link expansion. Put in a link from a URL shortening/redirection service, e.g. bit.ly, and see where it redirects to. Works with a slew of popular link-shorteners, including the house brands goo.gl and nyti.ms.

Some of the shortening services do offer a way to see the link target before you visit it, but they're all different; this presents a simple unified interface to that feature.

There's a bookmarklet too. If you have someone in your online life who frequently bombards you with, say, mile-long eBay links, tell them about it.

http://urlworks.appspot.com/

August 29, 2010

Trabajar con slugs en URLs

From Django en Español on August 29, 2010 10:16 AM

Una de las cosas que los buscadores tienen en cuenta para establecer la relevancia de sus resultados son las palabras que aparecen en las URLs. Por eso conviene que las URLs de nuestros objetos incluyan un slug que los represente.

Django Template Tag Namespaces Now Possible

From Cody Soyland on August 29, 2010 08:20 AM

I've been interested in the Django template language for some time now, and I've admired much of its simplicity and extendibility. I even wrote a shell for it (screencast) and a two-phase template renderer. Having spent the time to understand how it works, I've also had my share of ideas on how to improve it (addition of "elif", mathematical operations in variable tags, namespaces). The pony that I've been wanting the most is probably namespaces. There has been talk of adding namespaces to Django templates for quite a while (including a ticket with patches and some various discussions on the mailing list (1, 2 and 3)). For years, this concept has sat dormant due to lack of discussion and interest. No pluggable solution had been offered (as far as I know), so I wrote a couple of templatetags that offer namespacing and other features while retaining backwards compatibility and not requiring a fork of Django. This code is available on Github as django-smart-load-tag.

Backwards compatibility

Django's policy is to remain backwards compatible, and the template language is certainly no exception. In order to give the "{% load %}" tag namespacing features, it needed to be extended in a way that allows current assumptions about its behavior to remain the same. In particular, the assumption that all tags will be loaded into the global namespace by default had to stay. This means that, given a template library named "lib1" containing "tag1" and "tag2", the following code must work:

{% load lib1 %}
{% tag1 %}
{% tag2 %}

Current proposals have suggested the backwards-incompatible syntax that assumes namespaces are on by default:

{% load lib1 %}
{% lib1.tag1 %}
{% lib1.tag2 %}

In my implementation, "load" works the same (as in the top example), but has a few keywords that control its behavior. For example, to load a library into a namespace, use "into":

{% load lib1 into lib1 %}
{% lib1.tag1 %}
{% lib1.tag2 %}

Other features

To load a specific tag (optionally renaming it with the "as" keyword):

{% load lib1.tag1 as my_tag %}
{% my_tag %}

Loading from a specific app can be done using the "from" keyword:

{% load lib1 from app1 into app1 %}
{% load lib1 from app2 into app2 %}
{% app1.tag1 %}
{% app2.tag1 %}

To make everybody happy

It has been suggested to write a separate "{% import %}" tag in order to enable namespaces by default while retaining backwards-compatibility with existing Django applications. I've also experimented with the following import syntax, and it's also included in django-smart-load-tag:

{% import lib1 %}
{% lib1.tag1 %}

Its namespace-on-by-default design can be subverted using "* from":

{% import * from lib1 %}
{% tag1 %}

The "as" and "from" keywords are also implemented:

{% import lib1 as my_lib %}
{% my_lib.tag1 %}

{% import lib1 from app1 %}
{% lib1.tag1 %}

Where to go from here

If template tag namespaces are to be accepted as a core part of Django, some discussion will need to take place on what is the most correct solution for moving forward. Your comments here or on the mailing list can make a difference, and with enough contribution from the community, perhaps all my ponies will one day run free.

(Source and documentation available here.)

Screencast - django-template-repl

From Cody Soyland on August 29, 2010 08:20 AM

Django-template-repl is a unique project aimed at providing debugging tools for the Django template language. I did a screencast to highlight its features.

In this video, I describe how to use Django-template-repl's management shell, template tag, and context-capturing features.

Please grab the source or simply run:

pip install django-template-repl

Thread-Safe Object-Oriented Views in Django

From Cody Soyland on August 29, 2010 08:20 AM

An emerging design pattern in Django is the usage of class-based views. Writing views as classes is made possible by Python features that allow classes and objects to behave as callables, just like functions. They can help organize view code and promote reusability by offering a greater level of customization. However, callable objects sometimes have thread safety issues that developers are often not aware of.

The most common approach to class-based views is to create a callable object, a class that has been written with a __call__ method, therefore making instances of the class callable. The view is instantiated either as a module-level variable in the views file or in the urlconf. Jacob Kaplan-Moss has written a series of class-based generic views that follow this model.

The Problem

When writing persistent class-based views, you must be careful not to introduce stateful information in your object. From what I can tell, Jacob's implementation seems thread-safe, because the view's state appears to only be altered on initialization. The problem arises when you store request-specific variables on the view object. Because the object is only instantiated once per Python process, it persists while running multiple HTTP requests, for the life of the process. Stateful information can cause side effects including security problems.

Here is a simple example illustrating this effect:

class MyView(object):
    thing = 0
    def __call__(self, request):
        self.thing += 1
        return HttpResponse('%s' % (self.thing,))

my_view = MyView()

URL patterns:

urlpatterns = patterns('',
    url(r'^my_view/$', 'my_app.views.my_view', name='my_view'),
)

Every time you refresh your browser, you will see a number incrementing. Even in single-threaded (prefork) environments, this bug is present. Multi-threaded environments are succeptible to even scarier problems with object state. For example, you might be tempted to set the request object as an attribute to the view class:

class MyView(object):
    def __call__(self, request):
        self.request = request
        return self.create_response()

    def create_response(self):
        return HttpResponse('Welcome, %s' % (self.request.user,))

my_view = MyView()

If a view is called twice at roughly the same time in two threads, the request object may be incorrect in one of the threads, because the view object is shared between the threads. This bug might manifest seldomly, but it would be difficult to track down and conceivably a security problem.

Reducing shared state

If you are having thread safety issues because of class-based views, the first thing you can do is enable a prefork worker module in your web server. This causes greater memory usage, but it creates new processes instead of threads to handle concurrency. This is only a workaround, however, and there are ways to maintain thread-safe code using class-based views. A quick fix for a code base that already suffers from this bug would be to drop in __new__ method that handles the creation of new view instances per request. This should be as simple as adding the following to your view class:

def __new__(cls, *args_static, **kwargs_static):
    def view_wrapper(request, *args, **kwargs):
        view = object.__new__(cls)
        view.__init__(*args_static, **kwargs_static)
        return view(request, *args, **kwargs)
    return view_wrapper

If this is added to MyView above, it magically becomes thread-safe because the view is wrapped in such a way that every time it gets called, a new MyView instance is created for the request.

This method may be useful for existing apps, but a better designed stateful class-based view might drop the usage of __call__ altogether in favor of using a class (not an instance) as the view itself.

Michael Malone suggested in this Django ticket using __init__ as an alternative to __call__. By subclassing HttpResponse, you can treat the class as a view because calling the class creates an HttpResponse object. This creates a new view object for every request coming in. His suggestion did not appeal to me because you lose control over the resulting HttpResponse object and it's not possible to instantiate the view without creating an HttpResponse object. An alternative I've come up with is to override __new__ to return an HttpResponse.

Creating the response in __new__

Remember that any callable can be used as a view. This can mean functions, callable objects, and even classes. When you call a class, it's __new__ method is called and returns something, usually an instance of that class. However, it can be anything you want, including an HttpResponse object. A new approach to thread-safe classed-based views might be something like this:

class BaseView(object):
    def __new__(cls, request, *args, **kwargs):
        view = cls.new(request, *args, **kwargs)
        return view.create_response()

    @classmethod
    def new(cls, *args, **kwargs):
        obj = object.__new__(cls)
        obj.__init__(*args, **kwargs)
        return obj

    def __init__(self, request, *args, **kwargs):
        raise NotImplementedError()

    def create_response(self):
        raise NotImplementedError()

class MyView(BaseView):
    template = 'path/to/template.html'

    def __init__(self, request):
        self.request = request

    def create_response(self):
        return render_response(self.request, self.template, {'title': 'Home'})

This would be added to your URLPatterns directly (no need to instantiate it):

urlpatterns = patterns('',
    url(r'^my_view/$', 'my_app.views.MyView', name='my_view'),
)

A key difference is that the view is not a subclass of HttpResponse, but when you attempt to instantiate it, it will create an HttpResponse. In case you want to create a view object for testing or otherwise, the BaseView class has a factory method called "new" that makes new view instances.

The main thing to get out of all of this is that you need to be careful and aware of shared state whenever it exists. I prefer to minimize the possibility of side effects by avoiding module-level variables, singletons, and globals as they are almost always the road to hell. But if you do have a persistent view object, make sure that you don't store request-specific data on it.

Evaluating Django Caching Options

From Cody Soyland on August 29, 2010 08:20 AM

Caching is one of the first things you can do when you need to start thinking about scaling. Among efforts such as query minimization, denormalization, code optimizations, compression, database tuning, indexing, and load balancing, caching remains one of the lowest hanging fruits in methods to lighten your server load and handle huge amounts of traffic. There are many options, and I chose to evaluate a few of the most interesting setups.

This is not intended to be a rigoriously scientific test, but more of a first impression of the different caching systems. For all the tests I'm describing, I'm using a single VPS on Rackspace Cloud with 320MB of RAM, a quad-core AMD Opteron 2350HE, and a bleeding edge server stack using Ubuntu Server 9.10, NGINX with UWSGI, Python 2.6, Django 1.1, and PostgreSQL 8.4. I'm serving the home page view of Django-Mingus, which provides a realistic amount of complexity to the Python side of things and gives us a 9387 byte response. I'm using 4 UWSGI processes and a single NGINX worker. All my tests are using ApacheBench, which I'm running on the same machine. Note that for all my cache tests I'm prepopulating the cache before running the benchmark. Here are the different setups I'm going to evaluate:

  1. No caching whatsover.
  2. Django's template caching templatetag.
  3. Django's two-part caching middleware.
  4. NGINX Memcached module.
  5. On-disk caching with Django-staticgenerator.
  6. Varnish as front-end load-balancing cache.

No Caching

For any content-driven website, this is probably the worst idea of them all, and as you'll find out, it is trivial to implement most of the above caching strategies. Clearly, my single server arrangement is not going to be representative of your large app server cluster, so I urge you to evaluate all the options if you are anticipating scaling. Finding the right recipe for your server setup is going to be the fun part.

For the purpose of establishing a baseline, I ran ApacheBench on my setup with no caching turned on. I'm running 10 concurrent requests for a 1000 requests using the following ApacheBench command:

ab -n 1000 -c 10 <server-name>

Here's a snipped version of the results:

Concurrency Level:      10
Time taken for tests:   68.619 seconds
Sent requests:          1000
Completed requests:     1000
Failed requests:        0
Total transferred:      9660000 bytes
HTML transferred:       9387000 bytes
Requests per second:    14.5732231597662
Transfer rate:          141.610400362873 kb/s received

Connnection Times (ms)
              min     avg   max
Connect:        0    0.12    10
Response:     309  681.65  1330

It's probably possible to tune this for shorter latency, but we got the main number we were looking for; we can push 14.57 requests/second without a cache. Not bad, until you get Slashdotted!

Django's template caching templatetag

Django provides an easy way to cache parts of your template using the "cache" template tag. Here is an example of usage:

{% load cache %}
{% cache 500 sidebar %}
    This goes into cache.
{% endcache %}

Django-Mingus makes good use of the cache template tag in the default templates. In this test, I enabled Memcache in Django and removed view caching so I could get an idea how segment caching affects performance. This page benefits from 10 template cache hits and 4 other Memcache hits used in some of Mingus's apps.

Concurrency Level:      10
Time taken for tests:   26.19 seconds
Sent requests:          1000
Completed requests:     1000
Failed requests:        0
Total transferred:      9479000 bytes
HTML transferred:       9387000 bytes
Requests per second:    38.1825124093165
Transfer rate:          353.449253054601 kb/s received

Connnection Times (ms)
              min     avg   max
Connect:        0    0.29    10
Response:      90  260.61   490

Enabling templatetag caching has given a significant speed boost to 38.18 requests/second. This is a 262% improvement over no cache. Response time is also improved, down from 682ms to an acceptable 260ms. Good, but there's still a lot of room for improvement.

The subtle increase in performance shouldn't deter you from implementing the tag though, as template caching bears the benefit that one segment can be cached and used across multiple pages (for example, a sidebar that is the same on different parts of the site).

Django's two-part caching middleware

Django comes equiped with middleware that provides frontend proxy-style full page caching with almost no configuration. Full page caching is clearly where you're going to find the greatest benefits. Something like Squid, Varnish, or NGINX is better suited for this job, but the ease of setup makes this middleware useful for environments where a minimal amount of complexity is desired. Because of the greater performance, I'm running 10,000 requests instead of 1,000 to get a better sample.

Concurrency Level:      10
Time taken for tests:   9.07 seconds
Sent requests:          10000
Completed requests:     10000
Failed requests:        0
Total transferred:      130040000 bytes
HTML transferred:       127560000 bytes
Requests per second:    1102.53583241455
Transfer rate:          14001.3437155458 kb/s received

Connnection Times (ms)
              min     avg   max
Connect:        0    0.15    10
Response:       0    9.02   470

This is about as fast as Django's going to run on this hardware without a more sophisticated caching proxy. We've revved Django's internal caching to give us 1103 requests/second, over 75 times as many as we had with no caching. However, we're still passing every request into Python, which gives us limits we cannot avoid without moving the caching layer into the frontend server. For this we'll need to explore NGINX or Varnish.

NGINX's Memcached module

NGINX has a very nice caching feature that most servers lack: it can serve an HTML document directly from Memcached without ever touching your Python code. Since we are already using NGINX, enabling the Memcached HTTP caching module was a trivial task. For this test, I will disable Django's caching middleware and add a custom cache update middleware that sets a cache key that NGINX can be configured to read. I used a modified version of the middleware from Oliver Weichold's blog post on using Django with NGINX+Memcached. Enabling the module in NGINX config was just adding a new location directive for Memcached and assigning the web app as a 404 handler for that location:

Before:

location / {
    uwsgi_pass  unix:///tmp/mingus.sock;
    include     uwsgi_params;
}

After:

location / {
    default_type  text/html;
    set $memcached_key nginx.$request_uri;
    memcached_pass 127.0.0.1:11211;
    error_page 404 = @cache_miss;
}
location @cache_miss {
    uwsgi_pass  unix:///tmp/mingus.sock;
    include     uwsgi_params;
}

Running the same benchmark as above, here are my results:

Concurrency Level:      10
Time taken for tests:   3.699 seconds
Sent requests:          10000
Completed requests:     10000
Failed requests:        0
Total transferred:      130640000 bytes
HTML transferred:       129190000 bytes
Requests per second:    2703.43336036767
Transfer rate:          34489.8959178156 kb/s received

Connnection Times (ms)
              min     avg   max
Connect:        0    0.36    30
Response:       0    3.66   109

Now we're getting serious! I was serving 2703 requests/second through memcache on my VPS. Now we're in Slashdotting territory. This is over 185 times as fast as vanilla Django. The important thing to note here is that we're accomplishing the same thing as Django's built-in two-part caching middleware, but now we are doing it 2.5 times faster.

On-disk caching with django-staticgenerator

Another approach is to use on-disk caching techniques to serve static files. This is made possible with django-staticgenerator, which has middleware that generates flat files that NGINX can serve directly. It was simple to set up, and here are my results:

Concurrency Level:      10
Time taken for tests:   2.78 seconds
Sent requests:          10000
Completed requests:     10000
Failed requests:        0
Total transferred:      131320000 bytes
HTML transferred:       129190000 bytes
Requests per second:    3597.12230215827
Transfer rate:          46130.2832733813 kb/s received

Connnection Times (ms)
              min     avg   max
Connect:        0    0.67    90
Response:       0    2.66   190

Now we're rocking 3597 requests/second. NGINX can serve static files like nobody's business.

Varnish

Varnish is a very powerful load balancing caching proxy that is made for heavy traffic. I'm configuring it as an HTTP proxy to my NGINX server to see how it stacks up.

Concurrency Level:      10
Time taken for tests:   2.76 seconds
Sent requests:          10000
Completed requests:     10000
Failed requests:        0
Total transferred:      131230000 bytes
HTML transferred:       129190000 bytes
Requests per second:    3623.1884057971
Transfer rate:          46432.716259058 kb/s received

Connnection Times (ms)
              min     avg   max
Connect:        0    0.60    20
Response:       0    2.74    90

Varnish is very competitive in raw speed, serving 3623 requests/second, an impressive number, nearly 250 times higher than if there was no cache. Varnish is also very configurable and built for extremely high traffic.

Conclusion

Every scaling problem has it's own variables that can greatly affect the types of decisions that need to be made to implement a good caching stategy. For example, a multi-server setup is likely to behave much different given the same benchmarks. There are also more complicated factors such as how to treat logged-in users and cookies. There are workarounds for cookie hashing problems (such as removing the "Vary: Cookie" response header) that can add complexity to certain environments, so there is more to consider than raw performance.

Also make note that not all of these are mutually exclusive. A good combination of caching might be internal template caching plus either Varnish or NGINX acting as a frontend cache. My best suggestion is to experiment and see what works best for your environment, and I hope this post was helpful for summarizing your options.

Django Template Debugging Made Easier with django-template-repl

From Cody Soyland on August 29, 2010 08:20 AM

When working with the Django template language, specifically writing template tags or trying out new template tags that are not well documented, it's easy to fall into a testing loop that involves modifying your code, saving the file, causing runserver to restart, which could take some time for large projects, switching to your web browser, hitting reload, and viewing the results. This workflow can be repetitive and unproductive. I decided to improve template interpreter interactivity by writing a REPL for it, and I released the project as django-template-repl, which is freely available on Github and PyPI. I was surprised and happy to see it was well accepted judging from the twitter chatter and github statistics, so I wrote this to better explain how to use it.

One of the greatest advantages of Python, Lisp, and other programming languages is the ease of debugging and understanding code behavior provided by REPLs. A REPL, which stands for Read-Eval-Print Loop, is a shell that gives you an interactive command line session with your language interpreter. The two REPLs most Django users should be familiar with are ipython and (i)pdb. These tools are incredibly helpful and really boost the usability of Python. I spend more time inside ipdb than I read code output in a web browser. django-template-repl provides this type of tool for the Django template language.

Invoking the REPL

There are two ways to invoke the REPL. One uses a management command and the other uses a template tag. Run "python manage.py templateshell" to open the REPL:

$ python manage.py templateshell
>>> test
test
>>> {% if 1 %}
... It is true!
... {% else %}
... It is false!
... {% endif %}

It is true!

This behaves almost exactly like a python shell. It uses the readline library, so it handles command history. It also detects when you are inside a block tag, giving you decent multi-line support.

Providing context

Context can be provided in a number of ways. You can pass a context dictionary with the command-line, for example:

$ python manage.py templateshell --context "{'testvar': 'this is a string'}"
>>> {{ testvar }}
this is a string

Note that this actually runs eval() on the context string. That's the first time I've ever used eval(), but it seems not so hacky for this purpose.

Context can also be extracted from a specific URL in your project. This is accomplished by using Django's test client to capture context used in a page and bootstrap a REPL with the captured context. For example:

$ python manage.py templateshell --url /admin/
>>> {{ title }}
Log in
>>> {{ user }}
AnonymousUser

If you need to capture context at a specific place in your page, like in a for loop, you can use the {% repl %} template tag. This will halt the template rendering process at a specific point and replace your runserver shell with a template REPL loaded with the context used in the calling template.

Use the same tools with PDB

The management command and template tag give you nice ways to capture context, so I've made it possible to use pdb or ipdb with them. The management command has an option, --pdb. If used in conjunction with --url or --context, you can load context into a PDB shell. For example:

$ python manage.py templateshell --url /admin/ --pdb
...
ipdb> vars
Out[0]: 
['app_path',
 'error_message',
 'root_path',
 'title',
 'MEDIA_URL',
 'LANGUAGES',
 'LANGUAGE_BIDI',
 'LANGUAGE_CODE',
 'perms',
 'messages',
 'user']
ipdb> title
Out[0]: u'Log in'
ipdb> user
Out[0]: <django.contrib.auth.models.AnonymousUser object at 0x1019d7490>

This functionality is also possible with the template tag, using {% repl pdb %}.

I do hope these tools help somebody out. I feel like it is much easier to debug issues with template context and try new template tags. If you want to try it out, just run pip install django-template-repl and add 'template_repl' to your installed apps.

August 28, 2010

Google Moderator NoSQL and Django Panel at DjangoCon 2010

From Django Dose on August 28, 2010 08:36 PM

The place to submit questions for the panel on NoSQL that will happen at DjangoCon 2010.

New features on Read The Docs

From Eric Holscher on August 28, 2010 07:20 PM

Since the Django Dash ended, We've been working on adding some requested new features to Read The Docs. There are a couple of major ones that we have added that I'd like to talk about.

hg and svn support

We've added support for all of the version control systems that people have requested. When you sign up or edit a project, you can now tell us which VCS you are using, and we'll use that to check out your code to build your documentation.

There are two libraries that I wish existed: One to smartly parse urls into the correct repository, and a standard VCS abstraction that lets me treat all VCS' as the same. These would integrated presumably, so I could do vcs clone <url> and vcs update, and it would "Just Work"

Whitelisting support

By default we don't execute any python code when you import your project. This is a security precaution that we take, so that means disabling all extensions by default. A lot of people are using autodoc and some other extensions, so we have added the ability to whitelist projects so that they are built without any sanitization on our part.

A sweet logo

Our designer Bobby made a sweet logo for the site, and has been adding lots of little visual tweaks that I'm not qualified to talk about :) However, it seems that whenever I look at the site, it gets a little prettier, that's how I usually know that design is being done.

Subdomain and CNAME support

This is a really exciting one for me, because I've been learning more and more about sysadminery lately, and this was a fun little mixing of the two. For any project, you can now access the projects documentation at <slug>.readthedocs.org, for example, pip's documentation is now at pip.readthedocs.org.

Now that we have subdomain support, this makes supporting CNAME's really simple. So if you have your own domain name, and you'd like those docs to point to us, it's simple. All you need to do is add a CNAME record for that domain in your DNS settings to point at your subdomain URL. Pip is another good example here, pip-installer.org now is hosted on RTD. Other notable examples are djangotesting.com and djangowoodies.com :)

All of this support is implemented in middleware and only ends up being about 25 lines of code. There are going to be some complications when we try to add multiple version support, and internationalization, as you can't really specify those well on subdomains. I see us having a "default" project version, as well as letting you have other versions hosted as well.

RTFD.org

"RTFM" is a well known term in the programming community. Luckily when we were scheming up names for our project, we noticed that rtfd.org was available. We went ahead and bought it, and now we're supporting <slug>.rtfd.org and rtfd.org/<slug> redirects that go to your RTD page. This is a nice little keystroke saver, as well as a fun was to refer people to your documentation. This is implemented simple in an Nginx server directive. I'm sure it can be improved upon, but it's working well at the moment.

I think adding the ability to have "smart slugs" here would be interesting, so it could actually perform a search or something, and return the top result, kinda like LMGTFY. This could be a neat feature to add on.

LaTeX support

LaTeX is a pain to get setup, so if you want to support rendering LaTeX, we now support that as well. The sympy has been testing their docs on RTD, and have helped me clean up a bunch of bugs. The Geometric Algebra section shows off some of the LaTeX goodness.

Update: Just for kicks, we currently have 120 users and 80 projects currently hosted on Read The Docs. At least a couple of these are using RTD as their official documentation host. I'm pretty happy with the uptake that's already happened in the last 2 weeks (wow, that little?!). Thanks everyone for checking it out!

Update 2 We also have a new IRC channel if you need help or have questions, it's #rtd on irc.freenode.net.

Announcing Django Packages!

From Daniel Greenfeld on August 28, 2010 06:12 PM

I'm part of a two person team that just launched that BETA site for http://djangopackages.com, a site that is designed to list all the Django Applications, Frameworks, and Packages created by the Django community. While there are already a few places to look for these things, it is quite easy to argue that they are challenging to navigate, don't give any hard metrics, or are woefully incomplete/unstable/closed. Our goal was to provide an attractive, easy-to-navigate, easy-to-add-data, stable site and share it with the world.

Also, this was our entry into Django Dash 2010, and was the culmination of a few days of brainstorming over paper, a lot of research, and two days of feverish coding/designing. The project was feature complete to our specifications at 5pm the second day, and the rest of the time was spent adding in UI tweaks, usability enhancements, and trying to deploy our creation.

Since then, we've cleaned up a the UI, improved the design, and got the site stable. The code is open source and on github, so fork and contribute!

Design Consideration: No 'Like' button or 'Rate my app' rating systems

We wanted hard metrics. So the package numbers are pulled from the repo sites such as Github, Bitbucket, and Google Code. Otherwise things get weighted funny. Sure, this system can be monkeyed with, but its a good metric for now. We've had suggestions from Django core developers of coming up with a quality check system, things like pypants and/or a formalized approval system.

Design Consideration: Grids

Early on we wanted to duplicate and improve upon the Django CMS Comparison page. There is also a version for Forums, but it would be nice to have a current one for blogs! In addition, recently I heard that 'tag clouds are the mullets of web 2.0'. This really struck a chord in my soul. Since we had metrics on packages, why not compare those metrics, and use those comparisons, which we call 'grids', instead of tags? In fact, we extended our idea and instead of traditional tabs we use grids in the top navigation area as seen below:


Design Consideration: Categories

The site groups packages into three categories, 'Apps' which are individual django applications. 'Frameworks' which are aggregates of apps and python modules. And 'Projects' which are implementations of Apps and Frameworks. We've thought about adding 'Tools' but weren't sure if there was anything that fit that concept, and we are leery about allowing regular Python efforts into the fold.

Design Consideration: Regex vs XML

Slurping data out of Github was easy, especially since I used python-github2. Bitbucket has a RESTful API as well that serves out JSON. I think Google Code does as well. PyPI does not and DOAP on PyPI seems to give little that is useful, so I was forced to do screen scrapes of version numbers and downloads. I'm much faster with Regex and string methods than XML juggling, and speed was of the issue this weekend. I'm not sure what benefit there is to redoing it in HTML5lib or lxml, since what I have works and appears to be stable.

Design Consideration: Leave caching and optimization for later

Besides a tiny bit of memory based template caching on the home page, there is/was no optimization. In time I plan to cache many things using a proper key/value store like redis or memcached. Perhaps not before more design and usability work is done.

Why scared of rabbits?

You wouldn't understand unless you live on the Kansas prairie.

Note: if you have any suggestions, issues, problems with Django Packages please use our issue tracker!

Django Dash Lessons Learned

From Daniel Greenfeld on August 28, 2010 06:12 PM

Our experience with Django Dash 2010 was that it was an wonderful exercise in classic Django development, cowboy/cowgirl coding, and drinking copious amounts of caffeinated beverages. The result, Django Packages, is something we are happy with, are continuing to improve, and hope will improve the community.

Lesson: Fixtures are a must

Django gives you this amazing Admin control panel. As soon as you get your models in place and are entering test data, start creating fixtures. For the dash we named them initial_data.json and loaded them into the individual app directories. This meant that every time we blew away the database we got a reload with records in place. Sometimes this means you have to hand-edit JSON (or YAML if you swing that way), but the alternative is to waste time re-entering the same data again and again. Don't forget to change the names of your test fixtures before you launch!

From the command-line how to save a fixture pydanny-style:

./manage.py dumpdata package > apps/packages/fixtures/initial_data.json

One nice thing about fixtures is that when you do have the time/need, you can use them to help you write tests. And it makes development easier for contributors.

Lesson: Research ahead of time

In the days before the contest, we researched to see if our target repos (Github, ;BitBucket, and Google Project Hosting) each had an API and a python library to speak to that API. Github has both an API and python library, Bitbucket has an API but no library. And as far as we can tell, Google Project Hosting lacks both API and library (someone please tell me I'm wrong about Google Project Hosting lacking an API).

This meant that when we commenced coding we knew which code base to work with - we weren't trying to look up this or that random package.

We did the same thing for rendering charts.

Lesson: Get it working then optimize

Looking at some of the code makes us wince a bit. But we got it working. Now we can go back and do some code cleanup, maybe use an XML parser instead of regex to try to scrape content from PyPI, and generally feel better about ourselves.

Lesson: Plan out system architecture in advance

In retrospect it was really amusing, but the night of launch the site was serving via the Django runserver command. We were so dead tired and neither of us are crack system administrators that we did what we had to do to score the contest launch point. The next day Audrey got the site running underApache, and next week we'll be giving someone else system access to increase reliability. Next year for the contest we'll probably use something like this or get continuous integration running in the first hour.

Lesson: Don't be afraid to chat with others after the contest starts

Share your ideas, selected packages and frameworks with your competitors. The break from coding helps clear the mind and they might counter with a better idea/package/framework you can use.

New features for Django Packages

From Daniel Greenfeld on August 28, 2010 06:11 PM

Since the Django Dash ended, the Django Packages team has been working to add new features and close out bugs.

"I Use This" added

Since we only want hard metrics on this site, we incorporated an "I Use This" button on the packages. This is so you can identify which packages you use. Please don't press this button for packages that you like, only the ones that are part of your coding efforts.

Added BitBucket support

We are still working out some of the kinks for coming up with stats from BitBucket. Most of the data we collect is fetched via the API, but a little is scraped off individual project pages.

Cache the commits

Originally the commit history was  fetched live. But Github only provides the last 35 commits and BitBucket limits you to the last 50 commits. So now we store the commit history and update it nightly. Which means that the sooner you post your packages the better your commit history will look on Django Packages.

Rebuilt the package updater

Limitations on how many API calls you can make against Github (60 a minute) meant that we had to write some fun code to get around that problem. I think the problem is solved now, but I'm worried I might get to eat my words.

Added a help section

As much as we wanted a completely intuitive site, this will hopefully make it easier for people to figure out how to participate on the site.

Package Add/Edit form refactor

We completely rebuilt the Package add/edit form to make it easier to add packages. So far the response has been entirely positive.

Page cleanup and CSS Reset

We've been slowly cleaning up the HTML and resetting the CSS. Everything is looking prettier. Our goal is to make things more readable, so a lot of the changes are subtle.

Email verification works

It works, and now you get an email to confirm your account.

August 27, 2010

Musings about django.contrib.auth.models.User

From Peter Bengtsson on August 27, 2010 08:15 PM

Dawned on me that the Django auth user model that ships with Django is like the string built-in of a high level programming language. With the string built-in it's oh so tempting to add custom functionality to it like a fancy captialization method or some other function that automatically strips whitespace or what not. Yes, I'm looking at you Prototype for example.

By NOT doing that, and leaving it as it is, you automatically manage to Keep It Simple Stupid and your application code makes sense to the next developer who joins your project.

I'm not a smart programmer but I'm a smart developer in that I'm good at keeping things pure and simple. It means I can't show off any fancy generators, monads or metaclasses but it does mean that fellow coders who follow my steps can more quickly hit the ground running.

My colleagues and I now have more than ten Django projects that rely on, without overriding, the django.contrib.auth.models.User class and there has been many times where I've been tempted to use it as a base class or something instead but in retrospect I'm wholeheartedly happy I didn't. The benefit isn't technical; it's a matter of teamwork and holistic productivity.

Centralized logging for fun and profit!

From Revolution Systems on August 27, 2010 12:12 AM

Setting up a centralized log server using syslog isn't as hard as many may believe. Whether it's logs from Apache, nginx, email services, or even from your own Python applications having a central log server gives you many benefits:

Benefits to a centralized logs

  • Reduces disk space usage and disk I/O on core servers that should be busy doing something else. This is especially true if you want to log all queries to your database. Doing this on the same disk as your actual database creates a write for every read and an extra write for every write.
  • Removes logs from the server in the event of an intrusion or system failure. By having the logs elsewhere you at least have a chance of finding something useful about what happened.
  • All of your logs are in one place, duh! This makes things like grepping through say Apache error logs across multiple webservers easier than bouncing around between boxes. Any log processing and log rotation can also be centralized which may delay your sysadmin from finally snapping and killing everyone.

Syslog Review

In case you aren't terribly familiar with how syslog works, here's a quick primer. Syslog separates out various logs using two items. Facilities and Levels. Here are the standard facilities:

  • 0 kernel messages
  • 1 user-level messages
  • 2 mail system
  • 3 system daemons
  • 4 security/authorization messages
  • 5 messages generated internally by syslogd
  • 6 line printer subsystem
  • 7 network news subsystem
  • 8 UUCP subsystem
  • 9 clock daemon
  • 10 security/authorization messages
  • 11 FTP daemon
  • 12 NTP subsystem
  • 13 log audit
  • 14 log alert
  • 15 clock daemon
  • 16 local use 0 (local0)
  • 17 local use 1 (local1)
  • 18 local use 2 (local2)
  • 19 local use 3 (local3)
  • 20 local use 4 (local4)
  • 21 local use 5 (local5)
  • 22 local use 6 (local6)
  • 23 local use 7 (local7)

For each facility logs are sent using a particular level, the levels are:

  • 0 Emergency: system is unusable
  • 1 Alert: action must be taken immediately
  • 2 Critical: critical conditions
  • 3 Error: error conditions
  • 4 Warning: warning conditions
  • 5 Notice: normal but significant condition
  • 6 Informational: informational messages
  • 7 Debug: debug-level messages

So for any given log message you set these two options to give a hint as to where the logs should be directed. For example, if an email server receives a new message it would likely be sent as mail.info and a kernel panic would be sent using kern.emerg

The receiving syslog server then can be configured to direct log messages of a certain facility and/or log level to various files. For example, a default Ubuntu system has some settings like this:

daemon.*        /var/log/daemon.log
kern.*          /var/log/kern.log
mail.*          /var/log/mail.log

But you can also do more granular separation for example you might want to log mail.err into a separate file from the main mail logs to make it easier to spot new errors with this:

mail.*        /var/log/mail.log
mail.err      /var/log/mail-errors.log

Setting up your central server

Configuring the master log server is pretty easy. On Ubuntu the default syslog server is rsyslog and that's what I'll be using as an example here. You'll need to edit /etc/rsyslog.conf and uncomment the UDP module. You could also use the TCP module, but that one binds to all of your interfaces so you will need to restrict access to it with iptables (or some other mechanism) in order to not allow hackers to fill up your disks remotely. So your configuration should now contain these uncommented lines, where 'x' is an internal protected IP address:

$ModLoad imudp
$UDPServerAddress x.x.x.x
$UDPServerRun 514

And then restart rsyslogd. See that wasn't so hard...

Setting up the remote log sending servers

Setting up your remote servers is even easier. If you want to send ALL of your logs to the central server it's just a matter of adding one line to the top of /etc/rsyslog.d/50-default.conf. That line is:

 
*.* @x.x.x.x:514

This will send all logs of any facility and any level to the server. Note that the local syslog will, as configured by default, still log locally. So if you don't want that be sure to remove all of the other configuration in this file.

You can also get fancy here and keep some logs on the local server and only send some things remotely. For most of your custom apps and logs you'll want to be using the LOCAL[0-9] facilities. Let's say we're going to want to centrally log our Python logs and Apache error logs. We'll be using LOCAL0 and LOCAL1 for them respectively. That config would look like:

local0.* @x.x.x.x:514
local1.* @x.x.x.x:514

Keep in mind however that most systems have *.info, *.debug, etc. configurations setup so you might be duplicating your data. If you poke around this file you'll see lots of configurations ending in .none, this instructs rsyslog to not include those facilities in this particular file. So for example, you'd want to edit your /var/log/syslog to resemble this:

 
*.*;auth,authpriv,local0,local1.none        /var/log/syslog

Additional help and features

While most applications are easy to setup for use with syslog, here are some pointers for more info on the subject:

  • Apache support sending error logs to syslog via the ErrorLog syslog:local1 configuration option. However, it does not support sending access logs directly. To do that you'll need a small script and pipe your access logs through it.
  • For more information on setting up your own Python code to use syslog, check out the logging.handlers.SysLogHandler handler for the logging module.

We've only really scratched the surface of the features of rsyslog with this setup. You can configure it to do some fairly advanced separation of logs based on the sending host, application name, and other various aspects of the message itself. Refer to the rsyslog documentation for more information on that.

Happy Logging!

August 26, 2010

Caktus Consulting Group Sponsors DjangoCon 2010

From Caktus Consulting Group on August 26, 2010 06:50 PM

DjangoCon 2010 is just around the corner, and I’m proud to announce that Caktus is sponsoring the conference again this year!

DjangoCon is the annual gathering of software developers who use the open source, Python-based Django web framework. We use the framework every day here at Caktus to create custom web applications and dynamic, content-rich web sites. Additionally, starting this year, we’ve put some of that knowledge to use extending and developing applications for the RapidSMS framework. For more information about why we use Django and think it’s so great, check out our blog post titled Why Caktus Uses Django.

This year, the conference is being held again the week of September 6th in the beautiful city of Portland, Oregon. We’ve grown a little since last year at this time; it looks like 6 Caktus team members—Colin, Alex, Karen, Mark, Mike, and myself—will be attending the conference. We’re positively thrilled to be going again this year and we hope to see you there!

Tips & Tricks for your Django Powered Database

From David Cramer on August 26, 2010 07:33 AM

Tonight was our Django meetup here in San Francisco. Four of us presented (three from Whiskey, outnumbering everyone as always , with some pretty great material. A few of my favorites were .update() from Andy McCurdy, (I have to love anyone that agrees), formfield_overrides (not sure how I didn’t know about this) for the admin [...]

August 25, 2010

A new blog for Excel Geeks

From Stuart Marsh on August 25, 2010 08:26 PM

My friend and fellow web developer Andrew Walker has just started a new blog all about advanced Excel and VBA techniques, called Excel Geek.  If you want to find out how to do all sorts of stuff you didn’t know Excel could do, this is the place to go.

Years ago, Andy took over from me in my role as a performance analyst.  I trained him well, but it seems that the student has become the Master.

Keep up the good work young Padawan.


August 24, 2010

What is the history of Django?

From Simon Willison on August 24, 2010 05:31 PM

What is the history of Django?. I’ve been playing with Quora—it’s a really neat twist on the question-and-answer format, which makes great use of friends, followers and topics and has some very neat live update stuff going on (using Comet on top of Tornado). I just posted quite a long answer to a question about the history of Django.

books in a stack (a stack of books) by austinevan

From Andrew Wilkinson on August 24, 2010 12:23 PM

books in a stack (a stack of books) by austinevanvirtualenv is a tool for simplifying dependency management in Python applications. As the name suggests, virtualenv creates a virtual environment which makes it easy to install Python packages without needing root privileges to do so.

To use the packages installed in a virtual environment you run the activate script in the bin directory of the virtual environment. This is fine when you’re working on the command line, but you don’t want to have to remember this step when running the debug server, and it’s hard to get that to work when the site is deployed under mod_wsgi.

To make things easier you can add the appropriate directory from the virtual environment to Python’s path as part of manage.py, or the appropriate fcgi or wsgi control script.

import os
import sys
import site

root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))

site.addsitedir(os.path.join(root_dir, 've/lib/python%i.%i/site-packages' % (sys.version_info[0], sys.version_info[1])))

Just add the code above to the top of your manage.py file and the ve virtual environment will always be activated when you run the script.


Photo of books in a stack (a stack of books) by austinevan.


nashvegas 0.1a1.dev2 Released

From Patrick Altman on August 24, 2010 05:30 AM

So like a Phoenix rising from the ashes, nashvegas has returned with a 0.1a1.dev2 release. PEO Pheonix

18 months ago, I posted about the initial release of a migration tool that I wrote and found useful. I then later abandoned it in my personal projects in favor of South.

However, I quickly wanted something not so complicated and allowed me to better manage exactly what was getting executed. This brought me full circle to just needing to finish off some outstanding features on nashvegas.

With this release you can:

  • Execute both sql and Python scripts
  • Generate migration scripts for new models that are introduced to your project, whether they be from reusable apps that you have installed, or apps that live within your project.
  • A Migration model tracks everything in the database instead of what was previously just a table.
  • This Migration model bootstraps itself into your database when executing any of the commands -- after adding it to INSTALLED_APPS it's ready to use.

Check it out. Let me know what you think!

You can find the source code on the project page on Github.

Enjoy!

Mixing Django with Jinja2 without losing template debugging

From Thierry Schellenbach on August 24, 2010 01:26 AM

At Fashiolista we’ve build nearly the entire site with Jinja instead of the Django template engine.

There are a lot of reasons for choosing Jinja2 over Django for us. Better performance (atleast… it was a lot better with previous Django versions), way more options (named arguments, multiple arguments for filters, etc), macros and simply easier to extend. Writing custom tags is simply not needed anymore since you can just make any function callable from the templates.

But… during the conversion there are always moments when you need  a Django function in a Jinja template or vice versa. So… I created a few template tags to allow for Jinja code in Django templates (I’ve also created code to run Django code from Jinja, but I haven’t seen the need for it so I omitted it here).

A Jinja Include tag to include a template and let it be parsed by Jinja from a Django template:

from django import template
from coffin import shortcuts as jinja_shortcuts                                                                                                                                                      

register = template.Library()                                                                                                                                                                        

class JinjaInclude(template.Node):
    def __init__(self, filename):
        self.filename = filename                                                                                                                                                                     

    def render(self, context):
        return jinja_shortcuts.render_to_string(self.filename, context)                                                                                                                              

@register.tag
def jinja_include(parser, token):
    bits = token.contents.split()                                                                                                                                                                    

    '''Check if a filename was given'''
    if len(bits) != 2:
        raise template.TemplateSyntaxError('%r tag requires the name of the '
            'template to be included included ' % bits[0])
    filename = bits[1]                                                                                                                                                                               

    '''Remove quotes if used'''
    if filename[0] in ('"', "'") and filename[-1] == filename[0]:
        filename = bits[1:-1]                                                                                                                                                                        

    return JinjaInclude(filename)

Usage:

{% jinja_include "some_template.html" %}

A couple of noop nodes to make sure that when you convert your Jinja templates to be executed from Django, they won’t break because of the missing Django tag.

from django import template

class Empty(template.Node):
    def render(self, context):
        return ''                                                                                                                                                                                    

@register.tag
def django(parser, token):
    return Empty()                                                                                                                                                                                   

@register.tag
def end_django(parser, token):
    return Empty()

And the Jinja tag to allow Jinja blocks in Django templates.

from django import template
from coffin.template import Template

register = template.Library()

class Jinja(template.Node):
    def __init__(self, template):
        self.template = template                                                                                                                                                                     

    def render(self, context):
        return self.template.render(context)                                                                                                                                                         

@register.tag
def jinja(parser, token):
    '''Create a Jinja template block                                                                                                                                                                 

    Usage:
    {% jinja %}
    Although you're in a Django template, code here will be executed by Jinja
    {% end_jinja %}
    '''                                                                                                                                                                                              

    '''Generate the end tag from the currently used tag name'''
    end_tag = 'end_%s' % token.contents.split()[0]                                                                                                                                                   

    tokens = []
    '''Convert all tokens to the string representation of them
    That way we can keep Django template debugging with Jinja and feed the
    entire string to Jinja'''
    while parser.tokens:
        token = parser.next_token()
        if token.token_type == template.TOKEN_TEXT:
            tokens.append(token.contents)                                                                                                                                                            

        elif token.token_type == template.TOKEN_VAR:
            tokens.append(' '.join((
                template.VARIABLE_TAG_START,
                token.contents,
                template.VARIABLE_TAG_END,
            )))                                                                                                                                                                                      

        elif token.token_type == template.TOKEN_BLOCK:
            if token.contents == end_tag:
                break                                                                                                                                                                                

            tokens.append(' '.join((
                template.BLOCK_TAG_START,
                token.contents,
                template.BLOCK_TAG_END,
            )))                                                                                                                                                                                      

        elif token.token_type == template.TOKEN_COMMENT:
            pass                                                                                                                                                                                     

        else:
            raise template.TemplateSyntaxError('Unknown token type: "%s"' % token.token_type)                                                                                                        

    '''If our token has a `source` attribute than template_debugging is
    enabled. If it's enabled create a valid source attribute for the Django
    template debugger'''
    if hasattr(token, 'source'):
        source = token.source[0], (token.source[1][0], token.source[1][1])
    else:
        source = None                                                                                                                                                                                

    return Jinja(Template(''.join(tokens), source=source))

Do note that I have modified the “coffin.template.Template” to enable debugging completely. Just replace the “Template” class in “coffin/template/__init__.py” to make it work.

def _generate_django_exception(e, source=None):
    '''Generate a Django exception from a Jinja source'''
    from django.views.debug import linebreak_iter                                                                                                                                                    

    if source:
        exception = DjangoTemplateSyntaxError(e.message)
        exception_dict = e.__dict__
        del exception_dict['source']                                                                                                                                                                 

        '''Fetch the entire template in a string'''
        template_string = source[0].reload()                                                                                                                                                         

        '''Get the line number from the error message, if available'''
        match = re.match('.* at (\d+)$', e.message)                                                                                                                                                  

        start_index = 0
        stop_index = 0
        if match:
            '''Convert the position found in the stacktrace to a position
            the Django template debug system can use'''
            position = int(match.group(1)) + source[1][0] + 1                                                                                                                                        

            for index in linebreak_iter(template_string):
                if index >= position:
                    stop_index = min(index, position + 3)
                    start_index = min(index, position - 2)
                    break
                start_index = index                                                                                                                                                                  

        else:
            '''So there wasn't a matching error message, in that case we
            simply have to highlight the entire line instead of the specific
            words'''
            ignore_lines = 0
            for i, index in enumerate(linebreak_iter(template_string)):
                if source[1][0] > index:
                    ignore_lines += 1                                                                                                                                                                

                if i - ignore_lines == e.lineno:
                    stop_index = index
                    break                                                                                                                                                                            

                start_index = index                                                                                                                                                                  

        '''Convert the positions to a source that is compatible with the
        Django template debugger'''
        source = source[0], (
            start_index,
            stop_index,
        )
    else:
        '''No source available so we let Django fetch it for us'''
        lineno = e.lineno - 1
        template_string, source = django_loader.find_template_source(e.name)
        exception = DjangoTemplateSyntaxError(e.message)                                                                                                                                             

        '''Find the positions by the line number given in the exception'''
        start_index = 0
        for i in range(lineno):
            start_index = template_string.index('\n', start_index + 1)                                                                                                                               

        source = source, (
            start_index + 1,
            template_string.index('\n', start_index + 1) + 1,
        )                                                                                                                                                                                            

    exception.source = source
    return exception          

class Template(_Jinja2Template):
    """Fixes the incompabilites between Jinja2's template class and
    Django's.                                                                                                                                                                                        

    The end result should be a class that renders Jinja2 templates but
    is compatible with the interface specfied by Django.                                                                                                                                             

    This includes flattening a ``Context`` instance passed to render
    and making sure that this class will automatically use the global
    coffin environment.
    """                                                                                                                                                                                              

    def __new__(cls, template_string, origin=None, name=None, source=None):
        # We accept the "origin" and "name" arguments, but discard them
        # right away - Jinja's Template class (apparently) stores no
        # equivalent information.
        from coffin.common import env                                                                                                                                                                

        try:
            return env.from_string(template_string, template_class=cls)
        except JinjaTemplateSyntaxError, e:
            raise _generate_django_exception(e, source)                                                                                                                                              

    def __iter__(self):
        # TODO: Django allows iterating over the templates nodes. Should
        # be parse ourself and iterate over the AST?
        raise NotImplementedError()                                                                                                                                                                  

    def render(self, context=None):
        """Differs from Django's own render() slightly in that makes the
        ``context`` parameter optional. We try to strike a middle ground
        here between implementing Django's interface while still supporting
        Jinja's own call syntax as well.
        """
        if not context:
            context = {}
        else:
            context = dict_from_django_context(context)                                                                                                                                              

        try:
            return super(Template, self).render(context)
        except JinjaTemplateSyntaxError, e:
            raise _generate_django_exception(e)                                                                                                                                                      

def dict_from_django_context(context):
    """Flattens a Django :class:`django.template.context.Context` object.
    """
    if isinstance(context, DjangoContext):
        dict_ = {}
        # Newest dicts are up front, so update from oldest to newest.
        for subcontext in reversed(list(context)):
            dict_.update(dict_from_django_context(subcontext))
        return dict_
    else:
        return context

And you’re done, now you can just mix your Django and Jinja templates like this:

{% ifequal foo bar %}
Django style if...
{% endif %}

{% jinja %}
{% if foo == bar %}
Jinja style if...
{% endif %}
{% end_jinja %}
Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google

August 23, 2010

Python and Django job in Utrecht

From Reinout van Rees on August 23, 2010 10:54 PM

Summary: yep, we (Nelen & Schuurmans) need a new Python/Django programmer! Location: Utrecht, the Netherlands, right in the center of town. The work subject? Geodjango and water management: that's the shortest summary.

Five months ago I wrote about an upcoming job opening. We got, if I remember correctly, nine reactions, eight of which mentioned "yeah, I saw on Reinout's blog/twitter...". The power of twitter and blogs wasn't demonstrated that vividly to my colleages yet :-) The end effect was that when we invited people to come round for a talk, they were introduced internally as "oh, some of Reinout's friends are coming"... That meant I had some explaining to do during lunch :-)

Business is booming and there's yet another big project coming our way. Help! We really need to crank up the output. At the moment we're three people doing the practical day-to-day Django coding with another one coming up in a week. Cooperative, productive, friendly. We could really use another pair of hands (and a brain, of course).

  • Django experience? Not needed per se, Django is friendly and modular enough that we can teach you. But on the other hand, we really want to get cracking turning out new websites!
  • Bonus points if you're any good in html/css/javascript. If I tell former colleague (and css wizard) Mirella that I'm now the local number one css expert, she'll probably laugh out loud. I'm not bad at all, but a bit of extra knowledge won't hurt. That reminds me: I've got to interrogate our new colleague on his level of css knowledge...
  • We want good software. As an illustration, I've set up a Hudson continuous integration server to monitor the tests, code coverage and the pep8/pyflakes score. We really need to get our act together to get our code coverage up to percentages that I dare mention here.
  • Packaging, automation, buildout, hudson, deployment: we've got that one right. Lots of automation. Buildout helps a lot. Repeatable. Deployments are boring (as they should be) as they're mostly flawless.
  • Windows experience would be helpful as we sometimes need to deploy on windows. In practice all three (and soon four) of us Django programmers are on Ubuntu.

The timeframe? What I heard is a closing date of 23 September. But we might start inviting people over sooner as there really are a couple of projects lining up. So don't wait too long.

The official stuff? The official vacature is on our website. Yep, that's in Dutch. I'm afraid that speaking Dutch is handiest to get settled into the company. We might be open for an experiment, though :-)

You can always mail me if you've got further questions, of course.

Company outing in Utrecht

August 22, 2010

Lessons Learned From The Dash: Nginx SSI

From Eric Holscher on August 22, 2010 09:21 PM

Continuing from my previous post about Django Dash, I will be talking about another thing that I learned from the dash. This isn't as big of a post, but just something that we ran into that caused us some trouble.

We are hosting documentation for other projects, and we needed a way to put a toolbar on the top of the pages so users can still get around our site. We started out by hacking this into the sphinx template as static html, which was annoying because it didn't let us determine if the user was logged in, owned the project, etc. So we decided to load the header dynamically.

Using Nginx Ghetto ESI

We were deploying on Nginx, and luckily this post about Ghetto ESI with Nginx laid it out pretty well. We only ran into one problem with this approach, and it was minor.

The implementation is that we are hacking the SSI tag into the Sphinx template's we are rendering at the top of the page.

{% block relbar1 %}
<!--# include virtual="/render_header/" -->
{{ super() }}
{% endblock %}

Then you simply add a ssi on; into your Nginx configuration for your site. This makes the page call /render_header/ to fill out the top of the page when the user hits a documentation page.

The problem

The problem with this is that this doesn't work in your local testing environment. So Joshua's post earlier has a piece of middleware that you can include in your Django project to emulate the Nginx include behavior.

We turned this on, but every once in a while our pages were getting randomly cut off halfway through the response. We looked into it a little bit, and figured out that it was because of the response's Content Length header was still set to the old value. So our updated middleware simply added one line to the reponse.

response['Content-Length'] = len(response.content)

This allowed our pages to render correctly in testing, and then in production Nginx will hit the include before Django sees it, so the middleware never processed. If you are changing the content of your response in middleware, make sure that you update the Content-Length header.

Using Sass with Django

From Trey Piepmeier on August 22, 2010 07:00 PM

Install django-css.

Install Sass.

sudo gem install haml

Add to settings.py:

INSTALLED_APPS = (
    ...
    'compressor',
    ...
)

...

COMPILER_FORMATS = {
    '.sass': {
        'binary_path':'sass',
        'arguments': '*.sass *.css'
    },
    '.scss': {
        'binary_path':'sass',
        'arguments': '*.scss *.css'
    }
}

Add to a template that you want to load a Sass file:

{% load compress %}

...

{% compress css %}
<link rel="stylesheet" href="{{ MEDIA_URL }}css/base.scss" media="screen">
{% endcompress %}

Deal with caching when you deploy.

Perhaps pip install python-memcached then put something like this in settings.py

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

Write some nice Sass.

$orange: #EE8529;

ul {
    font-size: 26px;
    li {
        color: $orange;
    }
}

Rails-like configuration style for Django

From Thomas Pelletier on August 22, 2010 05:19 PM

Django’s default settings system is not very suitable for multiple configuration profiles — development, testing, production and so on: you have the settings.py, and that’s it. As far as I’m concerned, my settings are sometimes very different between my notebook and for instance my production server. First of all, it is obviously out of the question to change the settings.py on deployment. Some people append at the end of their settings.py a simple from deploy import *, surrounded by a try / except ImportError clause. In my opinion this workaround is neither sexy nor flexible. I finally decided to use the Rails way to manage configuration in my latest pet project.

The principle is simple: replace the content of the settings.py by the one provided further, create a config/ directory, as if it is an application, and in this folder, put the default configuration in init.py, and add a Python module per configuration profile. You should come up with something like that:

Files tree

Files tree

The default profile is ‘development’ (read the settings.py). If you want to use a specific profile, prepend DJANGO_ENV=”theprofile” to your shell command. It must become very long to type this at each command, that’s why I recommend you to add this line to the ‘bin/activate’ script of the corresponding virtualenv, or in your ~/.bash_profile.

August 21, 2010

Agrupar mediante el templatetag {% regroup %} con una función propia

From Django en Español on August 21, 2010 07:16 PM

El templatetag {% regroup %} del lenguaje de plantillas de Django nos permite agrupar un listado de objetos por una propiedad. Generalmente utilizamos un campo por el que agrupar pero también podemos utilizar una función propia para agrupar los objetos como queramos. Vamos a ver un ejemplo sencillo en el que agruparemos un listado de artículos por su fecha de publicación, dividiéndolos en artículos publicados "Esta semana", "La semana pasada" y "Hace X semanas" ...

August 19, 2010

Windows installation went OK

From Reinout van Rees on August 19, 2010 06:38 PM

My (geo)django installation on the customer's windows 2008 box went way better than I expected. "Mostly smooth" is what I'd call it. My windows experience is a tad rusty after not using windows productively for about 13 years or so. And I ran into all sorts of problems when trying it out locally. So I expected much worse.

Trying everything out beforehand and assembling the right set of packages and killing off all DLL problems helps. See my earlier article about all the weird problems I encountered). It is hard work to find the cause for some obscure DLL problem, but it beats having to debug it when the customer is sitting next to you :-)

The actual installation went without a hitch. Everything installed just fine. I had written down what I had to do beforehand: that paid out in spades. I only saw two serious problems in the end:

  • Oracle doesn't like django's TextFields when they're used with an .distinct() query. They also cannot be used for indexes or primary keys. Oracle stores them as a "NCLOB" field: some sort of character blob. I fixed the one part in the code where I had a .distinct(): no, that didn't help. In the end a "name" field (the most important field in the model) was a TextField instead of a CharField just like all the other models... Ok, that's a regular bug in our code. And another reminder that all our code needs to be tested: we'd have caught this bug for sure in that case.
  • Apache seems to die after a while. Apache with a django-behind-mod_wsgi setup. I got the impression it died after some 100 requests. But during the last test it seemed to die instantly. Every request seems to have no effect after the "crash" or whatever it is: there's no clear error message. Every request just seems to disappear in a black hole. I'll have to google that one, as I couldn't figure it out in time.

In any case: the software installed just fine. And the oracle database connection works just fine. And the customer was happy! And I'm too :-)

Simplejson, python2.6 and windows

From Reinout van Rees on August 19, 2010 06:16 PM

There was one additional problem I saw with yesterday's windows installation: simplejson!

Totally unexpected. The good thing is that simplejson is shipped with python 2.6 and higher. The sane thing is that they named it "json" instead of "simplejson".

The bad or at least unexpected thing: I cannot find a windows installer for simplejson for python 2.6! Which means my "import simplejson" won't work anymore. And those have to be in as the software gets installed on python 2.5 sometimes.

For windows + python 2.6 compatibility I now have the following in at least 10 different files:

try:
    import json  # Python 2.6+
except ImportError:
    import simplejson as json  # Python 2.5-

And I've had to remove simplejson as a dependency. I don't yet know a good solution for that. I don't think you can have an optional dependency based on the python version with the current distribute/setuptools. Of course you could do some more if/else trickery in there as a setup.py` is just a python file after all.

Anyone got a good hint? Is there something I overlooked?

August 18, 2010

Geodjango with oracle on windows: surprises and fixes

From Reinout van Rees on August 18, 2010 07:44 PM

One of our customers demanded a windows installation of his geodjango application. None of our active developers are well versed in that. My last personal well-maintained windows installation was windows 3.1. There.

So I spend two days getting it working on a temporary windows 2008 test machine and on a local virtualbox. Those were two days I'd rather been spending on some productive coding, to be honest. Finding problems and solving them have their own merit and somebody needs to so it so I girded myself and got down to work. I'll list some of the bad and good things below, including solutions.

Actual Django project: well-deployable

I did not have to worry about the actual Django project we were shipping. We're using buildout to make our builds and deployments repeatable. On a regular ubunbu box, deploying a new site is a non-event: easy as pie and faultless, mostly. You know which "aptitude install" commands to run to get the external dependencies. And buildout grabs all the regular python packages you specified and prepares the Django instance, apache config and all the rest.

Buildout works the same on windows: it grabs all the stuff and configures everything I told it to configure. And as I created the project from our internal skeleton/code template, it came out of the box with that at-least-working apache config for windows I distilled from the last windows project. Happy.

Problem: no internet access on the server. Intranet. Well-protected. So I need to bring a CD with all the code. Easy to do with buildout if you use zc.sourcerelease. The documentation isn't clear, but once you installed zc.sourcerelease somewhere, you can call bin/buildout-source-release https://url/to/your/project/tags/1.7 buildout.cfg. So the url is your svn url. That works best in my experience. And the second parameter is the buildout config (in our case deploy.cfg as that's the production config). And give it a -n project-1.7 parameter to get a better name than 1.7.

The result is: project-1.7.tgz. Your complete buildout including all the eggs and downloads it would otherwise have to grab from the internet. Extract it and run python install.py (instead of the normal python bootstrap.py and bin/buildout calls). Ready. Wow.

Installing, installing

Installing on windows starts easy. I looked at the geodjango install page and ask a colleague for the set of .exe and .zip files from his last windows deployment. Ok: python-2.6.4.exe and double-click. Same for apache. And matplotlib. And numpy. And an oracle connector.

And lxml: ouch, the latest version doesn't have a python2.6 installer! I found a slightly older one that did. Another problem solved.

Some stuff is different. Some oracle "instantclient" (a set of DLLs) must be placed in some directory that you then add to the system path. Same with mapnik.

And the apache mod_wsgi module is shipped as a .so file that you have to copy somewhere deep in apache's directory structure. Manually.

Ok, fine so far. Let's fire up apache. Boom. Ok, let's try django itself on the terminal. Boom. Uh oh...

cx_Oracle DLL problem

When firing up django, it would complain that it couldn't load the oracle database connector. Some dll had a hickup. Dependencywalker is a great, simple tool that inspects a DLL for you and tells you which DLLs that it depends on are missing. I don't remember whether it was linkinfo.dll or msvcr90.dll or msvcr71.dll anymore. The last two are redistributable files from microsoft visual studio. You can actually grab a .exe installer for them. Only, which of them? The 2008 or the 2009 or 2010? It doesn't say which one includes which number...

I'm actually not sure anymore what solved this issue. I installed at least two of those redistributable file installers. And I swapped around between an oracle 10 and 11 connector and matching "instantclient" libraries. I think the 11 one worked in the end. I sure hope it goes well on the client's server!

Oracle/geodjango GEOS DLL problem

Geodjango depends on the GEOS library. Also in combination with oracle. When trying to connect to the spatial database, I'd get an ImproperlyConfigured: Could not import user-defined GEOMETRY_BACKEND "geos". error. Well, that's just another library to install, right?

Problem: there's no windows installer for geos. Only compile instructions. What? Can't be true. So I asked around on two mailinglists and on irc and got great help. What the situation is?

  • Use the geodjango windows installer. It installs django, but you can ignore that (just zap the django.pth from your site-packages directory). It also installs the Proj projection files (ok, I already installed those by hand, but that's fine anyway). It installs gdal (good thing as I almost forgot that one). The gdal install also included some geos .so. Unfortunately, that's not enough.
  • Incredible when you want to use a remote oracle database: you need to install postgresql and postgis. Say again? Yes, to get the geos libraries you need to install postgis. You can probably copy/paste some DLLs around, but I thought it safer to just install the whole thing cleanly.

Matplotlib DLL problem

Next DLL problem on the list: matplotlib. After finally getting over the oracle + geo database hurdle, I could really start up Django. Only to have it die almost instantly once a graph had to be shown. Without a clear error message:

from matplotlib._path import affine_transform
ImportError: DLL load failed: The specified module could not be found

Well, which DLL?!? I had to go into the source code to get it to print a better error message. The ImportError exception was silently swallowed and a different import tried which also failed with a different error... The real error message gave me more information about which DLL was missing.

Dependencywalker to the rescue again. This DLL was missing a linkinfo.dll. Another one of those microsoft DLLs. Not available, though. And googling it gives a metric ton of virus links and warnings.

Some more googling showed that more recent matplotlib releases were compiled in a more reliable way with ms visual studio instead of the mingw compiler. So I grabbed a 1.0.0 release which even worked.

PIL problems

The python imaging library isn't clear on how it is supposed to be imported. "import Image", "import PIL.Image", "from PIL import Image". You see all of them. But that's apparently a problem on windows!

Python (and even apache plus mod_wsgi) completely died with a AccessInit: hash collision: 3 for both 1 and 1 error message. It turned out to be PIL. "Image" and "PIL.Image" are the same object, but with a different name. I have no idea why this would be different between windows and unix, but whatever.

The solution is to put this hack in your django's settings.py (see the comments on that article linked above:

import sys
import PIL.Image
sys.modules['Image'] = PIL.Image

Conclusion

  • Even when starting with a brand new windows install, you can still run into DLL hell.
  • Expect unexpected weirdness.
  • Reliably compiling for windows is hard, especially when you depend on a lot of things.
  • Don't complain toooooooo hard. Get to work. It needs doing, sadly.
  • Buildout and zc.sourcerelease work just fine. They're a island of tranquility in a sea of muck.

August 17, 2010

Announcing Read The Docs

From Eric Holscher on August 17, 2010 03:21 AM

This year's Django Dash just came to an end, and I'm really excited about the project that we built. I'm sure the other teams are feeling just as stoked, because there is an amazing amount of awesome work that was done in the last 48 hours.

I'm really happy with the work we did, I think it is close to production quality. Last years project didn't even get a blog post because it was "almost done". This year I'm putting it out there because I think it is genuinely useful and pretty damn awesome.

Our team consisted of, besides myself, Charlie and Bobby who are an amazing dev and designer, respectively.

Read The Docs

Our Django Dash project solves a real problem in the Open Source community I think. I have a love affair with Sphinx, and it's really started to catch on as a cross-platform documentation tool. Our idea was to provide hosting for people's documentation, in a central place with nice tools built around it.

I know whenever I create a project, I have this moment where I think about documentation, and hosting it is a problem that is hard to solve. I currently have a cron job running on my server pulling my docs every 5 minutes, this is no way to host documentation.

We created Read The Docs to solve this problem. It will automatically build your documentation for you, if you put in a github or bitbucket URL. You can also use it to create Sphinx documentation on the site with some basic editing tools that we created.

Cool Features

Open Source

This year's Django Dash required submissions to be open source, which I think is great. It gets a lot of knowledge from smart people into the world, and I think focuses the projects more on community problems. Feel free to take a look (again, written in 48 hours, be kind) our source and contribute, or laugh at us :)

Build your own docs

If you have a really basic project that doesn't need a whole bunch of documentation, you can use our documentation builder to create the docs right on our site. We will host them for you and automatically rebuild them whenever you update them. This solves a problem for people with simple documentation needs. My teammate Charlie made some docs for his own project on the site

Host existing docs

If you already have documentation in your project, but are hosting it in a crappy way, let us host it for you. We will update it in real time (see below) whenever your update it, and we have lots of neat features planned that will make it silly not to use our hosting. For example, here is a mirror of pip's docs.

Web Hooks

Web hooks are pretty amazing, and help to turn the web into a push instead of pull platform. We have support for hitting a URL whenever you commit to your project and we will try and rebuild your docs. This only rebuilds them if something has changed, so it is cheap on the server side. As anyone who has worked with push knows, pushing a doc update to your repo and watching it get updated within seconds is an awesome feeling. If you're on github, simply put http://readthedocs.org/github as a post-commit hook on your project. Otherwise your project detail page has your post-commit hook on it.

Bookmarking

I have a problem with Django's documentation, and it's that it is so big, I often find a page and then forget where I was when I need that information again. We added simple bookmarking so that you can find pages that you were on before. Check out the recently bookmarked pages

View Tracking

Another feature is that we track which doc pages are viewed the most. This is a great hueristic to what pages are important and useful, and I think will be an interesting UI feature once we hopefully get more and bigger projects on the site. Not suprisingly, the project's own docs are currently the most viewed

Planned cool features

Full Text Search

Once we have a critical mass of documentation, being able to search across all of it, based on tags and other attributes will be a killer feature. I'm really excited about the possiblities here, and think this will be the first big new feature that we will implement.

Quick browser editing

When I find a typo in your documentation, there should be a 1-click process to be able to make an edit and send you a diff. We want to be able to support this with a nice UI. I think it will really increase the quality of documentation if there is a super easy way to update and edit existing documentation from it's rendered interface.

Mobile View

With we will be able to make a mobile theme that we can serve based on user agent. This will be another killer feature to hosting your docs on our site, because you'll get a kick ass mobile version for free.

Future

I really hope that this utility becomes used by the community, because I think it is needed. I understand that large projects would want to and should host their own documentation, but for the 90% of projects that are small, I think this is a great solution.

Lessons Learned From The Dash: Easy Django Deployment

From Eric Holscher on August 17, 2010 02:21 AM

This is going to be a series of posts that talk about what I learned from the Django Dash. I think it's a really fun competetion that is also a great learning experience. I hope that this series catch on, and other people write about some of the things that they learned in the Django Dash.

What I learned

The thing that I learned about during my dash project was the awesomeness that is Gunicorn. It is an awesome HTTP server that I think has really solved the "how do I deploy Django" problem.

Here are the steps involved in deploying a site using the gunicorn:

  • pip install gunicorn
  • Add 'gunicorn' to your installed apps
  • ./manage.py run_gunicorn -b 127.0.0.1:1337 --daemon

It really is that simple. Gunicorn is the fastest way to having a production ready web server serving your site that I've found in the Django realm. However, Gunicorn by itself isn't production ready. It is recommended to deploy something in front of it. We used Nginx, which is another super simple web server.

Here is basically the simplest possible configuration of nginx that will work for your gunicorn backend server.

server {
        listen 80;
        server_name  example.com;
        access_log  /var/log/nginx/example.log;

        location / {
                proxy_pass   http://127.0.0.1:1337;
        }
}

After you restart Nginx, you should be able to hit your server at port 80 and have it be serving your Django web app. This allowed us to get our application into production during the dash in about 10 minutes, which was a great time saver.

I'd be curious if people have had any trouble with Gunicorn in deployment, because as far as I've seen its production ready. As a "first Django deployment" set up I think it's hard to beat. I've also noticed that is uses significantly less RAM than an Apache/mod_wsgi set up (I know this can be configured away, but by default it's much better). This is great for the memory constrained deployment platforms a lot of us are running on.

August 14, 2010

Crear gráficas estadísticas con Django y Google Visualization API

From Django en Español on August 14, 2010 11:16 PM

Vamos a ver cómo combinar django-qsstats para generar estadísticas y Google Visualization API para representarlas gráficamente. django-qsstats permite obtener estadísticas agregadas sobre querysets de forma sencilla. Vamos a crear una vista que nos dé el número de usuarios registrados cada día para los últimos 14 días...

August 13, 2010

Descargar y guardar archivos por su URL

From Django en Español on August 13, 2010 11:15 PM

Vamos a ver cómo guardar un archivo a partir de su URL. Este método nos permitirá descargar y guardar cualquier archivo de Internet y nos servirá tanto para campos ImageField como FileField...

Aplicar filtros a annotate() al usar Count()

From Django en Español on August 13, 2010 08:15 PM

A veces queremos utilizar la función de aggregación annotate() y aplicar filtros al modelo que se encuentra dentro de la misma, pero no es posible hacerlo con el ORM de Django actualmente. En este caso la solución pasa por usar la función extra() del ORM de Django para incluir SQL propio...

South 0.7.2

From Andrew Godwin on August 13, 2010 04:32 PM

After a slightly too long hiatus, I've released the second bugfix release of the 0.7 series.There's not a whole lot new and exciting in this one, just lots of small fixes - bugfixes, removal of some unused variables, a few more command line arguments where they should have been already, that sort of thing. You can read more in the release notes. In other news, I'll be giving a talk at DjangoCon US 2010 which, for once, isn't about South, but instead covers a more broad spectrum of databases and schemas, in particular encompassing schemaless databases (which do still need migrations, in some form), so if you're going to be at the conference, come along, or just catch me outside of a session if you want to talk. I apologise for the slightly slow rate of work on South as of late, but I'm in the middle of moving cities. After next week, I'll be settled in London, and hopefully work will progress faster. I'll also be looking for contract/consulting work after next week, so if you'd like to hire me, see my 'corporate site'.

August 12, 2010

Caktus Consulting Group Welcomes Lead Developer Karen Tracey

From Caktus Consulting Group on August 12, 2010 08:10 PM

I’m delighted to welcome Karey Tracey to our growing team of web developers here at Caktus. Karen is a core developer of the Django web framework and specializes in the development and testing of applications for the web. She is also the author of Django 1.1 Testing and Debugging, published by Packt Publishing in April, 2010.

Caktus is a seasoned team of web developers that creates interactive, content-rich sites and applications with the Django web framework. We put a strong emphasis on best practices, employ an agile method, and also actively participate in the Django development community.

For more information about Caktus and our team, check out our newly updated team page!

Python Edinburgh

From Dougal Matthews on August 12, 2010 11:06 AM

Python Edinburgh is user group for Pythonistas in (surprisingly) Edinburgh. This is a kick start of a group that died out unfortunately and only met once this year.

Find out more at the new (but basic) website and follow us on twitter or join the mailing list.

The group is going to be meeting on the 4th Tuesday of each month. The next being on the 24th August at Bert's Bar.

Hope to see some of you there - please let us know if your coming so we can get the numbers right.

August 11, 2010

Conference Fun

From Eric Holscher on August 11, 2010 05:21 AM

It's conference season and I realized that I haven't talked about any of the ones that I've been to or am going to, so I figured it would be a good post.

Djangocon US

Djangocon US is just around the corner, and I'm getting excited about going back to Portland for another year. The conference is being held in the second week of September (7-9th) with sprints afterwards. I will be speaking again at Djangocon, talking about the awesome applications that the Python community has put together. Modeled and named after my blog series "Large Problems, Mostly Solved", and the full desciption is available on the Djangocon site. The full schedule of speakers has a little bit for everyone.

My friend Danny has been highlighting some of the talks and other reasons he's also excited about going this year on his blog. It's a great oppertunity to meet up with all the members of the community and see them present ideas that they have been working on, and it really makes you appreciate the scope of the Django world. I highly recommend going for anyone who is interested in or doing Django developement.

Strange Loop

Strange Loop is a conference right around the corner in St Louis, Mo. It is a general developer conference, without any kind of specific focus besides being awesome. The keynotes that I'm super excited about are Douglas Crockford (Author of Javascript: The Good Parts) and Guy Steele (whos Growing a Language talk is excellent).

I think a crew of us from Lawrence are going to go, and the full speaker list doesn't fail to disappoint. I think it will be a really interesting experience, with lots of different communities coming together.

Djangocon EU

Djangocon EU is the Django communities conference in Europe. It was held in Berlin this year, and was a smashing success. It was the first conference run by the community, and I think the organizers did a fantastic job with the food, talks, venue, and general galavanting around Berlin that ensued.

I realized that I forgot to post the slides to my talk afterwards, so I'll go ahead and post it here for posterity. It is a talk about Continuous Integration, Testing, and some of the tools and utils in that realm. If you haven't seen it before, I hope you learn something:

I know that I love software conferences, especially Open Source oriented ones (I've never been to another kind :). So I hope to see (or that I saw you) at one of these awesome events.

August 09, 2010

Django open inviter – contact importer – python

From Thierry Schellenbach on August 09, 2010 10:41 PM

Django open inviter is a python port of the PHP api client for openinviter.com’s contact importer to work with Django. I build it for our fashion community, Fashiolista.com, where it is currently in production usage and fully functional. If you are a member of Fashiolista (which I highly doubt given the different audiences) you can test it by clicking find friends in your profile.

Usage is extremly straight forward:

from django_open_inviter.open_inviter import OpenInviter
o = OpenInviter()
contacts = o.contacts('example@example.com', 'test')

Get the code here.

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google

August 08, 2010

More reasons to go to DjangoCon!

From Daniel Greenfeld on August 08, 2010 01:05 PM

I was thinking some more about DjangoCon next month.

Capoeira - I'm known for doing one-handed cartwheels. But at last year's DjangoCon I got a chance to try my hand at Capoeira. Since then I've managed to get in some actual regular Capoeira training. My hope is to tag up with Oregon Capoeira and see how my Angola matches their Regional. Even if you've never done a cartwheel, it should still be a blast to try out!

Okay, now on to listing some more talks I'm looking forward to having at the conference:

  • Why Django sucks and how we can fix it - While Django is the king of Python web framewrks, Eric Florenzano is going to slam it hard. But he isn't going to just troll the conference, he's also going to provide us some possible solutions. The best thing of all, is that since Django is open source, we can all contribute to make it better!
  • Pony Pwning - Django is a nicely secure framework thanks to the security focus of the community, but Adam Baldwin shows us how as developers we can make compromising mistakes. If you want to avoid being caught with your pants down on a day where you didn't wear clean underwear, go to this talk!
  • State of Pinax - Pinax is a platform for rapidly building websites in Django that I've made contributions to off-and-on since about December 2008. Heck, my talks and tutorial at DjangoCon and Pycon were about Pinax! Anyway, Brian Rosner, arguably the second tallest man in the Pinax community is going to tell us the past, present, and future of this incredible tool set.
  • From Slice to Site: Django in 30 minutes - Tired of hearing about how its easier to deploy a <insert-competing-but-sucky-language> site? Katie Cunningham of NASA SMD fame is going to show you how you can do it fast and easy. A great beginner talk from someone skilled at speaking and educating!
  • How to sell Django - So your trying to convince the customer/boss that Django is the way to go. But they read somewhere that XML + Shell scripts is how to build an application used by millions or talk to a well-dressed salesperson selling antiquated/broken technology. How do you get past this issue? Well, I'm hosting a panel on how to convince the people who sign the checks that Django is the way to go. I've got Steve Holden, James Tauber, and either Jacob Kaplan-Moss or Frank Wiles, plus another person to discuss this on a panel.
  • Large Problems in Django, Mostly Solved - Unfortunately scheduled at the same time as my own talk, my good friend Eric Holscher is giving a talk on some of the best applications written for Django. These are tools that solve a lot of problems for you. Ever wonder what is the best REST application? How to do proper searches? Migrate data? Also, want to make your pet application endorsed by the Django ecosphere? Then attend this talk! 

Getting excited about DjangoCon US!

From Daniel Greenfeld on August 08, 2010 12:19 PM

Djangocon starts in just a month. I'm looking forward to this event because of so many reasons. Lets go over some of them!

Friends - I'll get to meet with old friends and make new ones. Rather than list names I'm going to mark a sheet of paper with the alphabet and check off letters as I meet/make a friend with the first name that matches an unmarked letter. Which makes me wonder if their is a Django app for that in the making...

Portland - Great food and awesome beer at cheap prices. The wonderful thing about Portland is that the base ingredients are really good. I found I liked the simpler/cheaper things there more than fancy foods. The food carts alone are worth visiting the city.

Oregon State - It is a beautiful state and since words can't do justice here is an image:

Me at Mutnomah falls!
 The conference schedule rocks! - The talks they lined up all look really good. They range from the basics to the advanced, and include things that go beyond the technical. Some of my favorite picks from just the first day:

Thats just four great talks described. And its just the tip of the iceberg! This is going to be great!

Edit: Fixed a geography mistake. I am so embarrassed.

Using oEmbed in Django

From Piotr Maliński on August 08, 2010 08:35 AM

Replacing links to multimedia websites with nice embedded players, thumbs and other content using django-oembed.

August 06, 2010

Merengue: el nuevo y sorprendente CMS basado en Django

From Django en Español on August 06, 2010 10:15 PM

Django es una excelente plataforma de desarrollo web, con una enorme comunidad de desarrolladores, siendo el framework web Python más popular. Sin embargo, Django no dispone de un CMS tan completo como otros muchos existentes en el mercado (Drupal, Plone, ModX, etc.) Merengue es un proyecto financiado por la Universidad de Málaga que pretende cubrir ese hueco existente, y que ha sido desarrollado reutilizando multitud de código proveniente de otros proyectos. También se ha reaprovechado el gran conocimiento previo que se poseía de otros CMSs existentes, principalmente Plone y Drupal, así como otras soluciones basadas en Django (Pinax, django-cms, feincms, LFC, etc.).

The kind of comments that will get ignored on Django's Trac

From Luke Plant on August 06, 2010 05:53 PM

Participating in any Open Source project can be frustrating if things do not move along as quickly as you would like. Django is a fairly popular project, whose developers are volunteers with limited resources, and with a pretty big commitment to stability and backwards compatibility, so there will always be people who get frustrated. This blog post is intended to help people in that situation be helpful, and to not actually make things worse.

To provide an example, a comment was posted on a bug 8426 yesterday. No offence to the poster — it's actually a fairly minor example — but it illustrates why comments like that one will be ignored, and may even work in the opposite direction:

  1. The bug is definitely in the category of minor, in the grand scheme of things, since it is only about how help text looks in forms, and the ability to style it — which you already have complete control of by subclassing Form. It is in fact a feature request. But the comment makes it sound like a dire problem. In fact, adding “I need it die hard!” just makes the commenter sound unreasonable.

  2. The previous comment had actually explained what needs to be done to move things on, but the commenter hasn't shown any evidence that they have read it. They want other people to invest time, but are not willing to invest time themselves.

  3. The main pain point from the commenter's point of view is probably the need to patch Django to include this fix. That doesn't really sway Django developers, for the following reasons:

    • For any bug/feature, it is always possible that someone may say it is a ‘must have’ — we quite understand that a customer may simply require certain behaviour. But this doesn't make it high priority for us — if it did, everything would be high priority, which makes ‘high priority’ meaningless.

    • If the main pain point is having to patch Django, that is true for any fix that is a ‘must have’. There are much bigger pain points with some bugs — like the lack of any available patch, or the need for a design decision to address the problem, where the design decision will make an impact on other areas of your code.

    • There are tools that make it easier to apply patches. I often run Django projects from patched versions of Django. The way I have done this is to fork the Mercurial BitBucket Django mirror, and add my own fixes, usually into named branches. Since I'm a committer, these usually get merged back into Django (though occasionally they are just thrown away). Merging from Django trunk is then extremely easy.

      I know that some people don't know how to do these things, and many people will wait for a release for everything. But if you can wait for a release, or if you can't learn the relatively simple tools that would allow you to maintain a fork, it can't be that high priority, can it?

      For reference, making your own Mercurial branch for this, assuming a clean patch and a Linux/Unix system, is as simple as:

      $ hg clone http://bitbucket.org/mirror/django
      $ cd django
      $ hg branch my_fixes
      $ patch -p1 < /path/to/correct/8426.diff
      $ hg commit -m "helptext CSS class for help text in forms"
      

      To pull in new changes from trunk into your repository and branch:

      $ hg pull && hg merge -r default && hg commit -m "Merged from trunk"
      

So, all in all, adding a comment like the above expresses frustration, but doesn't persuade us that it is high priority, and doesn't help us move forward. With this bug, it was on Version1.1Features as a 'small bug' that doesn't really need discussion. So, a helpful thing to post would be something like:

It's now post 1.2 so we are out of feature freeze, and this was procedurally accepted for 1.1 as a small bug, so I don't think it needs discussion. I've checked the patch, and it still applies to trunk, and the test suite passes with no failures. Is there any reason this can't go straight in?

As it happens, the latest patch did not apply cleanly to trunk, I had to manually apply some bits.

In addition, there are other things I would check as a committer. For instance, backwards compatibility: is this going to interfere with styling in the admin, or the other way around? Answer: probably not, because the admin doesn't use those methods to render forms, and other pages are unlikely to include admin CSS files. However, the admin CSS does use a 'help' class — shouldn't we stick with that name rather than add another for exactly the same use case? Or, is a 'help' class too generic, and will it end up clashing with existing HTML on people's pages? (In the end, I decided to stay with 'helptext': 1) for that latter reason, 2) because that's how it was in the accepted ticket, and 3) because that will make people who have already patched their source happy.)

These are all things that need to be checked, and which take time, and someone who was really desperate would have done some of these things. I have now committed it, mainly to avoid the complaint “it took longer to explain why you didn't commit it than it would have taken to commit”, but it should be noted that comments like the one above usually have the opposite effect on committers.

Anyway, I hope this post helps give some insight into why some comments are less than helpful, and what are the more constructive avenues. I won't claim that we always have the excuses laid out here — there are many more important tickets that have languished for way too long for no good reason. But the first thing to check is that you have followed our contributing guidelines and made things as easy as possible for the core developers.

Code coverage analysis in Django

From Marcin Mierzejewski on August 06, 2010 05:52 AM

August 05, 2010

Getting Started with Django Screencast Series

From Django Dose on August 05, 2010 01:43 PM

If you're an active Forrst.com then user you'd know Kenneth Love is active in almost all the Django discussions going on in the Forrst community. He's recently taken that discussion to his blog and started a screencast series titled "Getting Started with Django". In his first of currently 3 episodes he covers getting yourself setup locally, then working with app settings, diving into models and introduces South. Go take a peek.

August 04, 2010

New checkoutmanager releases

From Reinout van Rees on August 04, 2010 06:47 PM

I released checkoutmanager three days ago. An easy tool for managing all your svn/hg/bzr/git checkouts. After mentioning it on my blog, I got quite some positive reactions, bug reports and feature requests. Yeah! Nice to see that it's useful!

Bug reports and feature requests mean I got to make some changes and fixes and new releases, of course :-) So 1.1 and 1.2 are out. Some of the bigger changes:

  • It works on windows now.
  • Better initial experience by printing the sample config file if you don't have it yet. And updated the documentation to make it clearer. (Seeing a colleage install it is often instructional!)
  • Added a -v/--verbose option that prints the commands and the directory where you execute them. Handy if you want to know exactly what's happening.
  • I'm checking the exit code of all executed commands. When there's an error, the command and working directory are printed and also the output. And the script stops right away. Errors (mostly svn conflict errors) could go unnoticed otherwise).