Aggregated blog posts about Django, updated every hour

January 27, 2012

Read the Docs

From Wraithan on January 27, 2012 01:19 AM

Over the last few months I've really started to love Read the Docs. Several of my favorite projects are hosted on there. Projects such as Fabric, Requests, and Celery are all hosted on there. The list keeps growing, which is great. Read the Docs is a fantastic project that is promoting and unifying documentation even across language boundaries, both human and programming.

I got involved with Read the Docs during the sprints of DjangoCon 2011 as I have mentioned before on my blog. So my opinion isn't entirely unbiased, but I wouldn't have gotten involved if I didn't believe in it and what goals it is pressing towards. I strongly urge others to also contribute, which can come in many forms, such as spreading the word on using Read the Docs, writing docs and posting them on there, filing bug reports on things you find wrong or feature requests for things you think could be done better, or finally by contributing your coding or design skills.

Something that is pretty interesting is there have been a number of projects that aren't software libraries that have found their way on to Read the Docs. There have been a few books written using Sphinx and then posted on Read the Docs. Notes from various talks and conferences. And recently I stumbled upon a resume that is hosted on there and it got me thinking that I should do the same.

My resume is now hosted on there, though a warning, it is a work in progress. It looks really nice though, and I have access to multiple file types of it that sphinx and Read the Docs have generated, including a PDF. I highly recommend it to anyone that wants to write their resume, have it stand out a bit, but still look really nice.

Basically, this post is just to talk about how much I have come to depend on Read the Docs and to also to encourage people to use and help out in various ways. I love this project and if have ever come across horrible documentation, so should you.

January 25, 2012

How to justify attending PyCon sprints

From Daniel Greenfeld on January 25, 2012 01:36 PM

Sad that the PyCon sprints fall on business days? Wishing you could stay but the boss/client won't let you and demands you back so 'you can work'? This is how you make it so that the sprints are something your management is demanding you attend every sprint ever.
  1. Make it foremost in your mind that the wonderful thing about the PyCon sprints is that the odds are that anyone who knows anything about whatever you are doing in Python will be there.
  2. Write up a list of the things that you are finding challenging, hard, or impossible to do with Python.
  3. Now go to the boss and say something like:
    "Because the experts and leaders of the open source tools we are using are going to be there, I want to attend PyCon sprints. All my time at the sprints will be focused on sitting around them and working on our tools. I'll focus on things that directly impact our agency / company / organization, specifically things I wrote down on this list."
  4. If the boss says,
    "Why not just use IRC or email?"
    Then you say something like,
    "Well, IRC/email is not the same as sitting next to these people. I'll be so much more productive there!"

Rinse and repeat.

Then, when you attend the PyCon sprints, follow through on what you said you were going to do. Sure, it might be more fun to work on project 'spam' even though your company uses project 'lumberjack', but if you prove how much you got done during the sprints, next year the boss will be much more encouraging. Even if a good boss says to go do what you want, at least spend some time sprinting on work related technology.

Also, once you get approval to go, consider sending your boss to this old rant of mine.

Don't forget to register for PyCon! Early bird rates end today (January 25th, 2012) which means today is the last day to get involved in the extremely unofficial PyCon Early Birds program!

Join the PyCon Early Birds program!

From Daniel Greenfeld on January 25, 2012 09:56 AM

First off, I want to say that me and my fiancee will be attending PyCon US this year! Hooray! Can't wait to see old friends and make new ones. I'll be chairing one of the Panels at the PyWeb Summit on March 8th. We're absolutely delighted to see all the great talks, hang out in the hallway, and just be in the middle of Python for well over a week.

Now on to the extremely unofficial PyCon Early Birds program!

PyCon early registration ends on January 25th. If you register at the early bird rate that gets you the benefit of joining the elite PyCon Early Birds group. Being a member of the PyCon Early Birds gets you all sorts of incredible rewards and benefits.
  • Most importantly, you get some serious bragging rights.
  • A custom ribbon that says 'Early Bird' that you get to attach to your conference badge.
  • A discounted rate from the regular ticket rate as according to the registration page.
  • The confidence of knowing you have a ticket before they sell out.
  • A tasty and rather edible store-bought cookie provided by myself and Audrey Roy.
  • If the PyCon Early Birds program gets enough members, I'm going to challenge PyCon chair Jesse Noller to stump me with Yoga poses! There's no way he'll even consider accepting a challenge like this unless the PyCon Early Birds membership roster is big enough. So join and help me find out if his Bikram will beat my Capoeira!
  • Other incredible things that are in the works!

Of course, PyCon has tons of other reasons to sign up besides the PyCon Early Birds program. Amazing tutorials, talks, and sprints, plus great hallway tracks, a vendor room filled with great schwag, poster sessions, and startup row. Sponsorship levels are unbelievably high, and since the event is non-profit that means the money just goes right back into the community - starting with PyCon itself. This year is going to be AWESOME!!!

So what are you waiting for? Sign up for the Pycon Early Birds before it's too late!

January 24, 2012

Forms Part 3: Model Based Forms

From GoDjango on January 24, 2012 10:30 PM

Model forms really helps to show the power behind Django. When all it takes is to define a model and then attach the model object to a form object and poof you have a form, I call that winning. In this video we will go over using your models you already have to create forms to save you time, heartache, and code.
Watch Now...

January 23, 2012

Django Generic Class Based View Tip

From Wraithan on January 23, 2012 02:06 AM

Those reading this via a feed reader will have to view the page on my blog as I am using embedded gists. I'll find a solution for that in the future.

So say you have a base template that looks something like:
And a template that looks like either of these:
It used to be that you could write something like this:
But generic function based views are deprecated and the world is being strongly urged to move to generic calls based views. If you would like to get extra_content working with the direct_to_template replacement TemplateView, you can use a view like the following:
And a urls.py like the following using it:
The code used in this blog post can be found in this gist.

January 22, 2012

Why Read the Docs matters

From Eric Holscher on January 22, 2012 11:22 PM

Documenting projects is hard, hosting them shouldn't be. Read the Docs was created to make hosting documentation simple. I think that we have solved this problem well, but now we need to start thinking about the larger picture.

Along with hosting, Read the Docs was created with 2 other main goals. One was to encourage people to write documentation, by removing the barrier of entry of hosting. The other was to create a central platform for people to find documentation. Having a shared platform for all documentation allows for innovation at the platform level, allowing work to be done once and benefit everyone. Having run the site for over a year now, I think there is a third thing that we should be striving for. That is to make the quality of documentation better.

I think that we can help a documentation culture flourish within the open source world. Django is a shining example of what a project with great documentation can do, and it has a community that values docs more than the norm. I think we can help spread this culture throughout the Python world, and beyond. This has already started, and I want to think about how something like RTD can help.

What we can do to help

I think that having a guide for writing useful documentation would be a great step towards helping people along the path of documentation enlightenment. Jacob Kaplan-Moss has started down this road with his blog series and Pycon 2011 talk on this subject. I think that we could start by collecting these into a section of the site.

We could build on top of that great start with simple guides for how to get started with Sphinx, best practices for documentation, and providing a general place to learn more about how to write good documentation. Since we host a lot of documentation, we could point to live examples of techniques, and provide helpers for people to enable the techniques.

I have started a reStructedText Philosophy document that is meant to help people understand the ideas behind how reST works, so that it isn't as mystifying. This reST cheatsteet also appears to have similar goals. These are a very basic start, and I think some more along these lines would really help a lot of people get over the barrier to starting and continuing to write good documentation.

I think that we could also help create contributors to projects, if we could find an easy way to provide patches to documentation. If you could go to the project documentation, and fix small typos, or help add a paragraph in the tutorial, it would lower the bar to helping.

However, it isn't a wiki. These changes would be represented to the project author as pull requests in their VCS, and they would still be responsible for tending the garden. This gets rid of the "Just Edit The Wiki" solution of documentation, and also helps new contributors get started in an easier fashion.

The Plone community has built a proof of concept, linking to Github's edit pages for the current document. I think we can integrate this at the platform level, and make it available to everyone.

Want to help?

Read the Docs is open source. You can help by writing docs for the site, writing code for the site, or just writing documentation in general. People can also help just by using the site, and reporting bugs. Telling us how to make the site better helps everyone in the long run. Come join us on Freenode in the #readthedocs channel as well.

Another area that we're hurting is in the design front. We have been adding features over time, and the design of the site is getting a bit strained. Having someone with a good sense of design help re-think and re-architect some of the features and ideas that we've been working on I think would help a lot.

A lot of the RTD contributors will be at Pycon 2012, where we will be having a sprint on the site. If you want to get started contributing, that is a great place to come and get started.

Scalable Vector Graphics (SVG) - Jeroen Dijkmeijer

From Reinout van Rees on January 22, 2012 10:02 PM

(Talk at the 2012-01-18 Dutch Django meeting)

Jeroen actually has zero Django experience, but he does nice things :-) That website of his is actually made with SVG.

Update 2012-01-23: his presentation is online at
http://www.iscriptdesign.com/svgopen/presentation/ .

SVG is more than 10 years old, Adobe published the first version in 1999. IE6 supported it with an adobe plugin, most other browsers started adopting it in 2003. IE9 fully supports it, btw. Nice: since 2011 (corrected, I said 2001...) it is even part of the html5 specification.

Scalable vector images are, well, scalable. Despite the screen size, you won't see the pixels as in an upscaled pixel image.

SVG as a format is just XML. You can draw lines, arcs, circles, boxes and so on on a canvas. The output can be styled with css, just like html. It are DOM elements. And you can manipulate the drawn SVG elements with javascript, too.

What you use often is pen movements. M or m for move, L or l to draw a line. The lower case is relative to the current location, the upper case for absolute location. M 10 10 l 10 20 l 20 20 l 20 10 l 10 10 would draw a rectangle. For more elaborate forms you can use inkscape, an open source SVG editor.

Nice: events on SVG objects can be processed by Javascript. You can also do animations with SVG. There are three ways:

  • CSS. Very promising, but browser support is spotty and it is still not really completely standardized.
  • SMIL. Synchronised multimedia language. It came from CWI, where Python also came from :-) It ran out of steam early on because of flash and so, but it is now back inside SVG.
  • With javascript. "Simple" DOM manipulation.

As browser compatibility is still a bit of a problem, you're probably best off by using libraries like raphaeljs if you want to use SVG. (Personal note: I used raphaeljs with success in a small Django project last week).

He showed a couple of demos. What was particularly good was the easy and on-the-fly integration between forms and the picture (like for a custom table that can hover over the couch). Put in a different height of the couch and the table would adjust. Good javascript integration.

He uses it for making designs for feeding them into CNC milling machines like they have at the fab lab in The Waag in Amsterdam.

Website changes: sphinx code, layout, twitter bootstrap, less

From Reinout van Rees on January 22, 2012 09:49 PM

I've made a couple of changes to my website (and thus weblog):

  • I moved from svn to git.
  • I made my weblog-related sphinx customizations public.
  • I'm using twitter bootstrap as the css framework instead of the yahoo one I've been using before.
  • I changed the layout. Bigger font, less clutter. More bare-bones. Most of the looks (font type, headers) is from the bootstrap css, but the ordering is mine. I especially like the bigger font.

Svn to git

I'm getting used to git at the office. And my website was in svn. The svn is on me and my brother's server. And we're going to re-build that server in the near future. And not having to set up svn there saves quite some time.

So I moved part of the code to github and the actual content to a simple directory in our own server. Git works just fine with a repository URL like ssh://vanrees.org/~/git/websitecontent. Just init a repo in ~/git/websitecontent on the server and off you go.

Sphinx as weblog: code is now available (somewhat)

My site is static html now for two years or so. Generated with http://sphinx.pocoo.org/. No Plone (which it was when I worked with Plone) and no Django (which I'd like, as I work with it, but my current text-only setup works real fine and Sphinx is an essential Django documentation ingredient, too).

I also make my weblog with Sphinx. I need a couple of custom restructuredtext tags for that (like ..tags:: for handling tags). I basically point a script at a sphinx directory that contains my entries and it generates index.txt files for all the directories with a list of entries. And it generates tags/TAGNAME.txt files for the tags. Afterwards, I just run Sphinx and everything gets generated OK.

I got some questions in the past about my code, but never followed up by publishing my terribly Reinout-specific code. It is still Reinout-specific, but I've published it now. It might provide some ideas. The full code is at https://github.com/reinout/reinout.vanrees.org . Again: this is not something you can pip install. Best you can do is borrow and copy at the moment. (Threat the Python code in there as public domain).

Most of interest are probably https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/rst.py which provides the custom restructuredtext (and thus sphinx) tags. And https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/weblog.py which creates all the index and tags files.

It is not well-documented and quite specific to my case. But there are some docstrings and the code is nicely PEP8 compliant :-) If you need any info, feel free to contact me.

Twitter bootstrap css

I have to re-visit the layout of the websites we make at my work (Nelen & Schuurmans). One of the things we want is to have a more attractive layout. And after some browsing, twitter bootstrap struck me as a good basis.

And, as I do often, I first experiment on myself. So I set out to change the layout of my own website by using twitter bootstrap. I'm not even using half of bootstrap's possibilities, but I'm quite happy with the result. The basic typography is fine. There are some handy helpers for common things like labels (which I used to liven up the tags below the weblog headers on my weblog). And I quite like the simple header at the top of the page: just a simple bar.

And there's another good thing about twitter bootstrap: less. See below.

Less

What coffeescript is for javascript, less is for css. A simpler syntax that helps you write better and less duplicated css.

I like it! Take for example the headings of this entry. Hover over one and you'll see a paragraph token at the end. Sphinx adds this so you can link to individual sections in a bigger file. Sphinx also has some css that hides the paragraph token normally and only shows it when you hover over the heading:

a.headerlink {
    visibility: hidden;
}

h1:hover > a.headerlink,
h2:hover > a.headerlink,
h3:hover > a.headerlink,
h4:hover > a.headerlink,
h5:hover > a.headerlink,
h6:hover > a.headerlink,
dt:hover > a.headerlink {
    visibility: visible;
}

I restricted that to:

.headerlink {
  visibility: hidden;
  }

h1, h2, h3, h4, h5, h6, dt {
  &:hover {
    .headerlink {
      visibility: visible;
      }
    }
  }

It looks just like css, as you see in that first .headerlink rule. The second one has the special less stuff. The ampersand means an extra condition, so the &:hover effectively adds :hover to each of the h1..``dt``. This really shines if you have a couple of other extra conditions on the same set of elements.

Anyway, head over to http://lesscss.org/ where they have a better explanation.

Closing off

I'm happy with the change. One extra change that I didn't mention yet: I've removed deliverance from the mix. Deliverance is a tool that I used to change the layout of a couple of inputs (sphinx, docbook2html, latex2html) into my vanrees.org layout. It did that on the fly. But the versions kept changing and the versions I had were getting old and hard to update. And I had to run a wsgi process to run it, even though the actual website was all plain html.

I'm back to plain html now. I've implemented the layout in a Sphinx template. Simple to do, as it is just jinja2, which has the same syntax as Django. And the docbook and latex html exports are just that. I'll leave them that way. It isn't that important to have my logo on those pages :-)

One extra advantage: the site ought to be faster!

Being a Contributor

From Wraithan on January 22, 2012 02:26 PM

When I attended DjangoCon I decided I needed to work a bit less on my own projects and start putting more time into contributing to other projects. I use so many free and open libraries both in my work and my personal code, and it is time to start contributing again.

The first project I decided to try to contribute to was Read the Docs. The choice was pretty easy since I know Eric Holscher and it is a project I benefit from on a regular basis. So at the DjangoCon sprints I sat down near him and asked him if there were any particularly valuable tickets I could take and start hacking on. He gave me a list of issues and I started in.

By the end of the day I had a patch for RTD, granted it didn't fully work but it was a start. Added some unit tests and fixed up the code and soon I had a complete contribution that was added and it made me smile. And smile even more to be added to the AUTHORS file. Couple more patches later and Eric asked if I wanted to help out with the servers and get access to them. I said yes and now I am part of the RTD ops team as well as a contributor.

I used to contribute to a couple projects a few years ago. In recent history I've spent my time scratching my own itches and working on proprietary code for work. Going through the process of contributing has made me excited to do it some more on various projects. And GitHub makes it so easy to just send a pull request, comment a couple times back and forth and get patches accepted.

I am far from the first to talk about how easy it is to contribute to projects compared to the old days of sending patches to mailing lists. I just want to be another voice in the crowd, if even one person decides to contribute because of this, or it makes a person happy and continues contributing this was a success.

bpython and django

From Wraithan on January 22, 2012 02:26 PM

There are a few posts out there to get basic support for django inside of bpython but I as far as I have seen they ignore the virtualenv stuff that most of us django users do.

virtualenv is a must when doing development on django projects, especially when you have several projects going on the same machine or you want to involve other developers. It allows for a clean environment with known versions of every package that is installed inside of the virtualenv.

The first part of this is a known hack or method (take your pick) to make bpython recognize the django project.

First you need to modify your ~/.bashrc or ~/.bash_profile (system dependent as to which is used) and add the following line to it:
export PYTHONSTARTUP=~/.pythonrc

What this does is it tells python to run '~/.pythonrc' before it does anything else in the interpreter, this is a python specific thing not bpython specific thing.

Next you need to create the ~/.pythonrc file and put the following into it:
try:
from django.core.management import setup_environ
import settings
setup_environ(settings)
except:
pass

Now if we didn't care about virtualenv we'd be completely done. If you go the less strict route and don't use --no-site-packages (the switch means that system wide packages are to be ignored in the virtualenv) then you will need to edit your bpython binary, most commonly located at /usr/bin/bpython or /usr/local/bin/bpython and change the shebang from:
#!/usr/bin/python
(it may be /usr/local/bin/python)
to:
#!/usr/bin/env python

This makes bpython use the value of the environment variable python instead of using a specific binary, the environment variable gets set by virtualenv so you are good to go after this.

If you use --no-site-packages then you will have to install bpython in each virtualenv that you want to use it in.

If you have any questions toss them in the comments or find me on freenode as Wraithan.

January 20, 2012

Setting Up a ZNC IRC Bouncer

From Dustin Davis on January 20, 2012 05:22 PM

First of all, I’ll skip the discussion on what ZNC and IRC are, as you likely know if you are reading this post. I use IRC with my development team at work. It’s helpful to stay logged in 24/7 to keep up on pertinent discussions. I could stay logged in 24/7 with any IRC client at work, but some days (and perhaps some evenings) I may be working and chatting from home. To solve this problem I was using tmux and irssi running on my work computer. If I ever needed to log in from home, I would just ssh into my work computer and connect to tmux.

Since switching jobs, I got a laptop. Obviously it is not connect 24/7. So I switched to using my home computer. In this process of changing, I decided to try out ZNC. The main benefit to ZNC over IRC to me is that I can use other IRC clients that have more features. So I’m going to explain how I set up ZNC and Limechat (OS X).

I was running ZNC on my home computer – currently a Mac Pro. But I find either my 1 yr old or 4 yr old keeps getting into my office and turning off my computer. So, just to try it out, I set up a Linode running Ubuntu. Here are the step by step instructions that took me all of 10 minutes to get set up.

If you don’t have a Linode account, create one. Please feel free to use my affiliate link.

  1. Add a Linode. I selected the cheapest one, which is a Linode 512 for $19.95 per month. (I understand that paying $20 is likely a waste of money just for an IRC bouncer, but I’ll talk about this later…)
  2. Create an instance of Ubuntu on your Linode. I used Ubuntu 11.10. Make note of your new IP address for your server. You may want to create an entry in your hosts file so you can reference it by name instead of remembering the IP address.
  3. Boot your Linode
  4. ssh into your Linode
  5. Create a new user:
    • adduser <username>
  6. Create an admin group
    • addgroup admin
  7. Add yourself to the admin group (so you can use sudo)
    • adduser <username> admin
  8. Log in to your user account
    • su <username>
  9. Install ZNC
    • sudo apt-get install znc
  10. Make a ZNC config file
    • znc –makeconf
    • Here are the options I selected:
      • listen on port 6667
      • SSL? no
      • ipv6? no
      • Listen host – left blank for all
      • No global modules
      • Username and password – I used the same as my IRC nick & password so it is easy to remember
      • Blind host – left blank
      • Number of lines to buffer per channel – 5000 (why not?)
      • Keep buffers after replay? no
      • Default channel modes – went with default [+stn]
      • yes on on modules
      • IRC server: irc.freenode.net, port 6667, no password
      • I added one IRC channel (#utahdjango), I’ll join the others later
      • I added my IRC user and password.
      • Then I started IRC at the end of the script. (If you don’t, just type ‘znc’ to start the znc server)
  11. Connect to your new server. I used Limechat with the following settings:
    • Network name: Linode ZNC
    • Server: Name I mapped to my linode IP address in my hosts file. You could just enter your IP address.
    • Port: 6667
    • Server Password: passoword I entered on my IRC makeconf script
    • Nickname: irc nick
    • Login name: irc nick (remember I used the same nick & password for znc that I use for freenode)
    • Real Name: Dustin Davis ;)
    • Nickserv Passoword: same as above

That was it! I was connected and up and running just like that. Now, I’m bar far an expert on this stuff, but if you have any questions on what I did, leave a comment below.

Now, as I mentioned, paying $20 per month for an IRC bouncer doesn’t seem like a great idea. So here are some things to consider…

  • Use your home computer. If you keep your computer on 24/7 and basically have a static IP this shouldn’t be a problem. If you have a router, you will have to set up port forwarding as well.
  • If you don’t want to leave your main computer or, or use you main computer in this manner, consider buying another computer. I have a friend that uses a plug computer for this purpose. This is actually very tempting for me. Paying for a linode for 5 months is about the same as buying a $99 plug computer. You would then have a small server running 24/7 on minimal power and it could also be used for a media server by plugging in an external hard drive among other uses. Another options is Tonido Plug, but these always seem to sell out so fast.
  • User your work computer. This may not be easy. At my old job I was given a static IP and subdomain to connect to my computer.
  • If you are paying for hosting elsewhere, consider moving your hosting to your new Linode server. This would save you money from hosting elsewhere, but you should be comfortable managing your own hosting server.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

January 19, 2012

Building secure Django websites - Erik Romijn

From Reinout van Rees on January 19, 2012 04:40 PM

(Talk at the 2012-01-18 Dutch Django meeting)

Security is about three things:

  • Integrity of your data. I should not be able to change it.
  • Confidentiality: can I access what I should not be able to access?
  • Availability: can I break your website?

A well-known bad case is DigiNotar. Malware on most of their computers, including the most secure once. Their digital certificates were compromised and the company went bankrupt in three weeks time.

One thing that we have to take care of regarding Django: cookies. Cookies are stored in your browser and are linked to a website. All requests to that website get the cookies in every request. In Django, your session ID is stored in your cookie. If you get hold of the session ID, you get hold of the user's session. Django stores the actual session's data in the database, so the actual data is safe.

In Django 1.4 there are cookie-based sessions, but they're a bad idea: never use them. He doesn't know why they even build it in. UPDATE: they're not that bad, see the comments.

CSRF: cross site request forging. Tricking the browser into posting requests to other websites. http://your-router/set_password?pwd=hamster . Django protects you by adding a secred csrf_token to every POST form and checking for it. So you're safe.

XSS injection. Cross site scripting. Happens when you don't filter/escape user submitted content. So you could send a <script> tag with a script that for instance steals every cookie on your server for stealing session IDs. Common is tricking a single user into running the script. A more powerful, but rarer, way is to get a script on some website and trick everyone visiting the site...

Django is mostly safe as it escapes template tag output by default. There is a rare vulnerability if you don't put double quotes around html attributes. Everything after a space in var in <p class={{ var }}> ends up as a separate attibute...

Cookie security is pretty good in Django. 1.4 sets SESSION_COOKIE_HTTPONLY to true. You ought to do this always. And if you always use https, set CSRF_COOKIE_SECURE and SESSION_COOKIE_SECURE to true.

And... don't trust in a regex that filters dirty tags out of your user-submitted code. There are SO many tricks and ways to circumvent any regex. Use markdown or something like that.

SQL injection is common, but Django has no know exploits. It is rock solid. But there are other ways. For instance LDAP injection. Don't put an unfiltered username into your LDAP query as you can hack the query. Or direct user input into filenames. And what about shell injection? some_filename.txt;rm -rf... Shell calls are always very risky, but Python's subprocess() helps a lot.

There's a difference between validation and escaping. Validation is done as early as possible. Escaping is done as late as possible. Both serve a different purpose. As always: whitelisting is better than blacklisting.

A new option is clickjacking that overlays a transparent website over another one, stealing your form fields and clicks. It works with iframes. Django 1.4 has a new X_FRAME_OPTIONS='DENY' that works with newer browsers.

And... Don't trust browsers. If you disable something in javascript, a user can work around it, for instance. The speaker could always book rooms in school before anyone else could as the only-two-weeks-in-advance limitation was implemented in javascript, not in the backend...

Regarding tests: check login-protected pages as anonymous first. Check actions for the admin user first as anonymous, then as a regular user. Check it.

And if you run some test site that's (temporarily) on an external IP address, you can be sure someone finds it. So watch out.

If you run backups, do you also run restores? Do you keep backups with a different provider? Once a provider was hacked and its servers were wiped. Including their backups. All customer data lost.

Django: don't ever run it in DEBUG mode in production. It gives away way too much information. Another tip: name the admin url something different than /admin. it is a dead give-away that it is a Django website.

Watch out for other apps on the same server. Your Django server can be compromised by running some faulty-configured PHP code on the same server.

Always restrict database access per app.

Summary. Django helps out with CSRF, SQL injection and XSS; you need to take care of all other injection attacs (LDAP, filesystem, etc). Use the build-in Django security features. And never trust what the browser sends. And secure the web server itself as well. Make backups.

January 18, 2012

Dutch Django meeting - lightning talks

From Reinout van Rees on January 18, 2012 08:17 PM

(Short 5 minute talks at the 2012-01-18 Dutch Django meeting)

Coffeescript & Django - Roald de Vries

Coffeescript is not really a language, but it is syntactic sugar for javascript. A nicer syntax. It compiles to javascript. It uses indentation (like Python) to indicate code blocks instead of using curly braces, for instance. And the way to declare functions is easier and less verbose.

Coffeescript also allows classes, including constructors, and inheritance.

Nice: on the coffeescript site you can type in some code and get it transformed in to javascript. Handy for trying it out.

The language isn't difficult. You can learn it within the hour (he said 'a bit more than 5 minutes'). You can compile it into javascript on the server with node.js. You can also include the coffeescript.js script in your page and let the browser handle the compiling automatically (at a performance penalty, but it is easier for development).

Django-coffeescript integrates the compiling into Django. And you can even put inline coffeescript in your template. But normally you'll just point at external files.

Bonus tip: django-less. Less is css + variables + nesting + inheritance. Integrated in the same way as django-coffeescript.

One drawback of coffeescript: it makes debugging harder as you have to debug the javascript that comes out of the compiler instead of the coffeescript code you just wrote.

Django-fluent-dashboard - Diederik van der Boor

"Django-fluent-dashboard - a fresh dashboard for your admin interface" was the title.

Django is a bit technical. His thought: how to make the Django admin more usable as a CMS that's friendly? Another core thing in his thinkwork was the idea of reusable apps.

If you look at the current admin, you directly see the huge number of apps. Not nice.

Somebody made Django admin tools. Nice. A good solution. It gives the admin a nicer layout and even a menu bar that helps you find things.

The core idea: django-fluent-dashboard only modifies the first page of the admin: instead of the big list of apps you get friendly, grouped, icons.

He made a modification (or plugin, or newer version, I don't know) that looks even nicer. More dashboard-based. See the example on the documentation site.

In the end, he hopes to turn it into a full CMS. There's already a django-fluent-contents and django-fluent-pages on github. But they're separate projects. django-fluent-dashboard works just fine with Django Fiber, for instance.

Who hates stdlib's logging? - Remco Wendt

A small teaser for a bigger talk at next month's Dutch python meeting.

Logbook is nicer logging for python than the standard library's default logging module.

And for Django: whatever logging module you use, use django-sentry to get a website that collects your various site's logs. He made a version that works especially nice with logbook, as logbooks gives you all logs of a web request if something goes wrong: it doesn't only give you just the last traceback. So use debug logging liberally, it will help you debug.

30loops.net Django hosting - Christo Buschek

http://30loops.net/ is a Python and Django hosting platform, located in Amsterdam (and a bit in Berlin since he moved recently).

They have years of experience in sysadmin and hosting, so they're building yet another django/python/wsgi hosting platform. What you normally get at such a "platform as a service:

  • Pay only for what you use.
  • Scale horizontally as needed.
  • You can concentrate on developpment, we care about the rest

They differ on two major points:

  • REST interface with json. You can build your own custom tools if needed.
  • One app can have multiple environments, like development/staging/production. You can clone environments. Handy for testing and running performance tests - all without interfering with production.

They actually abstracted away quite a lot, so you can run your apps on 30loops' own hardware, but also on other cloud providers. Handy if you have to deploy to multiple physical locations.

You can access all logs and metrics with REST: you own all your data.

They're also on github: https://github.com/30loops

You can sign up for private beta now, btw.

What does is take to be a professional developer? - Nicolaas Heyning

Asking around gave the usual list of "good tools", "passion" and "emacs". He mentioned:

  • Clients. Online like freelance.nl, linkedin, network meetings, by reference. From the audience: organize stuff; run a blog.
  • Administration. Taxes, budgetting, investments, staff. From the audience: outsourcing (but be very, very careful); having a partner in the company that wants to do this.
  • Professional work. Provide documentation. Add tests. Internationalization. Data migrations. Etcetera.

Why all these questions? Reason: there is a big need for professional software developers. You see lots of mails on the mailinglist asking for personell.

Would it be a good idea to start a "django academy"? Certification? Learning about acquiring clients? Create a network for supply and demand of professional developers?

There was quite some discussion afterwards (and I didn't agree with some of the points, for the record).

Django Fiber - 11.000 downloads later - Dennis Bunskoek

Django fiber is a simple, user friendly CMS for your Django projects. It is getting bigger and bigger. 11k downloads since April 7, 2011. That's about 1250 downloads per month. That's only counted on pypi, though. Django CMS has 60k downloads since the beginning of time, as a comparison. So it is turning out to be pretty big!

What did we do since early 2011?

  • Performance and admin improvements.
  • Translations. (Via transifex, they got quite some contributed translations!)
  • Lots of spit&polish. Better WYSIWIG editor.

And... they build a lot of sites and apps with it (he said to me they build some 60 sites!)

Near-future plans:

  • Better integration of your models.
  • UI and performance improvements. They are having a sprint session tomorrow with some designers that are also users of a Fiber website. Nice.
  • Better page creation (widgets, lists and views of your model instances).
  • Versioning of pages and content.
  • More tests and even better documentation. Sass/coffeescript.

They are realizing that they need to build a community around it if they want to get it off the ground further:

  • Create a real django fiber website.
  • Better documentation: read the docs website. And videos.
  • More examples in the Django fiber example.
  • Get more use out of github. More interaction via issues and pull requests.

Nice: a couple of days ago, Luke Plant mentioned some big db optimizations for Django. They will look at that tomorrow, too.

(I wrote about an earlier 30 minute talk he gave on Django Fiber for if you want more information.)

Back

From Reinout van Rees on January 18, 2012 12:48 PM

Despite my back, I'm back.

Ouch, my back, ouch

Since the summer my right leg is acting up. We moved to a new house and I probably did too much hard work. My right leg muscles started contracting the whole time, making me limp a bit, giving me muscle pain. Sometimes it felt like my leg was partially on fire.

I went to the doctor and she told me it was probably a lower back problem. A lower back problem can squeeze out another muscle or nerve by constricting it or limiting oxygen supply or so. The problem then radiates from the lower back into other body parts: for instance, a leg. Of course, limping a bit on that leg isn't too good for your posture, potentially leading to other back problems. The verdict: nothing major, just irritating. She suggested me to go visit a physiotherapist.

Four days later I bend over the wrong way while standing in the bath. It took me five minutes to crawl out of the bath: I severely wrecked my back muscles. I had to lie down in bed or on the sofa for four days. On my left hand side. Sitting or lying or standing or walking were painful. Going to the toilet was a major undertaking.

After three days I started taking stronger-than-aspirine painkillers (ibuproven), which helped getting the pain a bit under control and allowed me to move again. Moving sore muscles is necessary for getting them back in order again.

At the end of the week I managed to sit down (yes! accomplishment!) at my parents' 40 year wedding anniversery dinner.

Back at work

I stayed home for more than two weeks. First I couldn't move or sit. Later I could actually sit down on the couch, but not in a regular office chair. And soaking your muscles in a nice hot bath halfway the day, that's not something I'm going to attempt at the office :-)

Luckily I'm back at work again. The going is slow, as I tire at the end of the day, even when I take regular walks through the office instead of sitting. So I go home earlier.

And I come into the office later, as the internal body repair functions apparently take a lot of energy, so I need to sleep more. And I still prefer to soak in a hot bath in the morning to get my muscles going.

Back at my book

One thing that really suffered during those weeks was my Django book. I wanted to finish three chapters by the end of the month, but that schedule was obviously shot. I'm still occasionally tired (and most book work is in the evening), but I've gotten back on track.

Good news: today and tomorrow I've taken a day off to work full-time on my book. Well, full-time apart from this evening as there's a Django meeting in Amsterdam tonight which I'll attend.

Back at the physiotherapist

I went to the physiotherapist a couple of years ago. And I'm back. With my back. The lower back this time: the last time it was more in my shoulder area.

He helps, as I can do much more than a few weeks ago. I still limp and walking is a chore, but I can at least move about again.

Back to the middle ages

Back to the medieval times of torture... Ouch, it hurts to have a physiotherapist massage a leg muscle that's been rock solid hard for a couple of months... But it helped!

Back to (Django) blogging

I haven't blogged in about a month; no wonder, seeing the above. I'm starting again. That Django meeting in Amsterdam tonight? Great time to make summaries again; the program looks great.

Last year was a good blogging year. 165 entries in one year: my second best year. I somehow managed 182 in 2004. I like statistics, so I have two graphs with statistics on my weblog.

Well, I'm back to writing my book now. Bye!

My SOPA boycott

From Daniel Greenfeld on January 18, 2012 11:13 AM

I'm against SOPA and PIPA.

I believe that those bills will kill not just free speech, but also business within the USA. Innovation will wither. I'm also of the belief that those companies trying to get SOPA into place don't realize that no idea is new and if SOPA passes they'll be hammered with an increasing amount of takedowns and suits against them for anything they do. Litigation based on SOPA won't be as easily handled as the current status quo.

I've signed the petitions, I've posted on Twitter, Facebook, and Google. That isn't enough. I have to be willing to make a sacrifice. And in this case I'm going to make the sacrifice my vote.

My vote sacrifice is a boycott. It's directed at any politician, local or otherwise:

My Boycott:

  • If you as a politician vote for SOPA/PIPA then you lose my vote. Regardless of whatever other opinions you have or party you belong to, you've lost me as a supporter.
  • If SOPA/PIPA passes you can get my vote back by voting for what bill that destroys SOPA/PIPA is nominated.
  • If SOPA/PIPA fails you can get my vote back by voting against whatever bills are resurrected to replace SOPA/PIPA.
  • I will ignore party boundaries. I will vote against my normal grain simply to get you removed from office.

Why?

Like campaign finance reform, controls of freedom of speech often have unpredictable repercussions. 

The terrible thing about these bills is that their supporters are bi-partisan. While it's wonderful to see political opponents working together, in this case, it's for a terrible cause. 

So I'm going to cross political boundaries too. I'm going to say that as a registered Democrat I'm going to vote Republican if a Democrat candidate at any level votes for SOPA/PIP.

About Me

I don't use pirated software. I don't read, watch, or listen to pirated content. I purchase everything legally or use open source equivalents. I make a pretty decent salary and am pretty much in the direct center of what is called the 'middle-class'.

I'm also pro-business and a rather patriotic citizen of the United States. I believe in our nation and what it represents, and I know these bills are going to be a dagger in the heart of what our founding fathers gave us.

Finally, as I said, I'm a Democrat willing to vote Republican, Green, Libertarian, or whatever to make my point. That's my sacrifice. My vote and role in this nation that took my family in over 100 years ago is now in the hands of politicians.

What's yours?

Some quick Django optimisation lessons

From Luke Plant on January 18, 2012 10:53 AM

I recently used django-fiber for a simple project.

It's a nice CMS, a bit more lightweight than django-cms, with a slightly slicker frontend editing experience, and a bit easier when it comes to sharing content between different pages.

I found however, that it was doing a rather large number of queries to render pages, and some that pulled back lots of data, especially when you were logged in (which enables frontend editing).

So, I set to work and created a query count reduction branch. Below are the results so far, and some lessons.

Results

I used the example django-fiber project for testing my changes (as well as my own project), and tried them out on both the home page and a more deeply nested page.

  • URL: /
    • Anonymous user:
      • Original: 30 queries
      • Optimised: 15 queries - factor 2 reduction
    • Staff user:
      • Original: 103 queries
      • Optimised: 28 queries - factor 3 reduction
  • URL: /products/product-b/downloads/
    • Anonymous user:
      • Original: 64 queries
      • Optimised: 16 queries - factor 4 reduction
    • Staff user:
      • Original: 150 queries
      • Optimised: 31 queries - factor 5 reduction

Lessons

As you can see, there was a lot of unnecessary work. Django makes it extremely easy to generate database queries, which is both a good and bad thing, and here it is bad.

Here are some lessons for avoiding this unnecessary work:

  • Use django-debug-toolbar when developing, right from the beginning.

  • Seriously, use django-debug-toolbar.

  • Use django-debug-toolbar and keep it turned on. And look at it regularly. OK, point made.

  • This project was quite a lot easier to fix than django-cms, because it is much newer, and so has less code, and - more importantly - far fewer compatibility issues to worry about with 3rd parties who depend on certain features.

    You should think about big O scaling issues fairly early, because you can easily put yourself into a situation where things are hard to fix, due to:

    • schema design.

    • promises you've made regarding functionality to 3rd parties.

    For example, django-fiber has a concept of 'current' pages (pages which would form part of the bread crumb for the page you are on) and, in addition to the obvious ones (the 'ancestors' of the active page in the tree of pages), it has a feature which allows any page in the database to be a candidate 'current page' for any other page, based on a regex field. And so you have to check all these pages when rendering any page.

    This does not scale well. Thankfully, it's not too much of a problem if you don't use this feature, since you can do DB level filtering to eliminate most of the pages as potential candidates for this.

    But if you do use it, you have a scaling problem - for every time you use it, then the amount of work you've got to do to render any page increases. (And, if the DB filtering isn't efficient, you may still be paying an increasing penalty for every page added to the system even if you never use the feature).

    EDIT: I should have mentioned on the positive side that django-fiber has obviously given thought to general scaling issues, and used django-mptt for the tree structure of their Page model. This made it relatively easy to fix the 'show_menu' template tag to do everything in 2 queries. Otherwise it would have been a nightmare.

  • Create tests with assertNumQueries, and add them fairly early. You'll then be alerted to performance regressions that affect scaling.

    The tests will need to include tests for whole pages, not just bits, if you are going to analyse Big O scaling correctly, because a set of objects might be retrieved from the DB efficiently, but each one can easily make more database calls when it is actually used.

Some more specific lessons:

  • Read and understand the Django docs on optimizing DB access.

  • Understand when QuerySets are evaluated (which is part of the above, but worth mentioning).

    There were some examples in the fiber code of really inefficient use of QuerySets, e.g. if obj in MyModel.objects.filter(foo=bar) (example). This code will load all the MyModel records with foo=bar and create MyModel instances. It does not do MyModel.objects.filter(foo=bar, pk=obj.pk).exists(), and it certainly does not do if obj.foo == bar.

    Although it could do the first of these, Django deliberately does not make this optimisation. Django's QuerySets are deliberately dumb. The rule of thumb is this: a QuerySet will only do one query - the query you have told it to do using methods such as filter(), order_by() and slicing. It does not respond 'intelligently' to any Python builtins such as len(), bool() or the in operator - these simply force the QuerySet to be evaluated. There is efficiency in the way it uses its cache, but there is no 'cleverness'.

    (BTW, I remember the discussions we had about this a long time ago on django-dev, and I'm convinced with hindsight we made the right discussion. It might seem like a nice opportunity to do some clever queries, but since the cleverness does not extend to actual mind-reading, it will fail and it will get in the way. For instance, in the template example in the docs, cleverness on the part of QuerySet would only result in unnecessary work, and it would be much harder to get it to do the right thing).

  • Don't do queries when you've already got the information you need. There were multiple examples of this.

    This will often mean that you will have some duplication of logic - a manager method that defines some filtering for a set of objects, and an instance level method which answers the question 'do I belong to that set of objects'.

    This duplication is often unavoidable if you want any kind of performance. Just put a comment on the two methods indicating that they need to be synced with each other, and never use one when the other is what you need.

    (I can conceive of a cleverer system that would allow one of these to be automatically created from the other, but it would be limited to the subset of what can be expressed in both SQL and Python, and it doesn't exist in Django).

  • When defining special values that you need to search for, ensure that you can do efficient DB filtering.

  • Beware this common bug - writing:

    if not foo:
    

    when what you mean is:

    if foo is not None:
    

    These are completely different. If None is being used as a sentinel value, the first will treat things like the empty list or empty dictionary incorrectly. If you mean if foo is None or if foo is not None, always write just that, never assume that no other false-y values will be passed in.

    This is not just a performance-related bug, but it can cause a massive amount of repeated work where None is a sentinel value meaning the work has not been done yet, which is very common. This bug resulted in dozens of unneeded queries (including UPDATES being sent with every request) in django-fiber.

January 17, 2012

Forms Part 2: Class Based Forms

From GoDjango on January 17, 2012 08:00 PM

Forms are generally annoying to deal with. Fortunately Django offers some great ways to work with forms to make your life easier. In this our second part on forms we walk you through Class Based Forms in Django on our way to understanding the breadth of what you can do with forms in Django.
Watch Now...

January 14, 2012

IP Street’s Senior Developer opening now more about Search, less about Python/Django

From John DeRosa on January 14, 2012 09:58 PM

After some job market feedback and chin-scratching, I’ve changed our Senior Developer opening’s job description. Now it’s less about Python or Django, and more about search technologies, specifically full-text and LSI search.

We hope candidates will have some experience with Python or Django, but search technology experience (e.g., tuning, tokenizers, parsers, relevancy rank tweaking, aggregates and pivots) in now more important, and emphasized, in the the job.

Here’s the new description:

———

Founded in 2009, IP Street develops and markets software to help corporations, law firms, financial research firms, and government agencies better analyze patent information.  Our goal is to make IP data easy to get, use, and understand, so everyone can have access to high quality and transparent information.

A significant facet of our application’s capabilities are derived from Solr and other search technologies. We’re seeking a great full-text Search developer with experience in:

  • Solr, Lucene, or other search engines
  • Full-text search schemas, tokenizers, parsers, and rules for returning statistics and meaningful analytics
  • Automated workflows that process millions of objects
  • Data quality metrics and repairs

You’ll be joining us at a great time! Revenue is coming in, and we’ve done two Angel funding rounds at increasing valuations.

Key Responsibilities.

  • Enhance our Solr engine to provide more statistics and meaningful analytics to the product
  • Enhance or tune our use of other search technologies, e.g., LSI
  • Enhance and extend the existing code base to add new product features. Our application is written in Django and Python, with an almost all open-source technology stack
  • Occasionally wear testing or devops “hats,” as the needs arise
  • Write unit tests for your code, and do performance analysis
  • Demonstrate technical leadership within the team
  • Communicate well with the team, in writing and orally

Qualifications.

  • Significant experience using and tuning Solr, Lucene, or other search engines with similar capabilities
  • 3+ years related experience in Python development
  • 1+ years experience in Django development, or a strong interest in learning
  • Experience using one or more of: MongoDB, CouchDB, or another NoSQL database; Celery; Redis; PostgreSQL or another SQL database
  • Experience using latent semantic indexing search technologies would be a plus
  • Experience integrating with open-source 3rd-party libraries
  • Experience creating customer-focused software to process data and generate statistics and analytics
  • Solid troubleshooting abilities, self-directed, and proactive
  • Enjoy all aspects of software product creation — design, implementation, and debugging
  • Familiarity with using OS X as a development environment, and Linux as a production environment
  • Bachelors Degree or equivalent in Computer Science or Software Engineering
  • Excellent communication skills

Salary is DOE.

Please send resume to johnd@ipstreet.com.


Tagged: Django, jobs, Python

January 13, 2012

Add a custom backend to django-registration

From Inka Labs on January 13, 2012 06:07 PM

django-registration is a wonderful django app that allows us to easily and safely register new users in our website. It generates registration keys and sends confirmation emails to finish with the user registration.

But django-registration only works with a simple registration form, the django registration form, including only username and password fields. So, what can we do if we want to define our custom registration form and/or do some actions between the process? Well, django-registration gives us the ease to do so. This is because it works with "backends", it has one that uses by default, but, you can easily define your custom backend (usually inherits from the default backend) including al the information and actions you want to do. Lets see how to do so.

  • First, I'll suggest you to do a fast peek at the whole django-registration code, to have a outline of how does this app works.
  • Let's create our own application, say "my_registration", and inside this app, we're going to add a "backends" folder.
  • Inside the backends folder, we're going to copy the "__init__.py" file from django-registration backends folder. Now, still inside the backends folder, create the folder for your custom backend, say "my_custom_backend".
  • Inside my_custom_backend, copy the "urls.py" folder from the django-registration default backend and create a "__init__.py" file.
  • Let's see what our "__init__.py" should have:
# Usually, my custom backend is going to inherit from DefaultBackend
from registration.backends.default import DefaultBackend

# Inside my custom backend, I will have to define 6 methods:
#        * register
#        * activate
#        * registration_allowed
#        * get_form_class
#        * post_registration_redirect
#        * post_activation_redirect
# You can change their behaviour or just keep them the way they are.
class MyCustomBackend(DefaultBackend:(
    def register(self, request, **kwargs:(
        user = super(MyCustomBackend, self).register(request, **kwargs)
        # Here I can get the additional fields from the new form I'm sending
        user.first_name = kwargs['first_name']        
        user.last_name = kwargs['last_name']
        user.save()
        # I could also create a UserProfile for the new user
        profile = UserProfile()
        profile.user = user
        profile.telephone = kwargs['telephone']
        profile.save()
        # Or I could do something else, e.g. send an email to the new user
        return user
    
    # If you don't want to change the behaviour of the method you still
    # have to define it, but just calling super (same for all 6 methods).
    def activate(self, request, activation_key:(
        return super(MyCustomBackend, self).activate(request, activation_key)

    def registration_allowed(self, request:(
        return super(MyCustomBackend, self).registration_allowed(request)
   ...
  • Now, let's modify the content of our "urls.py" file to do what we need:
# ... After all the imports, we probably want to import the form we will
# use in our custom registration
from my_registration.forms import MyRegistrationForm

urlpatterns = patterns('',
       url(r'^activate/complete/$',
           direct_to_template,
           # You can change the template if you need to
           {'template': 'registration/activation_complete.html'},
           name='registration_activation_complete'),
       url(r'^activate/(?P<activation_key>\w+)/$',
           activate,
           # Here, we change the backend to our custom backend
           {'backend': 'my_registration.backends.my_custom_backend.MyCustomBackend'},
           name='registration_activate'),
       url(r'^register/$',
           register,
           # Here, in the registration part, we change the backend and, if
           # you need so, add the form to be used:
           {'backend': 'my_registration.backends.my_custom_backend.MyCustomBackend',
            'form_class': MyRegistrationForm},
           name='registration_register'),
       url(r'^register/complete/$',
           direct_to_template,
           # You can change the template if you need to
           {'template': 'registration/registration_complete.html'},
           name='registration_complete'),
       url(r'^register/closed/$',
           direct_to_template,
           # Same, change the template if you need to
           {'template': 'registration/registration_closed.html'},
           name='registration_disallowed'),
       (r'', include('registration.auth_urls')),
)
  • Finally, having set up our "__init__.py" and "urls.py" files, the only thing left to do is to assign a url to our new custom backend in our project "urls.py" file:
    ...
    url(r'^accounts/', include('my_registration.backends.my_custom_backend.urls')),

January 11, 2012

Inclusion tag for rendering forms

From Inka Labs on January 11, 2012 09:30 PM

Forms are widely used in almost all kind of Django proyects. Generally, we define specific views to handle different type of forms, but all of them usually do the same work:

  1. Verify if the request method is correct. If not, render an empty form.
  2. Validate the sent form. If it is not valid, render form with errors.
  3. Save the form.
  4. Send a success message.

So, instead of repeating this block of code again and again for each type of form, we can use this inclusion tag directly in the template:

from django import template
from django.contrib import messages
from django.utils.translation import ugettext_lazy as _

register = template.Library()

@register.inclusion_tag('render_form.html', takes_context=True)
def render_form(context, app_name, form_class_name, message=None,
                             method_='POST', additional_data={}):
    request = context['request']
    app = __import__(app_name)
    form_class = getattr(app.forms, form_class_name)
    # This is going to raise an AttributeError if form_class_name
    # doesn't exist inside app_name.forms 
    if request.method == method_
        form = form_class(data=getattr(request, method_), **additional_data)
        if form.is_valid():
            form.save()
            form = form_class()
            if message is not None:
                messages.info(request, _(message))
            return {'form': form, 'method': method_, 'data': additital_data}
        return {'form': form, 'method': method_, 'data': additital_data}
    form =  form_class()
    return {'form': form, 'method': method_, 'data': additital_data}

Now, you can render and validate a form inside of a template just adding a single line. This is specially useful in Django-CMS templates.

Example of usage:

{% render_form 'users' 'CreateUser' %}
{% render_form 'shop' 'ProductSelection' 'Selection completed successfully' %}

January 10, 2012

Configuring a Jenkins Slave

From Caktus Consulting Group on January 10, 2012 01:50 PM

We're pretty avid testers here at Caktus and when one of our Django projects required upgrading to Python 2.7, we also needed to upgrade our Jenkins build environment. Luckily, Jenkins supports distributed builds to allow a master install to delegate tasks to slaves instances. This way we can continue to run our primary build system on Ubuntu 10.04, which defaults to Python 2.6, and delegate tasks to an Ubuntu 11.04 environment running Python 2.7. The setup is fairly easy, but since I didn't find much out there already, I figured I write up a quick post outlining what we did.

To start, we'll need a new machine. I setup an Ubuntu 11.04 instance on Linode. Then SSH in, upgrade the packages, and install a Java Runtime Environment:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install default-jre

That's the only package Jenkins needs by default. Next we'll setup a user for Jenkins to SSH as. To do this, we'll add a new user to the system and copy the master's SSH public key:

$ sudo useradd -m jenkins
$ sudo -u jenkins mkdir /home/jenkins/.ssh
$ sudo -u jenkins vim /home/jenkins/.ssh/authorized_keys2

Now the master Jenkins client can ssh to the slave without a password. Next we need to configure the Jenkins master to connect to the slave. Head over to the Master environment and navigate to "Manage Jenkins" and then "Manage Nodes". Click "New Node" in the sidebar and add a Dumb Slave. On the following page, fill in the following fields:

  • # of executors: 2 (controls the number of concurrent builds)
  • Remote FS root: /home/jenkins
  • Labels: python27 natty
  • Usage: Leave this machine for tied jobs only
  • Launch method: Launch slave agents on Unix machines via SSH. Also fill in the Host field with the address of your slave machine.

Hit save and your Jenkins master should open a connection to your slave machine. To use the new slave machine, update an existing Jenkins job and set the "Restrict where this project can be run" Label Expression to "python27". You'll need to install any project dependencies on the slave for it to build properly, but that's basically it!

January 09, 2012

Release 0.6.4

From Lightning Fast Shop on January 09, 2012 07:19 PM

We just released LFS 0.6.4. This is a yet another bugfix release. 

Changes

  • Bugfix: fixed price calculation of configurable products.

News:

Information

You can find more information and help on following locations:

 

Einladung zur Django-UserGroup Hamburg am 11. Januar

From Arne Brodowski on January 09, 2012 08:51 AM

Das nächste Treffen der Django-UserGroup Hamburg findet am Mittwoch, den 11.01.2012 um 19:30 statt. Wie bei den letzten Malen treffen wir uns wieder in den Räumen der CoreMedia AG in der Ludwig-Erhard-Straße 18 in 20459 Hamburg (Anfahrtsbeschreibung auf Google Maps).

Bitte am Eingang bei CoreMedia AG klingeln, in den 3. Stock fahren und oben am Empfang nach der Django-UserGroup fragen.

Da wir in den Räumlichkeiten einen Beamer zur Verfügung haben hat jeder Teilnehmer die Möglichkeit einen kurzen Vortrag (Format: Lightning Talks oder etwas länger) zu halten. Neben Projektvorstellungen werden dieses mal sicher auch die Neuerungen in Django 1.4 ein Thema sein. Die meisten Vorträge ergeben sich erfahrungsgemäß vor Ort.

Als kleine Besonderheit wird bei diesem Treffen unter allen Teilnehmern eine Lizenz für PyCharm verlost, die uns der Hersteller JetBrains zur Verfügung stellt.

Eingeladen ist wie immer jeder der Interesse hat sich mit anderen Djangonauten auszutauschen. Eine Anmeldung ist nicht erforderlich, aber hilfreich für die Planung: Doodle Kaldender.

Weitere Informationen über die UserGroup gibt auf unserer Webseite www.dughh.de.

January 08, 2012

DjangoRedmineAdmin 1.0 released

From Thomas Capricelli on January 08, 2012 08:21 AM

I recently needed (again) to ‘browse’ a redmine database, and I used my DjangoRedmineAdmin application to do so. I took this opportunity to update the code and doing some more tests/fixes. As a result I decided to tag this as 1.0. The main modifications are: updated to comply with current redmine version (1.2.1 and 1.3.0) [...]

January 06, 2012

Django render_to_response

From Inka Labs on January 06, 2012 11:22 PM

It is important to see how django doesn’t automatically appends the context variables to the request object.

Say you want to access MEDIA_URL in your templates, using the following code will not work since the context is not created and it is not passed to the render_to_response function

def my_view(request):
    return render_to_response('my_template.html',
                              my_data_dictionary)

The first solution is to build the context and pass it to the render_to_response function, like this:

def my_view(request):
    return render_to_response('my_template.html',
                              my_data_dictionary,
                              context_instance=RequestContext(request))

But, this is kind of repetitive and boring. So a django snippet was proposed for that in http://www.djangosnippets.org/snippets/3/

from django.shortcuts import render_to_response
from django.template import RequestContext

def render_response(req, *args, **kwargs):
    kwargs['context_instance'] = RequestContext(req)
    return render_to_response(*args, **kwargs)

That works, but django just implemented almost the same thing in generic views. So, the solution I prefer is using the direct_to_template generic view

def my_view(request):
    # View code here...
    return direct_to_tempalte(request,
                                        template_name='my_template.html',
                                        extra_context=some_dict)

Getting Random objects from a Queryset in Django

From Inka Labs on January 06, 2012 11:17 PM

When providing related or suggested info to the user in a website, it’s a common practice to obtain a random set of items. To do this in django the "usual" way is:

Book.objects.all().order_by('?')[:10]

The above code, sorts all books in a random order and then picks the first 10 objects. This approach is not really efficient en MySQL. Since the "ORDER BY ?" clause is really expensive. So, a good way to obtain random elemets (actually a random slice) is the following:

import random
count = Book.objects.all().count()
slice = random.random() * (count - 10)
Book.objects.all()[slice: slice+10]

The above code selects a random position for starting a slice (the next items are going to be consecutive). So it’s not really random.

Another approach is to get the random ids from Python:

from random import sample
count = Book.objecs.all().count()
rand_ids = sample(xrange(1, count), 10)
Book.objects.filter(id__in=rand_ids)

This last solution works fine for consecutive ids, where there are no missing elements. But you can always re call it until you have all the items you want.

How pagination affects performance

From Inka Labs on January 06, 2012 11:14 PM

I am working on a kind of medium-size website where we have many posts (when I say many I talk about millions of posts).

Because posts are shown to users according to their preferences; the list of posts changes for every user. We used a Digg style pagination with the paginators provided by django, but the problem was that those paginators perform "SELECT COUNT(*) …" queries and when you have complex querysets and many rows; that queries are really expensive, making the complete application very slow.

Since we knew in this case users don’t care about the total number of objects or pages there was no need to use that "counts" so we decided to use django endless pagination http://code.google.com/p/django-endless-pagination/. Now our database is not exhausted anymore and the site is very quick now. Also, the installation and configuration was so easy that just took some minutes.

I’m really glad these guys built endless pagination, you should definitely give it a try.

January 04, 2012

The Django gunicorn fabfile project

From Alexis Bellido on January 04, 2012 10:21 PM

Since I read about Green Unicorn, better known as gunicorn, a Python WSGI HTTP server that goes along very well with Django, I knew I had to try it, and trying it I did. And I liked it. There were many steps to make a server work but overall it was cleaner than other approaches, including the Django deployment using Nginx, Apache and mod_wsgi that I described long time ago. And many reports indicate that a Django setup with gunicorn performs better than one using Apache.

But when you have to setup many servers with the same configuration, the process gets boring and repetitive, and when you're bored you make mistakes. This is the perfect scenario for automating with Python and a Fabric script.

I wanted the initial version of my fabfile to do the following:

  • Start with a clean install of Ubuntu 11.10, just with an openssh-server running.
  • Install all the Ubuntu packages needed for running a basic Django site, these include Nginx, virtualenv, some Python development tools, and PostgreSQL.
  • Create virtual environments and manage them with virtualenvwrapper.
  • Install a few Python packages in the virtual environments: Django, gunicorn, iptyhon, psycopg2, to name a few.
  • Create the configuration files to manage both staging and production sites with upstart.
  • Grab the code for a Django project from a git repository.
  • Keep the project's specific settings.py outside of version control.
  • Make everything work with just one or two commands from the shell.

And that's what the first version of this fabfile is doing. I don't have the time to explain all the details of the setup, Senko does a good job with his Django setup using Nginx and gunicorn and I was inspired by him (thanks dude), but I can give you the code I wrote and commited to GitHub.

I've called this the Django gunicorn fabfile project, so go get it and play with it.

January 01, 2012

Using Signals in your Django App

From Patrick Altman on January 01, 2012 09:57 PM

Yesterday, I posted a fairly popular post (for my blog anyway) dealing with how I write reusable Django apps. There were a couple of comments on Hacker News asking for some more details about some of thing suggestions mentioned in my post. Therefore, I'd like to start by digging into the use of signals to provide better site level integration.

What is a signal anyway?

A signal is defined by the Django documentation as:

“Django includes a “signal dispatcher” which helps allow decoupled applications get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place.”

A good example to highlight this is a recent upgrade I made to biblion which is the blog app that I use to drive this site. After my previous article was getting hammered from being on page 1 of HackerNews for an extended period of time, I noticed the page response times were starting to suffer.

I took a more careful look at the detail view serving up the page and discovered it was calling post.inc_views() which would be making an update to a count on the row that was just selected. I wasn't using these counts at all in my site and even if I did, I'd prefer to use redis or some form of a background task.

But how was I going to have access to do that at the site level?

The answer was to replace the post.inc_views() call with a signal. So I upgraded biblion to allow this by adding the following code:

Changes in biblion

biblion/signals.py

import django.dispatch

post_viewed = django.dispatch.Signal(providing_args=["post","request"])

biblion/views.py

In this module, I replaced the post.inc_views() call with:

post_viewed.send(sender=post, post=post, request=request)

Since I didn't care about incrementing the view count in my site, I could safely just ignore wiring up a receiver for this signal and it would become a no-op call in biblion. All I did was upgrade biblion in my requirements file and redeploy (thanks to Gondor this was super easy to redeploy).

However, for the sake of illustration if I did want to record the view count in a way that was not impeding the performance of heavy traffic, I could do something like the following.

Changes in your site

receivers.py

from django.dispatch import receiver

from biblion.signals import post_viewed

@receiver(post_viewed)
def handle_post_viewed(sender, **kwargs):
    post = kwargs.get("post")
    request = kwargs.get("request")
    if post:
        # Increment a redis key or kick off a background task or whatever here

urls.py

import receivers

Keep in mind that when handling that receiver that you are in the middle of a request/response cycle so you want to be as lightweight as possible. In fact, you should really consider putting any activity here in a background task via celery.

How I Write Django Reusable Apps

From Patrick Altman on January 01, 2012 01:21 AM

Over the past year I have written quite a number of reusable apps or have made significant contributions to previously existing ones (most if not all of these are under Eldarion or Pinax organizations on GitHub). A sort of a pattern has developed as I have learned little bits from trial and error so I thought I'd capture them in a single post as they might be useful for other app authors.

In summary they are:

  • Start with a Real Project
  • Provide Signals
  • Properly Package and Document Your App
  • Maintain a CHANGELOG
  • Provide injection points in Views

Start with a Real Project

I find it very difficult to develop good useful apps with features that are meaningful and work well, when built in the abstract. I also need a real project (whether an active project under development or a sample project doing real work for the purpose of proving out said app). All of the apps I have written to date started in a project being built for real. I got it working in the project, abstracted away domain specific and project specific bits and replaced the in project code with a requirements listing for the packaged version of the code.

This is one area that Pinax helps tremendously. Encouraging app development through the project structure of apps living in their own apps package helps to remind me to keep things isolated from the get-go.

Provide Signals

At first I didn't think much about signals as I thought the site developer could always register receivers for the model signals (e.g. post_save). However, I started running into situations where I wanted to know things were happening at the user interaction level which occurs in the views. Most importantly, I wanted access to the session when some event was occurring. Therefore, I started making a habit out of adding signals for the different sources of user interaction with the app and sending them in the views. And in doing so one of the arguments I started providing is the `request` object. By providing the `request` object, you allow the site developer to add their own language using apps like `django.contrib.messages` framework.

Properly Package and Document Your App

If you app isn't packaged it's going to be very hard to publish and if it's not published (on PyPI), it's going to be difficult for users to find and install it. Likewise, if you app isn't documented, then users are going to have to crawl through your code for reference instead of just glancing through some well-written documentation.

In order to speed the process along of releasing new apps, I started a repo that I base all new apps on. It includes a setup.py and a docs/ directory with a skeleton set of docs. Just copy these bits into a new folder and add your app code. Modify the variables in setup.py and the docs/Makefile; write the documentation appropriate for your app, and it's really that easy.

I then go to ReadTheDocs, create a project, go to GitHub and add a post receive hook for ReadTheDocs so that docs get rebuilt on every commit. Lastly, I build and upload an sdist to PyPI.

Maintain a CHANGELOG

Part of the docs is a changelog.rst whereby I maitain a list of changes for each versioned release. This is a good place to call out backwards incompatible changes and even provide example schema migrations (I disagree with putting South migrations in resuable apps).

Provide injection points in Views

If your apps has views, think through various ways of extending them. If using functional based views you can have them take a form class for instance to use instead of a default form. Then you can pass this in at the site level in the urls.py definition. If using class-based views you can provide even greater extensibility as a site developer can just subclass your view and override methods you provide as extension points. Another way of providing extensibility with class-based views is to allow mixins to be defined in settings.py that your app will mixin to override or augment built in functionality.

December 31, 2011

2011 At A Glance

From Muhammet S. Aydin on December 31, 2011 07:07 PM

I can call 2011 quite a good year on my behalf however when I look at things I've done, I feel I've done much less than I could. Here is a list of what I have done in 2011.

Things I've Done in 2011

  • Executed an idea we had with my brother, http://compector.com.

  • Built the start-up above with Ruby on Rails so I became more familiar with Ruby world.

  • I have made the switch to PostgreSQL from MySQL.

  • Got a job in one of Turkey's top web sites where we deal with thousands of concurrent users...

Release 0.6.3

From Lightning Fast Shop on December 31, 2011 09:18 AM

We just released LFS 0.6.3. This is a yet another bugfix release. 

Changes

  • Fixed update of prices if a configurable product is for sale.
  • Fixed calculation of property prices for configurable products.
  • Fixed saving of property data (added missing csrf token).
  • Fixed removing products / properties from a group.
  • Fixed filtering for float field steps.

News:

Information

You can find more information and help on following locations:

 

December 30, 2011

Deferred foreign keys with django-dfk

From Dan Fairs on December 30, 2011 05:25 PM

django-dfk is a project that I developed for a recent project to allow foreign keys to be declared on models without an explicit target ('dfk' stands for 'deferred foreign key'). It provides an API to 'point' these foreign keys to a concrete target at a later date, and also allows you to forcefully 'repoint' foreign keys that have already been set up. This last facility should be used with caution - it's essentially akin to monkey-patching.

You can use GenericForeignKeys for this, and these are slightly more flexible in that each model instance foreign key may point to a different model. However, there is a performance cost associated with them, and joining can be problematic.

(The project actually rarely uses django-dfk directly - instead, it uses it as a basis for abstract foreign keys, which have a greater awareness of the application environment - however, that's a topic for another day.)

Before we go much further though - rather than using this package, a better long-term investment would be to look at the app-loading branch that Arthur Koziel and Jannis Leidel have been working on - testing it, and helping them to get it into a state to be merged into trunk. I think that should provide a more holistic approach to solving this kind of problem. Until then...

Deferred foreign keys are useful in applications where you know that a model will require a foreign key, but don't know what the target will be at the time you're writing the application. Taking an example from the project that django-dfk was created for, let's say you have a Django app that is an online game. Users are entered into the game by way of an 'Entry' model, and the Entry contains a foreign key back to a model which contains user information.

However, you will have several instances of this game deployed, and each game may have its own source of player data - hence, its own Player model.

Let's say that there are two applications involved: one called 'core', which contains the core game logic, and one called 'mygame1', which contains models and logic specific to an individual game deployment.

core/models.py:
from django.db import models

class Entry(models.Model):
created = models.DateTimeField(defaults=datetime.datetime.now)
player = models.ForeignKey( …. uh, what do I type here?

Remember - the core application is going to be deployed in several places, and there are several possible places that FK might need to point.

One approach solving this would be to introduce a model which relates an entry to the game-specific Player model, resulting in models that look something like this:

core/models.py:
from django.db import models

class Entry(models.Model):
    created = models.DateTimeField(defaults=datetime.datetime.now)


mygame1/models.py:

from django.db import models

class MyPlayer(models.Model):
    name = models.CharField(max_length=50)
    entry = models.OneToOneField(Entry)

This works fine - however, there are some games which share player data with each other. This means that the key needs to live on the Entry model - but, we don't know which player model to point the FK at, as this model might be deployed in multiple places.

We can solve this using django-dfk.

core/models.py:
from django.db import models
from dfk.models import DeferredForeignKey

class Entry(models.Model):
    created = models.DateTimeField(defaults=datetime.datetime.now)
    player = DeferredForeignKey(unique=True)


mygame1/models.py:

from django.db import models
from dfk import point
from core import Entry

class MyPlayer(models.Model):
name = models.CharField(max_length=50)

point(Entry, 'player', MyPlayer)

The first thing to notice is that our Entry model now sports a DeferredForeignKey instance. It's important to realise that this isn't a real field, it's just a placeholder. Any arguments (except for the special 'name' argument, more on this below) are simply stored.

The action happens during the call to 'point', in mygame1's models.py. As the name implies, this points the DFK called 'player' on Entry to the MyPlayer class. (Actually, under the hood, it simply replaces the DeferredForeignKey instance with a real ForeignKey instance complete with the arguments which were originally passed to the DFK). Note that we do this at the module level in models.py - all your pointing (and repointing) needs to be done before the application is ready to use to ensure that syncdb outputs the correct SQL.

After all this is done, the definition of Entry will effectively look like this (although of course your code won't have changed!):

class Entry(models.Model):
created = models.DateTimeField(defaults=datetime.datetime.now)
player = models.ForeignKey('mygame1.MyPlayer', unique=True)

Other game applications (say, mygame2 and mygame3) would point the foreign key to the Player model that is appropriate for their game.

It's quite common to need to point a number of these keys at once - there might be several models which refer to a player. Rather than writing lots of 'point' statements (and having to add to them if a new key is added), django-dfk allows deferred foreign keys to be named:

core/models.py
class Entry(models.Model):
created = models.DateTimeField(defaults=datetime.datetime.now)
player = DeferredForeignKey(name='Player', unique=True)

class StatusUpdate(models.Model):
text = models.CharField(max_length=140)
player = DeferredForeignKey(name='Player')


mygame1/models.py:

from django.db import models
from dfk import point
from core import Entry

class MyPlayer(models.Model):
name = models.CharField(max_length=50)

point_named('core', 'Player', MyPlayer)

Roughly translated, this means 'point all the deferred foreign keys in all models in the core app which have the name 'Player' to the MyPlayer model'. This will affect both the 'Entry' and 'StatusUpdate' models above.

Finally, django-dfk also provides a 'repoint' function. This is a big hammer, and is not to be used lightly. 'point' only works on DeferredForeignKey instances, by design - it's meant to prevent you making mistakes. 'repoint' works on regular foreign keys too. It's useful if you've used 'point' to change the destination of a DFK, but later need to change it. (However, if you find yourself in this position, you should probably refactor your code to just use 'point'). Both 'point' and 'repoint' take care of cleaning up various internal Django caches to ensure things life filtering on related fields work properly after a point operation.

django-dfk can be found on PyPI, and the source is on github - forks, bug reports and patches with docs and tests are welcome. django-dfk is in production on several high-volume sites.

December 29, 2011

Class-based views in Django 1.3

From Caktus Consulting Group on December 29, 2011 11:00 AM

Django class-based views


Introduction

Django 1.3 added class-based views, but neglected to provide documentation to explain what they were or how to use them. So here's a basic introduction.


Example of a very basic class-based view

Let's start with an example of a very basic class-based view.

urls.py:

...
url(r'^/$', MyViewClass.as_view(), name='myview'),
...

views.py:

from django.views.generic.base import TemplateView

class MyViewClass(TemplateView):
    template_name = "index.html"

    def get(self, request, *args, **kwargs):
        context = # compute what you want to pass to the template
        return self.render_to_response(context)

This will render your template index.html with the context you computed and return it as the content of an HttpResponse.


Introduction to class-based views

Now that we've seen the obligatory example, how about some instructions?

  • To create a class-based view, start by creating a class that inherits from django.views.generic.View or one of its subclasses.

  • In your URLconf, specify the view method as the name of the new class, plus .as_view():

    url(r'urlpattern', MyViewClass.as_view(), ...)

  • In your class, write a get method that takes as arguments self (as always), request (the HttpRequest), and any other arguments from the request as specified in your URLconf.

  • In your get method, use the same logic you'd have used in an old view, except that you can assume the request method is GET. Return an HttpResponse as usual.

  • If you need to handle POST, write a post method, just like your get method except that you can assume the request method is POST.

  • Any request method that you don't write a handler method for will automatically get back a "method not allowed" response; you don't have to do anything special.

Example:

from django.views.generic import View
from django.shortcuts import render

class MyViewClass(View):
    def get(self, request, arg1, keyword=value):
        return do_something()
    def post(self, request, arg1, keyword=value):
        return do_something_else()

Handy subclasses of View

Django comes with a number of useful subclasses of View that provide some of the function that often ends up as boilerplate in views, just by inheriting from them. You saw TemplateView being used already. You'll probably want to base your views on TemplateView almost anytime you're generating the content for a response.

Another useful one is RedirectView. This can be used to redirect all requests. Example:

from django.core.urlresolvers import reverse
from django.views.generic import RedirectView

class MyRedirectView(RedirectView):
    url = reverse(...)

That is a complete view, and will return a redirect to url on any GET, POST, or HEAD request.

You can optionally set permanent = False to return a temporary redirect instead of the default permanent redirect, and query_string = True to include any query string from the incoming request on the redirect URL:

from django.core.urlresolvers import reverse
from django.views.generic import RedirectView

class MyRedirectView(RedirectView):
    url = reverse(...)
    permanent = False
    query_string = True

Decorators

Unfortunately, using decorators with class-based views isn't quite as simple as using them with the old method-based views.

Maybe you're used to doing this:

from django.contrib.auth.decorators import login_required

@login_required
def myview(request):
    context = ...
    return render(request, 'index.html', context)

With class-based views, you have to decorate the .dispatch() method of the class view, which means you have to override it just to decorate it. And you need to decorate the decorator, because the decorators provided by Django expect to be decorating method-based views, not class-based ones:

from django.contrib.auth.decorators import login_required
from django.views.generic.base import View
from django.views.utils.decorators import method_decorator

class MyViewClass(View):

    def get(self, request, **kwargs):
        context = ...
        return render(request, 'index.html', context)

    @method_decorator(login_required)
    def dispatch(self, *args, **kwargs):
        return super(MyViewClass, self).dispatch(*args, **kwargs)

This is an area of class-based views that could use some improvement.

You could apply the decorator in urls.py without needing so much extra code:

urls.py:

from django.contrib.auth.decorators import login_required
...
    url(r'^/$', login_required(MyViewClass.as_view()), name='myview'),
...

but that moves the policy from the view code to the URLconf, which is not where people will be expecting to have to look for it, so I wouldn't recommend it.


Passing arguments to the view

The method signature for get(), post(), etc. in a view class is:

def get(self, request, *args, **kwargs)

Any unnamed values captured in the URLconf regular expression are passed in args, and any named values are passed in kwargs, just like before.

You can pass extra arguments to your view using the third element of your URLconf, the same as before, or using a new technique -- passing them to the .as_view() call in your url settings. E.g.

...
    url(r'^/$', MyViewClass.as_view(extra_arg=3), name='myview'),
...

One warning - don't accidently write MyViewClass(extra_arg=3).as_view(). That'll still appear to work, but that extra_arg is just thrown away.


Where's the beef?

So far, all we've done is the same behavior, written using a different syntax. But class-based views enable a whole new level of function.

Suppose you've got a view that displays some data on a web page, and you write it as a class-based view. Maybe something like this:

from django.views.generic.base import TemplateView

class MyViewClass(TemplateView):
    template_name = 'index.html'

    def get(self, request, **kwargs):
        # Lots of complex logic in here to compute 'context'
        self.render_to_response(context)

Now you're asked to provide an HTTP API that returns the same data in json.

Start by refactoring your existing class slightly, moving your business logic out of the get() method:

from django.views.generic.base import TemplateView

class MyViewClass(TemplateView):
    template_name = 'index.html'

    def compute_context(self, request, **kwargs):
        # Lots of complex logic in here to compute 'context'
        return context

    def get(self, request, **kwargs):
        self.render_to_response(self.compute_context(request, kwargs))

Now, write a new class that subclasses your original class, uses the same method to compute the data, but overrides get() with different rendering code:

class MyJsonViewClass(MyViewClass):
    def get(self, request, **kwargs):
        data = self.compute_context(request, **kwargs)
        # Very naive way to put your data into json, but a good starting place
        content = json.dumps(data)
        return HttpResponse(content, content_type='application/json')

Add a new URL to urls.py pointing to your new class-based view, and you're done. All the logic you worked out earlier is still in use, and the power of subclassing let you provide the data in a new format almost effortlessly.


Class-based views for common policy

The previous example was still something you could have done almost as easily with method-based views, by refactoring your code into separate methods and calling them from all your views.

A more powerful use of the new class-based views is to provide common function for many views. If you have a site with many views, and they all inherit from a common view, then you have the potential to change behavior across the site by changing that one view.

Previously, you would probably have used middleware for this kind of thing. The problem with middleware is that it's completely hidden from the view code. When working on your view, you won't even know middleware is affecting things unless you go look at the settings and track down each piece of middleware configured there.

Furthermore, middleware affects every request, not just the views you really wanted it for.

With a common class-based view, every view affected is declared to inherit from that view, making it obvious that we're inheriting behavior from elsewhere. With a good IDE, you can even jump straight to that superclass to inspect it. Any view that doesn't need the common behavior doesn't have to inherit it.


References

The only documentation page that really discussed class-based views in Django 1.3 is this one:

https://docs.djangoproject.com/en/1.3/topics/class-based-views/

Some of the rationale for the current design of class-based views, and pros and cons of some alternatives that were considered, are documented here:

https://code.djangoproject.com/wiki/ClassBasedViews

Beyond that, the best advice I can give is to go read the code. The code for the base View is surprisingly small, and can be found at django/views/generic/base.py.

December 28, 2011

OpenBlock Geocoder, Part 3: External Geocoders

From Caktus Consulting Group on December 28, 2011 01:23 PM

The OpenBlock geocoder is powerful and robust. It uses PostGIS for spacial queries, can extract addresses from bodies of text, and can understand block and intersection notation. We've run into a few issues with it, however, including a low geocoding success rate. This is a tough problem to solve and depends on a lot of factors (the extent of street and block data in OpenBlock, format of the street addresses, etc.), so your mileage may vary. Below I constructed a simple test using Google's Geocoding API to have as an alternative.

Disclamer: This is the third post in our OpenRural series reviewing OpenBlock and it's geocoder. You may wish to read Part 1: Data Model and Geocoding and Part 2: Text Parsing and Entity Extraction before proceeding.

Adding news with OpenBlock's geocoder

The Schema and NewsItem models provide OpenBlock with a generic data model to associate news with geographic locations. You can find a fairly extensive introduction in the official documentation, so we won't go into too much detail here.

Since a NewsItem requires a geographic point, let's use the OpenBlock geocoder to fine 123 East Franklin Street:

>>> from ebpub.geocoder import SmartGeocoder
>>> geocoder = SmartGeocoder()
>>> location_name = '123 East Franklin Street'
>>> point = geocoder.geocode(location_name)['point']
>>> point.wkt
'POINT (-79.0553588124999891 35.9133110937499964)'
You'll notice that point has a wkt attribute. wkt, or Well-known text, is a text markup language for representing geometry objects. Here we have a POINT, but the language can represent many geometries, including LineString and Polygons.

We'll use the "Local News" schema in this example as it is pre-loaded in OpenBlock:

>>> from ebpub.db import models as ebpub
>>> schema = ebpub.Schema.objects.get(name='Local News')

Using this schema, we'll add a new NewsItem with the point created above:

>>> import datetime
>>> news = schema.newsitem_set.create(
...     title='Incident downtown',
...     description='Something happend downtown today!',
...     item_date=datetime.date.today(),
...     location=point,
...     location_name=location_name,
... )
>>> news.location.wkt
'POINT (-79.0553588124999891 35.9133110937499964)'

That was easy. Now we have a NewsItem that OpenBlock is aware of and can be plotted on a map. However, what do we do if we can't geocode the address?

Using an External Geocoder

If we already have a geographic point, then we can circumvent the geocoder entirely:

>>> from django.contrib.gis.geos import Point
>>> manual_point = Point(-79.0553588124999891, 35.9133110937499964)
>>> news = schema.newsitem_set.create(
...     title='Incident downtown',
...     description='Something happend downtown today!',
...     item_date=datetime.date.today(),
...     location=manual_point,
...     location_name=location_name,
... )
>>> news.location.wkt
'POINT (-79.0553588124999891 35.9133110937499964)'

This means we can also use an external geocoder. For example, we can use Google's Geocoding API with geopy. First, you'll need a Google Maps API key, which we'll use with geopy:

>>> GOOGLE_MAPS_API_KEY = '' # your Google Maps API key

Then we can use geopy to construct a new geocoder:

>>> from geopy import geocoders
>>> g = geocoders.Google(GOOGLE_MAPS_API_KEY)

And we can geocode our address:

>>> address = '123 East Franklin Street, Chapel Hill, NC'
>>> place, (lat, lng) = g.geocode(address)
>>> point = Point(lng, lat)
>>> point.wkt
'POINT (-79.0549350000000004 35.9136495999999994)'

You can even tap into OpenBlock's internals and build a Geocoder that OpenBlock can use:

from django.conf import settings
from django.contrib.gis.geos import Point

from geopy import geocoders
from geopy.geocoders.google import GQueryError

from ebpub.geocoder import Geocoder, DoesNotExist


class GoogleGeocoder(Geocoder):

    def __init__(self, *args, **kwargs):
        kwargs['use_cache'] = False # haven't implemented cache yet
        super(GoogleGeocoder, self).__init__(*args, **kwargs)
        self.geocoder = geocoders.Google(settings.GOOGLE_MAPS_API_KEY)

    def _do_geocode(self, location_string):
        try:
            place, (lat, lng) = self.geocoder.geocode(location_string)
        except (GQueryError, ValueError), e:
            raise DoesNotExist(unicode(e))
        location = {'point': Point(lng, lat)}
        return location

This is an proof-of-concept geocoder we're using with OpenRural. You can find it on GitHub. Using this geocoder with a sample dataset from the North Carolina Secretary of State Corporation Filings, I was able to increase the geocoding success rate from about 37% to 95%. Again, your mileage will vary, but it can be useful to test out. We can't use Google's API for everything though. Normal users are limited to 2,500 requests per day. Business accounts are allotted 100,000 requests. Additionally, Google requires you to display any points geocoded with their API on a Google Map. So you'll need to evaluate your needs before deciding on using Google's API.

December 27, 2011

Django Caching in the Real World: Part 1

From Peter Baumgartner on December 27, 2011 06:20 PM

When you develop a sizable content heavy web site you quickly learn, hopefully not the hard way, that caching is a very important piece of your infrastructure. The database servers are the typical bottleneck in high volume website. In this series, we’ll explore the a caching strategy we recently implemented.