Tag Archives: python

The Funky Data Model (Again, Again!)

Just as I was thinking MongoDB was the bees knees. This pops up…

pickled-object-database – Google Code

… which despite being a clever fudge in which objects are picked (aren’t we all) into a SQLite database does have the advantage of being a SQLite database. Which means that you can run it on a shared server and use existing tools to browse the data you create.

This definitely wins the have your cake and eat it category for storing data this week.

The Holy Grail of the Funky Data Model

I’ve blogged about the “Funky Data Model” before…

When I was working with some very clever Oracle database dudes I came up with what I thought was a great idea. The idea was that rather than having database tables for users, pages, blog posts etc that we could have just two tables called “things” and “types of things”.

The Oracle guys rolled their eyes and said everyone goes through this stage of thinking sometime and they even had a name for it… “the funky data model”. They told me that The Funky Data Model, although cool, never, ever works. They were probably right.

But there’s something that keeps pulling me back to The Funky Data model simply because working with relational databases is a bit of a pain. Relational databases really are wonderful things, you put your data in, and if you’ve put it in properly, you can get your data out… quickly.

It would seem like that saving data would be an easy thing for a computer to do, wouldn’t it? It’s not. Well actually, saving data is an easy thing to do but if you ever want to get it back before next tuesday then you are probably going to find that a relational database or two will be in the mix somewhere.

The problem with relational databases is that they are pretty “fixed”. For example, if you want to add a “recipe” that has a title, then all recipes have to have a title, a recipe can’t have two titles. You have to pretty much decide what you are going to put in your database before you put it in. Life (and data) often isn’t like that… it’s messier.

So last week, I thought I’d explore what are called schema-less databases for a day and see how far I got. These databases are less “fixed” than relational ones, closer to my idea of what the funky data model is all about and probably don’t work but I wanted to see if I could maybe take one small step towards the funky data model for the hell of it.

My “plan” was to stick a load of data in and then see how easy (and quick) it was to get out using python (my programming language of choice).

I gave dbxml a wide berth simply because the means of getting your data out, called xQuery seems to require brains I don’t have.

In the morning I tried working with CouchDB. I used MacPorts to install all the dependencies and started adding data. Things were going swimmingly until I discovered that you get your data out by writing snippets of JavaScript. My JavaScript isn’t great, but then JavaScript itself isn’t great. The whole JavaScript-iness of it put me off.

In the afternoon, slightly disappointed with CouchDB, Andy suggested I give the ZODB a whirl. The ZODB is what underpins both Zope and Plone. It’s an object database with a long track record and as you know, very python-friendly. After working with a database for a while I struggled to get the “server part” running on MacOS X called the ZEO. And because I base lots of technological decisions on gut-feelings and tea leaves and because I give up quickly I gave up the ZODB quickly and thought the “there has to better way” thought beloved of many a funky data model hunter.

In the evening, thinking that relational databases weren’t so bad after all, I gave a last ditch trial run to MongoDB. Installation was easy enough and the documentation is very pretty (one of the most important things for geek stuff).

MongoDB is a database that stores its data in dictionaries. A dictionary is a complex enough to store anything, for example a recipe might look like this…

recipe = {'title': "Eggs on Toast",
'ingredients': ["eggs", "butter", "toast"],
'yumminess': 7.5}

… but interestingly, because Mongo is “schema-less”,  I could add extra data to my recipe, an image or “preparation instructions” without requiring all my other recipes to have an image.

Even more interestingly, you can create “collections” simply adding something to a collection. This too is unlike the relational model, where you have to make the container BEFORE you put things in it and you have to decide exactly WHAT A THING is before you can put it into your container. With MongoDB you can simply type…

db.recipes.save( {'title': 'Bacon Sandwich"})

… and not only have you added a recipe, you’ve created the collection called “recipes”. This is staggeringly “fall off the chair” close to what I think of a Funky Data Model.

Now, I know what you’re thinking. You are thinking “Any fool can put data INTO a database, it’s getting it OUT that counts”. And you, as ever, are totally right. How do you do it?

I was pleasantly suprised that you get data out of MongoDB by creating a query that is itself a dictionary. So using my example above, to write a query that gets my bacon sandwich out of the database, I simply …

cursor = db.recipes.find( {'title': "Bacon Sandwich"} )
bacon_sandwich = cursor.find_one( )
print bacon_sandwich['title']

… There! How funky, easy and simple was that? So, I then set about tweaking my engagement engine crawlers to fill up a database to see how it performed in the “getting data out” tests. Within minutes I’d added my database to Django and although I’d only added a few thousand records, it seemed to be able to get them out very quickly and easily indeed. I’d expected my crawlers, which run multiple threads would have blown MongoDB up, but it seemed to cope fine.

Of course, one of the wonderful thing about relational databases is that you can write pretty complex queries that bring you data back quickly (normally). I can see that MongoDB is probably going to struggle when I start looking for recipes that include “bacon” and have a yumminess factor of 8 or above (which is most of them as it happens).

My big problem now, is that when looked in the eye by the Funky Data Model, I realise that, like a blank piece of paper, a totally fluid database is a very daunting thing. Would you create a collection called “recipes” or one called “sandwiches”? Would you create a collection called “ingredients” or, as in the real world, put the ingredients in the sandwich itself? Or both?

So… here I am, really impressed with MongoDB and realising that my brain is still vaguely stuck in relational mode. It seems that with MongoDB I might have to spend a lot more time thinking about how to get my data out, time that you wouldn’t need to spend with a relational database.

I’m planning to give MongoDB a more thorough test in the next few weeks, I’m both excited and scared it’s capabilities, it really might be the funky data model that has eluded me for so long and I may not have the abilities to be able to deal with it.

I’m also planning to be a bit more careful about what I wish for.

Extending the Django Admin Interface

This blog post is just a collection of notes I took along the way that might help you if you’ve had a play with the Django Admin and now are wondering how to get a little more out of it.

I have just spent a few weeks working on the first phase of an anlaytics dashboard application that is built using a mixture of Django, jQuery and Flash with the Isotoma crew. The project needed some way to set up the different sites and regions and rather than create these screens “by hand” we decided that the default Django Admin functionality would do this job well.


When I first saw the Django Admin screens I was shocked at how much you get “for free”. Having designed your database you can start editing it using screens that look better and work better than many CMSs.  And as a developer, the thought of writing code to do very simple tasks, such as adding data, deleting data, finding data is painfully boring and Django provides this functionality, along with user management, memberships, searching, filtering and permissions … all out-of-the-box.  And did I say it looks nice too?!

The example above shows a “Region listing” screen, that with a few quick tweaks has links to the Region on the site, lovely icons, shows a list of sites in that region and even has a handy dandy plus button that preopulates the site creation form with the relevant region.

Despite Django Admin’s obvious abilities it is often thought of as being “only for the developer” and not something that you’d let a client loose on. And if it was the case that a client wanted bespoke features you would of course need to create your application from scratch, but I believe that the default Django Admin can get you a lot further than you might first assume.

My Approach

All of the Django Admin application is hackable, but I am loathe to hack it willy-nilly because I like to be able to swap in a later version of Django and know that it’ll just work without having to remember a whole heap of tweaks I’ve made to Django Admin files. For this reason, I’m uncomfortable with the gorgeous Django Grappelli project, which adds a shade more visual pzazz to the Django Admin screens because it requires me to alter files in the Django Admin application ( I have a cludge that fixes this though ).

I like things to be simple. Lots of the solutions for extending the Django Admin screens you can find on Django Snippets are a step too far for me, requiring too many inter-dependent settings to work in combination, or worse.

And, with the 3 or 4 Advanced approaches I’ve used (below) you can add functionality to lists of objects, change the way the editing form behaves, add extra functionality to the editing forms AND make it look nice.

The Django Way Of Doing Things

One of the things that irks me about the Django community is that there isn’t the Django recommended way of doing things. The community assumes that I’ll have my own ideas and to force me to accept their arbitrary design decisions would be wrong.

Of course, they are kind of right, but at times I’d like a little guidance on best practice about where to keep your image files, your CSS files etc. I’d like there to be default way of doing things designed by people cleverer than me. I’d like someone to build a version of Django that included WYSIWYG editing on the admin side (with JQuery) and a load of examples of how to use jQuery built into the public site. I want jam on it!

Despite lots of Django applications advertising themselves as pluggable, they rarely are. When I’ve browsed the source code for lots of these applications I’ve found different ways of arranging templates, urls, libraries and graphics, making it almost impossible to drop a Django app into your project and have confidently make the changes that will guarantee it will work.

There a huge list of so-called pluggable apps here. It’s worth exploring how other apps do things just for the hell of it.

The Starter Project Poisoned Chalice?

I have seen Starter Projects being mooted and failed again and again. And whilst I doubt the use of them for anything too complex they are great for sharing a House Style (without dictating it) and for plonking copy & paste example of code for those things you can never quite remember how to do. There are quite a few starter projects on Google Code too.

My Starter Project  is one that I use to do the boring things that always need, often forget to implement but shouldn’t really waste time developing. You the kind of thing, favicons, robots.txt, RSS feeds… but also things like jQuery and the like.

Pretty much every Django project I create has a “templates” folder in which there is a base.html, index.html (which extends base.html). Every project I create has a CSS file somewhere in the mix. I wish that ./manage.py was more “on rails” than it currently is.

A good example of Django’s “hands off” approach is media handling. For example, Django kind of pushes you towards using a “/media/” URL for your Admin media and  folder called “static” for images,css and .js files. Early on I decided that I’d prefer it if this folder was called “media” and set about fixing up the URL handling to “work my way” – which was fine until I tried to integrate other peoples’ code. Other people assumed I wouldn’t be so stupid as to go against the grain and do things my way. Other peoples’ code didn’t work.

I tend to have a folder structure that looks a bit like this…


… with css, images and js folders being used for shared items. In this case my application is called “main” so have a folder called “main” and in that put the css, images and js files used only by this application. The only reason for doing this is that it makes applications very easy to re-use. You simply add them to your settings file, move the media and then wire in the URLs.

There are a number of real  “starter” projects for Django such as this one, which have examples of how to create a blog, events or book databases.

And yes, I appreciate the lunacy of calling my application “main” but it does mean that I can often drop an application folder into Django and know that main.views.index will automatically map onto “/”.

My Misc Application

A frequent pain in the arse (for me) with Django was getting a WYSIWYG editor ( such as TinyMCE ) to play nicely with Django Admin. Ideally, I often want to create an editing interface that both looks and works as good as WordPress (without having to use PHP).

And with this in mind, I’ve created an application called “Misc” that has WYSIWYG editing installed by default that I can use when setting up a project to check that everything is in the right place and working before I start work on my application proper.

This app can have data bundled with it (in the form of fixtures) and contain a heap of code that I use for reference. For example, it’d be great if Django would auto-generate admin.py files but it doesn’t and I can never remember how to define an Admin class, so having an example to hand makes development quicker.

I always create a constant called LOCAL_FOLDER with the full path to django project for when I’m running it locally and on a server.

I tend to create a “libs” folder and a “scripts” folder at root level and add their paths to sys.path in settings.py

My Standard Imports

As part of my Starter Project approach, there are a number of “must have” additions.

  • Django Evolution. This is, for me, the most essential Django app because it let’s me change my database models and update the database without exporting and re-importing all the data. Whilst it might be useful as a database versioning tool, I’ve never used it to rollback to a previous database schema.
  • Django Filebrower. This is essentially a popup window that let’s you chose files from a given folder integrated into TinyMCE.
  • Django Tagging. This is wonderful. Lots of external apps use it so it’s worth adding it just for that reason.
  • Django Pagination. Great code that lets you do pagination in the template.

Honorable mentions go to…

  • DjangoLogging. This puts debugging information into the response object. Very cool (but watch out when it adds stuff to your Alax responses).
  • Django Extensions. This adds extra commands to manage.py. I already love it for ./manage.py shell_plus alone.

Standard Django Admin Stuff

Getting started with Django admin is great. The latest version of Django has lots of lovely new features to explore. The admin functionality has a raft of features you might not have tried yet… including…

  • Listing viewings list_display can take a function (but a list_filter can’t)
  • Filters are lovely. I almost always create a “modified” field so that I can just see the objects I’ve edited today. Because I can.
  • Default ordering is handy.
  • Slug fields are great (but need a little care) for creating textual urls (or slugs).
  • You can use Fieldsets to re-organise the editing form, and hide things a bit.
  • You can define a function for upload_to in FileFields to make personalised upload folders so that uploaded images don’t overwrite each other.
  • I just discovered the CurrentUserField ( very useful )
  • You can make the names of database classes more friendly with the class Meta: verbose_name_plural in models.py
  • You can put the buttons at the top AND the bottom of a change form.
  • You can add images or links in the listings view by defining a function then adding my_func.allow_tags = True

Warning. Admin.py hasn’t totally evolved from the “old way” of doing things yet. Somethings are still defined in models.py such as the help_text attribute.

Rebranding Your Django Admin Site

Every client wants their admin screens to be branded.They are like that.

And, I’ve found that if you are running lots of Django sites, then for your own sanity making each admin site different really helps. One of the first things I do is to install a proper favicon to make jumping between Firefox tabs more intuitive. Trust me this really helps.

Given that I don’t like hacking the Django Admin code… my solution was to hack the Django Admin code. I did try to copy only the admin files I’d hacked, sort of  overshadowing the templates.  And whilst this is a good idea, it doesn’t work. Django isn’t Zope.

What I do is copy the templates folder from django.contrib.admin into the root of my project and the add it to my settings.py like this…

TEMPLATE_DIRS = ( ‘templates’, )

This then means I can hack a logo into admin screens but I always have the fallback of removing my hacked folder and using the vanilla admin screens if it all goes horribly wrong. I can also easily update django in one hit.

In general, you can quickly theme your site simply by editing these files.

  • login_form.html
  • base_site.html
  • base.html

Whether you choose to play with Grappelli, which makes the admin screens even more beautiful is up to you, but I found I could get most of visual eye-candy by simply using their css file and avoid some of the issues they have when it comes to displaying collapsed fieldsets (trust me they’re lovely).

Adding this to my base_site.html

<link rel=”stylesheet” type=”text/css” href=”/static/css/grappelli.css” >

…after having fixed up the paths to media in grappelli.css creates something like this…

grappelli version

Advanced Django Admin Stuff

The features in the standard Django Admin are enough for most projects but I frequently come across issues where I wish there was a way of pushing Django a just that bit further.

As ever, any admin hacking has to take a “light touch” approach (not requiring lots of changes or patches) and not be about jQuery manipulation ( I’m not a JavaScript programmer! ).

Custom Django Forms

When you start using forms.ModelForm then you a whole heap of new features, such as being able to define actions (see below). My only issue with django forms is that when searching for examples you tend to get lots of old newforms examples.

Whilst you can create a Django admin site without touching ModelForms (which is nice)… the more powerful features come when you use these forms.

1. Custom Actions in Object Listings

New in Django 1.0 is the ability to create “actions”. Adding custom actions to the list view (for multiple items) means that you can do things like add an “approve” method to work list of objects, like this..

def approve(modeladmin, request, queryset):     print queryset # the selected objects     msg = "Items have been approved"     self.message_user(request, "%s" % msg)     approve.short_description = "Approve selected items"
class CountryAdmin( ButtonableModelAdmin ):
    list_display = ['name', 'code', 'small_image_img']
    actions= [approve]

See the http://docs.djangoproject.com/en/dev/ref/contrib/admin/actions/#ref-contrib-admin-actions.

custom actions

The example above shows both CustomActions and adding HTML to the list_display attribute to show the flag image.

2. Custom Display Widgets in the Change Form

To be frank, editing uploaded images in Django Admin is weak. I tried a number of approaches to improve on how images are handled in and editing form but found that the AdminImageWidget worked for me. To use it, I simple add the class to my admin file and overshadow the render() method. Simple! I can then add this line to my form.

large_image = forms.ImageField(widget=AdminImageWidget() )

You can see a simple example of this in the large and small image above, where I have changed the image widget to also display the image.

3. Custom buttons to the Change form

Lastly, it’s great to be able to add buttons to the editing form (next to the history button). Again, I simply add the  ButtonableModelAdmin class to my admin file and then add a function to my admin class and bingo, my admin forms have new features.

class CountryAdmin( ButtonableModelAdmin ): def my_button(self, obj):          "do something here"          url = "/admin" + "/".join(str( obj._meta ).split(".")) + "/" + str(obj.id) + "/"           return HttpResponseRedirect( url )
my_button.url = "my_button"    #puts this on the end of the admin URL. my_button.short_description='My lovely button' buttons = [ my_button,]

You can see “My Lovely Button” in one of the forms above. Isn’t it lovely?

4. Collapsed Fieldsets

Collapsed fieldsets are for when you want to edit lots of messy related objects. I first came across them in the Grappelli theme, and having tried them, there’ no going back.

Collapsed fieldsets really tidy up the interface, but I found another way of achieving the same idea here  http://www.djangosnippets.org/snippets/1492/

In the example below, the Statistics have been hidden by using ‘classes’:[‘collapse’], in the fieldets attribute, whilst  the Status Messages (a collection of related objects) are hidden using collapsed fieldsets. Both approaches are essential for tidying up a complicated interface.

collapsed fieldsets

A Common Django Admin Gotcha

Apart from old documentation lying around the internet, the most common admin gotcha that still gets me is the one whereby you make a change to your admin.py file, the server noticing the changes, reloads. You then get the error that your model has already ready been registered. Bizarrely, if you then make another request to the server you get a 404. Rebooting the server fixes this.


Having dug around a little I found that it was easier than I thought to…

  • Add actions to lists of objects
  • Add buttons to the edit form of individual objects
  • Create custom widgets for editing individual rows (such as images)
  • Theme the Django Admin without knackering your django install
  • Display related Inlines nicely (ish)

… which for most admin needs, is all you need.

Growling Tweets

It’s nice to see that John has taken my ropey old AppleScript code to display tweets with Growl and improved upon it here, Growling Tweets – John’s World Wide Wall Display, so that it doesn’t need to use Vienna (why didn’t I think of that?).

And then as if by magic, Pete (I think) shares his Python version to do the same thing.

One of my pet hobby horses (I have a stable full of them) is about how the presentation of dialogue alters what kinds of dialogues we have. The simple version of this might be, “Aren’t most forum tools totally rubbish?”…. or “No wonder twitter visualisation projects get so much attention!”

More later

DjangoCon 2008: Reusable Apps

I watched a few of the DjangoCon videos. I still marvel at how wonderful it is to be able to dip into a conference without stumping up the air fare. Of course Cal was the funniest, but this one, despite being a bit dry, was undoubtedly the most useful, and inspirational but in a geek way.

If you develop with Django and wonder why your apps get bloat, this video may help you. It did me. It’s sort of a mini philosophy of how to develop using Django that sort of echos the “do one thing only and do it well” idea and it helped me to decide where to draw the line with regards to Django applications.

YouTube – DjangoCon 2008: Reusable Apps

My Life Passing Before My Eyes…

This was a weird moment. With a few minutes to explore some python tools, but not enough time or energy to do real work (it IS Saturday!)  I wondered how quickly I could write a script (fired off by cron) that took a picture once an hour with the isight camera and upload it to Flickr with my external IP as a title. I thought I’d kick myself that I didn’t do this and my computer ever got left on a train or stolen.

You can see it took roughly 30 minutes and less than a page o’ code (the first 10 minutes produced no photos and I had to sign up for a Flickr API key).

I think I have a huge market for this software down at the Tax Office, Benefits Office, Insert-Your-Favourite-Government-Dept-here.

Semantic Hacker in Python (Code)

Following on from my rant on data…. here’s a quick example I made of using the Semantic Hacker API in Python. You can pass in a chunk of text or a URL and get back a list of categories that data should be in. Very very clever.

I’m weaving this in with other semantic toolsets and getting excited by the potential. You know Paul, “this could change everything”, and if not it might come a close second-best, which is to change something. It’s a start eh?
Now let’s all go win the million dollars before bedtime.