All Buttons Pressed

Using dev_appserver with Python 2.7 on App Engine

2011-10-18T15:29:00+00:00

Starting with SDK 1.6.0 the App Engine fully supports Python 2.7. This article originally described a workaround for getting Python 2.7 running on SDK 1.5.5. Now it only describes how to port your app.yaml and request handlers to Python 2.7.

The following instructions work with any WSGI-compliant web framework. I'll explain how to use Python 2.7 with djangoappengine/Django-nonrel at the end of this post. Yes, Django-nonrel works with Python 2.7 on App Engine. This blog (All Buttons Pressed) is already running on Python 2.7 with threadsafe: yes.

Let's start with a simple app.yaml for Python 2.5:

application: yourappid
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: handler.py

The handler.py file looks like this:

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

# This can be any WSGI handler. Here we just use webapp.
application = webapp.WSGIApplication(...)

def main():
    run_wsgi_app(application)

if __name__ == '__main__':
    main()

In order to use Python 2.7 you have to change your app.yaml like this:

application: yourappid
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /.*
  script: handler.application

Here we've changed the runtime to python27, added threadsafe: yes and modified the last line (script) to point to the WSGI handler directly instead of just the handler.py module.

Also, our handler.py can be simplified like this:

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

application = webapp.WSGIApplication(...)

In other words, we've just removed main(). Note, webapp users would normally port their code to webapp2, but that's not really relevant for this post, so let's just ignore that detail.

That's all you need to use Python 2.7 on both the production server and dev_appserver.

Using djangoappengine with Python 2.7

If you're a Django-nonrel / djangoappengine user you can just change your app.yaml to look like this:

application: yourappid
version: 1
runtime: python27
api_version: 1
threadsafe: yes

builtins:
- remote_api: on

inbound_services:
- warmup

libraries:
- name: django
  version: latest

handlers:
- url: /_ah/queue/deferred
  script: djangoappengine.deferred.handler.application
  login: admin

- url: /_ah/stats/.*
  script: djangoappengine.appstats.application

- url: /.*
  script: djangoappengine.main.application

That's it. djangoappengine will take care of the rest. Note that the libraries section is currently necessary due to a bug in SDK 1.6.0. Since you're already deploying a custom django package it shouldn't be necessary to enable Django in the libraries section. The App Engine team will fix this bug in the next SDK release.

Happy porting!

F() objects and QuerySet.update() support in djangoappengine

2011-07-07T14:55:00+00:00

As some of you already noticed, we added supported for QuerySet.update() and F() objects as part of some features we currently need for our startup pentotype. Finally, it's possible to use simple transactions on App Engine via Django-nonrel :) Let's see how it works.

Django's beauty

In order to show what we gain by using Django's QuerySet.update() method and F() objects let me illustrate the code differences between the old djangoappengine version and the new one using a simple transaction which increments the value of a counter by a specific amount. In both cases we have the following model definition:

# models.py
from django.db import models

class Accumulator(models.Model):
    counter = models.IntegerField()
    name = models.CharField(max_length=500)

The old version looks like this:

from google.appengine.ext import db

def increment_counter(pk, amount):
    obj = Accumulator.objects.get(pk=pk)
    obj.counter += amount
    obj.save()

db.run_in_transaction(increment_counter, 1, 5)

whereas the new one looks like this:

from django.db.models import F
Accumulator.objects.filter(pk=1).update(counter=F('counter') + 5)

Obviously, it's more elegant to update a counter using QuerySet.update() and F() objects because it's much clearer and we have to write less code, thus resulting in more productivity.

Additionally, it's possible to do more complicated transactions, for example, updating multiple entities using Django's elegant syntax:

Accumulator.objects.filter(name__startswith='simple').update(counter=F('counter') + 1)

Note, that this will update each entity in its own transaction. With the old version we would have to iterate over each entity explicitly and start the transaction:

counters =  Accumulator.objects.filter(name__startswith='simple')

for counter in counters:
   db.run_in_transaction(increment_counter, counter.pk, 5)

Again, we benefit from writing less code.

However, one of the main benefits of QuerySet.update() and F() objects is that the code becomes even more portable between backends than before. For example, the MongoDB backend also has support for F() objects and QuerySet.update(), so the same code can be used for transactional updates on App Engine and MongoDB. Moreover, the same code runs transactionally safe on SQL databases, too!

What's next

Support for QuerySet.update() and F() objects is the foundation of portable transactions over different databases. In combination with the django-dbindexer it now becomes possible to add support for automatically sharded counters as described in Sharding with Django on App Engine! In other words, we could add an index definition in order to tell django-dbindexer to automatically shard specific updates. Stay tuned!

Minor updates and API changes

2011-05-05T10:18:00+00:00

Maybe you noticed that we've started to work on our projects again. ;) We unsurprisingly (:P) passed our final physics exams and are ready to work on NoSQL development again. Waldemar has already started to do some work in order to get our efforts into Django trunk. In the meantime I've started to work on some bug reports concerning django-dbindexer and nonrel-search. Both now have a dependency on django-autoload which ensures the loading of indexes or signal handlers before any request is processed. As a consequence, you have to adapt your code if you make use of django-dbindexer or nonrel-search. The documentation has been changed in order to reflect these changes. Maybe we'll take a few days off to go on vacation. Then, we'll come back and continue improving the way of NoSQL development.

SimpleDB backend for Django-nonrel

2011-05-03T07:16:00+00:00

Dan Fairs has started developing a SimpleDB backend. That's the sixth backend for Django-nonrel! Check out django-simpledb from github. The code is experimental and far from finished, but in a recent tweet Dan mentioned that he thinks he's very close to having the Django admin running on SimpleDB. That's huge, Dan! :) We're very close to supporting every popular NoSQL database. Guys, if you want to use Dango-nonrel with SimpleDB please help Dan with the django-simpledb backend (or help Mirko with the Redis backend if you prefer that).

Redis backend for Django-nonrel in development

2011-04-16T05:48:00+00:00

Mirko Rossini has finally made his Redis backend public. It's called django-redis-engine and according to him it's still in pre-alpha stage. From a quick glance at the code it looks like you'll be able to define database indexes in order to support queries that go beyond Redis' (meager ;) native query capabilities (much like with django-dbindexer). This is awesome because you won't have to maintain indexes by hand. If you want to see a stable and fully featured Redis backend please help Mirko. Just clone the repository from github and start playing with the code. Enjoy!

P.S.: Sorry for the long period of silence. We're still not finished with our final diploma exams, but we're getting close. Soon we can restart blogging more frequently and take the results of the previous survey into account.

JOINs for NoSQL databases via django-dbindexer - First steps

2011-02-01T07:27:00+00:00

At the moment, we have a lot to do in terms of finishing our diploma thesis. However we're so excited about the early results of the currently refactored django-dbindexer that we couldn't hold back and keep django-dbindexer's simple JOIN support for non-relational databases a secret anymore. Yes, you didn't read wrongly, this post is about JOIN support for NoSQL databases! We'll show how to use in-memory JOINs and how to get JOINs working if the joined field's value doesn't change. So let's unpack our delayed Christmas present. ;)

Let's rock!

Let's take the example of a photo-user relationship, just like in our post JOINs via denormalization for NoSQL coders, Part 1:

# photo/models.py:

from django.contrib.auth.models import User
from django.db import models

class Photo(models.Model):
    owner = models.ForeignKey(User, null=True, blank=True)
    popularity = models.IntegerField(choices=[(0, 'low'), (1, 'medium'), (2, 'high')],
                                     default=0)
    published = models.DateField(auto_now_add=True)
    title = models.CharField(max_length=500)

Here, each photo entity is linked to a user via a many-to-one relationship and has a popularity tag assigned to it. Additionally we store the date when the photo has been published and give the photo a title. Now let's say you want to get all of a user's photos. In order to do so, NoSQL databases force us to use ugly workarounds. One possible way is to first get the user (or better the corresponding primary key in order to avoid fetching the whole user entity out of the database) and thereafter the user's photos:

def get_user_photos(first_name, last_name):
    user_pk = User.objects.values('pk').get(first_name=first_name, last_name=last_name)
    return Photo.objects.filter(owner__pk=user_pk)

However with our reborn baby, django-dbindexer, you can just use Django's spanning relationship syntax (double underscores) to which you are used to:

def get_user_photos(first_name, last_name):
    return Photo.objects.filter(owner__first_name=first_name, owner__last_name=last_name)

Feels a lot more comfortable, right? :) To get this working all you have to do is to install the django-dbindexer and add the following index definition:

# photo/dbindexes.py:

from models import Photo
from dbindexer.lookups import StandardLookup
from dbindexer.api import register_index

register_index(Photo, {'owner__first_name': StandardLookup(),
                       'owner__last_name': StandardLookup(),
})

and one of the new join resolvers (InMemoryJOINResolver or ConstantFieldJOINResolver) to DBINDEXER_BACKENDS in your settings:

# settings.py:

DBINDEXER_BACKENDS = (
    'dbindexer.backends.BaseResolver',
    'dbindexer.backends.FKNullFix',
    'dbindexer.backends.InMemoryJOINResolver',
)

That's it. django-dbindexer will handle anything else magically for you. :) Under the hook the refactored django-dbindexer now uses backends to resolve lookups. The BaseResolver is responsible for resolving lookups like __iexact or __regex for example and the FKNullFix backend will make __isnull queries work correctly on ForeignKey. In this example we choose the InMemoryJOINResolver which is used to resolve JOINs in-memory, just like we did manually in the example above. On the other hand, the ConstantFieldJOINResolver would denormalize the user's first_name and last_name into the Photo model and use these denormalized properties to execute the query. Anything described in JOINs via denormalization for NoSQL coders, Part 1 is then done automatically by the ConstantFieldJOINResolver for you. :)

JOINs aren't limited to simple examples like the one above. Of course you can combine a query with filters on the Photo model:

def get_popular_user_photos(first_name, last_name):
    return Photo.objects.filter(popularity=2, owner__first_name=first_name,
                                owner__last_name=last_name)

You can even span filters over multiple relationships if you have more complex models, just use multiple double underscores. If the photo model contains a ForeignKey to a group which itself contains a ForeignKey to a creator, a query getting all photos whose group creator is a specific user would look like this:

def get_creators_groups_photos(first_name, last_name):
    return Photo.objects.filter(group__creator__first_name=first_name,
                                group__creator__last_name=last_name)

Note that this query can return photos belonging to more than just one group because a user could have created multiple groups.

And as a nice side effect of the refactoring process, you can combine JOINs with other index definitions like an __iexact lookup on the user's first_name and last_name in combination with a __month lookup on published:

def get_user_photos_created_in_month(first_name, last_name, month):
    return Photo.objects.filter(popularity=2, published__month=month,
                                owner__first_name__iexact=first_name,
                                owner__last_name__iexact=last_name)

Just add the needed index definitions to your index module:

# photo/dbindexes.py:

from models import Photo
from dbindexer.lookups import StandardLookup
from dbindexer.api import register_index

register_index(Photo, {'owner__first_name': (StandardLookup(), 'iexact'),
                       'owner__last_name': (StandardLookup(), 'iexact'),
                       'published': 'month',
})

It couldn't be easier than that, right? :) First we hardly believed our own eyes when we tested JOINs on App Engine on production but it really works. :P Just give it a try and download the django-dbindexer-testapp. :) If you wanna see even more complex examples then take a look at the unit tests of django-dbindexer.

`InMemoryJOINResolver` or `ConstantFieldJOINResolver`?

At the moment you have to choose globally between either the InMemoryJOINResolver or the ConstantFieldJOINResolver in you settings. All queries involving JOINs will then use the join resolver chosen. This will be changed in a future release of django-dbindexer. For now, you may wonder which join resolver to use. In general, in-memory JOINs are useful for simple cases for which you know that you get only a small set of entities on the joined side i.e. the User side. This includes examples like the one above as well as one-to-one relationships. If you know that you have to deal with larger sets of entities on the joined side you have to use the ConstantFieldJOINResolver because in-memory JOINs don't scale in such cases.

Currently you can't combine in-memory JOINs with OR-queries and exclude-queries. Anything else should work. In-memory JOINs will try to query as efficiently as possible. On App Engine for example dbindexer will batch-get and use efficient values('pk') filters (which become a keys_only filter) where possible. Additionally, fields on the joined side won't be created for StandardLookup.

The ConstantFieldJOINResolver is more efficient than the InMemoryJOINResolver and doesn't have limitations neither for OR-queries nor for exclude-queries i.e. you can execute the following query for example:

return Photo.objects.exclude(Q(owner__first_name__iexact='Danzo') |
                             Q(owner__last_name__iexact='Shimura'))

Additionally it is useful for more than one-to-one-relations. However, at the moment it only works for constant fields i.e. if the user's first_name and the last_name doesn't change. In the future we will add support for non-constant values too.

For both resolvers, JOINs only work in the ForeignKey direction. No reverse lookups supported at the moment. :(

Expressive NoSQL - Beyond scalability

So the first steps towards JOIN support for NoSQL databases are done. We plan to extend the backends to handle JOINs for non-constant fields in a scaleable way, just like described in our JOINs via denormalization for NoSQL coders series. Additionally, django-dbindexer's API will be extended to allow to specify a join resolver on a per field/filter bases. With the refactored django-dbindexer and its backend system even support for aggregates becomes possible. This will allow you to write complex queries in a few minutes instead of hours (including unit tests and debugging ;). No more hand-written denormalization and map/reduce and aggregate code. Just tell the indexer what you want to do and it'll handle it for you in a way you specify (via a backend)! Developers still have to think about how to design their code. However, django-dbindexer allows them to use Django's expressive ORM and thus being much more productive. Another possible extension is a backend for nonrel-search which could be used in order to add full-text search functionality for the admin interface. :) It's hard to think of something that would not be possible! :P If you want to help with the next exciting phase of development you can drop us a mail: http://groups.google.com/group/django-non-relational

Merry Christmas

2010-12-25T15:41:00+00:00

Merry Christmas and a happy 2011, everyone! Actually we wanted to finish a little present, but December was just too full with other stuff. Looks like Thomas will have to play Santa in January. ;)

This blog started more or less one year ago. Back then we still were on a blogspot subdomain. A few months later Django-nonrel was ready for hosting a simple site, so we moved everything on a custom domain (allbuttonspressed.com) hosted on App Engine with Django-nonrel. Now this blog has more than 700 subscribers and we really need to get some feedback from you, so we can improve our articles in 2011. Please help us by filling out this short survey.

Thanks a lot! Have a nice Christmas and a successful 2011.

Porting from App Engine's webapp to django-nonrel

2010-11-30T17:19:00+00:00

Hey, you may have noticed it already, we've finally finished our top-secret article called Running Pure Django Projects on Google App Engine :). We thank Wesley Chun for his help, expertise and time so we could get the article published on App Engine's articles section.

Summarized the article goes through several steps explaining in depth how to port an App Engine webapp app over to django-nonrel with some additional notes on useful Django features. At the end of the article we summarize the advantages of using django-nonrel over other approaches and what awaits you in the future. But enough words, just read the article :)

HTML5 offline manifests with django-mediagenerator

2010-11-23T15:17:00+00:00

This is actually part 3 of our django-mediagenerator Python canvas app series (see part 1 and part 2), but since it has nothing to do with client-side Python we name it differently. In this part you'll see how to make your web app load without an Internet connection. HTML5 supports offline web apps through manifest files.

Manifest files

First here's some background, so you know what a manifest file is. A manifest file is really simple. In its most basic form it lists the URLs of the files that should be cached. Here's an example.manifest:

CACHE MANIFEST
/media/main.css
/media/main.js

The first line is always CACHE MANIFEST. The next lines can list the files that should be cached. In this case we've added the main.css and main.js bundles. Additionally, the main HTML file which includes the manifest is cached, automatically. You can include the manifest in the <html> tag:

<html manifest="example.manifest">

When the browser sees this it loads the manifest and adds the current HTML and manifest file and all files listed in the manifest to the cache. The next time you visit the page the browser will try to load the manifest file from your server and compare it to the cached version. If the content of the manifest file hasn't changed the browser just loads all files from the cache. If the content of the manifest file has changed the browser refreshes its cache.

This is important, so I repeat it: The browser updates its cache only when the content of the manifest file is modified. Changes to your JavaScript, CSS, and image files will go unnoticed if the manifest file is not changed! That's exactly where things become annoying. Imagine you've changed the main.js file. Now you have to change your manifest file, too. One possibility is to add a comment to their manifest file which represents the current version number:

CACHE MANIFEST
# version 2
/media/main.css
/media/main.js

Whenever you change something in your JS or CSS or image files you have to increment the version number, manually. That's not really nice.

django-mediagenerator to the rescue

This is where the media generator comes in. It automatically modifies the manifest file whenever your media files are changed. Since media files are versioned automatically by django-mediagenerator the version hash in the file name serves as a natural and automatic solution to our problem. With the media generator a manifest file could look like this:

CACHE MANIFEST
/media/main-bf1e7dfbd511baf660e57a1f36048750f1ee660f.css
/media/main-fb16702a27fc6c8073aa4df0b0b5b3dd8057cc12.js

Whenever you change your media files the version hash of the affected files becomes different and thus the manifest file changes automatically, too.

Now how do we tell django-mediagenerator to create such a manifest file? Just add this to your settings.py:

OFFLINE_MANIFEST = 'webapp.manifest'

With this simple snippet the media generator will create a manifest file called webapp.manifest. However, the manifest file will contain all of the assets in your project. In other words, the whole _generated_media folder will be listed in the manifest file.

Often you only want specific files to be cached. You can do that by specifying a list of regular expressions matching path names (relative to your media directories, exactly like in MEDIA_BUNDLES):

OFFLINE_MANIFEST = {
    'webapp.manifest': {
        'cache': (
            r'main\.css',
            r'main\.js',
            r'webapp/img/.*',
        ),
        'exclude': (
            r'webapp/img/online-only/.*',
        )
    },
}

Here we've added the main.css and main.js bundles and all files under the webapp/img/ folder, except for files under webapp/img/online-only/. Also, you might have guessed it already: You can create multiple manifest files this way. Just add more entries to the OFFLINE_MANIFEST dict.

Finally, we also have to include the manifest file in our template:

{% load media %}
<html manifest="{% media_url 'webapp.manifest' %}">

Manifest files actually provide more features than this. For example, you can also specify FALLBACK handlers in case there is no Internet connection. In the following example the "/offline.html" page will be displayed for resources which can't be reached while offline:

OFFLINE_MANIFEST = {
    'webapp.manifest': {
        'cache': (...),
        'fallback': {
            '/': '/offline.html',
        },
    },
}

Here / is a pattern that matches all pages. You can also define NETWORK entries which specify allowed URLs that can be accessed even though they're not cached:

OFFLINE_MANIFEST = {
    'webapp.manifest': {
        'cache': (...),
        'network': (
            '*',
        ),
    },
}

Here * is a wildcard that allows to access any URL. If you just had an empty NETWORK section you wouldn't be able to load uncached files, even when you're online (however, not all browsers are so strict).

Serving manifest files

Manifest files should be served with the MIME type text/cache-manifest. Also it's critical that you disable HTTP caching for manifest files! Otherwise the browser will never load a new version of your app because it always loads the cached manifest! Make sure that you've configured your web server correctly.

As an example, on App Engine you'd configure your app.yaml like this:

handlers:
- url: /media/(.*\.manifest)
  static_files: _generated_media/\1
  mime_type: text/cache-manifest
  upload: _generated_media/(.*\.manifest)
  expiration: '0'

- url: /media
  static_dir: _generated_media/
  expiration: '365d'

Here we first catch all manifest files and serve them with an expiration of "0" and the correct MIME type. The normal /media handler must be installed after the manifest handler.

Like a native iPad/iPhone app

Offline-capable web apps have a nice extra advantage: We can put them on the iPad's/iPhone's home screen, so they appear exactly like native apps! All browser bars will disappear and your whole web app will be full-screen (except for the top-most status bar which shows the current time and battery and network status). Just add the following to your template:

<head>
<meta name="apple-mobile-web-app-capable" content="yes" />
...

Now when you're in the browser you can tap on the "+" icon in the middle of the bottom toolbar (update: I just updated to iOS 4.2.1 and the "+" icon got replaced with some other icon, but it's still in the middle of the bottom toolbar :) and select "Add to Home Screen":

Then you can enter the name of the home screen icon:

Tapping "Add" will add an icon for your web app to the home screen:

When you tap that icon the canvas demo app starts in full-screen:

We can also specify an icon for your web app. For example, if your icon is in img/app-icon.png you can add it like this:

{% load media %}
<head>
<link rel="apple-touch-icon" href="{% media_url 'img/app-icon.png' %}" />
...

The image should measure 57x57 pixels.

Finally, you can also add a startup image which is displayed while your app loads. The following snippet assumes that the startup image is in img/startup.png:

{% load media %}
<head>
<link rel="apple-touch-startup-image" href="{% media_url 'img/startup.png' %}" />
...

The image dimensions should be 320x460 pixels and it should be in portrait orientation.

Summary

The manifest file just lists the files that should be cached
Files are only reloaded if the manifest file's content has changed
The manifest file must not be cached (!) or the browser will never reload anything
django-mediagenerator automatically maintains the manifest file for you
Offline web apps can appear like native apps on the iPad and iPhone
Download the latest canvas drawing app source which is now offline-capable

As you've seen in this post, it's very easy to make your web app offline-capable with django-mediagenerator. This is also the foundation for making your app look like a native app on the iPhone and iPad. Offline web apps open up exciting possibilities and allow you to become independent of Apple's slow approval processes for the app store and the iOS platform in general because web apps can run on Android, webOS, and many other mobile platforms. It's also possible to write a little wrapper for the App Store which just opens Safari with your website. That way users can still find your app in the App Store (in addition to the web).

The next time you want to write a native app for the iOS platform, consider making a web app, instead (unless you're writing e.g. a real-time game, of course).

Offline HTML5 canvas app in Python with django-mediagenerator, Part 2: Drawing

2010-11-16T19:12:00+00:00

In part 1 of this series we've seen how to use Python/pyjs with django-mediagenerator. In this part we'll build a ridiculously simple HTML5 canvas drawing app in Python which can run on the iPad, iPhone, and a desktop browser.

Why ridiculously simple? There are a few details you have to keep in mind when writing such an app and I don't want these details to be buried under lots of unrelated code. So, in the end you will be able to draw on a full-screen canvas, but you won't be able to select a different pen color or an eraser tool, for example. These extras are easy to add even for a newbie, so feel free to download the code and make it prettier. It's a nice exercise.

Reset the viewport

With a desktop browser we could start hacking right away, but since we also support mobile browsers we need to fix something, first: One problem you face with mobile browsers is that the actual screen size is not necessarily the same as the reported screen size. In order to work with the real values we have to reset the viewport in the <head> section of our template:

<head>
<meta name="viewport" content="initial-scale=1.0, width=device-width,
     minimum-scale=1.0, maximum-scale=1.0, user-scalable=no" />
...
</head>

Now the reported and actual screen size will be the same.

Start touching the canvas

Before we look at mouse-based drawing we first implement touch-based drawing because that has a greater influence on our design. We'll implement a simple widget called CanvasDraw which takes care of adding the canvas element to the DOM and handling the drawing process:

from __javascript__ import jQuery, window, setInterval

class CanvasDraw(object):
    # The constructor gets the id of the canvas element that should be created
    def __init__(self, canvas_id):
        # Add a canvas element to the content div
        jQuery('#content').html('<canvas id="%s" />' % canvas_id)
        element = jQuery('#' + canvas_id)

        # Get position of the canvas element within the browser window
        offset = element.offset()
        self.x_offset = offset.left
        self.y_offset = offset.top

        # Get the real DOM element from the jQuery object
        self.canvas = element.get(0)

        # Set the width and height based on window size and canvas position
        self.canvas.width = window.innerWidth - self.x_offset
        self.canvas.height = window.innerHeight - self.y_offset

        # Load the drawing context and initialize a few drawing settings
        self.context = self.canvas.getContext('2d')
        self.context.lineWidth = 8
        self.context.lineCap = 'round'
        self.context.lineJoin = 'miter'

        # ... to be continued ...

The last two lines configure the way lines are drawn. We draw lines instead of individual points because when tracking the mouse/finger the individual positions are too far apart and thus need to be connected with lines.

In case you've wondered: The lineCap property can be butt, round, or square:

The lineJoin property can be round, bevel, or miter:

(Note: both images are modifications of images used in a Mozilla tutorial.)

Next, let's add the event handlers and the mouse/finger tracking code. The problem here is that we can't receive any touch movement events while we're drawing something on the canvas. The touch events get lost in this case and the user will experience noticeably slower drawing and bad drawing precision in general. The solution to this problem is to record the touch events in an array and then draw the the lines asynchronously via a timer which gets executed every ca. 25ms and to limit the drawing process to ca. 10ms per timer event. You can fine-tune these numbers, but they worked well enough for us. CanvasDraw stores the mouse/finger positions in self.path. Here's the rest of the initialization code:

class CanvasDraw(object):
    def __init__(self, canvas_id):

        # ... continued from above ...

        # This variable is used for tracking mouse/finger movements
        self.path = []

        # Add asynchronous timer for drawing
        setInterval(self.pulse, 25)

        # Register mouse and touch events
        element.bind('mouseup', self.mouse_up).bind(
                     'mousedown', self.mouse_down).bind(
                     'mousemove', self.mouse_move).bind(
                     'touchstart touchmove', self.touch_start_or_move).bind(
                     'touchend', self.touch_end)

So far so good. The actual mouse/finger movement paths are represented as a list. The path is ordered such that old entries are at the end and new entries are added to the beginning of the list. When the touch event ends we terminate the path by adding None to the list, so multiple paths are separated by None. Here's an example of what self.path could look like (read from right to left):

self.path = [..., None, ..., [x1, y1], [x0, y0], None, ..., [x1, y1], [x0, y0]]

OK, let's have a look at the actual touch tracking code. We use only one method for both touchstart and touchmove events because they do exactly the same thing:

class CanvasDraw(object):
    # ... continued from above ...

    def touch_start_or_move(self, event):
        # Prevent the page from being scrolled on touchmove. In case of
        # touchstart this prevents the canvas element from getting highlighted
        # when keeping the finger on the screen without moving it.
        event.preventDefault()

        # jQuery's Event class doesn't provide access to the touches, so
        # we use originalEvent to get the original JS event object.
        touches = event.originalEvent.touches
        self.path.insert(0, [touches.item(0).pageX, touches.item(0).pageY])

When the finger leaves the screen we terminate the path with None. Note that the touchend event only contains the positions currently being touched, but it's fired after your finger leaves the screen, so the event.originalEvent.touches property is empty when the last finger leaves the screen. That's why we use event.originalEvent.changedTouches in order to get the removed touch point:

class CanvasDraw(object):
    # ... continued from above ...

    def touch_end(self, event):
        touches = event.originalEvent.changedTouches
        self.path.insert(0, [touches.item(0).pageX, touches.item(0).pageY])

        # Terminate path
        self.path.insert(0, None)

Drawing it

Now we can implement the actual asynchronous drawing process. Remember, we use a timer to periodically (every 25ms) draw the mouse/finger path in self.path. We also limit the drawing process to 10ms per timer event. This is our timer:

import time

class CanvasDraw(object):
    # ... continued from above ...

    def pulse(self, *args):
        if len(self.path) < 2:
            return
        start_time = time.time()
        self.context.beginPath()
        # Don't draw for more than 10ms in order to increase the number of
        # captured mouse/touch movement events.
        while len(self.path) > 1 and time.time() - start_time < 0.01:
            start = self.path.pop()
            end = self.path[-1]
            if end is None:
                # This path ends here. The next path will begin at a new
                # starting point.
                self.path.pop()
            else:
                self.draw_line(start, end)
        self.context.stroke()

First we have to call beginPath() to start a set of drawing instructions, then we tell the context which lines to draw, and in the end we draw everything with stroke(). The paths are processed in the while loop by getting the oldest (last) two points from the path and connecting them with a line. When we reach the path terminator (None) we pop() it and continue with the next path.

The actual line drawing instructions are handled by draw_line(). When we receive mouse/touch events the coordinates are relative to the browser window, so the draw_line() method also converts them to coordinates relative to the canvas:

class CanvasDraw(object):
    # ... continued from above ...

    def draw_line(self, start, end):
        self.context.moveTo(start[0] - self.x_offset, start[1] - self.y_offset)
        self.context.lineTo(end[0] - self.x_offset, end[1] - self.y_offset)

Finally, we have to instantiate the widget and also prevent scrolling on the rest of our page:

canvas = CanvasDraw('sketch-canvas')

# Prevent scrolling and highlighting
def disable_scrolling(event):
    event.preventDefault()
jQuery('body').bind('touchstart touchmove', disable_scrolling)

Style

We need a minimum amount of CSS code. Of course, Sass would be nicer and the media generator supports it, but then you'd also have to install Ruby and Sass which makes things unnecessarily complicated for this simple CSS snippet in reset.css:

body, canvas, div {
  margin: 0;
  padding: 0;
}

* {
  -webkit-user-select: none;
  -webkit-touch-callout: none;
}

The first part just resets margins, so we can fill the whole screen. The last part disables text selection on the iPad and iPhone and also turns off pop-ups like the one that appears when you hold your finger on a link. That's fine for normal websites, but we normally don't want that behavior in a web app.

Of course we also need to add the file to a bundle in settings.py

MEDIA_BUNDLES = (
    ('main.css',
        'reset.css',
    ),
    # ...
)

and the bundle must be added to the template:

<head>
...
{% load media %}
{% include_media 'main.css' %}
</head>

What about mouse input?

Now that the infrastructure is in place the mouse tracking code is pretty straightforward:

class CanvasDraw(object):
    # ... continued from above ...

    def mouse_down(self, event):
        self.path.insert(0, [event.pageX, event.pageY])

    def mouse_up(self, event):
        self.path.insert(0, [event.pageX, event.pageY])
        # Terminate path
        self.path.insert(0, None)

    def mouse_move(self, event):
        # Check if we're currently tracking the mouse
        if self.path and self.path[0] is not None:
            self.path.insert(0, [event.pageX, event.pageY])

Summary

Whew! As you've seen, writing a canvas drawing app with touch support is not difficult, but you need to keep more things in mind than with a mouse-based app. Let's quickly summarize what we've learned:

Touch events get lost if the drawing instructions block the browser for too long
Draw the image asynchronously and limit the amount of time spent on drawing
You have to prevent scrolling in touch events via event.preventDefault()
On touchend the event.touches property doesn't contain the removed touch point, so you should use event.changedTouches
jQuery doesn't provide direct access to event.touches, so you have to use event.originalEvent.touches (same goes for event.changedTouches)
You have to convert mouse/touch coordinates to coordinates relative to the canvas element
Download the latest sample code to try it out yourself

In the next part we'll make our app offline-capable and allow for installing it like a native app on the iPad and iPhone. You'll also see that django-mediagenerator can help you a lot with making your app offline-capable. Update: Part 3 is published: HTML5 offline manifests with django-mediagenerator

Offline HTML5 canvas app in Python with django-mediagenerator, Part 1: pyjs

2010-11-09T11:38:00+00:00

This is the first part in a short series (see also part 2 and part 3) on building a simple client-side offline-capable HTML5 canvas drawing app in Python via Pyjamas/pyjs, using django-mediagenerator. Canvas drawing apps are the "Hello world" of the web apps world, so why not make it more interesting and throw in a few neat buzz word technologies? ;)

In this part we'll take a look at running Python in the browser via pyjs, the Pyjamas framework's Python-to-JavaScript compiler. Note that we won't describe the Pyjamas framework, here. Instead, we only use the compiler itself and jQuery. You can build your own client-side Python framework on top of this. We don't use Pyjamas because we believe that a framework optimized for Python can be a lot simpler and more expressive than a simple 1:1 translation from Java/GWT to Python (which the Pyjamas framework is, mostly).

Also, in case you wondered, it's important to not mix your backend (server-side) and frontend (client-side) code. Keep your backend and frontend code cleanly separated. Otherwise your app will become an unmaintainable, cryptic mess. We'll focus on the client-side in this series. For the backend you might want to use a simple REST API based on Piston, for example.

If you don't know django-mediagenerator, yet, you should have a look at the intro tutorial. The media generator is an asset manager which can combine and compress JavaScript and CSS code. Beyond that, it also has many advanced features which can help you with client-side code. We're going to use several of the advanced features in this short series.

Prerequisites

First, let's install a recent pyjs version from the repo:

git clone git://pyjs.org/git/pyjamas.git
cd pyjamas/pyjs
python setup.py develop

Then, download Django and django-mediagenerator and install both via setup.py install. Django is only needed for running the media generator and serving a simple HTML file.

Finally, download the project source and extract it. The project already comes with jQuery, so the previous three dependencies are all that we need.

Compiling Python to JavaScript

Obviously, you can't run Python directly in your browser. So, how does it work? You write normal Python modules and packages. Then, pyjs is used to convert those modules into JavaScript files. This JavaScript code can then be integrated into your templates.

Normally, you'd use a the Pyjamas build script to compile your Python files. However, it's very difficult to make offline-capable apps that way. Also, the build script generates a single huge HTML file with all dependencies, which makes in-browser debugging more complicated. Also, the generated output can't get cached efficiently.

Instead, we use django-mediagenerator to handle the build process. Basically, you tell django-mediagenerator to build your app's main Python module and the media generator automatically collects all other modules which are required by your main module and its dependencies. This means that only modules which actually get imported by your app get compiled and later combined, compressed, and uploaded to the server. All other modules will be ignored.

So, all you need to do is add the main module, let's call it canvas_main.py, to MEDIA_BUNDLES in your settings.py:

MEDIA_BUNDLES = (
    ('main.js',
        'jquery.js',
        'canvas_main.py',
    ),
)

This will combine jquery.js and the whole canvas drawing app with all its dependencies into a main.js bundle. It's really that simple. :)

Then, we can include it in the template via

<head>
...
{% load media %}
{% include_media 'main.js' %}
</head>

Just a reminder: When developing via manage.py runserver the files will of course stay uncombined and uncompressed. They only get combined and compressed for production. Also, during development all Python files will contain extra debugging code, so tracebacks can be generated. This makes the files a lot larger and a little bit slower, though. The production code (via manage.py generatemedia) doesn't contain this debugging code, so it's faster and much smaller.

Look, ma, Python in my browser!

What's so great about Python in the browser? Python is a much more powerful language than JavaScript and its design makes a lot more sense in large projects. Python gives you a real module system with explicit import statements for better readability. You can have nice getters and setters. The object system makes a lot more sense. Note, I do like prototype-based programming, but the way it works in JavaScript is a disaster when compared to Self, Io, and other languages. Enough talk. Let's create a simple widget which should react on a mouse click in pure Python:

# Import jQuery. Note: JavaScript variables have to be
# imported from the fake __javascript__ module.
from __javascript__ import jQuery

# Create a simple widget class
class ClickWidget(object):
    def __init__(self, base_elem):
        # Add some initial HTML code
        base = jQuery(base_elem)
        base.html('<div class="clickme">Click me!</div>'
                  '<div class="change">Then this will change.</div>')
        self.output_div = jQuery('.change', base)

        # Bind the click event to the on_click method
        jQuery('.clickme', base).bind('click', self.on_click)

    # This is our click event handler
    def on_click(self, event):
        self.output_div.append(' It clicked!')

# Install our widget
widget = ClickWidget('body')

See, it's really that simple. :) Did you notice something neat about this code? You didn't have to bind the on_click method to self! Well, as a Python developer you might say "that's totally normal", but the world can be harsh once you leave your comfy Python home. ;)

So, what would you do in JavaScript, instead? When you want to pass a method of some JavaScript object to some other function the method's this gets lost. So, in JavaScript you would install the click handler like this:

// ...
// JavaScript equivalent, e.g. using Prototype
var ClickWidget = Class.create({
    initialize: function(base_elem) {
        // ...
        this.output_div = jQuery('.change', base);
        jQuery(input_div).bind('click', this.on_click.bind(this));
    },
    on_click(event) {
        this.output_div.append(' It clicked!');
    }
});
// ...

Did you see the strange this.on_click.bind(this)? It returns a new function which calls on_click bound to this, so the on_click method can access this.output_div with the correct this. The fact that you need do this this, at all, is unintuitive and there's no reason why developers should have to keep nonsense like this in mind. If you've only worked with jQuery you might never have written code like this because your app probably didn't write object-oriented code. However, once you enter the realm of complex web apps you should certainly begin to think about solutions that allow for better code reuse and maintainability.

By the way, you can also use many Python standard modules like time, base64, math, urllib, cgi, etc. It's not the complete standard library, but enough to make you smile. :)

Look, ma, Python talking to JavaScript!

When you interact with Python/pyjs objects everything should work as expected. Basically, you can write normal Python code as long as you don't use advanced features like __metaclass__. With every new release pyjs gets closer to being fully Python-compatible, so even the advanced features will be supported at some point.

However, we won't always get around interacting with a little bit of JS code. You'll at least want to use jQuery and probably also the document variable. You've already seen above how to import JavaScript variables and display something on the page:

from __javascript__ import jQuery, document

jQuery('body').html('Hello world!')
doc = jQuery(document)
# ...

That's pretty simple. When you want to access JavaScript variables you have to import them from the fake __javascript__ module. Then, you can interact with JavaScript objects almost like with Python objects. With the default compilation settings you can pass in strings and numbers unmodified to JavaScript functions. However, dicts and lists must be converted because pyjs uses custom classes for those data types and they're incompatible with JavaScript.

The biggest problem with JavaScript arrays is that you can't access them via [] (e.g., x[0]) because both Python and pyjs convert this to a call to __getitem__(), but that method isn't defined for JavaScript arrays. Also, Python lists can't be passed directly to JavaScript functions because they expect that you call __getitem__() instead of using []. Similarly, len() is only available in Python and .length is only available in JavaScript. So, we have to convert lists when interacting with JavaScript. We can do this in two ways. Either we insert raw JavaScript or we convert the data types:

# Import some JavaScript function which takes an array
from __javascript__ import some_js_func

# The JS() function allows inserting raw JavaScript code
from __pyjamas__ import JS

# Create a JavaScript array via JS()
js_list = JS('[1, 2, 3]')

# Now let's access the first element in the array

# 1. possibility: access array via JS.
item = JS('@{{js_list}}[0]')

# 2. possibility: convert to Python list
py_list = list(js_list)
item = py_list[0]

# Now pass the list to a JavaScript function

# 1. possibility: use the raw JavaScript array
some_js_func(js_list)

# 2. possibility: get the Python list's internal JavaScript array.
# Changes to the internal array will also affect the Python list.
# E.g., when you add an element it also becomes part of the Python list.
some_js_func(py_list.getArray())

# 3. possibility: convert the Python list into a standalone JavaScript array.
# Changes to that array have no effect on the Python list.
from __javascript__ import toJSObjects
some_js_func(toJSObjects(py_list))

Most of the time you'll probably want to use list() and getArray().

There's one important detail, here. When using JS() you have to escape Python variables via @{{}}. That's because pyjs might internally rename a variable, so you have to tell the JS() function which variables you need.

When working with dicts you also have to convert the data types:

# Import some JavaScript function which takes an object
from __javascript__ import other_js_func, toJSObjects

from __pyjamas__ import JS

js_object = JS('{a: 3, b: 5}')
py_dict = dict(js_object)
other_js_func(toJSObjects(py_dict))

In the case of dicts there's really just one way to do it. The dict() constructor knows how to convert JavaScript objects. The universal toJSObjects() function takes care of converting the Python dict back to a JavaScript object.

That's all you need to know when dealing with JavaScript.

Summary

Python code works as you would expect. E.g. you don't have to bind functions to self.
You can also use many modules from the standard library like time, base64, etc.
JavaScript variables can be imported from the __javascript__ module.
When interacting with JavaScript you have to convert list and dict objects.
Download the sample code to see the widget demo in action.

In the next part we'll take a look at integrating the HTML5 canvas element into our little Python app, so we can draw something with the mouse. Update: Part 2 is out.

Cassandra and ElasticSearch backends for Django-nonrel in development

2010-10-22T09:43:00+00:00

This is a quick update: Rob Vaterlaus has started working on a Cassandra backend and Alberto Paro is working on an ElasticSearch backend for Django-nonrel.

The Cassandra backend is still experimental and lacks support for ListField (from djangotoolbox.fields), but overall it already looks very very interesting. This backend comes with experimental secondary indexes support for Cassandra and requires a recent Cassandra 0.7 build. Secondary indexes allow to efficiently query the DB by attributes other than the primary key which makes Cassandra more similar to App Engine and MongoDB than low-level key-value stores. This feature is disabled, by default, though. Currently, without secondary indexes all queries are filtered in memory. The repository contains the installation instructions, so head over and play with the code, if you're fearless. Keep in mind: it's not production ready and it depends on the latest bleeding-edge Cassandra code. Any help with the backend is highly appreciated.

The ElasticSearch backend is also in alpha state. Not all unit tests pass, yet, and for now it only supports a simple subset of ElasticSearch. Basically, you can use string operations like __contains, __istartswith, __regex and you can compare integers via __gt, __lt, etc. and you can order results. Currently, OR queries are not supported, but Alberto is working on that. He also plans to add support for aggregates! If you want to have a stable and feature-complete backend please grab the source from the repo and contribute. Alberto would be happy to accept patches.

So, we now have backends for App Engine, MongoDB, Cassandra, and ElasticSearch. The first two are stable and get used in production. Who would've thought that Django-nonrel would get this far? That's pretty awesome, isn't it? :) If you want to build some other NoSQL backend please read our post about writing a non-relational Django backend which explains our nonrel backend API. If you need more help please ask on the Django-nonrel discussion group. Now that SimpleDB has a free tier it would be an interesting candidate for Django-nonrel, wouldn't it?

JOINs via denormalization for NoSQL coders, Part 3: Ensuring consistency

2010-10-06T13:43:00+00:00

In part 1 and part 2 we introduced the concept of denormalization, materialized views and background tasks in order to emulate JOINs in the to-one direction on NoSQL databases. Now we'll talk about all remaining little but important snippets of the puzzle left over and discuss how to ensure that this method works in real world situation like server crashes.

When to start background tasks?

Let's first remember our current situation:

We have a materialized view (i.e. the model PhotoUserView) containing denormalized properties of users (i.e. gender) and denormalized properties of photos (i.e. popularity). This materialized view is used to emulate JOINs in the to-one direction. Instances of PhotoUserView have to be kept up to date.
If a user edits properties of a photo we have to start a background task in order to update all denormalized fields of the corresponding PhotoUserView entity
If a user changes his/her gender (or her hair color) we have to start background tasks in order to update the denormalized gender of all PhotoUserView entities which point to that specific user

Given this we have to answer when to start background tasks while keeping in mind that the connection / database / web server can fail for many reasons. A straightforward way is to start background tasks right after having saved changes to photos or users (via Django's post_save signal for example). However this solution comes with some problems. Let's take the following evil failing scenario when trying to edit an already existing photo:

User edits a photo
Corresponding submit_view saves these changes
Web server crahses

In this scenario we succeeded in saving the user's changes to one of his/her photos but we didn't start a background task yet in order to update the corresponding materialized view. A query (like the first one from section "Materialized views" in our last post) using this materialized view would fail to find the changed photo because it doesn't contain up-do-date denormalized properties of its corresponding photo.

Another problem with starting background tasks after a save() is that background tasks and a save() can come into update-conflicts. To see this, let's take a closer look at the following example:

At time t1 an update to a photo is being saved (via a submit from an user for example)
At t2 the background task 1 starts right after the save
Background task 1 fetches the data to be denormalized out of the database (photo and user). For some reason we have some delay after this
At t3 another update happens to the same photo
At t4 background task 2 starts and finishes before background task 1 does
Background task 1 finishes its updates to the materialized view using old data resulting in overwriting background task 2's updates

As a result we have an incorrectly updated materialized view because the first background task fetched the data before the second background task saved its updates. This problem exists in general if the delay between a save() and the corresponding background tasks is smaller than the biggest time interval a background task is allowed to be executed in. In such cases overlaps between background tasks can happen.

One way out of both problems is to start background tasks right before a save() using a delay larger than the longest time interval for an update to the materialized view (longest time for getting the data + longest time for saving the data). On App Engine a request can't take longer than 30 seconds for example. This will ensure that background tasks get executed after the saving process is finished and avoid the update-conflicts discussed above because overlaps between background tasks and another save() and its corresponding background task like in the example above can't happen (see figure below). Additionally crashes right after a save() won't stop updates of materialzed views because we've already started the background tasks before. Background tasks will get the photo out of the database via the photo's primary key and update its corresponding materialized view.

Using big delays avoids update-conflicts and ensures correct updates

Multiple updates to photos almost at the same time don't represent any problem for updates of materialized views because they'll start in delayed background tasks i.e. they will get the latest version of the photo out of the database in order to update the corresponding materialized view.

Apart from that, if the database crashes right before saving the user's changes but after starting the background task, the background task will still be executed resulting in updating the materialized view with the same data already saved in the materialized view. This case doesn't represent any problem.

What about inserts?

Now we still have to consider the situation in which a user creates a new photo. Because we start background tasks before a save() we don't have the primary key of the photo i.e. we can't pass the primary key of the photo to the background task. One way to solve this is to mark newly created photos with a unique UUID. The corresponding background task will use this UUID to get the newly inserted photo and create the corresponding materialized view.

How to update materialized views

Until now we discussed when to start background tasks but not how to update materialized views. It's important that each update to a materialized view will rebuild the affected entities of the materialized view from scratch (but not unaffected entities of the materialized view) because otherwise it can result in incorrect updates. Let's assume that changing a user's gender only updates the denormalized gender for the corresponding materialized view and that changes to a photo's title only update the denormalized title for the corresponding materialized view. Given this we can get into the following situation:

User's gender (hair color) changes
Photo's title changes
Background task 1 starts and only gets the user and the corresponding materialized view out of the database (in order to update denormalized_gender)
Background task 2 starts and only gets the photo and the same materialized view out of the database (in order to update denormalized_title)
Background task 1 saves updates to the materialized view i.e. changes denormalized_gender
Background task 2 saves updates to the materialized view i.e. changes denormalized_title overwriting changes by background task 1 i.e. the denormalized_gender

As a result we have an incorrectly updated materialized view because the background task 2 fetched the data before background task 1 saved its updates. To avoid this we always have to completely rebuild the affected entities of the materialized view. Even if only the photo's title changes we have to update the denormalized_gender too! I.e., we have to get the corresponding photo and the user! This ensures correct updates even if overwrites happen because each update rebuilds the affected entities completely i.e. the materialized view is kept up-to-date with the latest data.

Best of both worlds

Using large delays for background tasks may be unsatisfying because user may notice them. For example, a user inserts a photo but can't find it right afterwards. So you may ask "why not start another background task right after we save a photo too?" And yes that's what we suggest. Starting background tasks right after having saved an entity will ensure fast updates so we can use up-to-date materialized views in queries almost immediately. Starting delayed background tasks before we save an entity will ensure the execution of background tasks as well as correct updates to the materialized view.

However we have to keep in mind that background tasks which start immediately after a save can get into conflicts as discussed above. So you might think: "Forget about starting background tasks after a save()", but remember we still have the background task started right before the save. This background task will clean up incorrectly updated materialized views.

The cleanup background tasks will fetch the latest photo and user info so that the materialized view will contain correctly denormalized properties afterwards even if the first background tasks got into update-conflicts.

Even if we get into the same conflict between a third update to a photo and a cleanup background task, this update starts its own delayed background task so that we've ensured a cleanup to come. So in the worst case scenario we have incorrectly updated materialized views which will get cleaned up soon. In a best case scenario we always have immediately updated materialized views. The important aspect here is that we'll always have correct materialized views.

Summarized, the following should be done:

Always start background tasks before saving entities using a large enough delay. This has the nice side effect of cleaning up wrongly updated materialized views
Start background tasks after saving entities to ensure fast updates to the materialized view

Summary

In the last three posts (including this one) we described one possible method to handle JOINs for the to-one side on non-relational databases. Let's summarize the most important points:

JOINs are achieved via denormalization using an additional model (the materialized view)
This solution comes with eventual consistency (in most cases no problem)
Doubles storage requirements (because of the materialized view) but storage is cheap
If the values of denormalized properties are allowed to change, we need background tasks

This method can be implemented on App Engine and MongoDB (in combination with celery for example) as well as on other NoSQL databases.

It becomes clear that using JOINs in the to-one direction on non-relational databases is a mess to deal with. From the view point of a developer, it's far from optimal for the following reasons:

For each model which requires JOINs you have to maintain materialized views by hand
You are forced to rethink your queries and to remember to formulate them using the denormalized fields on a different model (materialized view). Then you have to get the model actually queried for
Materialized views create dependencies between models (photos and users for example) which make them less reusable
Thus you are predestined to make bugs. The result is less productivity and much more pain while coding

So is there a better solution than setting up the whole process by hand each time we need JOINs? We believe that there is a much more elegant way to do so. Our answer is django-dbindexer, which will use the method described in this blog post series so you can use JOINs without having to rethink your queries or to add denormalized fields to your models manually! Just tell django-dbindexer which JOINS you want to use and the indexer will take care of everything else. Stay tuned!

JOINs via denormalization for NoSQL coders, Part 2: Materialized views

2010-09-27T12:13:00+00:00

In part 1 we discussed a workaround for JOINs on non-relational databases using denormalization in cases for which the denormalized properties of the to-one side don't change. In this post we'll show one way to handle JOINs for mutable properties of the to-one side i.e. properties of users.

Let's summarize our current situation:

We have users (the to-one side) and their photos (the to-many side)
Photos contain their users' gender in order to use it in queries which would need JOINs otherwise

It's obvious that a solution for the problem of mutable properties on the to-one side has to keep denormalized properties up to date i.e. each time the user changes his/her gender (or more likely her hair color ;) we have to go through all of the user's photos and update the photos' denormalized gender. It is clear that we get into problems here if a user has thousands of photos to update because such an update would take too long. We need a scalable way to deal with such updates.

Background tasks to the rescue

One way to solve the update-problem is to start a background task each time a user changes his/her gender. It's clear that this solution comes with eventual consistency i.e. changes won't be visible immediately. Nonetheless in most cases that's acceptable and normally background tasks are freaking fast.

Using background tasks in order to update all photos of a given user isn't as simple as it seems. The devil is in the details. Let's assume that a background task tries to update a photo (i.e. some denormalized property of the user) while a user is editing some property of the same photo at the same time i.e. the photo's title for example. In such a scenario it can happen that changes of the user will be overwritten by the background task (or vice versa). To see this take the example of a user changing a photo's title:

Background task gets a photo out of the database in order to update the denormalized gender (task holds version A of the photo)
submit_view gets the same photo out of the database in order to update its title (view holds version A of the photo too)
Background task saves the photo (version B of the photo is saved i.e. denormalized gender updated)
submit_view finished denormalization updates and saves the photo (version C of the photo is saved i.e. photo's title updated)

The problem here is that version C doesn't contain the background task's changes i.e. updates to the denormalized gender (contained in version B) because the submit_view fetched the photo (version A) before the background task saved its changes to the user's denormalized gender. Thus the submit_view never knows about the background task's changes and overwrites them.

One way out of this problem is to use transactions. However this would force us to use transactions in background tasks as well as for all saves to photos in order to avoid such situations. This can slow down our high-traffic web site and forces us to remember to use transactions whenever we want to update a photo. Additionally Django's transactions aren't optimistic so we have to remember to use QuerySet.update(). Also only few non-relational databases support optimistic transactions or atomic UPDATE operations.

Materialized views

Another way to solve this problem is to introduce a third model containing a one-to-one relation to the Photo model. The basic idea behind this is to separate information used for querying (i.e. properties of photos used in queries) and the entities actually containing the information we care about (i.e. the photos itself).

class PhotoUserView(models.model):
    # denormalize all data of the photo
    denormalized_photo_title = models.CharField(max_length=100)
    denormalized_photo_popularity = models.CharField(max_length=100)
    denormalized_photo_user = models.ForeignKey(User)
    ....
    # copy the user's gender into denormalized_gender
    denormalized_user_gender = models.CharField(max_length=10)
    # one-to-one relation used as primary key
    photo = models.OneToOneField(Photo, primary_key=True)

As you can see all fields of the Photo model are being denormalized into the PhotoUserView as well as the gender of the user. This doubles the amount of storage used because we have to store an additional entity for each photo , but storage is cheap so this doesn't represent any disadvantage.

Of course PhotoUserView points to User too because it contains the denormalized foreign key denormalized_user from the photo model. The figure doesn't contain this link because it doesn't help in understanding the concept of materialized views.

Given this model we can use the following efficient query to get photos for which we would've needed JOINs before

photo_pks = PhotoUserView.objects.filter(
    denormalized_user_gender='female',
    denormalized_photo_popularity='high'
    ).values('pk')
female_user_photos = Photo.objects.filter(pk__in=photo_pks)

The trick here is that we use the one-to-one field as the primary key for entities of PhotoUserView so we only need to get their primary keys in order to fetch photos we are interested in. This technique is similar to the relation index (see Building Scalable, Complex Apps on App Engine, same technique used in nonrel-search too). Additionally on App Engine and most other NoSQL databases the pk__in filter can efficiently batch-get all users. Queries which wouldn't need JOINs can still be done on the Photo model directly

popular_photos = Photo.objects.filter(popularity='high')

The important advantage of materialized views is that we don't have conflicts between users editing properties of their photos and background tasks updating denormalized properties anymore. Let's take a closer look at why that's the case: if a user changes his/her gender, the corresponding background task has to update the denormalized gender of all entities on the PhotoUserView model now and not on the Photo model. As a result changes by users editing photos at the same time won't get into conflicts with background tasks updating PhotoUserView entities anymore.

However changes to photos have to start their own background tasks too now in order to keep all denormalized properties in PhotoUserView of the Photo model up to date.

So basically all comes down to the following situations:

User edits properties of a photo => We have to start a background task to update all denormalized properties of the corresponding PhotoUserView entity
User changes its gender => We have to start background tasks to update the denormalized gender of all PhotoUserView entities who point to that specific user

As a result materialized views solve the problem of having to use transactions whenever we want to update a photo.

Now you might object that background tasks will eat too much bandwidth and that map/reduce would be more efficient. However, unless you use CouchDB views, map/reduce isn't incremental. What we've built here is a materialized view which updates only the affected entities. In contrast, map/reduce implementations like in MongoDB rebuild the whole index and that requires a lot more resources if you have a large and popular website. As an optimization, if your database supports transactions or atomic UPDATE operations you can get rid of materialized views, but then you have to be disciplined about using transactions/QuerySet.update() absolutely everywhere in your code. This becomes a problem if you want to reuse existing Django apps. Most of them use Model.save() which isn't safe. Materialized views don't have these disadvantages.

In the next post we'll explain in detail when to start background tasks in order to avoid update conflicts and how to handle evil situations like database crashes.

JOINs via denormalization for NoSQL coders, Part 1: Intro

2010-09-21T11:49:00+00:00

Non-relational databases like App Engine or MongoDB natively don't support operations like JOINs because they don't scale. However in some situations there just exists the need to use such operations. This is the first part of a series presenting one way to handle JOINs (at first in the to-one direction) on NoSQL databases while maintaining scalability. Additionally you'll learn some useful NoSQL coding techniques throughout this series.

Why would I need JOINs?

Let's take the example of users and their photos. Here users represent the to-one side and photos the to-many side:

It's common that users have the possibility to search for pictures of other users. While searching for photos of a specific user is easy to achieve on non-relational databases via

Photo.objects.filter(user=some_user)

for example, searching for photos of users given some specification like

Photo.objects.filter(user__gender='female', popularity='high')

isn't equally simple to achieve. Obviously we need JOINs here (which aren't supported on NoSQL databases) because we span a filter over two models. A straightforward solution to this problem is to get all female users first and then to get all popular photos whose user property points to these users:

user_pks = User.objects.filter(gender='female').values('pk')
female_user_photos = Photo.objects.filter(user__in=user_pks, popularity='high')

Getting only the primary keys of users here is just an optimization so we don't have to fetch whole user instances out of the database. However this solutions comes with the problem of scalability. To demonstrate that, let's take a look at the following situation:

We have a high-traffic website
We want to display 20 photos per result page
But only 1 in 5000 users has popular photos

In such a situation we need to fetch more users in order to get more popular photos. It's easy to see that this doesn't scale because we have to fetch too much content out of the database at the same time. In addition to the scalability problem the site would become very slow and the database would get overloaded. On App Engine we would run into timeouts too.

Denormalization is the answer

Another way to solve the problem is to denormalize the users' gender into their photos i.e. to copy the user's gender into the Photo model.

class Photo(models.model):
    ....
    # copy the user's gender into denormalized_gender
    denormalized_gender = models.CharField(max_length=10)
    ...

Each time we create a new photo we have to denormalize the corresponding user's gender:

new_photo = Photo(user=some_user, denormalized_gender=some_user.gender)
new_photo.save()

Once done, a query for popular photos whose user is female becomes a simple and efficient exact filter on the denormalized gender:

Photo.objects.filter(denormalized_gender='female', popularity='high')

But what if you need more than just the user's gender for some of your queries? Maybe we need the user's age too. Following the example above we just denormalize it into the Photo model. That's it.

If the user's gender doesn't change we've worked around the need for JOINs while maintaining scalability. However we have to keep in mind that changes to a user's gender can happen :P (if you refuse to agree with that replace the user's gender with his hair color :). In such cases we'll get wrong result sets because the query doesn't match the user's denormalized gender on the Photo model anymore. In the next post we'll discuss how to handle these situations.

Get SQL features on NoSQL with django-dbindexer

2010-09-02T12:53:00+00:00

With the just released django-dbindexer you can use all Django lookup types ("iexact", "month", "day", etc.) on non-relational DBs like App Engine and MongoDB, even if they're not natively supported by that DB! Also, django-dbindexer can help you with optimizing your code. For instance, case-insensitive filters on MongoDB can't use indexes, so they're not very efficient. With django-dbindexer they can be handled as efficient case-sensitive filters which utilize an index. Sounds too good to be true? Keep reading.

Non-relational databases have rather limited query APIs. For example, on App Engine you can't do a case-insensitive "iexact" match. Developers on these platforms have to use ugly workarounds to implement unsupported filters. Poor guys. ;) This is how you'd emulate "iexact" and "month" on App Engine (using djangoappengine):

# models.py:

class MyModel(models.Model):
    name = models.CharField(max_length=64)
    lowercase_name = models.CharField(max_length=64, editable=False)
    last_modified = models.DateTimeField(auto_now=True)
    month_last_modified = models.IntegerField(editable=False)

    def save(self, *args, **kwargs):
        self.lowercase_name = self.name.lower()
        self.month_last_modified = self.last_modified.month
        super(MyModel, self).save(*args, **kwargs)

def run_query(name, month):
    MyModel.objects.filter(lowercase_name=name.lower(),
                           month_last_modified=month)

This has several problems:

You have to remember to use lowercase_name and lower() and month_last_modified
You can't reuse existing Django code without analyzing and modifying all models and queries
You can't easily utilize this workaround in the admin interface
Even if you modify an existing app you still have to keep your patches up-to-date
It makes your models and queries unnecessarily complicated

The model above had merely two fields: "name" and "last_modified". It's easy to imagine that in larger projects the models can become a real mess because of all the workarounds. This is just the wrong way to write DB code. Can't we automate this? Yes, we can! Enter django-dbindexer. It allows you to specify index definitions separately from the model, similar to the "index.yaml" file on App Engine and the djangoappengine index definitions feature. Let's see what the example from above looks like with the indexer:

# models.py:

class MyModel(models.Model):
    name = models.CharField(max_length=64)
    last_modified = models.DateTimeField(auto_now=True)

def run_query(name, month):
    MyModel.objects.filter(name__iexact=name, last_modified__month=month)

Looks exactly like with SQL. Nice, isn't it? All you need to make this work is this index definition:

# dbindexes.py:

from models import MyModel
from dbindexer.api import register_index
register_index(MyModel, {'name': 'iexact', 'last_modified': 'month'})

With this little index definition we solve all of the problems mentioned above. Many Django apps like django-registration can be brought to life on non-relational DBs without any modifications to their source. Also, the authors of reusable Django apps can add a simple "index.py" file to their project to make sure that NoSQL developers can use it out-of-the-box. At the same time the code will continue to work on SQL DBs. Even projects that target only non-relational DBs can benefit from the indexer because their code becomes simpler and portable across many NoSQL DBs. The indexer allows the different NoSQL communities to share the same code, sometimes even with the SQL community!

Moreover, the index definition makes the "iexact" filter work efficiently on MongoDB because it converts the string to lowercase, so the "iexact" filter becomes an "exact" filter. This way it can be executed efficiently by using a MongoDB index.

You can also define multiple filters on a single field by passing a tuple of filters:

register_index(MyModel, {'name': ('iexact', 'istartswith', ...)})

Finally, if you want to see a complete example you can download the sample project for App Engine.

With great power comes responsibility ;)

If you're new to NoSQL all this might sound like the Holy Grail that can magically solve all problems. Keep in mind that this is primarily a utility to make advanced developers more productive. You have to understand what's happening internally, so you can make wise design decisions. The internal implementation details of every filter are documented in the reference.

For example, the "contains" filter stores a ListField with all substrings of the indexed field. When querying, it uses "startswith" on that ListField. On App Engine startswith gets converted to ">=" and "<" filters.

Installation

This might sound strange, but the indexer is implemented as a database backend which proxies your actual database backend. For example, if you're on App Engine you have to specify your App Engine backend (here: "gae") and the indexer (here "default") and then you tell the indexer via 'TARGET' which backend should be indexed:

# settings.py:

DATABASES = {
    'default': {
        'ENGINE': 'dbindexer',
        'TARGET': 'gae',
    },
    'gae': {
        'ENGINE': 'djangoappengine.db',
    },
}

MIDDLWARE_CLASSES = (
    # This has to come first
    'autoload.middleware.AutoloadMiddleware',
    ...
)

INSTALLED_APPS = (
    ...
    'autoload',
    'dbindexer',
)

Note that the settings.py in django-testapp already auto-detects and configures the dbindexer somewhere at the bottom of settings.py, so you might not need to change your settings. The middleware has to come first because it must load all index definitions before any other code tries to interact with the database. For more information on auto-loading of specified modules see django-autoload.

Next we have to define a site configuration module which loads the required index definitions. Let's put it in a dbindexes.py in our project root folder. First, we add it to settings.py:

# settings.py:
AUTOLOAD_SITECONF = 'dbindexes'

The actual site configuration module looks like this:

# dbindexes.py:
from dbindexer import autodiscover
autodiscover()

The autodiscover() function will load all modules named dbindexes.py from your settings.INSTALLED_APPS, so they can register their indexes. Alternatively, you can just import the desired index definition modules directly without calling autodiscover().

The future

This is just the beginning. At the moment the indexer "only" adds support for all Django filters, but the plan is to also add support for JOINs by automatically denormalizing your models. We could also support aggregates and distinct() and values() and nested queries and much much more. It won't be 100% all of SQL, but we want to get really close as long as it can be done in a scalable way. This will allow you to write complex queries in a few minutes instead of hours (including unit tests and debugging ;). No more hand-written denormalization and map/reduce and aggregate code. Just tell the indexer what you want to do and it'll handle it for you. Most importantly, it'll do this in a scalable way and internally it'll use the DB exactly like your hand-written code would (or even better ;). The first step is done. If you want to help with the next phase you can drop us a mail: http://groups.google.com/group/django-non-relational.

Permissions with Django-nonrel

2010-08-31T10:44:00+00:00

Quick update: Florian Hahn has implemented a solution for permission handling with Django on non-relational databases. Django's own permission system unfortunately requires JOIN support and thus doesn't work. After his original announcement the code has been optimized, so a permission check can be done with just two database operations. Also, his backend now scales with the number of users. Florian has posted installation and usage instructions on his blog. Check it out if you need permission support in your project.

Final, official GSoC Django NoSQL status update

2010-08-22T10:15:00+00:00

Alex Gaynor has posted a final status update on his Google Summer of Code (GSoC) project which should bring official NoSQL support to Django. Basically, Django now has a working MongoDB backend (not to be confused with the MongoDB backend for Django-nonrel: django-mongodb-engine) and (after lots of skepticism) the ORM indeed needed only minor changes to support non-relational backends (surprise, surprise ;). There are still a few open design issues, but probably the ORM changes will be merged into trunk and the MongoDB backend will become a separate project.

The biggest design issue (in my opinion) is how to handle AutoField. In the GSoC branch, non-relational model code would always need a manually added NativeAutoField(primary_key=True) because many NoSQL DBs use string-based primary keys. As you can see in Django-nonrel, a NativeAutoField is unnecessary. The normal AutoField already works very well and it has the advantage that you can reuse existing Django apps unmodified and you don't need a special NativeAutoField definition in your model. Hopefully this issue will get fixed before official NoSQL support is merged into trunk.

Another issue is about efficiency: In the GSoC branch, save() first checks whether the entity already exists in the DB by doing ...filter(pk=self.pk).exists() and then it decides whether to do an insert() or update() on the DB. Since non-relational DBs normally don't need to distinguish between inserts and updates we could just always call insert(). That would remove an unnecessary query from every save().

The final issue primarily affects App Engine's transaction support: When you delete() an entity Django will also delete all entities that point to that entity (via ForeignKey). This won't work in an App Engine transaction because it would access multiple entity groups. Also, this operation can take very long when batch-deleting multiple entities (via QuerySet.delete()). In the worst case it will cause DeadlineExceededErrors. The solution would be to allow the backend to handle the deletion. This way the App Engine backend (djangoappengine) could delegate the deletion to a background task.

For Django 1.3 it's probably sufficient to only handle the AutoField issue. This doesn't affect App Engine, though, so independent of that we'll port our App Engine backend to Django trunk once the GSoC branch has been merged. This means you will only need Django-nonrel if you want to use App Engine transactions. In all other cases you can use djangoappengine with the official Django release! Isn't this exciting? Maybe some of you have waited for official NoSQL support before porting their model code and now the time has come? What do you think? I'd love to hear your comments.

Using Sass with django-mediagenerator

2010-08-17T12:05:00+00:00

This is the second post in our django-mediagenerator series. If you haven't read it already, please read the first post before continuing: django-mediagenerator: total asset management

What is Sass?

Great that you ask. :) Sass is a high-level language for generating CSS. What? You still write CSS by hand? "That's so bourgeois." (Quick: Who said that in which TV series?) Totally. ;)

Sass to CSS is like Django templates to static HTML. Sass supports variables (e.g.: $mymargin: 10px), reusable code snippets, control statements (@if, etc.), and a more compact indentation-based syntax. You can even use selector inheritance to extend code that is defined in some other Sass file! Also, you can make computations like $mymargin / 2 which can come in very handy e.g. for building fluid grids. Let's see a very simple example of the base syntax:

.content
  padding: 0
  p
    margin-bottom: 2em
  .alert
    color: red

This produces the following CSS code:

.content {
  padding: 0;
}
.content p {
  margin-bottom: 2em;
}
.content .alert {
  color: red;
}

So, nesting can help reduce repetition and the cleaner syntax also makes Sass easier to type and read. Once you start using the advanced features you won't ever want to go back to CSS. But please do yourself a favor and don't use the ugly alternative SCSS syntax. :)

Do I have to convert my CSS by hand?

Nah, you don't have to do it by hand. One solution is to just go to the css2sass website and paste your CSS code there. The website will convert everything to nice Sass source. Alternatively, you can convert the CSS file using the sass-convert command line tool which comes pre-installed with Sass. Let's pretend you want to convert "style.css" to "style.sass":

sass-convert style.css style.sass

It's that easy.

Yes, I want to use it now!

Enough talk. Let's say you have a Sass file called "css/design.sass" and just for the fun of it you also have a CSS file from a jQuery plugin called "css/jquery.plugin.css". Let's combine everything into a "main.css" bundle:

# settings.py

MEDIA_BUNDLES = (
    ('main.css',
        'css/design.sass',
        'css/jquery.plugin.css',
    ),
)

It couldn't be easier. The media generator detects Sass files based on their file extension, so you just have to name the file. But here comes the best part: In your Sass files you can use the handy @import statement to import other files and the media generator will automatically keep track of all dependencies. So, whenever you change one of the dependencies the main Sass file gets recompiled on-the-fly. This is very much like the "sass --watch" development mode, but with the nice advantage that you don't even have to remember to start the Sass command. Barney Stinson would say: "It's legendary!" :)

So, if you want to feel legendary don't wait for it and quickly install Sass and give it a try with django-mediagenerator. It'll boost your CSS productivity to new heights.

nonrel-search updates: auto-completion and separate indexing

2010-08-11T14:50:00+00:00

It was planned already very long to add some remaining features from gae-search to nonrel-search and since we stopped developing gae-search we decided to make some of the premium features open-source. So let's see what changed.

New Features

We basically changed two things in nonrel-search: first it's possible to index a model via a separate definition i.e. without having to modify the model's source itself and second you can use our auto-completion feature from the good old gae-search days. :)

Separate indexing

So let's say you want to index some of your models. With the old version of nonrel-search you had to add a SearchManager to each model you want to search for. With the latest version of nonrel-search you have to define these indexes separately from your model like this:

# post.models
from django.db import models

class Post(models.Model):
    title = models.CharField(max_length=500)
    content = models.TextField()
    author = models.CharField(max_length=500)
    category = models.CharField(max_length=500)

# post.search_indexes
import search
from search.core import porter_stemmer
from post.models import Post

# index used to retrieve posts using the title, content or the
# category.
search.register(Post, ('title', 'content','category', ),
    indexer=porter_stemmer)

As you can see we use the new register function to make posts searchable by title, content and category leaving the author aside. The first parameter defines the model you want to index. The remaining parameters are just the same as for the old SearchManager. The register function automatically adds an index called 'search_index'. Of course it's possible to add multiple such search indexes, just register more of them and pass in a name for the index:

# post.search_indexes
...
search.register(Post, ('category', 'title' ),
    indexer=porter_stemmer,
    search_index='second_search_index')

Here we define a new index called 'second_search_index'.

In addition to defining the indexes we have to make sure that the register function will be executed. Nonrel-search provides a function called autodiscover which automatically searches the INSTALLED_APPS for "search_indexes.py" modules and registers all search indexes.

# search for "search_indexes.py" in all installed apps
import search
search.autodiscover()

You should call autodiscover in your settings.AUTOLOAD_SITECONF module to make sure that your indexes get loaded. See django-autoload for more information on auto-loading modules (so don't forget to install django-autoload). Now it's possible to search for models using the newly added search function:

from search.core import search
posts = search(Post, 'Hello world')

search takes two arguments: the first specifiing the model to search for and the second argument specifies the query used for searching. search automatically uses the index 'search_index'. If you want to use a different index just pass in the name of the desired index:

from search.core import search
# use the auto-completion index explained in the next section
posts = search(Post, 'Hello world', search_index='second_search_index')

Defining indexes separately from the model definition is especially useful for already existing Django apps so you can make them searchable without having to modify their source code. For example, it's possible to make users searchable via their first name and last name just by adding a search index in a separate module.

Auto-completion or "suggest-as-you-type"

Auto-completion is the first premium feature we make open-source. Let's say you want to add auto-completion for category names while creating posts. Nonrel-search makes this an easy job. In order to do so you first have to register a search index which uses the startswith indexer:

# post.search_indexes
...
# auto-completion index used to suggest categories
search.register(Post, ('category', ), indexer=startswith,
    search_index='autocomplete_index')

Then you can use the LiveSearchField which can be integrated into forms to define an auto-completion form:

# post.forms
from django import forms
from post.models import Post
from search.forms import LiveSearchField

class CreatePostForm(forms.ModelForm):
    category = LiveSearchField('/post/live_search/')

    class Meta:
        model = Post

Just pass LiveSearchField the URL to the auto-complete view which retrieves your posts. You can also configure the auto-completion behavior with additional parameters. See the documentation for more information. You can then use this form in your templates to display an auto-completed input field. So, the only thing left is a view returning the data required for auto-completion and to include the necessary Javascript / CSS files into your html:

# post.views
from post.models import Post
from search.views import live_search_results

def live_search(request):
 return live_search_results(request, Post, search_index='autocomplete_index',
     result_item_formatting=
         lambda post: {'value': u'<div>%s</div>' % (post.category),
         'result': post.category, })

Here, we use the function live_search_results and pass in the model class to search on, the name of the index to use for searching (default is 'search_index'), and a formatting function which specifies how your auto-completed posts will be displayed. Note that this function returns a dictionary having two items: 'value' specifies how to display your posts and 'result' specifies what to put into the input field when selecting a post from the auto-completed results list.

If you don't specify any result_item_formatting function, 'value' will be the escaped value of the first indexed property and 'result' will be the unescaped value of the same property.

Note that request has to include a GET parameter 'query'.

Here is the remaining code you have to add so that all necessary Javascript / CSS files will be included using the django-mediagenerator:

# settings
# list your css and js data here
MEDIA_BUNDLES = (
    ('main.css',
        ...,
        'search/jquery.autocomplete.css',
        'search/search.css',
    ),
    ('main.js',
        ...,
        'jquery.js',
        'jquery.livequery.js',
        'search/jquery.autocomplete.js',
        'search/autocomplete_activator.js',
    ),
)

and in your templates add this:

...
{% block css %}
  {% include_media 'main.css' %}
{% endblock %}

{% block js %}
  {% include_media 'main.js' %}
{% endblock %}

You can download the nonrel-search-testapp to get started and play around with nonrel-search. If you create some nice app using nonrel-search please let us know.

All Buttons Pressed

Using dev_appserver with Python 2.7 on App Engine

Using djangoappengine with Python 2.7

F() objects and QuerySet.update() support in djangoappengine

Django's beauty

What's next

Minor updates and API changes

SimpleDB backend for Django-nonrel

Redis backend for Django-nonrel in development

JOINs for NoSQL databases via django-dbindexer - First steps

Let's rock!

InMemoryJOINResolver or ConstantFieldJOINResolver?

Expressive NoSQL - Beyond scalability

Merry Christmas

Porting from App Engine's webapp to django-nonrel

HTML5 offline manifests with django-mediagenerator

Manifest files

django-mediagenerator to the rescue

Serving manifest files

Like a native iPad/iPhone app

Summary

Offline HTML5 canvas app in Python with django-mediagenerator, Part 2: Drawing

Reset the viewport

Start touching the canvas

Drawing it

Style

What about mouse input?

Summary

Offline HTML5 canvas app in Python with django-mediagenerator, Part 1: pyjs

Prerequisites

Compiling Python to JavaScript

Look, ma, Python in my browser!

Look, ma, Python talking to JavaScript!

Summary

Cassandra and ElasticSearch backends for Django-nonrel in development

JOINs via denormalization for NoSQL coders, Part 3: Ensuring consistency

When to start background tasks?

What about inserts?

How to update materialized views

Best of both worlds

Summary

JOINs via denormalization for NoSQL coders, Part 2: Materialized views

Background tasks to the rescue

Materialized views

JOINs via denormalization for NoSQL coders, Part 1: Intro

Why would I need JOINs?

Denormalization is the answer

Get SQL features on NoSQL with django-dbindexer

With great power comes responsibility ;)

Installation

The future

Permissions with Django-nonrel

Final, official GSoC Django NoSQL status update

Using Sass with django-mediagenerator

What is Sass?

Do I have to convert my CSS by hand?

Yes, I want to use it now!

nonrel-search updates: auto-completion and separate indexing

New Features

Separate indexing

Auto-completion or "suggest-as-you-type"

`InMemoryJOINResolver` or `ConstantFieldJOINResolver`?