We really weren't posting often enough recently. Now we'll make up for it with an advanced new asset manager called django-mediagenerator. Those of you who used app-engine-patch might still remember a media generator. This one is completely rewritten with a new backend-based architecture and muchos flexibility for the shiny new HTML5 web-apps world (see the feature comparison table). In this post I'll give you a quick intro and after that I'll make another post about some crazy stuff you can do with the media generator.

Why oh why?

What is an asset manager and why do you need one? Primarily, asset managers are tools for combining and compressing your JS and CSS files into bundles, so instead of many small files your website visitors only need to download a single big JS file and a single big CSS file. This is important because request latency has a much bigger impact on your site's load times than file size. You should definitely read Yahoo's Exceptional Performance and Google's Speed pages to learn more about how to improve your site's performance.

The second important task of an asset manager is to help you with handling HTTP caches. This is done by renaming your files, such that they contain a version tag. For example main.css could be renamed to main-efe88bad66a.css. Whenever the file's contents change the version tag is updated, so the browser will not use the cached version of your file, but download the updated file.

Django already has lots of existing solutions for managing your JS and CSS files and images, so why oh why do we make yet another one? Well, they don't provide the flexibility we need:

• Integration with Sass, PyvaScript, pyjs (the Python->JS compiler used in Pyjamas), etc.
• Flexible backend system for other converters (CleverCSS, etc.)
• Versioning for everything (including images)
• Support for image spriting
• Uncompressed and uncombined output during development with runserver for easier debugging
• Works in sandboxed environments like App Engine

Similar to django-compress, django-mediagenerator stores bundles in the file system. The bundles are defined in settings.py. Some people prefer to define bundles in their templates. Why don't we define bundles in templates?

• It doesn't work in sandboxed hosting environments like App Engine because all files have to be statically pre-generated in advance
• It can lead to unnecessary bundles and thus slow page loads if different pages have only slightly different scripts
• The configuration is not very flexible (you can only list a few JS/CSS files)
• It adds unnecessary checks to every request whether file contents have changed

Even if you'd say that these definitions belong into the templates the disadvantages are much bigger than that little increase in "comfortability".

Let's install it!

We tried our best to make the media generator easy to use while keeping it flexible. On the development server we provide a middleware for serving files. If you want to generate files for production you just run manage.py generatemedia. We also provide two simple template tags for referencing media files in your templates.

So, let's install the media generator. Just download and extract the source code and run setup.py install. Then, go to your project and edit your settings.py:

MIDDLEWARE_CLASSES = (
    # Media middleware has to come first
    'mediagenerator.middleware.MediaMiddleware',
    # ...
)

INSTALLED_APPS = (
    # ...
    'mediagenerator',
)

MEDIA_DEV_MODE = DEBUG
DEV_MEDIA_URL = '/devmedia/'
PRODUCTION_MEDIA_URL = '/media/'

GLOBAL_MEDIA_DIRS = (os.path.join(os.path.dirname(__file__), 'static'),)

It's important that the middleware is the very first middleware in the list. Otherwise media files won't be cached correctly on the development server.

During development via manage.py runserver media is served at DEV_MEDIA_URL by MediaMiddleware. The media generator stores production media in a _generated_media folder. In production, the _generated_media folder should be served from PRODUCTION_MEDIA_URL by your web server. MediaMiddleware is automatically disabled in this case, so Django will not serve any media in production. The MEDIA_DEV_MODE setting specifies whether you're in development or production mode and whether to use development or production URLs (more on that in the templates section).

In the last line we've added the static folder in the project's root directory to the media search path. The media generator will look for media files in that search path. Additionally, all static folders in your INSTALLED_APPS are added to the media search path. Only the admin app is removed, by default. Note that app-specific media should be placed in a subfolder: app/static/app/.... This is very similar to Django templates which are placed in app/templates/app/....

Defining JS and CSS bundles

All required JS and CSS files must be explicitly listed via bundles in settings.py:

MEDIA_BUNDLES = (
    ('main.css',
        'css/reset.css',
        'css/design.css',
    ),
    ('main.js',
        'js/jquery.js',
        'js/jquery.autocomplete.js',
    ),
)

Bundles are defined as tuples where the first entry is the bundle name and the remaining entries list the file names that should be combined. Here, we have a main.css bundle which combines css/reset.css and css/design.css. The second bundle is named main.js and it combines js/jquery.js and js/jquery.autocomplete.js.

Generating media

If you want to generate the media files for production you can just run manage.py generatemedia. This will store all generated files and copy all images into the _generated_media folder. Also, this creates a _generated_media_names.py module which stores the mapping from unversioned file names to versioned file names.

Compressing media

You can tell the media generator to minify your JS and CSS files via YUICompressor in settings.py:

ROOT_MEDIA_FILTERS = {
    'js': 'mediagenerator.filters.yuicompressor.YUICompressor',
    'css': 'mediagenerator.filters.yuicompressor.YUICompressor',
}

YUICOMPRESSOR_PATH = os.path.join(os.path.dirname(os.path.dirname(__file__)),
                                  'yuicompressor.jar')

In your project you'll need a convention where to find the YUICompressor jar file. In this case we assume that it's in the project's parent folder and called yuicompressor.jar.

Let's add media to our templates

Now you know how to define bundles, but that's pretty useless if you can't reference those bundles from within your templates. Now this time I'll make it really simple. :)

In your template you first need to add the media generator template library via

{% load media %}

Then you can include JS and CSS directly using e.g.:

<head>
...
{% include_media 'main.css' %}
...
</head>

This will automatically generate <link> and <script> tags. In development mode (MEDIA_DEV_MODE = True) your files are not combined, so this will generate multiple <script> tags instead of a single one. This is very useful because your JS tracebacks will point directly to the file that caused an exception instead of a huge spaghetti code soup file (in that case only grep can save you).

Optionally, you can also specify the CSS media type via:

{% include_media 'main.css' media='screen,print' %}

Image URLs can be generated using e.g.:

<img src="{% media_url 'some/image.png' %}" />

URLs in CSS files

Whenever you write an URL via url(some/relative/path...) in your CSS files the URL gets rewritten to the actual generated file name. This is only done with relative URLs. Absolute URLs stay untouched (e.g., those that start with / or http(s)://).

Example: If you have a CSS file in css/style.css and you want to access img/icon.png you would write url(../img/icon.png).

Sass is treated differently (we'll explain Sass in the next post). In Sass you'd write url(img/icon.png). In other words, you always write the full media path to the file, without the leading DEV_MEDIA_URL/PRODUCTION_MEDIA_URL, of course. Why the difference? When you @import a Sass module we lose all information about the imported code's original location. Thus, we can't support relative URLs like in CSS. In order to make your code easier to understand and more reusable we decided to use full paths, instead.

Web server cache settings

In order to get the maximum possible performance out of the media generator you have to configure proper caching for your _generated_media folder:

• Disable ETags because they cause unnecessary If-modified-since requests.
• Use Cache-Control: public, max-age=31536000

This will guarantee that your media files are only fetched once on the first visit. All subsequent requests will retrieve the media files from the browser's cache which will help make your website blazingly fast!

Installation on App Engine

Add the following handlers to your app.yaml:

- url: /media/admin
  static_dir: django/contrib/admin/media/
  expiration: '0'

- url: /media
  static_dir: _generated_media/
  expiration: '365d'

If you use Django-nonrel that's all you need.

If you use some alternative Django setup (app-engine-patch, Django helper, etc.) you'll also need to add this at the top of your main.py handler (or whatever you've called your handler in app.yaml):

import os
if os.environ.get('SERVER_SOFTWARE', '').lower().startswith('devel'):
    try:
        from google.appengine.api.mail_stub import subprocess
        sys.modules['subprocess'] = subprocess
        import inspect
        frame = inspect.currentframe().f_back.f_back.f_back
        old_builtin = frame.f_locals['old_builtin']
        subprocess.buffer = old_builtin['buffer']
    except Exception, e:
        import logging
        logging.warn('Could not add the subprocess module to the sandbox: %s' % e)

This will enable Python's subprocess module which is needed by some media generator backends. Note that Django-nonrel uses a safer solution, but the code above is easier to integrate into the other Django helpers.

How to reload pages utilizing the browser cache

When you press CTRL+R or F5 in your browser all cached files get invalidated and refreshed. This can take a while if you have many media files. Since the media generator automatically takes care of versioning your media files even on the local development server you can use a more efficient technique to reload your pages: Click in the URL bar (or use CTRL+L) in your browser and press enter. That will reload the page without invalidating the cache. That way, only modified files will be reloaded. All other files will be retrieved from the cache.

Update: Current versions of Google Chrome have a bug in the preconnect handling code. Chrome creates multiple connections although the development server is only single-threaded. This causes lockups during reloads because one connection is blocking the development server while the other is trying to communicate with the blocked instance. You can fix this by starting Chrome with --disable-preconnect. This way only one connection is created and the lockups are gone.

Now it's your turn

In the repository you can find a sample project with a little CSS example. If you have installed Django it should work out-of-the-box.

This post should get you started with the most common use-cases. There's a lot more that can be done with the media generator. We'll talk about the really exciting stuff in the next media generator post.

Update: The next post is Using Sass with django-mediagenerator