nonrel-search - Full-text search for App Engine & NoSQL on Django

Key Features

Separate and multiple independent indexes on specific fields

You can specify which fields to index and create multiple independent indexes (each with its own fields) for one single model in separate modules.

Porter stemmers and word prefix search

Porter stemmers normalizes your words such that, for example, it doesn't matter whether you search a word in plural or singular. We include stemmers for English and German. Word prefix search allows for searching your data using an arbitrary beginning of a word (e.g. 'ho' matches 'house).

jQuery Auto-completion

Our auto-complete feature provides direct feedback about its functionality. We've taken care of handling all the little details that improve usability: For example, the user move his mouse around while holding down the mouse button, so he can make a correction if he mistakenly clicked on the wrong element.

Installation

Download nonrel-search or clone it via 'hg clone http://bitbucket.org/twanschik/nonrel-search' and put the contained folder 'search ' into your project. Add 'search' to your INSTALLED_APPS in settings.py. Nonrel-search uses ListField such that it is dependend on djangotoolbox. Clone djangotoolbox via 'hg clone http://bitbucket.org/wkornewald/djangotoolbox' and put its contained 'djangotoolbox' folder into your project. Add 'djangotoolbox' to your INSTALLED_APPS in settings.py. The same procedure has to be done for django-autoload since nonrel-search makes use of it. Note that you have to add Javascript / CSS files to your templates if you want use the auto-completion feature.

Indexing and searching your data

One way to make your entities searchable is to index them. The documentation will show you how to create such a search index for your model and how to use this index in order to search your entities. Throughout the documentation we'll use a model called Post to demonstrate the use of nonrel-search. So let's get started.

A post consists of a title, a content, an optional author, a category, and a post rating:

# post.models

from django.db import models

class Post(models.Model):
    title = models.CharField(max_length=500)
    content = models.TextField()
    author = models.CharField(max_length=500)
    category = models.CharField(max_length=500)
    rating = models.IntegerField(choices=[(0, 'zero'), (1, 'one'), (2, 'two'),
        (3, 'three'), (4, 'four'), (5, 'five'), ], default=0)

As you can see, there is nothing special here. Let's say you want to make posts searchable only by title, content and category leaving the author aside. This can be done by adding an index definition in a separate file called "post/search_indexes.py":

# post.search_indexes
import search
from search.core import porter_stemmer
from post.models import Post

# index used to retrieve posts using the title, content or the
# category.
search.register(Post, ('title', 'content','category', ),
    indexer=porter_stemmer)

Use the register function to add a search index for your posts. The first argument specifies the model to search for, the second argument specifies which fields to index. All remaining arguments are optional. The indexer argument specifies that we want to use the Porter Stemmer algorithm which normalizes all indexed words such that, for example, it doesn't matter whether you search a word in plural or singular. You can add multiple indexes for one model. To do so just register a new index and pass in the name for that index via the keyword argument 'search_index'.

You have to make sure that search.register will be executed. Therefor, you have to install django-autoload. nonrel-search provides a function called autodiscover which automatically searches the INSTALLED_APPS for "search_indexes.py" modules and registers all search indexes. autodiscover should be called from within your settings.AUTOLOAD_SITECONF module.

# AUTOLOAD_SITECONF module

# search for "search_indexes.py" in all installed apps
import search
search.autodiscover()

For more information on auto-loading see django-autoload.

Let's assume we have a post with the title 'Greetings' and the content 'Hello little world' saved in our database, then we can find it by calling

from search.core import search
results = search(Post, 'hello greeting')
# results[0] should be our first "Hello little world" post

nonrel-search automatically uses the search index 'search_index'. Here, "greeting" (without "s") matched the title because of the porter stemmer and obviously "hello" matched the content. The result is a Query like object you can add additional filters to:

# ...
results.filter(author='Itachi')
# now results should only contain entities with author set to 'Itachi'

Another possible indexer is 'startswith' which allows you to use word prefix search.

# post.search_indexes
import search
from search.core import startswith
from post.models import Post

# index used to retrieve posts via startswith
search.register(Post, ('title', 'content','category', ),
    indexer=startswith, search_index='autocomplete_index')

# post.views.py
...
results = search(Post, 'he', search_index='autocomplete_index')

Here we pass in the optional argument 'search_index' to tell nonrel-search to use the 'autocomplete_index' index for searching. Searching for 'hel' obviously matched the content because 'Hello' starts with 'hel'.

You should use word prefix search carefully because of some limitations on many nonrelational databses. Basically long text cannot be indexed this way.

Relation index and index backends

By default nonrel-search uses a relation index for storing indexed words in order to optimize later used gets from the database by only getting entities searched for, leaving the index aside. This is done by storing indexed words in a seperate entity. You can disable this behavior by setting 'relation_index' to 'False' when registering your search index:

# post.search_indexes
import search
from search.core import porter_stemmer
from post.models import Post

# index used to retrieve posts using the title, content or the
# category.
search.register(Post, ('title', 'content', 'category'),
    indexer=porter_stemmer, relation_index=False)

Disabling the relation_index results in storing the index in saved entities itself.

When using the relation index you can specify which fields to integrate into the relation index giving the possibility to make complexer queries. By default nonrel-search integrates all fields but you should optimize this otherwise you will store unnecessary data!

Again when using the relation index nonrel-search provides the possibility to use several behaviors in order to create / update indexes. By default nonrel-search creates / updates the index directly when saving entities. But let's say you want to create / update this index using a background task. This can be done by implementing a method 'update_relation_index'. Nonrel-search will search your settings for a property called 'SEARCH_BACKEND' which should hold a string containing the module of where your custom method is implemented. Here is an example of how to set a custom backend:

# settings.py
...
SEARCH_BACKEND = 'search.backends.gae_background_tasks'
...

Nonrel-search provides two build-in backends. search.backends.immediate_update updates the index directly when saving entities and can be used for any database e.g. MongoDB, App Engine, ... . search.backends.gae_background_tasks is an backend which uses App Engine's background tasks to update the index. Therefor it only can be used with App Engine set as your database.

If you want to see a more complete example of how to use nonrel-search take a look at the demo application. There you can find all remaining things related to a complete Django app e.g. the urls, templates and views. Don't forget to copy / clone nonrel-search, djangotoolbox, djangoappengine and django-nonrel into the demo application. See Native Django on App Engine for installation instructions for djangoappengine and django-nonrel.

Now you should have an understanding of how to use nonrel-search. For more information see the reference.

Reference

from search import register
register(model, fields_to_index, indexer, splitter, relation_index,
    integrate, filters, language, search_index)

Take a look at the documentation to see an example.

This is the heart of the search engine. The register function takes care of creating / adding an index to your Model that is to create / add a subclass of ListField.

from search.core import search
results = search(model, query, search_index, language)

settings.SEARCH_BACKEND = 'search.backends.immediate_update'

from search.forms import LiveSearchField
LiveSearchField(src, multiple_values, select_first, auto_fill,
    must_match, match_contains)

from search.views import live_search_results
live_search_results(request, model, index, filters, chain_sort, limit,
        result_item_formatting, query_converter, converter, redirect)

[ // List of people
  { // Result 1
    'result': 'Joe Hacker<injected-tag>', // String that should be copied into
                                // input when field item is selected (not escaped)
    'value': 'Joe Hacker&lt;injected-tag&gt;', // String as displayed in drop-down
                                // results (should be escaped)
    'data':  {'link': '/user/joe-hacker/'}  // Optional; specifies where user should
                                // be redirected when selecting a result
  },
  { // Result 2
    'result': 'Joe User',
    'value': 'Joe User',
    'data':  {}  // When clicking this result you don't get redirected. Instead, the
           // 'result' is copied into the input field.
  },
  ...
]

# in gae-search
filters=('title =', 'hello', 'author =', 'Itachi')
# in nonrel-search
filters={'title':'hello', 'author': 'Itachi'}

# in gae-search
search(Post, 'hello', filters=('author =', 'Itachi'))
# in nonrel-search
search(Post, 'hello').filter(author='Itachi')