The Braindonor Network

The Braindonor Network » sql http://www.braindonor.net Just another WordPress weblog Tue, 01 Jun 2010 17:42:10 +0000 en hourly 1 http://wordpress.org/?v=3.0 Custom Field Searching WordPress Using Sphinx http://www.braindonor.net/coding-blog/custom-field-searching-wordpress-using-sphinx/199/ http://www.braindonor.net/coding-blog/custom-field-searching-wordpress-using-sphinx/199/#comments Wed, 20 May 2009 19:15:59 +0000 John Hoff http://www.braindonor.net/?p=199 previous post on implementing a custom field search in WordPress, I showed how to modify the internal SQL LIKE search that is the default search engine of WordPress. There are other search engines that WordPress can make use of, most notably Sphinx Search.

Sphinx is a great open-source full-text search engine. I have used it several times to implement search functionality in content management systems. I didn't go into using Sphinx with WordPress initially because it is a lot more work. The default WordPress search is an SQL query performed by PHP. Sphinx is composed of an indexer, a search daemon, and a client library and the installation of the package is unavailable in shared-hosting environments.

Since more and more people are using WordPress on dedicated servers, I'm not surprised that questions on implementing custom field searching using Sphinx has arrisen. The good news is that it is very easy to implement! All you have to do is update the sphinx.conf to pull out the custom fields as fields in the sql_query of your sources.

In my previous example, I worked with the fields: bio, byline, kicker, and deck. I'll use these same fields again for continuity sake. I'm also using those fields because I have a site that uses these fields for custom search and I could test the changes needed for Sphinx.

In order to add each field, we have to do three things. Add the column to the select on the wp_post, add a left join to the select on the wp_post, and add a placeholder to the select on wp_comments.

Example Columns:

p.post_content as body, \
t.name as category, \
bio_meta.meta_value as bio, \
byline_meta.meta_value as byline, \
kicker_meta.meta_value as kicker, \
deck_meta.meta_value as deck, \
IF(p.post_type = 'post', 1, 0) as isPost, \
0 as isComment, \

Example Joins:

inner join \
{wp_terms} t on (tt.term_id = t.term_id) \
left join \
{wp_postmeta} bio_meta on (p.ID = bio_meta.post_id and bio_meta.meta_key = 'bio') \
left join \
{wp_postmeta} byline_meta on (p.ID = byline_meta.post_id and byline_meta.meta_key = 'byline') \
left join \
{wp_postmeta} kicker_meta on (p.ID = kicker_meta.post_id and kicker_meta.meta_key = 'kicker') \
left join \
{wp_postmeta} deck_meta on (p.ID = deck_meta.post_id and deck_meta.meta_key = 'deck') \
where \

Example Placeholders:

c.comment_content as body, \
'' as category, \
'' as bio, \
'' as byline, \
'' as kicker, \
'' as deck, \
0 as isPost, \

I have also attached the sphinx.conf distributed with the WordPress plugin so that you can see a more complete example of the implementation.

I hope everyone enjoys this as much as they did the last!]]> http://www.braindonor.net/coding-blog/custom-field-searching-wordpress-using-sphinx/199/feed/ 6 SQLAlchemy and memcached http://www.braindonor.net/coding-blog/sqlalchemy-and-memcached/151/ http://www.braindonor.net/coding-blog/sqlalchemy-and-memcached/151/#comments Thu, 12 Mar 2009 03:27:54 +0000 John Hoff http://www.braindonor.net/?p=151

Making Sure Everything Is Ready
Nearly all of the trouble I encountered with SQLAlchemy boiled down to how I initialized my objects. For a given mapped object, how I am doing things in my projects now is:

Create the engine
Create the schema metadata
Bind the engine to the MetaData
Declare the orm class to be mapped
Create the schema table to be mapped
Create the orm mapping
Bind and the orm mapping to the schema table
Compile the bound orm mapping

Leaving The Session Behind
One of the first things I attempted to do in my web application with SQLAlchemy was detach a mapped object from the session and store it accross page requests. I wanted to detach the object from the session context to separate class inheritance from the context of the child and avoid the thread-safety dangers of the session.

class DetachedORMObject(object):
    @classmethod
    def fetch_by_field(cls, field, value):
        session = SESSION()
        try:
            class_object = session.query(cls).filter(field == value).one()
        except sqlalchemy.orm.exc.NoResultFound:
            class_object = None
        finally:
            session.close()
        return class_object

The above code works when a common sessionmaker, SESSION, has been defined in the module. It provides the ability to create a whole collection of utility tasks to be used in implementation. For my purposes, it allowed for a common path for fetching orm objects from the database that could be overridden in the inheritance chain.

The issue I ran into with the above code dealt with related tables. The default behavior for SQLAlchemy is for related table attributes to be lazy-loaded. Once the orm object has been detached from session, the related attribute no longer has the ability to fetch the related data from the database. This can be fixed by disabling the lazy loading of relations in mappings that are going to be disconnected.

UserTable.mapper = mapper(User, UserTable, \
    properties = { 'user_status': relation(UserStatus, lazy=False)})

Serial Killer
Now that I had objects being detached from the session successfully, I needed to start serializing them. My goal for serialization is to store the orm objects in memcached. We can avoid the extra complexity of introducing memcached into the mix at this point by using cpickle for testing. This is the same package that python-memcached uses to serialize objects as it interacts with a memcached server.

Initially, I thought pickling worked like a charm. I was able to pickle an object to a string and load it again without encountering errors. Once I tried to interact with the attributes of the object, I started getting AttributeError exceptions. After doing some digging, I discovered that the mapping was broken when we attempted to unpickle the object. The solution was to explicitly compile the mapping.

UserTable.mapper = mapper(User, UserTable, \
    properties = { 'user_status': relation(UserStatus, lazy=False)})
UserTable.mapper.compile()

The compilation step is not included in the tutorials in the SQLAlchemy documentation. It is implicitly invoked when python interacts with the mapped object through the SQLAlchemy API. In fact, if program that is un-pickling loads a seperate object first, an AttributeError exception will not be thrown. By explicitly compiling the mapping, we ensure that pickling can occur successfully before the SQLAlchemy API is called.

Who's Object is it Anyway?
The final issue I encountered was in finding a method to generate a instance key in a generic fashion. While including abstract methods forcing child classes to provide instance keys to the parent class would work, I wanted a more elegant solution. Investigating the SQLAlchemy internals pointed me in the direction of the class manager for the mapped objects. It links the orm objects to the mapper, with in turn links it to the metadata. I can update the fetch_by_field method as follows:

    @classmethod
    def fetch_by_field(cls, field, value):
        """Fetch the requested object from the cache and database"""
        orm_object = None
        matched_primary_key = True
        for key in cls._sa_class_manager.mapper.primary_key:
            if field.key != key.key:
                matched_primary_key = False
        if matched_primary_key:
            orm_object = cls.get_cached_instance('(' + str(value) + ')')
        if orm_object is None:
            orm_object = super(MemcachedORMObject, cls). \
                fetch_by_field(field, value)
            if orm_object is not None:
                orm_object.set_cached_instance()
        return orm_object

The method first compares the field to the primary key collection in the mapper. If it is found, it attempts to fetch the cached value. If not found, it fetches it using the parent fetch_by_field and adds it to the cache. It should be noted that I am converting the field value to a string when creating the instance key. A long integer column will append the letter L to the end of __repr__, but not __str__. Because of this discrepancy, keys may not match between the initial set and subsequent get unless the conversion is explicitly handled.

Putting it All Together
with all of the issues handled, it's time include memcached in the mix. My solution was to a set of classes that mapped objects can inherit to include the necessary caching behavior.

DetachedORMObject
Implements a generic fetch_by_field as well as the necessary db synchronization tasks. All session handling in the dependency chain is done here.
MemcachedObject
Abstract class that has the ability to save and restore itself from memcached.
MemcachedORMObject
Implements both DetachedORMObject and MemcachedObject. Contains the instance key management logic.

Final Notes
I have included a tarball of example scripts that show the full interaction with memcached. I pulled the database classes straight out of my application, and verified that all of the issues were repeatable using SQLite instead of PostgreSQL.

In my own implementation, I included classes to implement an optional second level of caching using resident memory. I also included a separate class hierarchy covering read-only database objects as well as object collections.]]> http://www.braindonor.net/coding-blog/sqlalchemy-and-memcached/151/feed/ 4 WordPress Custom Field Search Plugin http://www.braindonor.net/coding-blog/wordpress-custom-field-search-plugin/102/ http://www.braindonor.net/coding-blog/wordpress-custom-field-search-plugin/102/#comments Tue, 27 Jan 2009 18:30:39 +0000 John Hoff http://www.braindonor.net/?p=102

Why custom fields in the first place? Using the custom fields allowed them to add deck, byline, bio, kicker text that their custom template was checking for and formatting appropriately. It really makes life easier when you are dealing with clients that come from the standard publishing background. Not to mention how much easier it is to apply a global formatting change to those elements since you don't have to manually edit each post body.

Here's where I came in. The client noticed that when they entered in a post authors' name(custom field, byline) no search results were being returned. So they called me up for help.

I did some quick digging online and found a few blog posts that had people writing custom queries to fetch the search information. Further digging revealed the WordPress filters that are responsible for building the SQL query for the search: posts_join, posts_groupby, posts_where, and posts_request. For those wanting to play around, the posts_request filter is likely the first place you will want to look. I tossed the following code into our plugin and was able to echo out the entire sql request that was being made for each page:

function custom_search_request($request) {
    echo($request);
    return($request);
}
add_filter('posts_request', 'custom_search_request');

Once that is done, a few page views will let you know what WordPress is doing internally to pull up the posts for each page. You can then update the above to work with the other filters to show you how everything is assembled.

From there, it was a matter of looking at the existing code in the get_posts funciton in wp-includes/query.php and figuring out what additional query information needs to be passed in the filters. Since that function is the densest code I've seen yet in WordPress, I won't go into all of the details. I'll leave that to folks who have good eyes and brave hearts.

When putting it all together, there are several things to note. Always check to make sure that you are currently in a search page and return the original query snippet if you are not. The example I have below adds the custom fields: bio, byline, kicker, and deck.

function custom_search_join($join) {
    if ( is_search() && isset($_GET['s'])) {
        global $wpdb;

       $join = " LEFT JOIN $wpdb->postmeta ON $wpdb->posts.ID = $wpdb->postmeta.post_id ";
    }
    return($join);
}
add_filter('posts_join', 'custom_search_join');

function custom_search_groupby($groupby) {
    if ( is_search() && isset($_GET['s'])) {
        global $wpdb;
        $groupby = " $wpdb->posts.ID ";
    }
    return($groupby);
}
add_filter('posts_groupby', 'custom_search_groupby');

function custom_search_where($where) {
    $old_where = $where;
    if (is_search() && isset($_GET['s'])) {
        global $wpdb;
        $customs = Array('bio', 'byline', 'kicker', 'deck');
        $query = '';
        $var_q = stripslashes($_GET['s']);
        if ($_GET['sentence']) {
            $search_terms = array($var_q);
        }
        else {
            preg_match_all('/".*?("|$)|((?<=[\\s",+])|^)[^\\s",+]+/', $var_q, $matches);
            $search_terms = array_map(create_function('$a', 'return trim($a, "\\"\'\\n\\r ");'), $matches[0]);
        }
        $n = ($_GET['exact']) ? '' : '%';
        $searchand = '';
        foreach((array)$search_terms as $term) {
            $term = addslashes_gpc($term);
            $query .= "{$searchand}(";
                        $query .= "($wpdb->posts.post_title LIKE '{$n}{$term}{$n}')";
            $query .= " OR ($wpdb->posts.post_content LIKE '{$n}{$term}{$n}')";
            foreach($customs as $custom) {
                $query .= " OR (";
                $query .= "($wpdb->postmeta.meta_key = '$custom')";
                $query .= " AND ($wpdb->postmeta.meta_value  LIKE '{$n}{$term}{$n}')";
                $query .= ")";
            }
            $query .= ")";
            $searchand = ' AND ';
        }
        $term = $wpdb->escape($var_q);
        if (!$_GET['sentense'] && Count($search_terms) > 1 && $search_terms[0] != $var_q) {
            $search .= " OR ($wpdb->posts.post_title LIKE '{$n}{$term}{$n}')";
            $search .= " OR ($wpdb->posts.post_content LIKE '{$n}{$term}{$n}')";
        }

        if (!empty($query)) {
            $where = " AND ({$query}) AND ($wpdb->posts.post_status = 'publish') ";
        }
    }

    return($where);
}
add_filter('posts_where', 'custom_search_where');

With all of that done, I added the above code into their plugin and the new fields were being used in the search. It should be noted that the code is adding significant overhead to the post searching. In the above example, each keyword in a search generates 5 SQL LIKE statements instead of 1. In addition, these LIKEs are being performed across joined tables. Because this was for a small website, I do not know the full performance trade-off of the changes. For larger sites, I'll almost always advocate using an external search engine such as Google or Sphinx.]]> http://www.braindonor.net/coding-blog/wordpress-custom-field-search-plugin/102/feed/ 30