The Braindonor Network » Coding Blog

jQuery and ASP UpdatePanel Integration

John Hoff — Tue, 01 Jun 2010 17:42:10 +0000

With my latest gig sending me back into the .NET world, I have been busy migrating quite a few of my tools and experience to play nice with ASP.NET sites. One of my favorite open-source tools that is gaining popularity in .NET is jQuery. For several years now, I have been using jQuery heavily to provide rich interaction on websites. When faced with the basic UpdatePanel in ASP.NET, I was presented with a challenge. I wanted a solution that would automatically call jQuery so that new content was properly parsed and updated. Because I had been managing all the AJAX calls in my other development, this was trivially easy to do. UpdatePanel hides all of this inside the ASP AJAX libraries and scripts—making it difficult to figure out where and how to structure my jQuery code. After some research and experimentation, I have a solution up and running on multiple sites now that integrates everything in a very clean manner.

Scoping jQuery Rewrites
The first step is to scope all of the jQuery rewrites. The typical method of integrating jQuery rewrites into a website is to add a document.ready function like so:

$(document).ready(function() {
	$('.content_tab_button').click(function(){
		$('.content_tab_container').load('/newcontent.html');
	});
});

Because the rewrites are only called when the document.ready event fires, integrating with new content pulled in from AJAX needs to be done differently. By placing our rewrites in a new function that recieves the scope being updated, we can have a single initialization handle all of the updating:

function my_initializer(data){
	data.find('.content_tab_button').click(function(){
		$('.content_tab_container').load('/newcontent.html');\
		my_initializer($('.content_tab_container').html());
	});
}
$(document).ready(function() {
	my_initializer($(document));
});

Everything is now scoped cleanly. The rewrites are applied to the document when the page is loaded, and the new content when the tab button is clicked. Even if you are not an ASP.NET developer, please take the above scoping strategy with you!

How The UpdatePanel Works
At the highest level, an UpdatePanel refreshes its entire contents when a specificied event occurs within the panel. Server-side, the event triggering the refresh is processed, and the entire page-stae is regenerated. Then, only the content inside the UpdatePanel is pushed out. This is important for any developer using UpdatePanel to understand: UpdatePanel is much more convenient than it is efficient. As a result, our solution must follow that model.

With that in mind, we look to javascript class that is already managing the UpdatePanel, the Sys.WebForms.PageRequestManager class. In this class, we see an event handler for the pageLoad event. In the documentation, you can see that we are able to fetch the update panels that were just updated. In the example, they are looking for a panel with a specific ID. In our case, we are looking at having our rewrites applied to any panel that was updated:

var panels = args.get_panelsUpdated();
if (panels.length > 0) {
    for (i = 0; i < panels.length; i++) {
        my_initializer($(panels[i]));
    }
}

Because we have already scoped our jQuery code, this will work like a charm.

Putting It All Together
We can put all of the code together now, and we'll have a script that will play cleanly with the UpdatePanel:

function my_initializer(data){
	data.find('.content_tab_button').click(function(){
		$('.content_tab_container').load('/newcontent.html');
		my_initializer($('.content_tab_container').html());
	});
}
function my_pageloaded_initializer(sender, args) {
    var panels = args.get_panelsUpdated();
    if (panels.length > 0) {
        for (i = 0; i < panels.length; i++) {
            my_initializer($(panels[i]));
        }
    }
}
$(document).ready(function() {
	my_initializer($(document));
	Sys.WebForms.PageRequestManager.getInstance()
	    .add_pageLoaded(my_pageloaded_initializer);
});

All of our rewrites are called once—and only once—during page load and UpdatePanel refresh. Enjoy!

New Language Direction With Erlang

John Hoff — Thu, 11 Mar 2010 23:30:26 +0000

In my two most recent posts, Always Learning and Coders at Work, I talked about my personal and professional need to keep learning. My most recent addition to my development arsenal has been Python. Why I considered learning the language a success, I didn't really feel that it a significant amount of new materials and features. As a result, I have chosen to dive into the functional programming language Erlang.

I know that I need to address the issue of Python not adding enough new material to my utility belt before I get anybody riled up. Nearly all of the language features such as its primarily oriented paradigm, dynamic typing, and automatic memory management are not new--I also have experience and a high degree of comfort with C, C++, C#, JavaScript, Perl and PHP. Throughout my introduction to Python, I was simply able to look at how I performed certain tasks in the other languages, and ported things over without any real difficulty. I enjoyed learning Python immensely and I am using it more and more in place of Perl.

Now that I have gotten that disclaimer out of the way, I have to pull us back to the subject at hand: What made me choose Erlang? I wanted to learn something completely different. All of the languages I listed above are procedural languages, while Erlang is a functional language. Also, one of my favorite topics of study is scalable web architecture--yes, I study it even when I don't have to for work or a client. Erlang boasts a concurrent, distributed, fault-tolerant paradigm...all of which are issues that make scalable architecture so challenging and interesting. My final reason is that it isn't a large-scale popular language. Yeah, I'm the guy who will zig while everyone else zags and have a good time doing it. As I am only spending my own time and money in this endeavor, no clients will be harmed during the filming of this episode.

A lot of the blame for getting fired up for Erlang has to be leveled at the interview of Joe Armstrong in Coders at Work. Especially considering the first post I see on his blog dealing with websockets in Chrome. It all really started clicking at that point. Want a highly interactive web application? Have the browser on one end, and an Erlang process on the other end. Want thousands of other people interacting as well? No problem! The recent opening of Facebook Chat to everyone clearly showed off the power of Erlang through ejabberd.

So how am I going about this? Book and interpreter in hand, that's how! I'm still working through the exercises and very thankful that none of the recursion and induction exercises from college fell out of my head during the last 10 years. Next up is getting Yaws up and running on a server and start playing around with web apps. I should have a fun example or two in the next couple of weeks, so stay tuned.

Time for me to head off and go hack up some more Erlang code!

Mako Authentication Required

John Hoff — Thu, 21 Jan 2010 21:11:55 +0000

I have been using my mod_python Mako handler for several months now in my personal projects. For the most part, I have been very happy with Mako and am finding it extremely useful. One issue I have had to wrap my head around has been the inability to halt template execution cleanly. A common practice in many a website has been to flush the current output buffer, display the required authentication information with a form or a redirect, and then end the request—making authentication required on a page of content.

Mako posed a significant challenge to my own thought process as I was designing page templates. I was very used to having the content buffered and being able to perform an internal redirect or transfer. Both mod_perl and ASP.NET offer this feature as part of the various frameworks. Luckily, my work in PHP has helped me pay much closer attention to how I am handling the internal flow of programs. Since most PHP configurations are set to be unbuffered, you have to pay particular attention to when and where you are delivering headers to the client browser.

One way to do solve this was to introduce exception handling to my mako handler. In ASP.NET, when an internal transfer or redirect is made an exception is thrown through the stack. In almost all cases, this exception can be cleanly ignored. To introduce this behavior, I would need to add acceptable exception conditions to my handler. Further, I would have to code my specific authentication methods into my generic handler. This was something I did not wish to do.

Instead of working with exceptions, I looked to the template inheritance of Mako to solve the problem. I start with a non-authenticated template which is in turn inherited by an authenticated template. The page requested inherits the template that corresponds to its need for authentication.

The following example pages illustrate how inheritance can be used to introduce the required authentication.

template.html

<%
    self.page_executed = False
%>

${next.body()}



Page Executed: ${self.page_executed}

require_auth_template.html

<%inherit file="template.html" />
<%!
    from my_library import Account
%>
<%
    self.account = Account.fetch_from_request(req)
%>
% if self.account is None:
    You must log in to continue.
% else:
    ${next.body()}
% endif

page1.html

<%inherit file="template.html" />
<%
    self.page_executed = True
%>
This page will always be displayed in the template.

page2.html

<%inherit file="require_auth_template.html" />
<%
    self.page_executed = True
%>
This page will only be displayed in the template
if the user has logged in.

Output of page1.html

This page will always be displayed in the template.

Page Executed: True

Output of page2.html

You must log in to continue.

Page Executed: False

Because the user is not logged in, the inheritance chain of page2.html ends the request and sends the login content to the top-level template without even calling the template of page2.html.

This technique can be further expanded to introduce exception handling by wrapping the next.body() call in a try block. This would keep all of the necessary exception and authentication logic outside of template handler and in the templates where they belong.

Custom Pages with WordPress Plugins

John Hoff — Tue, 18 Aug 2009 20:16:43 +0000

One of the more interesting challenges I have faced with WordPress is offering custom pages. These could be pages such taking a survey, asking a question, or suggesting a topic. Previously, I had gone about coding these kinds of pages by just sticking the php file in the web root. The problem with that strategy is twofold. The page cannot be disabled without going to the file system. The page also doesn't respect theme changes cleanly.

The solution comes in three parts. The first step is updating the rewrite rules to map a url to an internal value. Next, we map this internal value to a template. Finally, we map this internal template to the theme template.

Most WordPress installations take advantage of URL rewriting on the part of the web server. The goal is to rewrite all blog page requests to index.php. From there, WordPress examines the original URL to determine what content to send. The WordPress rewrite rules allow us to modify this URL. In our case, we want to map a custom page to query variable that we are going to pass into index.php, sending it down the $wp_query.

add_filter('generate_rewrite_rules', 'custom_page_generate_rewrite_rules');
function custom_page_generate_rewrite_rules($wp_rewrite) {
    $custom_page_rules = array(
        'survey' => 'index.php?custom_page=survey',
    );
    $wp_rewrite->rules = $custom_page_rules + $wp_rewrite->rules;
}

Now that we are able to map a url to the $wp_query, we need to resolve that into a template. For this, we utilize the template redirect of WordPress. This allows our plugin to resolve the request into a template call and exit before wordpress attempts to resolve the request.

add_action('template_redirect', 'custom_page_template_redirect');
fnction custom_page_template_redirect() {
    global $wp_query;
    $custom_page = $wp_query->query_vars['custom_page'];
    if ($custom_page == 'survey') {
        include(ABSPATH.'wp-content/plugins/custom_page/survey_page.php');
        exit;
    }
}

For the final piece, we will take a look at just one of the custom pages, survey_page.php. The template redirect hands off the request processing to this page and exits. It is up to this page to interact with the existing theme and display our custom page. The only catch is that the theme must utilize the the_content() template function of WordPress. Otherwise, the proper hooks will not be called and our page will not display correctly.

Fetching the proper file to use from the existing theme is very simple. To do so, we resolve the page template first, single post template second, and finally default to the index.

if (file_exists(TEMPLATEPATH.'/page.php')) {
    include(get_page_template());
}
elseif (file_exists(TEMPLATEPATH.'/single.php')) {
    include(get_single_template());
}
else {
    include(TEMPLATEPATH.'/index.php');
}

Updating the content involves inserting filters onto the_title and the_content. However, we only want these filters to be called during the_loop. Surpressing the filters from get_header, get_sidebar, and get_footer will prevent the custom filters from overrideing page template content.

add_action('get_header', 'custom_page_remove_filters');
add_action('get_sidebar', 'custom_page_remove_filters');
add_action('get_footer', 'custom_page_remove_filters');
function custom_page_remove_filters() {
    remove_filter('the_title', 'custom_page_title');
    remove_filter('the_content', 'custom_page_content');
}

add_action('loop_start', 'custom_page_add_filters');
function custom_page_add_filters() {
    add_filter('the_title', 'custom_page_title');
    add_filter('the_content', 'custom_page_content');
}

I went ahead and put all of this together into a plugin on this site. You can see it in action with the following pages:

I have also attached the source code for the plugin as a full example. You can download it here.

Hijack the WordPress Media Gallery

John Hoff — Thu, 06 Aug 2009 21:05:57 +0000

I have a friend that wants to update a banner on his WordPress blog homepage without changing the template files. I tell him I'd be glad to help him out and start working on a plugin that provides a custom administration page to allow for this. As I am a firm believer in the power of a lazy coder to get things done faster and better, my first thought was: "Why don't I just see if I can hijack the 'Select Image From Gallery' page to do this?" After all, it just pops up into an Iframe page and tells post editer to insert an image. Hijacking that process for my own ends shouldn't take me but a few minutes...and a lot less time than trying to build my own media selector. Like nearly all the content on my site, if you are reading this, I wasn't able to find my answer elsewhere.

First, we will examine how the media library iframe is launched from the post editor and how we can duplicate that in our plugin. So lets crack open the page source and see what we can find:

Looks like a standard hyperlink, with some extra information in the tag. What first jumps out at me is the onclick="return false;". By having an onclick return false in a hyperlink, you ensure that a browser with javascript will never open the link when it is clicked. Sounds pretty silly until you remember that WordPress is a big user of JQuery. From there, we know to look at the class of the hyperlink, class="thickbox". This is a reference to the Thickbox 3.1 library found at /wp-includes/js/thickbox/thickbox.js. This is a library to load an iframe inside an overlay with the background greyed out. Further, it uses the href of the anchor tag to know where to pull the iframe content from.

A quick check of the source on our custom page shows that Thickbox is being loaded in the footer, so we do not have to call it directly from our custom wp-admin page. I can just call up the media library with some html code:


" width="450" height="25" />

The snippet displays an image with the source of $banner_image that can be clicked on to open the Media Gallery iframe. I have also added a hidden form input to store the resulting information passed back from the media library. Once done, I now have an image on my admin page that I can click on to call up the media library.

We're half-way there now! All we have to do next is figure out how the media library sends an image to the post editor. This part isn't as easy. Thanks to thickbox, the media library opens up in an inline-iframe. This means that we can't just view the page source and find our answer. Instead, the only clue we have to hunt for is the "Insert into Post" button. A quick file search turns that up in /wp-admin/includes/media.php. From there, we discover the onclick of the button, onclick="addExtImage.insert();", which is thankfully also in the media.php file. At the end of the function, we see exactly what we are looking for:

var win = window.dialogArguments || opener || parent || top;
win.send_to_editor(html);

The media library calls the function send_to_editor in the scope of the window that opened the media library--our admin page! We can add our own function into the admin page and collect the html that the media library sends us:

function send_to_editor(html) {
    alert(html);
    tb_remove();
}

Once done, we can open the media library from our admin page and select an image. When we hit the "Insert into Post" button, we get an alert with the following content:

All we have to do now, is strip out everything but the image source and updates the form. First, the function eliminates all of the text returned except the image source. It then updates the banner image and the hidden input value that I am using in my form:

function send_to_editor(html) {
    var source = html.match(/src=\".*\" alt/);
    source = source[0].replace(/^src=\"/, "").replace(/" alt$/, "");
    $("#banner_image").attr('src', source);
    $('#banner_image_input").attr('value', source);
    tb_remove();
}

I fire up the admin page a final time, and everything works! I now have a custom admin page that allows my friend to update a banner on his wordpress site using the media gallery.

Apache Optimization and NGINX

John Hoff — Wed, 29 Jul 2009 18:50:51 +0000

Once again, my WordPress friends had to hit me up for some help. These are the same friends that I helped with a Custom Field Search Plugin. They have been maintaining a community site, SuccessNet Online™, and an email mailing list of several hundred thousand participants for Business Networking International. They send out an email every month with a quick update on the community and directing participants to visit the site. For several months they had been using a Lyris server in their office to send out the mailing list. This past month, they started using Lyris HQ to send out the newsletter. When the first newsletter was sent out using the Lyris HQ, the community site quickly ground to a halt and nearly crashed the server. They quickly gave me a call to figure out why the site was having problems and what we could do to prevent it from happening again.

The reason for the site performance issues was immediately apparent once they told me that they were no longer using the Lyris server in their office. They went from an 3-year old server sitting behind a T1 to a managed service sitting behind who knows how big of a pipe. Instead of taking 7-8 hours to deliver the email newsletter using the office server, the new service was able to do it in about 15 minutes! As a result, the peak traffic to the site was compressed from 12 hours to a single hour. Using the default Apache configuration, there was no way they could handle the traffic. With about an hour of work, I was able to optimize their server configuration to handle the traffic. Because I was able to do this without increasing the resource footprint of their server, my friends were very happy.

This same story is repeated time-and-again by businesses large and small that encounter sudden spikes of traffic. The reason for the spike can range from a Slashdotting to the removal of a resource bottleneck. The result is nearly always the same and exposes a fundamental weakness in common Apache configurations.

Anatomy of an Apache Failure
The default configuration of Apache is set up to handle the delivery of static files. This default configuration is able to deliver massive amounts of content within a small footprint. When you begin adding an application layer and a database to the mix, you introduce the ability for Apache to bring even high-end servers to their knees.

Most web applications that run under Apache require the use of the PMP prefork configuration. These include web applications written in PHP, mod_perl, mod_python, etc. The critical section of the Apache configuration that we are going to look follows:


    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0

When this configuration is placed under load, Apache will create up to 150 worker processes to handle the requests. A typical worker process that handles a WordPress site will weigh in at about 30MB of memory. In order for a web server to handle 150 of these workers, it would need at least 4GB of memory! This is also a very optimistic estimate. I've seen quite a few site applications that end up with workers using 200MB of memory and up. The typical web server has no where near enough memory to handle 150 workers.

At idle, this Apache configuration will have ten idle worker processes. Now we start adding traffic, measured in concurrent requests. Once there are 6 concurrent requests, Apache creates an additional worker because it is required to have a minimum of 5 spare workers. The creation of a new worker process is not free and it does have an impact on the load of the server. As load continues to increase, Apache will continue to create worker processes. Once all physical memory is exhausted, additional workers will begin to hit swap memory. The overhead required to use the swap memory imparts even more load on the system. Once this point is reached, worker creation will snowball out of control. One of two things will typically happen—and happen fast: the server enters swap-death; or the database crashes under too many connections.

We have all seen both on many occasions. In the case of swap-death, the browser connection times out. In the case of a crashed database, we get the standard 'unable to connect to database' error. So how do we avoid that happening on our own site?

Strict Apache Resource Limits
The only sure-fire method to preventing load from disabling the web server is to configure Apache so that it will not consume more resources than are available. In the case of the WordPress site I was called in to help on, the server had 1GB of memory and was running both Apache and MySQL. My plan called for having Apache consume no more than 2/3 of the physical memory of the system, leaving the remaining 1/3 for MySQL and the OS. Using the previous 30MB estimate, this allows for a maximum of 20 worker processes:


    StartServers          10
    MinSpareServers       10
    MaxSpareServers       10
    MaxClients            20
    MaxRequestsPerChild    0

As you can see, I also increased the start, min, and max number of processes. I did this to further reduce the frequency of worker creation and destruction—reducing overhead as load increases and decreases. For high traffic sites that utilize a dedicated database server, I recommend setting the start, min, and max servers to max clients.

With the changes made, it is time to test things out. My personal preference is to fire up several looping wget scripts that also download the images, javascript, and stylesheets for the page:

#!/bin/bash
while true; do
wget -pq http://www.server.com/
done

With the resource limits in place, the system load will remain at acceptable levels even with several scripts generating traffic. However, as the number of traffic scripts are increased, the responsiveness of Apache decreases at an alarming rate. When I used this method on the WordPress site, I was experiencing delays with as few as two scripts generating traffic.

Introducing a Reverse Proxy
Now we have Apache in a configuration that no longer runs away with our resources, but the site appears to be significantly slower to respond to requests. The solution is to have separate web server instance to serve all static content. The easiest way to accomplish this is through the use of a reverse proxy. The reverse proxy responds to all web requests to the site. If the request is for static content, it delivers the content. If the request is for dynamic content, it requests the dynamic content from Apache. This ensures that the heavyweight Apache worker processes are only being used to respond to dynamic content requests and a lightweight process is being used to respond to static requests. Otherwise, the heavyweight workers will spend most of their time responding to static requests.

I cannot stress the importance of a reverse proxy enough for high volume sites! Without a reverse proxy, our small pool of heavyweight processes spend a significant amount of time just sending the raw data for the static content. By having a separate web server send the data for static content, you minimize the impact that data delivery will have in tying up worker processes.

My server of choice for the role of reverse proxy is NGINX. NGINX is easy to configure, fast, and stable. The first step of installation is to reconfigure Apache to listen to a port other than 80. Port 8000 or 8080 are commonly used. Once that change has been made and confirmed, NGINX can be installed.

I prefer to set up NGINX initially to proxy all content from port 80 to port 8000. This allows me to verify that everything is working correctly before I set up the static handlers. I then add a proxy for the specific hosts that I need. An edited version of the configuration that I use on this site follows:

nginx.conf

user                    www-data;
worker_processes        5;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;
worker_rlimit_nofile    8196;

events {
    worker_connections    1024;
}

http {
    include         /usr/local/nginx/conf/mime.types;
    include         /usr/local/nginx/conf/proxy.conf;
    default_type    application/octet-stream;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;

    include /usr/local/nginx/conf/sites/*;
}

proxy.conf

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;

sites/000-default:

server {
    listen 80;
    server_name _;

    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

sites/braindonor.net:

server {
    listen 80;
    server_name braindonor.net *.braindonor.net;

    # serve static files directly
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
        root /var/www/braindonor.net/htdocs/;
    }

    # proxy the rest
    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

Be sure to check both the Apache and NGINX log files to ensure that requests are being handled by the appropriate server.

Wrapping Things Up
With the reverse proxy in place and running, there are a few tasks to wrap up.
The LogFormat in apache.conf needs to be updated to reflect the use of a reverse proxy:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

If you are using any log analysis programs, you will need to update them to use the logs from NGINX instead of Apache.



Don't hesitate to leave a comment if you have questions or need some help!



Custom Field Searching WordPress Using Sphinx
John Hoff — Wed, 20 May 2009 19:15:59 +0000
In my previous post on implementing a custom field search in WordPress, I showed how to modify the internal SQL LIKE search that is the default search engine of WordPress.  There are other search engines that WordPress can make use of, most notably Sphinx Search.



Sphinx is a great open-source full-text search engine.  I have used it several times to implement search functionality in content management systems.  I didn't go into using Sphinx with WordPress initially because it is a lot more work.  The default WordPress search is an SQL query performed by PHP.  Sphinx is composed of an indexer, a search daemon, and a client library and the installation of the package is unavailable in shared-hosting environments.



Since more and more people are using WordPress on dedicated servers, I'm not surprised that questions on implementing custom field searching using Sphinx has arrisen.  The good news is that it is very easy to implement!  All you have to do is update the sphinx.conf to pull out the custom fields as fields in the sql_query of your sources.



In my previous example, I worked with the fields: bio, byline, kicker, and deck.  I'll use these same fields again for continuity sake. I'm also using those fields because I have a site that uses these fields for custom search and I could test the changes needed for Sphinx.



In order to add each field, we have to do three things.  Add the column to the select on the wp_post, add a left join to the select on the wp_post, and add a placeholder to the select on wp_comments.



Example Columns:

p.post_content as body, \
t.name as category, \
bio_meta.meta_value as bio, \
byline_meta.meta_value as byline, \
kicker_meta.meta_value as kicker, \
deck_meta.meta_value as deck, \
IF(p.post_type = 'post', 1, 0) as isPost, \
0 as isComment, \


Example Joins:

inner join \
{wp_terms} t on (tt.term_id = t.term_id) \
left join \
{wp_postmeta} bio_meta on (p.ID = bio_meta.post_id and bio_meta.meta_key = 'bio') \
left join \
{wp_postmeta} byline_meta on (p.ID = byline_meta.post_id and byline_meta.meta_key = 'byline') \
left join \
{wp_postmeta} kicker_meta on (p.ID = kicker_meta.post_id and kicker_meta.meta_key = 'kicker') \
left join \
{wp_postmeta} deck_meta on (p.ID = deck_meta.post_id and deck_meta.meta_key = 'deck') \
where \


Example Placeholders:

c.comment_content as body, \
'' as category, \
'' as bio, \
'' as byline, \
'' as kicker, \
'' as deck, \
0 as isPost, \


I have also attached the sphinx.conf distributed with the WordPress plugin so that you can see a more complete example of the implementation.



I hope everyone enjoys this as much as they did the last!


SQLAlchemy and memcached
John Hoff — Thu, 12 Mar 2009 03:27:54 +0000
I think it is safe to say that my personal goal of learning and using Python this winter has been a huge success!  I have now found myself hard at work on a large Python-based project in my spare time and investigating issues that I haven't been able to track down much documentation on.  How to use Memcached in conjunction with SQLAlchemy was one of those issues.



Making Sure Everything Is Ready

Nearly all of the trouble I encountered with SQLAlchemy boiled down to how I initialized my objects.  For a given mapped object, how I am doing things in my projects now is:

Create the engine
Create the schema metadata
Bind the engine to the MetaData
Declare the orm class to be mapped
Create the schema table to be mapped
Create the orm mapping
Bind and the orm mapping to the schema table
Compile the bound orm mapping


Leaving The Session Behind

One of the first things I attempted to do in my web application with SQLAlchemy was detach a mapped object from the session and store it accross page requests.  I wanted to detach the object from the session context to separate class inheritance from the context of the child and avoid the thread-safety dangers of the session.



class DetachedORMObject(object):
    @classmethod
    def fetch_by_field(cls, field, value):
        session = SESSION()
        try:
            class_object = session.query(cls).filter(field == value).one()
        except sqlalchemy.orm.exc.NoResultFound:
            class_object = None
        finally:
            session.close()
        return class_object


The above code works when a common sessionmaker, SESSION, has been defined in the module.  It provides the ability to create a whole collection of utility tasks to be used in implementation.  For my purposes, it allowed for a common path for fetching orm objects from the database that could be overridden in the inheritance chain.



The issue I ran into with the above code dealt with related tables.  The default behavior for SQLAlchemy is for related table attributes to be lazy-loaded.  Once the orm object has been detached from session, the related attribute no longer has the ability to fetch the related data from the database.  This can be fixed by disabling the lazy loading of relations in mappings that are going to be disconnected.



UserTable.mapper = mapper(User, UserTable, \
    properties = { 'user_status': relation(UserStatus, lazy=False)})


Serial Killer

Now that I had objects being detached from the session successfully, I needed to start serializing them.  My goal for serialization is to store the orm objects in memcached.  We can avoid the extra complexity of introducing memcached into the mix at this point by using cpickle for testing.  This is the same package that python-memcached uses to serialize objects as it interacts with a memcached server.



Initially, I thought pickling worked like a charm.  I was able to pickle an object to a string and load it again without encountering errors.  Once I tried to interact with the attributes of the object, I started getting AttributeError exceptions.  After doing some digging, I discovered that the mapping was broken when we attempted to unpickle the object.  The solution was to explicitly compile the mapping.



UserTable.mapper = mapper(User, UserTable, \
    properties = { 'user_status': relation(UserStatus, lazy=False)})
UserTable.mapper.compile()


The compilation step is not included in the tutorials in the SQLAlchemy documentation.  It is implicitly invoked when python interacts with the mapped object through the SQLAlchemy API.  In fact, if program that is un-pickling loads a seperate object first, an AttributeError exception will not be thrown.  By explicitly compiling the mapping, we ensure that pickling can occur successfully before the SQLAlchemy API is called.



Who's Object is it Anyway?

The final issue I encountered was in finding a method to generate a instance key in a generic fashion.  While including abstract methods forcing child classes to provide instance keys to the parent class would work, I wanted a more elegant solution.  Investigating the SQLAlchemy internals pointed me in the direction of the class manager for the mapped objects.  It links the orm objects to the mapper, with in turn links it to the metadata.  I can update the fetch_by_field method as follows:



    @classmethod
    def fetch_by_field(cls, field, value):
        """Fetch the requested object from the cache and database"""
        orm_object = None
        matched_primary_key = True
        for key in cls._sa_class_manager.mapper.primary_key:
            if field.key != key.key:
                matched_primary_key = False
        if matched_primary_key:
            orm_object = cls.get_cached_instance('(' + str(value) + ')')
        if orm_object is None:
            orm_object = super(MemcachedORMObject, cls). \
                fetch_by_field(field, value)
            if orm_object is not None:
                orm_object.set_cached_instance()
        return orm_object


The method first compares the field to the primary key collection in the mapper.  If it is found, it attempts to fetch the cached value.  If not found, it fetches it using the parent fetch_by_field and adds it to the cache.  It should be noted that I am converting the field value to a string when creating the instance key.  A long integer column will append the letter L to the end of __repr__, but not __str__.  Because of this discrepancy, keys may not match between the initial set and subsequent get unless the conversion is explicitly handled.



Putting it All Together

with all of the issues handled, it's time include memcached in the mix.  My solution was to a set of classes that mapped objects can inherit to include the necessary caching behavior.

DetachedORMObject
Implements a generic fetch_by_field as well as the necessary db synchronization tasks.  All session handling in the dependency chain is done here.


MemcachedObject
Abstract class that has the ability to save and restore itself from memcached.


MemcachedORMObject
Implements both DetachedORMObject and MemcachedObject.  Contains the instance key management logic.


Final Notes

I have included a tarball of example scripts that show the full interaction with memcached.  I pulled the database classes straight out of my application, and verified that all of the issues were repeatable using SQLite instead of PostgreSQL.



In my own implementation, I included classes to implement an optional second level of caching using resident memory. I also included a separate class hierarchy covering read-only database objects as well as object collections.


Updated mod_python Mako Handler
John Hoff — Thu, 26 Feb 2009 17:43:43 +0000
I have been using the mod_python handler to parse Mako templates for about a month now in a personal project.  As I have done more and more development on my project, I have naturally encountered shortcomings and errors in my handler.



The first thing I encountered when using the handler was the incorrect status being returned after a template was parsed.  Initially, I was not aware of the 500 status attached to all of the responses because the content came through correctly.  Once I started to implement some AJAX features, everything came crashing down.  While content was indeed returned, the ajax request looked at the status in the response header first, and started erring out all of my AJAX calls.  I tracked this down to how I was previously setting the status so that it could be updated inside of the template:



req.status = apache.OK
req.write(template.render(req=req))
return req.status


After some digging, I discovered that the reason for the 500 status was that I was attempting to set the status of the request.  Instead of attempting to use the request status, I needed to make a temporary place-holder:



req.mako_status = apache.OK
req.write(template.render(req=req))
return req.mako_status


After that fix, it was smooth sailing until I started integrating SQLAlchemy into my application.  The moment I imported SQLAlchemy into my template file, mod_python started throwing error pages concerning the directory to cache eggs into.  Because I had already been testing my ORM objects through the command line, this came as a surprise.  The quick fix was to create a directory for it, and add it to the environment from within the handler.  Once that was done, everything imported cleanly.



Now that I had my ORM objects imported, it was time to use them.  As i was implementing a form-handling routine, I ran into another issue.  When I passed in the raw value from the util.FieldStorage object provided by mod_python, I started receiving ProgrammingError exceptions from SQLAlchemy indicating that it was unable to adapt the select statements.  After some digging, I found one little reference pointing back to the behavior of FieldStorage.  I found the reason for the error here.  FieldStorage was returning StringField objects instead of raw strings when I parsed the form inputs.  When I placed those into the orm calls, it was unable to perform the needed type-matches.  For the solution, I created a wrapper object for FieldStorage to implement some additional methods that did return string values.




mod_python Handler for Mako
John Hoff — Sat, 31 Jan 2009 06:32:49 +0000
One of my personal goals this Winter has been to learn and start using Python.  Learning new languages and putting them to use is something I try to do about every year.  Like so many others out there, messing around with Perl in college really has had an impact on my professional career.  Being a professional web developer who uses Perl—especially through Apache/mod_perl—it was only natural that I began looking at building out a small site using mod_python. Little did I understand how much of a minefield I was stepping into!



Whenever I'm learning a new language, I inevitably explore how others are using it.  This means trying out many of the Python-based web development frameworks that are out there.  One of the biggest surprises for me was the reliance that nearly all of the frameworks place on custom Python application servers, bypassing Apache.  This also extends to most of the template libraries.  It all really depends on a pure Python application server.  That would be fine if my goal was to learn Django or Zope. I'm interested in learning Python...and learning a new language for me has always been very dependent on reinventing a known wheel.  For me, that really meant focusing on mod_python.



So I dive in, get mod_python installed on a system of mine and start working.  Time from apt-get install mod_python to a working Hello World handler: about 15 minutes.  All confident in my ability to get things moving, I start taking another look at Python template libraries.  Everything grinds to a halt again for me almost immediately.  The libraries all have documentation on how to call them from within python programs, but very little information on how to call them from within mod_python.  Two exceptions stood out: Cheetah and Mako.  Cheetah had a couple of handler examples on their wiki...but I was never able to get any of them to work well.  Mako had a working WSGI handler that I was able to study and learn very quickly how to use their library from within mod_python.



Now, everything is up and running, and I'm building a couple of small web applications using mod_python and Mako.  To help fill some of the vacuum out there with respect to mod_python handlers, I decided that I wanted to share the handler I have been working with.  I wouldn't call it production-worthy, but I am enjoying using it a lot.  The Apache configurations required are below and I have attached the handler Python script.  Enjoy!



Apache Configuration:


    PythonPath "sys.path+['/var/www/braindonor.net/lib']"
    PythonDebug On
    SetHandler mod_python
    PythonHandler MakoHandler