The Braindonor Network » wordpress

Custom Pages with WordPress Plugins

John Hoff — Tue, 18 Aug 2009 20:16:43 +0000

One of the more interesting challenges I have faced with WordPress is offering custom pages. These could be pages such taking a survey, asking a question, or suggesting a topic. Previously, I had gone about coding these kinds of pages by just sticking the php file in the web root. The problem with that strategy is twofold. The page cannot be disabled without going to the file system. The page also doesn't respect theme changes cleanly.

The solution comes in three parts. The first step is updating the rewrite rules to map a url to an internal value. Next, we map this internal value to a template. Finally, we map this internal template to the theme template.

Most WordPress installations take advantage of URL rewriting on the part of the web server. The goal is to rewrite all blog page requests to index.php. From there, WordPress examines the original URL to determine what content to send. The WordPress rewrite rules allow us to modify this URL. In our case, we want to map a custom page to query variable that we are going to pass into index.php, sending it down the $wp_query.

add_filter('generate_rewrite_rules', 'custom_page_generate_rewrite_rules');
function custom_page_generate_rewrite_rules($wp_rewrite) {
    $custom_page_rules = array(
        'survey' => 'index.php?custom_page=survey',
    );
    $wp_rewrite->rules = $custom_page_rules + $wp_rewrite->rules;
}

Now that we are able to map a url to the $wp_query, we need to resolve that into a template. For this, we utilize the template redirect of WordPress. This allows our plugin to resolve the request into a template call and exit before wordpress attempts to resolve the request.

add_action('template_redirect', 'custom_page_template_redirect');
fnction custom_page_template_redirect() {
    global $wp_query;
    $custom_page = $wp_query->query_vars['custom_page'];
    if ($custom_page == 'survey') {
        include(ABSPATH.'wp-content/plugins/custom_page/survey_page.php');
        exit;
    }
}

For the final piece, we will take a look at just one of the custom pages, survey_page.php. The template redirect hands off the request processing to this page and exits. It is up to this page to interact with the existing theme and display our custom page. The only catch is that the theme must utilize the the_content() template function of WordPress. Otherwise, the proper hooks will not be called and our page will not display correctly.

Fetching the proper file to use from the existing theme is very simple. To do so, we resolve the page template first, single post template second, and finally default to the index.

if (file_exists(TEMPLATEPATH.'/page.php')) {
    include(get_page_template());
}
elseif (file_exists(TEMPLATEPATH.'/single.php')) {
    include(get_single_template());
}
else {
    include(TEMPLATEPATH.'/index.php');
}

Updating the content involves inserting filters onto the_title and the_content. However, we only want these filters to be called during the_loop. Surpressing the filters from get_header, get_sidebar, and get_footer will prevent the custom filters from overrideing page template content.

add_action('get_header', 'custom_page_remove_filters');
add_action('get_sidebar', 'custom_page_remove_filters');
add_action('get_footer', 'custom_page_remove_filters');
function custom_page_remove_filters() {
    remove_filter('the_title', 'custom_page_title');
    remove_filter('the_content', 'custom_page_content');
}

add_action('loop_start', 'custom_page_add_filters');
function custom_page_add_filters() {
    add_filter('the_title', 'custom_page_title');
    add_filter('the_content', 'custom_page_content');
}

I went ahead and put all of this together into a plugin on this site. You can see it in action with the following pages:

I have also attached the source code for the plugin as a full example. You can download it here.

Hijack the WordPress Media Gallery

John Hoff — Thu, 06 Aug 2009 21:05:57 +0000

I have a friend that wants to update a banner on his WordPress blog homepage without changing the template files. I tell him I'd be glad to help him out and start working on a plugin that provides a custom administration page to allow for this. As I am a firm believer in the power of a lazy coder to get things done faster and better, my first thought was: "Why don't I just see if I can hijack the 'Select Image From Gallery' page to do this?" After all, it just pops up into an Iframe page and tells post editer to insert an image. Hijacking that process for my own ends shouldn't take me but a few minutes...and a lot less time than trying to build my own media selector. Like nearly all the content on my site, if you are reading this, I wasn't able to find my answer elsewhere.

First, we will examine how the media library iframe is launched from the post editor and how we can duplicate that in our plugin. So lets crack open the page source and see what we can find:

Looks like a standard hyperlink, with some extra information in the tag. What first jumps out at me is the onclick="return false;". By having an onclick return false in a hyperlink, you ensure that a browser with javascript will never open the link when it is clicked. Sounds pretty silly until you remember that WordPress is a big user of JQuery. From there, we know to look at the class of the hyperlink, class="thickbox". This is a reference to the Thickbox 3.1 library found at /wp-includes/js/thickbox/thickbox.js. This is a library to load an iframe inside an overlay with the background greyed out. Further, it uses the href of the anchor tag to know where to pull the iframe content from.

A quick check of the source on our custom page shows that Thickbox is being loaded in the footer, so we do not have to call it directly from our custom wp-admin page. I can just call up the media library with some html code:


" width="450" height="25" />

The snippet displays an image with the source of $banner_image that can be clicked on to open the Media Gallery iframe. I have also added a hidden form input to store the resulting information passed back from the media library. Once done, I now have an image on my admin page that I can click on to call up the media library.

We're half-way there now! All we have to do next is figure out how the media library sends an image to the post editor. This part isn't as easy. Thanks to thickbox, the media library opens up in an inline-iframe. This means that we can't just view the page source and find our answer. Instead, the only clue we have to hunt for is the "Insert into Post" button. A quick file search turns that up in /wp-admin/includes/media.php. From there, we discover the onclick of the button, onclick="addExtImage.insert();", which is thankfully also in the media.php file. At the end of the function, we see exactly what we are looking for:

var win = window.dialogArguments || opener || parent || top;
win.send_to_editor(html);

The media library calls the function send_to_editor in the scope of the window that opened the media library--our admin page! We can add our own function into the admin page and collect the html that the media library sends us:

function send_to_editor(html) {
    alert(html);
    tb_remove();
}

Once done, we can open the media library from our admin page and select an image. When we hit the "Insert into Post" button, we get an alert with the following content:

All we have to do now, is strip out everything but the image source and updates the form. First, the function eliminates all of the text returned except the image source. It then updates the banner image and the hidden input value that I am using in my form:

function send_to_editor(html) {
    var source = html.match(/src=\".*\" alt/);
    source = source[0].replace(/^src=\"/, "").replace(/" alt$/, "");
    $("#banner_image").attr('src', source);
    $('#banner_image_input").attr('value', source);
    tb_remove();
}

I fire up the admin page a final time, and everything works! I now have a custom admin page that allows my friend to update a banner on his wordpress site using the media gallery.

Apache Optimization and NGINX

John Hoff — Wed, 29 Jul 2009 18:50:51 +0000

Once again, my WordPress friends had to hit me up for some help. These are the same friends that I helped with a Custom Field Search Plugin. They have been maintaining a community site, SuccessNet Online™, and an email mailing list of several hundred thousand participants for Business Networking International. They send out an email every month with a quick update on the community and directing participants to visit the site. For several months they had been using a Lyris server in their office to send out the mailing list. This past month, they started using Lyris HQ to send out the newsletter. When the first newsletter was sent out using the Lyris HQ, the community site quickly ground to a halt and nearly crashed the server. They quickly gave me a call to figure out why the site was having problems and what we could do to prevent it from happening again.

The reason for the site performance issues was immediately apparent once they told me that they were no longer using the Lyris server in their office. They went from an 3-year old server sitting behind a T1 to a managed service sitting behind who knows how big of a pipe. Instead of taking 7-8 hours to deliver the email newsletter using the office server, the new service was able to do it in about 15 minutes! As a result, the peak traffic to the site was compressed from 12 hours to a single hour. Using the default Apache configuration, there was no way they could handle the traffic. With about an hour of work, I was able to optimize their server configuration to handle the traffic. Because I was able to do this without increasing the resource footprint of their server, my friends were very happy.

This same story is repeated time-and-again by businesses large and small that encounter sudden spikes of traffic. The reason for the spike can range from a Slashdotting to the removal of a resource bottleneck. The result is nearly always the same and exposes a fundamental weakness in common Apache configurations.

Anatomy of an Apache Failure
The default configuration of Apache is set up to handle the delivery of static files. This default configuration is able to deliver massive amounts of content within a small footprint. When you begin adding an application layer and a database to the mix, you introduce the ability for Apache to bring even high-end servers to their knees.

Most web applications that run under Apache require the use of the PMP prefork configuration. These include web applications written in PHP, mod_perl, mod_python, etc. The critical section of the Apache configuration that we are going to look follows:


    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0

When this configuration is placed under load, Apache will create up to 150 worker processes to handle the requests. A typical worker process that handles a WordPress site will weigh in at about 30MB of memory. In order for a web server to handle 150 of these workers, it would need at least 4GB of memory! This is also a very optimistic estimate. I've seen quite a few site applications that end up with workers using 200MB of memory and up. The typical web server has no where near enough memory to handle 150 workers.

At idle, this Apache configuration will have ten idle worker processes. Now we start adding traffic, measured in concurrent requests. Once there are 6 concurrent requests, Apache creates an additional worker because it is required to have a minimum of 5 spare workers. The creation of a new worker process is not free and it does have an impact on the load of the server. As load continues to increase, Apache will continue to create worker processes. Once all physical memory is exhausted, additional workers will begin to hit swap memory. The overhead required to use the swap memory imparts even more load on the system. Once this point is reached, worker creation will snowball out of control. One of two things will typically happen—and happen fast: the server enters swap-death; or the database crashes under too many connections.

We have all seen both on many occasions. In the case of swap-death, the browser connection times out. In the case of a crashed database, we get the standard 'unable to connect to database' error. So how do we avoid that happening on our own site?

Strict Apache Resource Limits
The only sure-fire method to preventing load from disabling the web server is to configure Apache so that it will not consume more resources than are available. In the case of the WordPress site I was called in to help on, the server had 1GB of memory and was running both Apache and MySQL. My plan called for having Apache consume no more than 2/3 of the physical memory of the system, leaving the remaining 1/3 for MySQL and the OS. Using the previous 30MB estimate, this allows for a maximum of 20 worker processes:


    StartServers          10
    MinSpareServers       10
    MaxSpareServers       10
    MaxClients            20
    MaxRequestsPerChild    0

As you can see, I also increased the start, min, and max number of processes. I did this to further reduce the frequency of worker creation and destruction—reducing overhead as load increases and decreases. For high traffic sites that utilize a dedicated database server, I recommend setting the start, min, and max servers to max clients.

With the changes made, it is time to test things out. My personal preference is to fire up several looping wget scripts that also download the images, javascript, and stylesheets for the page:

#!/bin/bash
while true; do
wget -pq http://www.server.com/
done

With the resource limits in place, the system load will remain at acceptable levels even with several scripts generating traffic. However, as the number of traffic scripts are increased, the responsiveness of Apache decreases at an alarming rate. When I used this method on the WordPress site, I was experiencing delays with as few as two scripts generating traffic.

Introducing a Reverse Proxy
Now we have Apache in a configuration that no longer runs away with our resources, but the site appears to be significantly slower to respond to requests. The solution is to have separate web server instance to serve all static content. The easiest way to accomplish this is through the use of a reverse proxy. The reverse proxy responds to all web requests to the site. If the request is for static content, it delivers the content. If the request is for dynamic content, it requests the dynamic content from Apache. This ensures that the heavyweight Apache worker processes are only being used to respond to dynamic content requests and a lightweight process is being used to respond to static requests. Otherwise, the heavyweight workers will spend most of their time responding to static requests.

I cannot stress the importance of a reverse proxy enough for high volume sites! Without a reverse proxy, our small pool of heavyweight processes spend a significant amount of time just sending the raw data for the static content. By having a separate web server send the data for static content, you minimize the impact that data delivery will have in tying up worker processes.

My server of choice for the role of reverse proxy is NGINX. NGINX is easy to configure, fast, and stable. The first step of installation is to reconfigure Apache to listen to a port other than 80. Port 8000 or 8080 are commonly used. Once that change has been made and confirmed, NGINX can be installed.

I prefer to set up NGINX initially to proxy all content from port 80 to port 8000. This allows me to verify that everything is working correctly before I set up the static handlers. I then add a proxy for the specific hosts that I need. An edited version of the configuration that I use on this site follows:

nginx.conf

user                    www-data;
worker_processes        5;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;
worker_rlimit_nofile    8196;

events {
    worker_connections    1024;
}

http {
    include         /usr/local/nginx/conf/mime.types;
    include         /usr/local/nginx/conf/proxy.conf;
    default_type    application/octet-stream;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;

    include /usr/local/nginx/conf/sites/*;
}

proxy.conf

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;

sites/000-default:

server {
    listen 80;
    server_name _;

    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

sites/braindonor.net:

server {
    listen 80;
    server_name braindonor.net *.braindonor.net;

    # serve static files directly
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
        root /var/www/braindonor.net/htdocs/;
    }

    # proxy the rest
    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

Be sure to check both the Apache and NGINX log files to ensure that requests are being handled by the appropriate server.

Wrapping Things Up
With the reverse proxy in place and running, there are a few tasks to wrap up.
The LogFormat in apache.conf needs to be updated to reflect the use of a reverse proxy:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

If you are using any log analysis programs, you will need to update them to use the logs from NGINX instead of Apache.



Don't hesitate to leave a comment if you have questions or need some help!



Custom Field Searching WordPress Using Sphinx
John Hoff — Wed, 20 May 2009 19:15:59 +0000
In my previous post on implementing a custom field search in WordPress, I showed how to modify the internal SQL LIKE search that is the default search engine of WordPress.  There are other search engines that WordPress can make use of, most notably Sphinx Search.



Sphinx is a great open-source full-text search engine.  I have used it several times to implement search functionality in content management systems.  I didn't go into using Sphinx with WordPress initially because it is a lot more work.  The default WordPress search is an SQL query performed by PHP.  Sphinx is composed of an indexer, a search daemon, and a client library and the installation of the package is unavailable in shared-hosting environments.



Since more and more people are using WordPress on dedicated servers, I'm not surprised that questions on implementing custom field searching using Sphinx has arrisen.  The good news is that it is very easy to implement!  All you have to do is update the sphinx.conf to pull out the custom fields as fields in the sql_query of your sources.



In my previous example, I worked with the fields: bio, byline, kicker, and deck.  I'll use these same fields again for continuity sake. I'm also using those fields because I have a site that uses these fields for custom search and I could test the changes needed for Sphinx.



In order to add each field, we have to do three things.  Add the column to the select on the wp_post, add a left join to the select on the wp_post, and add a placeholder to the select on wp_comments.



Example Columns:

p.post_content as body, \
t.name as category, \
bio_meta.meta_value as bio, \
byline_meta.meta_value as byline, \
kicker_meta.meta_value as kicker, \
deck_meta.meta_value as deck, \
IF(p.post_type = 'post', 1, 0) as isPost, \
0 as isComment, \


Example Joins:

inner join \
{wp_terms} t on (tt.term_id = t.term_id) \
left join \
{wp_postmeta} bio_meta on (p.ID = bio_meta.post_id and bio_meta.meta_key = 'bio') \
left join \
{wp_postmeta} byline_meta on (p.ID = byline_meta.post_id and byline_meta.meta_key = 'byline') \
left join \
{wp_postmeta} kicker_meta on (p.ID = kicker_meta.post_id and kicker_meta.meta_key = 'kicker') \
left join \
{wp_postmeta} deck_meta on (p.ID = deck_meta.post_id and deck_meta.meta_key = 'deck') \
where \


Example Placeholders:

c.comment_content as body, \
'' as category, \
'' as bio, \
'' as byline, \
'' as kicker, \
'' as deck, \
0 as isPost, \


I have also attached the sphinx.conf distributed with the WordPress plugin so that you can see a more complete example of the implementation.



I hope everyone enjoys this as much as they did the last!


WordPress Custom Field Search Plugin
John Hoff — Tue, 27 Jan 2009 18:30:39 +0000
Some friends of mine that have started offering custom WordPress designed sites and consulting hit me up a couple of weeks ago for some help.  They had a site for a client that was utilizing custom fields to provide some additional CMS-like functionality to WordPress and had hit a snag.  Their custom fields were not being searched.



Why custom fields in the first place?  Using the custom fields allowed them to add deck, byline, bio, kicker text that their custom template was checking for and formatting appropriately.  It really makes life easier when you are dealing with clients that come from the standard publishing background.  Not to mention how much easier it is to apply a global formatting change to those elements since you don't have to manually edit each post body.



Here's where I came in.  The client noticed that when they entered in a post authors' name(custom field, byline) no search results were being returned.  So they called me up for help.



I did some quick digging online and found a few blog posts that had people writing custom queries to fetch the search information.  Further digging revealed the WordPress filters that are responsible for building the SQL query for the search: posts_join, posts_groupby, posts_where, and posts_request.

For those wanting to play around, the posts_request filter is likely the first place you will want to look.  I tossed the following code into our plugin and was able to echo out the entire sql request that was being made for each page:


function custom_search_request($request) {
    echo($request);
    return($request);
}
add_filter('posts_request', 'custom_search_request');




Once that is done, a few page views will let you know what WordPress is doing internally to pull up the posts for each page.  You can then update the above to work with the other filters to show you how everything is assembled.



From there, it was a matter of looking at the existing code in the get_posts funciton in wp-includes/query.php and figuring out what additional query information needs to be passed in the filters.  Since that function is the densest code I've seen yet in WordPress, I won't go into all of the details.  I'll leave that to folks who have good eyes and brave hearts.



When putting it all together, there are several things to note.  Always check to make sure that you are currently in a search page and return the original query snippet if you are not.  The example I have below adds the custom fields: bio, byline, kicker, and deck.



function custom_search_join($join) {
    if ( is_search() && isset($_GET['s'])) {
        global $wpdb;

       $join = " LEFT JOIN $wpdb->postmeta ON $wpdb->posts.ID = $wpdb->postmeta.post_id ";
    }
    return($join);
}
add_filter('posts_join', 'custom_search_join');

function custom_search_groupby($groupby) {
    if ( is_search() && isset($_GET['s'])) {
        global $wpdb;
        $groupby = " $wpdb->posts.ID ";
    }
    return($groupby);
}
add_filter('posts_groupby', 'custom_search_groupby');

function custom_search_where($where) {
    $old_where = $where;
    if (is_search() && isset($_GET['s'])) {
        global $wpdb;
        $customs = Array('bio', 'byline', 'kicker', 'deck');
        $query = '';
        $var_q = stripslashes($_GET['s']);
        if ($_GET['sentence']) {
            $search_terms = array($var_q);
        }
        else {
            preg_match_all('/".*?("|$)|((?<=[\\s",+])|^)[^\\s",+]+/', $var_q, $matches);
            $search_terms = array_map(create_function('$a', 'return trim($a, "\\"\'\\n\\r ");'), $matches[0]);
        }
        $n = ($_GET['exact']) ? '' : '%';
        $searchand = '';
        foreach((array)$search_terms as $term) {
            $term = addslashes_gpc($term);
            $query .= "{$searchand}(";
                        $query .= "($wpdb->posts.post_title LIKE '{$n}{$term}{$n}')";
            $query .= " OR ($wpdb->posts.post_content LIKE '{$n}{$term}{$n}')";
            foreach($customs as $custom) {
                $query .= " OR (";
                $query .= "($wpdb->postmeta.meta_key = '$custom')";
                $query .= " AND ($wpdb->postmeta.meta_value  LIKE '{$n}{$term}{$n}')";
                $query .= ")";
            }
            $query .= ")";
            $searchand = ' AND ';
        }
        $term = $wpdb->escape($var_q);
        if (!$_GET['sentense'] && Count($search_terms) > 1 && $search_terms[0] != $var_q) {
            $search .= " OR ($wpdb->posts.post_title LIKE '{$n}{$term}{$n}')";
            $search .= " OR ($wpdb->posts.post_content LIKE '{$n}{$term}{$n}')";
        }

        if (!empty($query)) {
            $where = " AND ({$query}) AND ($wpdb->posts.post_status = 'publish') ";
        }
    }

    return($where);
}
add_filter('posts_where', 'custom_search_where');




With all of that done, I added the above code into their plugin and the new fields were being used in the search.  It should be noted that the code is adding significant overhead to the post searching.  In the above example, each keyword in a search generates 5 SQL LIKE statements instead of 1.  In addition, these LIKEs are being performed across joined tables. Because this was for a small website, I do not know the full performance trade-off of the changes.  For larger sites, I'll almost always advocate using an external search engine such as Google or Sphinx.


New Site Launched
John Hoff — Wed, 12 Nov 2008 16:36:04 +0000
The new Braindonor Network site is up and running! I took down the static place-holder page that I have been running for about five years and finally have some content for general consumption.



The site is a WordPress blog with a custom written theme.  With all the time and effort I have been putting in to creating custom WordPress sites, it only made sense to get one of my own up and running.  I should be releasing my theme soon to share some of the tips and tricks I've learned with WordPress along the way.