The Braindonor Network » Apache

Mako Authentication Required

John Hoff — Thu, 21 Jan 2010 21:11:55 +0000

I have been using my mod_python Mako handler for several months now in my personal projects. For the most part, I have been very happy with Mako and am finding it extremely useful. One issue I have had to wrap my head around has been the inability to halt template execution cleanly. A common practice in many a website has been to flush the current output buffer, display the required authentication information with a form or a redirect, and then end the request—making authentication required on a page of content.

Mako posed a significant challenge to my own thought process as I was designing page templates. I was very used to having the content buffered and being able to perform an internal redirect or transfer. Both mod_perl and ASP.NET offer this feature as part of the various frameworks. Luckily, my work in PHP has helped me pay much closer attention to how I am handling the internal flow of programs. Since most PHP configurations are set to be unbuffered, you have to pay particular attention to when and where you are delivering headers to the client browser.

One way to do solve this was to introduce exception handling to my mako handler. In ASP.NET, when an internal transfer or redirect is made an exception is thrown through the stack. In almost all cases, this exception can be cleanly ignored. To introduce this behavior, I would need to add acceptable exception conditions to my handler. Further, I would have to code my specific authentication methods into my generic handler. This was something I did not wish to do.

Instead of working with exceptions, I looked to the template inheritance of Mako to solve the problem. I start with a non-authenticated template which is in turn inherited by an authenticated template. The page requested inherits the template that corresponds to its need for authentication.

The following example pages illustrate how inheritance can be used to introduce the required authentication.

template.html

<%
    self.page_executed = False
%>

${next.body()}



Page Executed: ${self.page_executed}

require_auth_template.html

<%inherit file="template.html" />
<%!
    from my_library import Account
%>
<%
    self.account = Account.fetch_from_request(req)
%>
% if self.account is None:
    You must log in to continue.
% else:
    ${next.body()}
% endif

page1.html

<%inherit file="template.html" />
<%
    self.page_executed = True
%>
This page will always be displayed in the template.

page2.html

<%inherit file="require_auth_template.html" />
<%
    self.page_executed = True
%>
This page will only be displayed in the template
if the user has logged in.

Output of page1.html

This page will always be displayed in the template.

Page Executed: True

Output of page2.html

You must log in to continue.

Page Executed: False

Because the user is not logged in, the inheritance chain of page2.html ends the request and sends the login content to the top-level template without even calling the template of page2.html.

This technique can be further expanded to introduce exception handling by wrapping the next.body() call in a try block. This would keep all of the necessary exception and authentication logic outside of template handler and in the templates where they belong.

Apache Optimization and NGINX

John Hoff — Wed, 29 Jul 2009 18:50:51 +0000

Once again, my WordPress friends had to hit me up for some help. These are the same friends that I helped with a Custom Field Search Plugin. They have been maintaining a community site, SuccessNet Online™, and an email mailing list of several hundred thousand participants for Business Networking International. They send out an email every month with a quick update on the community and directing participants to visit the site. For several months they had been using a Lyris server in their office to send out the mailing list. This past month, they started using Lyris HQ to send out the newsletter. When the first newsletter was sent out using the Lyris HQ, the community site quickly ground to a halt and nearly crashed the server. They quickly gave me a call to figure out why the site was having problems and what we could do to prevent it from happening again.

The reason for the site performance issues was immediately apparent once they told me that they were no longer using the Lyris server in their office. They went from an 3-year old server sitting behind a T1 to a managed service sitting behind who knows how big of a pipe. Instead of taking 7-8 hours to deliver the email newsletter using the office server, the new service was able to do it in about 15 minutes! As a result, the peak traffic to the site was compressed from 12 hours to a single hour. Using the default Apache configuration, there was no way they could handle the traffic. With about an hour of work, I was able to optimize their server configuration to handle the traffic. Because I was able to do this without increasing the resource footprint of their server, my friends were very happy.

This same story is repeated time-and-again by businesses large and small that encounter sudden spikes of traffic. The reason for the spike can range from a Slashdotting to the removal of a resource bottleneck. The result is nearly always the same and exposes a fundamental weakness in common Apache configurations.

Anatomy of an Apache Failure
The default configuration of Apache is set up to handle the delivery of static files. This default configuration is able to deliver massive amounts of content within a small footprint. When you begin adding an application layer and a database to the mix, you introduce the ability for Apache to bring even high-end servers to their knees.

Most web applications that run under Apache require the use of the PMP prefork configuration. These include web applications written in PHP, mod_perl, mod_python, etc. The critical section of the Apache configuration that we are going to look follows:


    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0

When this configuration is placed under load, Apache will create up to 150 worker processes to handle the requests. A typical worker process that handles a WordPress site will weigh in at about 30MB of memory. In order for a web server to handle 150 of these workers, it would need at least 4GB of memory! This is also a very optimistic estimate. I've seen quite a few site applications that end up with workers using 200MB of memory and up. The typical web server has no where near enough memory to handle 150 workers.

At idle, this Apache configuration will have ten idle worker processes. Now we start adding traffic, measured in concurrent requests. Once there are 6 concurrent requests, Apache creates an additional worker because it is required to have a minimum of 5 spare workers. The creation of a new worker process is not free and it does have an impact on the load of the server. As load continues to increase, Apache will continue to create worker processes. Once all physical memory is exhausted, additional workers will begin to hit swap memory. The overhead required to use the swap memory imparts even more load on the system. Once this point is reached, worker creation will snowball out of control. One of two things will typically happen—and happen fast: the server enters swap-death; or the database crashes under too many connections.

We have all seen both on many occasions. In the case of swap-death, the browser connection times out. In the case of a crashed database, we get the standard 'unable to connect to database' error. So how do we avoid that happening on our own site?

Strict Apache Resource Limits
The only sure-fire method to preventing load from disabling the web server is to configure Apache so that it will not consume more resources than are available. In the case of the WordPress site I was called in to help on, the server had 1GB of memory and was running both Apache and MySQL. My plan called for having Apache consume no more than 2/3 of the physical memory of the system, leaving the remaining 1/3 for MySQL and the OS. Using the previous 30MB estimate, this allows for a maximum of 20 worker processes:


    StartServers          10
    MinSpareServers       10
    MaxSpareServers       10
    MaxClients            20
    MaxRequestsPerChild    0

As you can see, I also increased the start, min, and max number of processes. I did this to further reduce the frequency of worker creation and destruction—reducing overhead as load increases and decreases. For high traffic sites that utilize a dedicated database server, I recommend setting the start, min, and max servers to max clients.

With the changes made, it is time to test things out. My personal preference is to fire up several looping wget scripts that also download the images, javascript, and stylesheets for the page:

#!/bin/bash
while true; do
wget -pq http://www.server.com/
done

With the resource limits in place, the system load will remain at acceptable levels even with several scripts generating traffic. However, as the number of traffic scripts are increased, the responsiveness of Apache decreases at an alarming rate. When I used this method on the WordPress site, I was experiencing delays with as few as two scripts generating traffic.

Introducing a Reverse Proxy
Now we have Apache in a configuration that no longer runs away with our resources, but the site appears to be significantly slower to respond to requests. The solution is to have separate web server instance to serve all static content. The easiest way to accomplish this is through the use of a reverse proxy. The reverse proxy responds to all web requests to the site. If the request is for static content, it delivers the content. If the request is for dynamic content, it requests the dynamic content from Apache. This ensures that the heavyweight Apache worker processes are only being used to respond to dynamic content requests and a lightweight process is being used to respond to static requests. Otherwise, the heavyweight workers will spend most of their time responding to static requests.

I cannot stress the importance of a reverse proxy enough for high volume sites! Without a reverse proxy, our small pool of heavyweight processes spend a significant amount of time just sending the raw data for the static content. By having a separate web server send the data for static content, you minimize the impact that data delivery will have in tying up worker processes.

My server of choice for the role of reverse proxy is NGINX. NGINX is easy to configure, fast, and stable. The first step of installation is to reconfigure Apache to listen to a port other than 80. Port 8000 or 8080 are commonly used. Once that change has been made and confirmed, NGINX can be installed.

I prefer to set up NGINX initially to proxy all content from port 80 to port 8000. This allows me to verify that everything is working correctly before I set up the static handlers. I then add a proxy for the specific hosts that I need. An edited version of the configuration that I use on this site follows:

nginx.conf

user                    www-data;
worker_processes        5;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;
worker_rlimit_nofile    8196;

events {
    worker_connections    1024;
}

http {
    include         /usr/local/nginx/conf/mime.types;
    include         /usr/local/nginx/conf/proxy.conf;
    default_type    application/octet-stream;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;

    include /usr/local/nginx/conf/sites/*;
}

proxy.conf

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;

sites/000-default:

server {
    listen 80;
    server_name _;

    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

sites/braindonor.net:

server {
    listen 80;
    server_name braindonor.net *.braindonor.net;

    # serve static files directly
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
        root /var/www/braindonor.net/htdocs/;
    }

    # proxy the rest
    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

Be sure to check both the Apache and NGINX log files to ensure that requests are being handled by the appropriate server.

Wrapping Things Up
With the reverse proxy in place and running, there are a few tasks to wrap up.
The LogFormat in apache.conf needs to be updated to reflect the use of a reverse proxy:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

If you are using any log analysis programs, you will need to update them to use the logs from NGINX instead of Apache.



Don't hesitate to leave a comment if you have questions or need some help!



Updated mod_python Mako Handler
John Hoff — Thu, 26 Feb 2009 17:43:43 +0000
I have been using the mod_python handler to parse Mako templates for about a month now in a personal project.  As I have done more and more development on my project, I have naturally encountered shortcomings and errors in my handler.



The first thing I encountered when using the handler was the incorrect status being returned after a template was parsed.  Initially, I was not aware of the 500 status attached to all of the responses because the content came through correctly.  Once I started to implement some AJAX features, everything came crashing down.  While content was indeed returned, the ajax request looked at the status in the response header first, and started erring out all of my AJAX calls.  I tracked this down to how I was previously setting the status so that it could be updated inside of the template:



req.status = apache.OK
req.write(template.render(req=req))
return req.status


After some digging, I discovered that the reason for the 500 status was that I was attempting to set the status of the request.  Instead of attempting to use the request status, I needed to make a temporary place-holder:



req.mako_status = apache.OK
req.write(template.render(req=req))
return req.mako_status


After that fix, it was smooth sailing until I started integrating SQLAlchemy into my application.  The moment I imported SQLAlchemy into my template file, mod_python started throwing error pages concerning the directory to cache eggs into.  Because I had already been testing my ORM objects through the command line, this came as a surprise.  The quick fix was to create a directory for it, and add it to the environment from within the handler.  Once that was done, everything imported cleanly.



Now that I had my ORM objects imported, it was time to use them.  As i was implementing a form-handling routine, I ran into another issue.  When I passed in the raw value from the util.FieldStorage object provided by mod_python, I started receiving ProgrammingError exceptions from SQLAlchemy indicating that it was unable to adapt the select statements.  After some digging, I found one little reference pointing back to the behavior of FieldStorage.  I found the reason for the error here.  FieldStorage was returning StringField objects instead of raw strings when I parsed the form inputs.  When I placed those into the orm calls, it was unable to perform the needed type-matches.  For the solution, I created a wrapper object for FieldStorage to implement some additional methods that did return string values.




mod_python Handler for Mako
John Hoff — Sat, 31 Jan 2009 06:32:49 +0000
One of my personal goals this Winter has been to learn and start using Python.  Learning new languages and putting them to use is something I try to do about every year.  Like so many others out there, messing around with Perl in college really has had an impact on my professional career.  Being a professional web developer who uses Perl—especially through Apache/mod_perl—it was only natural that I began looking at building out a small site using mod_python. Little did I understand how much of a minefield I was stepping into!



Whenever I'm learning a new language, I inevitably explore how others are using it.  This means trying out many of the Python-based web development frameworks that are out there.  One of the biggest surprises for me was the reliance that nearly all of the frameworks place on custom Python application servers, bypassing Apache.  This also extends to most of the template libraries.  It all really depends on a pure Python application server.  That would be fine if my goal was to learn Django or Zope. I'm interested in learning Python...and learning a new language for me has always been very dependent on reinventing a known wheel.  For me, that really meant focusing on mod_python.



So I dive in, get mod_python installed on a system of mine and start working.  Time from apt-get install mod_python to a working Hello World handler: about 15 minutes.  All confident in my ability to get things moving, I start taking another look at Python template libraries.  Everything grinds to a halt again for me almost immediately.  The libraries all have documentation on how to call them from within python programs, but very little information on how to call them from within mod_python.  Two exceptions stood out: Cheetah and Mako.  Cheetah had a couple of handler examples on their wiki...but I was never able to get any of them to work well.  Mako had a working WSGI handler that I was able to study and learn very quickly how to use their library from within mod_python.



Now, everything is up and running, and I'm building a couple of small web applications using mod_python and Mako.  To help fill some of the vacuum out there with respect to mod_python handlers, I decided that I wanted to share the handler I have been working with.  I wouldn't call it production-worthy, but I am enjoying using it a lot.  The Apache configurations required are below and I have attached the handler Python script.  Enjoy!



Apache Configuration:


    PythonPath "sys.path+['/var/www/braindonor.net/lib']"
    PythonDebug On
    SetHandler mod_python
    PythonHandler MakoHandler