The Braindonor Network » Python

New Language Direction With Erlang

John Hoff — Thu, 11 Mar 2010 23:30:26 +0000

In my two most recent posts, Always Learning and Coders at Work, I talked about my personal and professional need to keep learning. My most recent addition to my development arsenal has been Python. Why I considered learning the language a success, I didn't really feel that it a significant amount of new materials and features. As a result, I have chosen to dive into the functional programming language Erlang.

I know that I need to address the issue of Python not adding enough new material to my utility belt before I get anybody riled up. Nearly all of the language features such as its primarily oriented paradigm, dynamic typing, and automatic memory management are not new--I also have experience and a high degree of comfort with C, C++, C#, JavaScript, Perl and PHP. Throughout my introduction to Python, I was simply able to look at how I performed certain tasks in the other languages, and ported things over without any real difficulty. I enjoyed learning Python immensely and I am using it more and more in place of Perl.

Now that I have gotten that disclaimer out of the way, I have to pull us back to the subject at hand: What made me choose Erlang? I wanted to learn something completely different. All of the languages I listed above are procedural languages, while Erlang is a functional language. Also, one of my favorite topics of study is scalable web architecture--yes, I study it even when I don't have to for work or a client. Erlang boasts a concurrent, distributed, fault-tolerant paradigm...all of which are issues that make scalable architecture so challenging and interesting. My final reason is that it isn't a large-scale popular language. Yeah, I'm the guy who will zig while everyone else zags and have a good time doing it. As I am only spending my own time and money in this endeavor, no clients will be harmed during the filming of this episode.

A lot of the blame for getting fired up for Erlang has to be leveled at the interview of Joe Armstrong in Coders at Work. Especially considering the first post I see on his blog dealing with websockets in Chrome. It all really started clicking at that point. Want a highly interactive web application? Have the browser on one end, and an Erlang process on the other end. Want thousands of other people interacting as well? No problem! The recent opening of Facebook Chat to everyone clearly showed off the power of Erlang through ejabberd.

So how am I going about this? Book and interpreter in hand, that's how! I'm still working through the exercises and very thankful that none of the recursion and induction exercises from college fell out of my head during the last 10 years. Next up is getting Yaws up and running on a server and start playing around with web apps. I should have a fun example or two in the next couple of weeks, so stay tuned.

Time for me to head off and go hack up some more Erlang code!

Mako Authentication Required

John Hoff — Thu, 21 Jan 2010 21:11:55 +0000

I have been using my mod_python Mako handler for several months now in my personal projects. For the most part, I have been very happy with Mako and am finding it extremely useful. One issue I have had to wrap my head around has been the inability to halt template execution cleanly. A common practice in many a website has been to flush the current output buffer, display the required authentication information with a form or a redirect, and then end the request—making authentication required on a page of content.

Mako posed a significant challenge to my own thought process as I was designing page templates. I was very used to having the content buffered and being able to perform an internal redirect or transfer. Both mod_perl and ASP.NET offer this feature as part of the various frameworks. Luckily, my work in PHP has helped me pay much closer attention to how I am handling the internal flow of programs. Since most PHP configurations are set to be unbuffered, you have to pay particular attention to when and where you are delivering headers to the client browser.

One way to do solve this was to introduce exception handling to my mako handler. In ASP.NET, when an internal transfer or redirect is made an exception is thrown through the stack. In almost all cases, this exception can be cleanly ignored. To introduce this behavior, I would need to add acceptable exception conditions to my handler. Further, I would have to code my specific authentication methods into my generic handler. This was something I did not wish to do.

Instead of working with exceptions, I looked to the template inheritance of Mako to solve the problem. I start with a non-authenticated template which is in turn inherited by an authenticated template. The page requested inherits the template that corresponds to its need for authentication.

The following example pages illustrate how inheritance can be used to introduce the required authentication.

template.html

<%
    self.page_executed = False
%>

${next.body()}



Page Executed: ${self.page_executed}

require_auth_template.html

<%inherit file="template.html" />
<%!
    from my_library import Account
%>
<%
    self.account = Account.fetch_from_request(req)
%>
% if self.account is None:
    You must log in to continue.
% else:
    ${next.body()}
% endif

page1.html

<%inherit file="template.html" />
<%
    self.page_executed = True
%>
This page will always be displayed in the template.

page2.html

<%inherit file="require_auth_template.html" />
<%
    self.page_executed = True
%>
This page will only be displayed in the template
if the user has logged in.

Output of page1.html

This page will always be displayed in the template.

Page Executed: True

Output of page2.html

You must log in to continue.

Page Executed: False

Because the user is not logged in, the inheritance chain of page2.html ends the request and sends the login content to the top-level template without even calling the template of page2.html.

This technique can be further expanded to introduce exception handling by wrapping the next.body() call in a try block. This would keep all of the necessary exception and authentication logic outside of template handler and in the templates where they belong.

SQLAlchemy and memcached

John Hoff — Thu, 12 Mar 2009 03:27:54 +0000

I think it is safe to say that my personal goal of learning and using Python this winter has been a huge success! I have now found myself hard at work on a large Python-based project in my spare time and investigating issues that I haven't been able to track down much documentation on. How to use Memcached in conjunction with SQLAlchemy was one of those issues.

Making Sure Everything Is Ready
Nearly all of the trouble I encountered with SQLAlchemy boiled down to how I initialized my objects. For a given mapped object, how I am doing things in my projects now is:

Create the engine
Create the schema metadata
Bind the engine to the MetaData
Declare the orm class to be mapped
Create the schema table to be mapped
Create the orm mapping
Bind and the orm mapping to the schema table
Compile the bound orm mapping

Leaving The Session Behind
One of the first things I attempted to do in my web application with SQLAlchemy was detach a mapped object from the session and store it accross page requests. I wanted to detach the object from the session context to separate class inheritance from the context of the child and avoid the thread-safety dangers of the session.

class DetachedORMObject(object):
    @classmethod
    def fetch_by_field(cls, field, value):
        session = SESSION()
        try:
            class_object = session.query(cls).filter(field == value).one()
        except sqlalchemy.orm.exc.NoResultFound:
            class_object = None
        finally:
            session.close()
        return class_object

The above code works when a common sessionmaker, SESSION, has been defined in the module. It provides the ability to create a whole collection of utility tasks to be used in implementation. For my purposes, it allowed for a common path for fetching orm objects from the database that could be overridden in the inheritance chain.

The issue I ran into with the above code dealt with related tables. The default behavior for SQLAlchemy is for related table attributes to be lazy-loaded. Once the orm object has been detached from session, the related attribute no longer has the ability to fetch the related data from the database. This can be fixed by disabling the lazy loading of relations in mappings that are going to be disconnected.

UserTable.mapper = mapper(User, UserTable, \
    properties = { 'user_status': relation(UserStatus, lazy=False)})

Serial Killer
Now that I had objects being detached from the session successfully, I needed to start serializing them. My goal for serialization is to store the orm objects in memcached. We can avoid the extra complexity of introducing memcached into the mix at this point by using cpickle for testing. This is the same package that python-memcached uses to serialize objects as it interacts with a memcached server.

Initially, I thought pickling worked like a charm. I was able to pickle an object to a string and load it again without encountering errors. Once I tried to interact with the attributes of the object, I started getting AttributeError exceptions. After doing some digging, I discovered that the mapping was broken when we attempted to unpickle the object. The solution was to explicitly compile the mapping.

UserTable.mapper = mapper(User, UserTable, \
    properties = { 'user_status': relation(UserStatus, lazy=False)})
UserTable.mapper.compile()

The compilation step is not included in the tutorials in the SQLAlchemy documentation. It is implicitly invoked when python interacts with the mapped object through the SQLAlchemy API. In fact, if program that is un-pickling loads a seperate object first, an AttributeError exception will not be thrown. By explicitly compiling the mapping, we ensure that pickling can occur successfully before the SQLAlchemy API is called.

Who's Object is it Anyway?
The final issue I encountered was in finding a method to generate a instance key in a generic fashion. While including abstract methods forcing child classes to provide instance keys to the parent class would work, I wanted a more elegant solution. Investigating the SQLAlchemy internals pointed me in the direction of the class manager for the mapped objects. It links the orm objects to the mapper, with in turn links it to the metadata. I can update the fetch_by_field method as follows:

    @classmethod
    def fetch_by_field(cls, field, value):
        """Fetch the requested object from the cache and database"""
        orm_object = None
        matched_primary_key = True
        for key in cls._sa_class_manager.mapper.primary_key:
            if field.key != key.key:
                matched_primary_key = False
        if matched_primary_key:
            orm_object = cls.get_cached_instance('(' + str(value) + ')')
        if orm_object is None:
            orm_object = super(MemcachedORMObject, cls). \
                fetch_by_field(field, value)
            if orm_object is not None:
                orm_object.set_cached_instance()
        return orm_object

The method first compares the field to the primary key collection in the mapper. If it is found, it attempts to fetch the cached value. If not found, it fetches it using the parent fetch_by_field and adds it to the cache. It should be noted that I am converting the field value to a string when creating the instance key. A long integer column will append the letter L to the end of __repr__, but not __str__. Because of this discrepancy, keys may not match between the initial set and subsequent get unless the conversion is explicitly handled.

Putting it All Together
with all of the issues handled, it's time include memcached in the mix. My solution was to a set of classes that mapped objects can inherit to include the necessary caching behavior.

DetachedORMObject
Implements a generic fetch_by_field as well as the necessary db synchronization tasks. All session handling in the dependency chain is done here.
MemcachedObject
Abstract class that has the ability to save and restore itself from memcached.
MemcachedORMObject
Implements both DetachedORMObject and MemcachedObject. Contains the instance key management logic.

Final Notes
I have included a tarball of example scripts that show the full interaction with memcached. I pulled the database classes straight out of my application, and verified that all of the issues were repeatable using SQLite instead of PostgreSQL.

In my own implementation, I included classes to implement an optional second level of caching using resident memory. I also included a separate class hierarchy covering read-only database objects as well as object collections.

Updated mod_python Mako Handler

John Hoff — Thu, 26 Feb 2009 17:43:43 +0000

I have been using the mod_python handler to parse Mako templates for about a month now in a personal project. As I have done more and more development on my project, I have naturally encountered shortcomings and errors in my handler.

The first thing I encountered when using the handler was the incorrect status being returned after a template was parsed. Initially, I was not aware of the 500 status attached to all of the responses because the content came through correctly. Once I started to implement some AJAX features, everything came crashing down. While content was indeed returned, the ajax request looked at the status in the response header first, and started erring out all of my AJAX calls. I tracked this down to how I was previously setting the status so that it could be updated inside of the template:

req.status = apache.OK
req.write(template.render(req=req))
return req.status

After some digging, I discovered that the reason for the 500 status was that I was attempting to set the status of the request. Instead of attempting to use the request status, I needed to make a temporary place-holder:

req.mako_status = apache.OK
req.write(template.render(req=req))
return req.mako_status

After that fix, it was smooth sailing until I started integrating SQLAlchemy into my application. The moment I imported SQLAlchemy into my template file, mod_python started throwing error pages concerning the directory to cache eggs into. Because I had already been testing my ORM objects through the command line, this came as a surprise. The quick fix was to create a directory for it, and add it to the environment from within the handler. Once that was done, everything imported cleanly.

Now that I had my ORM objects imported, it was time to use them. As i was implementing a form-handling routine, I ran into another issue. When I passed in the raw value from the util.FieldStorage object provided by mod_python, I started receiving ProgrammingError exceptions from SQLAlchemy indicating that it was unable to adapt the select statements. After some digging, I found one little reference pointing back to the behavior of FieldStorage. I found the reason for the error here. FieldStorage was returning StringField objects instead of raw strings when I parsed the form inputs. When I placed those into the orm calls, it was unable to perform the needed type-matches. For the solution, I created a wrapper object for FieldStorage to implement some additional methods that did return string values.

mod_python Handler for Mako

John Hoff — Sat, 31 Jan 2009 06:32:49 +0000

One of my personal goals this Winter has been to learn and start using Python. Learning new languages and putting them to use is something I try to do about every year. Like so many others out there, messing around with Perl in college really has had an impact on my professional career. Being a professional web developer who uses Perl—especially through Apache/mod_perl—it was only natural that I began looking at building out a small site using mod_python. Little did I understand how much of a minefield I was stepping into!

Whenever I'm learning a new language, I inevitably explore how others are using it. This means trying out many of the Python-based web development frameworks that are out there. One of the biggest surprises for me was the reliance that nearly all of the frameworks place on custom Python application servers, bypassing Apache. This also extends to most of the template libraries. It all really depends on a pure Python application server. That would be fine if my goal was to learn Django or Zope. I'm interested in learning Python...and learning a new language for me has always been very dependent on reinventing a known wheel. For me, that really meant focusing on mod_python.

So I dive in, get mod_python installed on a system of mine and start working. Time from apt-get install mod_python to a working Hello World handler: about 15 minutes. All confident in my ability to get things moving, I start taking another look at Python template libraries. Everything grinds to a halt again for me almost immediately. The libraries all have documentation on how to call them from within python programs, but very little information on how to call them from within mod_python. Two exceptions stood out: Cheetah and Mako. Cheetah had a couple of handler examples on their wiki...but I was never able to get any of them to work well. Mako had a working WSGI handler that I was able to study and learn very quickly how to use their library from within mod_python.

Now, everything is up and running, and I'm building a couple of small web applications using mod_python and Mako. To help fill some of the vacuum out there with respect to mod_python handlers, I decided that I wanted to share the handler I have been working with. I wouldn't call it production-worthy, but I am enjoying using it a lot. The Apache configurations required are below and I have attached the handler Python script. Enjoy!

Apache Configuration:


    PythonPath "sys.path+['/var/www/braindonor.net/lib']"
    PythonDebug On
    SetHandler mod_python
    PythonHandler MakoHandler