Home > Blogs > Slotted Classes

Slotted Classes

By  Jul 23, 2008

Topics: Programming

For me, the most surprising OOP feature of Python is slotted classes. I was so excited about this feature, that I immediately re-wrote some existing code to use it. And then the problems started ... Luckily they were not too hard to fix.
It should not be a surprise that slotted classes were of such interest to me. Being able to statically declare what attributes are allowed in a class has a lot of appeal to folks coming from a statically typed background. There are obvious performance benefits. The most obvious is a memory footprint, and that gets multiplied when you start creating type hierarchies. Theoretically, the Python VM should be able to do a lot of optimizations around slotted classes. I don't know if it does this or not, but it could.

So going to the App Engine project that I have been working on, I had a class that seemed perfect for slotting. It is a class I use for entries from an RSS or Atom feed. As such, a very large number of these instances get created per user. They aren't kept around too long (just the lifetime of a request) so they can be garbage collected. However, extremely spiky memory is usually trouble when your virtual machine has a limited allocation of memory. In fact, such applications are often memory-bound, i.e. more users mean higher spikes until you hit your memory limit. So to handle more users, you have to add memory (or add machines to spread users out) or you need to make the spiker per user smaller. Slotting will do just that.

For my app, these entries were also being stored in memcache and they were being put into lists that were then being sorted. This caused problems. Putting the objects in memcache required pickling the objects, and pickling relies on the presence of a __dict__, i.e. the underlying dictionary. For a slotted class, there is no __dict__. For sorting, I was using the operator module to sort by an attribute of my Entry class. This also relied on __dict__.

So I went back to Core Python to investiage further. I also wrote a smaller test program to experiment with pickling and sorting on slotted objects. Here is the class that I wrote:

class Toaster(object):
    __slots__ = ['brand', 'model', 'capacity']
    def __init__(self, brand, model, capacity=2):
        self.brand = brand
        self.model = model
        self.capacity = capacity
    def __getstate__(self):
        dict = {}
        for key in self.__slots__:
            dict[key] = self.__getattribute__(key)
        return dict
    def __setstate__(self, dict):
        for key in dict.keys():
            self.__setattr__(key, dict[key])
    def __cmp__(self,other):
        if self.brand != other.brand:
            return cmp(self.brand, other.brand)
        if self.model != other.model:
            return cmp(self.mode, other.model)
        return cmp(self.capacity, other.capacity)
    def __str__(self):
        return '(' + self.brand + ',' + self.model + ',' + str(self.capacity) + ')'
    def __repr__(self):
        return self.__str__()

The beginning of the class defines it slots, and provides a constructor for it. Next up are the two methods that are the keys to pickling a slotted class: __getstate__ and __setstate__. The first produces a dictionary to pickle, and the second takes a dictionary and restores the attributes. Notice the for loops in each method. For the getstate I used self.__getattribute__(key). For setstate I used a corresponding self.__setattr__(key). I must admit that the naming of these methods really bothered me because they seem inconsistent. Oh well.

Next is a custom __cmp__ method. You don't have to have this for sorting, but if you have it then it will be used for default sorting. When my operator based sorting was failing, I thought that I might have to rely on the default sorting, but as you will see later, I found a way around that too. In fact here is the test script:

t1 = Toaster('BrandA', 'X100', 2)
print t1

t2 = Toaster('BrandC', 'CAT', 4)
print t2

#default sorting
items = [t1, t2]
print items
print items

s1 = pickle.dumps(t1)
s2 = pickle.dumps(t2)

o1 = pickle.loads(s1)
o2 = pickle.loads(s2)

#check if sorting still works
l = [o1, o2]
print l
print l

#custom sorting
l.sort(key = lambda t:t.capacity)
print l

Most of this is self explanatory. The last section is the most interesting, custom sorting. Instead of using the operator module (which relies on the presence of __dict__ and thus is incompatible with slotted classes) I simply used a lambda expression.