Manipulating zope.schema field's properties dynamically

Why do we want to do that in first place? Lets consider an use-case where we get loginid and user's name from an AddForm. But, in EditForm we don't want the user to change the login ID. One way to implement this, is to, omit the loginid field in the edit form's form_fields attribute. Suppose we want to display the loginid as read only field in EditForm alone, then we need to manipulate the schema field's default properties.

from zope import interface, schema

class IUser(interface.Interface):
    loginid = schema.BytesLine(title=u"Login ID", required=True)
    name = schema.TextLine(title=u"User name", required=True)

The zope.schema allows us to set a property named readonly to schema fields in the interface. For our case, if we specify them in the interface we can't be able to get the loginid from the user in AddForm. So, we are going to manipulate this at runtime in EditForm.

Let's see its implementation using Grok

import grok

class EditUser(grok.EditForm):
    grok.context(IUser)
    grok.name('edit')
    form_fields = grok.AutoFields(IUser)

    def setUpWidgets(self, ignore_request=False):
        # This method is responsible for constructing the widgets from
        # form_fields. So, we override this method to manipulate the
        # schema field
        self.form_fields['loginid'].field.readonly = True
        super(EditUser, self).setUpWidgets(ignore_request)
        # Once we change the field's properties, its effect will be
        # seen across forms. i.e even the AddForm after viewing an
        # EditForm will have the 'loginid' as readonly field. So, we
        # revert them back to their originals
        self.form_fields['loginid'].field.readonly = False

    @grok.action('Save')
    def edit(self, **data):
        self.applyData(self.context, **data)
        self.redirect(self.url(self.context.__parent__, 'list'))

Adding custom validation in zope.formlib

If you are using ZTK, you would be familiar with zope.formlib and zope.schema libraries. These libraries provide a powerful mechanism to generate CRUD forms. In this, basic validation options can be specified in the interface via schema fields and by the constraint and invariant functions. I am not going to dive into those details as the documentation for it already exists on the Internet.

Let us consider a trivial example using Grok, in which, we shall try to check the validity of a form field

import grok
from zope import interface, schema

class IUser(interface.Interface):
    id = schema.BytesLine(title=u'User ID', required=True)

class AddUser(grok.AddForm):
    grok.name('add_user')
    form_fields = grok.AutoFields(IUser)
    
    @grok.action('Add')
    def add(self, **data):
        user = User()
        self.applyData(user, **data)
        self.context.add(user)
        self.redirect(self.url(self.context, 'list'))

The context for AddUser and the add() method of it are left to the reader's imagination ;)

As per the schema definition, zope.formlib will not allow the User's 'id' field in the add form to be empty. But, unfortunately it will accept spaces as a valid entry. This can be validated by adding an invariant to check the field's striped value. But, for the sake of simplicity we will use this example to enforce our custom validation.

Under the hood

Form's input and display elements are represented by formlib's widgets. There are default set of widgets associated with each schema field. On form submission, formlib tries to get the values from these widgets during which error checking is done based on the schema field's definition and then invariant validation is done. For our case, it checks whether the 'id' field is empty or not. If it is empty, ValidationError is raised. This ValidationError in turn is converted to WidgetInputError and appended to an error-list. Thereby, this error-list contains the list of all validation errors found in the form. An empty list indicates that there were no errors.

Error categories

If the form contains errors, the input fields will not be cleared and the error messages will be displayed. The errors are divided into two categories

1. Widget level errors, which are associated with each widget and displayed along with them.
2. Top level errors, which are usually displayed on the top of the form just below the status message. It contains both the invariant error messages and widget error messages.

Displaying top-level errors in form

In order to display the top-level error messages, formlib takes each error in the error-list displays them, as it is, if it is a string. If the error is not of type string, it gets the MultiAdapter view for the browser request and error to the interface IWidgetInputErrorView.

view = component.getMultiAdapter((error, self.request), 
                                 IWidgetInputError)

The error is displayed by calling the snippet() method on this view, which in turn, returns the error message as HTML snippet. 

Displaying widget-level errors in form

Widgets on the other hand, have an error state, which can be rendered in a form using its error() method. Whenever the error() method is called, it checks its internal error state, if there is an error it gets the multi-adapter view as mentioned above and displays the error by calling the snippet() method.

Implementation

As there is already an InvalidErrorView object which implements IWidgetInputErrorView and adapts interface.Invalid and IBrowserRequest, we shall replace the Widget's error() method with a callable error string for our induced errors.

Enough of Zope's internals, lets override the validate method of zope.formlib's FormBase class and write a helper function to set the error.

def set_form_error(widgets, wgt_name, emsg, elist):
    wgt = widgets.get(wgt_name)
    if wgt is None:
        raise interface.Invalid("Invalid widget (%s)" % (wgt_name,))
    if isinstance(elist, list) is False:
        raise interface.Invalid("Error list invalid")

    # We override the Widget's 'error' method
    wgt.error = lambda : emsg
    # Here we append the error message as string in the list.
    # As this corresponds to top-level error messages we
    # mention the widget's label in it so that the error
    # message can be meaningful.
    elist.append("%s: %s" % (wgt.label, emsg))

class AddUser(grok.AddForm):
    grok.name('add_user')
    form_fields = grok.AutoFields(IUser)

    def validate(self, action, data):
        ret = super(AddUser, self).validate(action, data)
        # If the form has basic errors we just display them
        if len(ret) != 0: return ret
        # If the form has no errors we do additional validation
        data['id'] = data['id'].strip()
        if len(data['id']) == 0:
            # Here we induce our custom error
            set_form_error(self.widgets, 'id', 'Invalid', ret)
        return ret

    @grok.action('Add')
    def add(self, **data):
        user = User()
        self.applyData(user, **data)
        self.context.add(user)
        self.redirect(self.url(self.context, 'list'))
 

I guess, the above piece of code is fairly simple now.

Creating standalone windows application with python

Here in this post we will see how to convert a python application into a standalone windows executable. So that, windows users can run them without installing windows binaries of python and its related libraries separately.

In this we will write a small python code which uses the pubsub and pygtk libraries. And we shall see, how to write the setup script for it using py2exe and cx_Freeze in order to produce a standalone windows application.

Introduction

The job of a packaging/distribution tool is to build a list of dependent modules that are needed for the python application and include those modules for packaging. The list of dependent modules are specified via packaging options provided in the conventional setup.py file and/or by recursively finding the modules from the import statements used in the code.

There are certain limitations when a packaging tool tries to find the dependent modules from the import statements. If the python code imports modules using the  __import__ statement or if the python code modifies the sys.path at runtime in order to import certain modules, then those modules will not be found successfully by the packaging tool.

Python pubsub package

The pubsub module supports two different messaging protocols namely args1 and kwargs. Choosing and switching between these protocols are done by modifying the module path dynamically. This can result in import error like this during runtime.

      from listenerimpl import Listener, ListenerValidator
      ImportError: No module named listenerimpl

This is the main reason I choose pubsub for my example. In the following sections we shall see how to package them successfully for windows.

Sample code

Lets consider a sample application consisting of a single file named say testpubsub.py with the following code

from pubsub import pub
import gtk

def listener1(msg):
    ms = gtk.MessageDialog(None, 0, gtk.MESSAGE_INFO, gtk.BUTTONS_CLOSE,
                           "The listener received the message : %s" % (msg, ))
    ms.run()
    ms.destroy()

pub.subscribe(listener1, 'test.pubsub')

def sender():
    pub.sendMessage('test.pubsub', msg="Hola! this is a test message")

if __name__ == "__main__":
    sender()

In the above application, the listener displays the message received from the sender using gtk's message dialog popup. Now, we shall see, how to write the conventional setup.py file for it, using py2exe and cx_Freeze.

Setup file using py2exe

The setup.py using py2exe would look something like this

from distutils.core import setup

try:
    import py2exe
except:
    pass

setup (
    name='TestPubSub',
    description="Script to test pubsub for packaging",
    version="0.1",
    windows=[{'script': 'testpubsub.py'}],
    platforms=["any"],
    options={ 'py2exe': {
            'packages': 'encodings, pubsub',
            'includes': 'cairo, pango, pangocairo, atk, gobject, gio'}
              },
    )

Here we can see the packages option specified for py2exe, to include the encodings and pubsub packages explicitly. This will include the entire pubsub (and encodings) modules from the installation location for packaging.

As the final package has the entire pubsub module, the windows executable will have no trouble in finding the dynamic protocol libraries during runtime.

To generate exe file, run the following in the build machine (i.e. a windows machine with python and necessary library dependencies installed)

   $ python setup.py py2exe

this will produce a standalone windows executable file named testpubsub.exe

Setup file using cx_Freeze

The setup.py using cx_Freeze would look something like this

from cx_Freeze import setup, Executable
import platform

if platform.system() == 'Windows':
    bse = 'Win32GUI'
else:
    bse = None

opts = { 'compressed' : True,
         'create_shared_zip' : False,
         'packages' : ['pubsub.core.kwargs', 'pubsub.core.arg1'],
         }

WIN_Target = Executable(
    script='testpubsub.py',
    base=bse,
    targetName='testpubsub.exe',
    compress=True,
    appendScriptToLibrary=False,
    appendScriptToExe=True
    )

setup(
    name='TestPubSub',
    description="Script to test pubsub for packaging",
    version='0.1',
    options={'build_exe' : opts},
    executables=[WIN_Target]
    )

The packages option slightly differs for cx_Freeze. Here we have to explicitly include the sub packages of pubsub library namely, pubsub.core.kwargs and pubsub.core.arg1 so that, all the modules under them gets included for packaging.

Run the following in the build machine to generate the exe file

   $ python setup.py build

We can safely ignore the missing modules warning in the build log

   Missing modules:
   ? core.publisher imported from pubsub.pub
   ? listenerimpl imported from pubsub.core.listener
   ? publishermixin imported from pubsub.core.topicobj
   ? topicargspecimpl imported from pubsub.core.topicargspec
   ? topicmgrimpl imported from pubsub.core.topicmgr

The above listed missing modules are under pubsub.core.kwargs and pubsub.core.arg1. In cx_Freeze, the list of dependent modules are gathered from the import statements used in the code. While constructing this list, cx_Freeze reports missing modules, when a particular module has been imported in the code but not found in sys.path. But, as we have included those modules directly via the packages option, these modules will get safely included in the package. So, there won't be any runtime error.

Voila! We have created a standalone windows application from a python script. :)

Quick fix for Linux power regression problem

Linux Power Regression is yet another flame topic within the open source community. Where one group says that the sky isn't falling. Whereas the other group claims that, starting from kernel versions 2.6.38 the battery's backup time is reduced by 50% and the CPU temperature runs all time high. But, no one mentions the optimal temperature range.

To check this on my T420 thinkpad which has the sandy bridge processor, I ran the sensors command, which reported the processor temperature as 73° in spite of, CPU being idle. This was very alarming.

On further digging, we can find that this is an issue partly related to BIOS, where it indicates erroneously to the kernel that the PCI Express Active State Power Management (ASPM) isn't supported for the board which has that feature.

Moreover, it seems that, in the intel graphics driver i915, RC6 feature (which allows the GPU to enter a lower power state when it is idling) is not enabled by default.

So, here is the quick fix

1. Open the /etc/default/grub file
2. Add these parameters to the variable GRUB_CMDLINE_LINUX_DEFAULT "quiet splash i915.i915_enable_rc6=1 pcie_aspm=force"
3. Then run the command update-grub as super user 
4. Reboot the system and see the difference.

After this, my CPU temperature fell in the range of 40 to 50+

Satisfied :)

Clearing recent documents history in GNOME 3

If you are using GNOME 3, have you ever wondered how to clear the history of recently used documents without fiddling around with ~/.local/share/recently-used.xbel file. Ok. Here is the solution for you.

1. Open the GNOME shell's integrated inspector tool and JavaScript console called Looking Glass by typing the command lg in the prompt invoked by Alt+F2
2. Then enter the following imports.gi.Gtk.RecentManager.get_default().purge_items()
3. Press Esc to exit from the looking glass console.

I hope that the GNOME guys will provide an easier way for this in the near future.

Have fun :)

random surge

Well, having a I won't blog resolution for a long time. Today I decided to break it, not because I have some cool information that the world needs to know. But, just to record stuffs that I do in my day to day life (and forget them as the time passes by). I hope this will save my time and effort when I come across the same issue later.

I'll be glad if that helps someone having a similar scenario. I tend to keep this blog more technical. Lets see how it goes :)