Peer Pressure

TiddlyWeb + IMAP

One of the more interesting but mostly unexplored areas of TiddlyWeb is its ability to use its storage system to adapt pretty much any datastore into something useful to the tiddlers, bags and recipes that TiddlyWeb uses. Last year some time I created tiddlywebweb, which lets you use another (remote) TiddlyWeb server as the storage for a different TiddlyWeb server. While cleaning it up last night, to get it more in line with contemporary plugin behaviors, I decided I need to make something similar to prove the point a bit stronger.

Python’s standard library includes some pretty useful tools for talking to IMAP servers and parsing email messages. Like many of the modules in the standard library, these aren’t exactly pretty or super clear, but they get the job done well.

I decided I’d start a TiddlyWeb store that uses an IMAP account on the backend. Rather than figuring out the mapping between email and tiddlers from the start, I figured I’d just get rolling and see where I could take things.

First I needed some code to talk to IMAP4 servers. imaplib does that. Second I need to figure out how to use it. So I started some code that logs me into my IMAP account, selects the default mailbox, and gets me some references to all the messages there:

imap = IMAP4_SSL(hostname)
imap.login(user, password)
imap.select()
typ, data = imap.search(None, 'ALL')

That code is straight out of the docs.

If I have some messages I want to be able to parse them. The email library does that just fine. When you fetch a specific message from the IMAP server it can be parsed from a string into a message object. For the sake of simplicity I’ve tossed aside any message that is MIME multipart:

messages = []
for num in data[0].split():
    typ, data = imap.fetch(num, '(RFC822)')
    message = message_from_string(data[0][1])
    if not message.is_multipart():
        messages.append(message)

Now it’s possible to do things like

print 'Subject: %s' % messages[1]['subject']

Okay, satisfactory. Now we know how to get at stuff on the IMAP server. Time to start making a store. We can use the same file. A store in TiddlyWeb is a subclass of StorageInterface. There is a set of methods it may support. No method is required. It’s really up to you what you want to handle. A store gets, puts and deletes various entities and lists collections thereof.

To get things rolling I decided my store was going to be readonly and work for just one user, at least at first. Making messages map to tiddlers, and folders/mailboxes map to bags seems to make sense. Recipes? Dunno. Maybe someday that would be a mechanism for handling multiple user accounts on the server.

The store methods we need to implement, then, are list_bags, bag_get and tiddler_get. List all the folders on the server, list all the messages in a folder, retrieve an individual message and map some fields to tiddler fields. Mostly straightforward, with some blips:

• I don’t know about you, but I have several hundred folders in my imap account, some with an astounding number of messages in them. Anything with “list all” in the description is kind of bad news. So the first hack was to only list folders beginning with ‘a’ (in IMAP-style pattern matching this is ‘/a%’).
• Any bag may only contain one tiddler with any one title. This is not true in an email folder, so when listing the contents of a folder/bag and creating a list of messages/tiddlers it is necessary to create a unique ID. I chose the union of message-id and subject. The message-id should be unique but is not particularly meaningful in lists. I thought about using the IMAP UID but these are not guaranteed to be the same between sessions.

From there it is pretty cool with the tiddlers: the body of the message is the tiddler text (we are only accepting non-multipart messages), the date of the message is the tiddler modified time, the email address part of the from is the tiddler modifier.

And so it all pretty much just works. If you go to /bags you get a list of all the folders starting with ‘a’. If you follow links, you eventually get a list of all the messages in a folder. If you click on one, you can read it. If you click on a tiddlywiki link, all those mail messages show up in a tiddlywiki, which I must say is quite wacky.

Things that would be useful:

• Some form of caching. Lord can it be slow. Even since some core performance changes to TiddlyWeb in the last few days.
• The same date used to set modified should set created too.
• Other message headers in the tiddler as extended fields.
• A TiddlyWiki plugin plus a TiddlyWeb plugin that lets you send mail from TiddlyWiki in response to messages.
• tiddler_put and tiddler_get for moving and removing messages.
• Some kind of role for recipes.
• A something, maybe called polystore, that lets one TiddlyWeb server access multiple different stores.

What else? Give it try if you are curious and hardy. Contact me if you want some help.

The code is in the TiddlyWiki subversion repository, in my tiddlyweb-plugins dir. I’ve also started keeping track of such things on github.