Peer Pressure

One of the more interesting but mostly unexplored areas of TiddlyWeb is its ability to use its storage system to adapt pretty much any datastore into something useful to the tiddlers, bags and recipes that TiddlyWeb uses. Last year some time I created tiddlywebweb, which lets you use another (remote) TiddlyWeb server as the storage for a different TiddlyWeb server. While cleaning it up last night, to get it more in line with contemporary plugin behaviors, I decided I need to make something similar to prove the point a bit stronger.

Python’s standard library includes some pretty useful tools for talking to IMAP servers and parsing email messages. Like many of the modules in the standard library, these aren’t exactly pretty or super clear, but they get the job done well.

I decided I’d start a TiddlyWeb store that uses an IMAP account on the backend. Rather than figuring out the mapping between email and tiddlers from the start, I figured I’d just get rolling and see where I could take things.

First I needed some code to talk to IMAP4 servers. imaplib does that. Second I need to figure out how to use it. So I started some code that logs me into my IMAP account, selects the default mailbox, and gets me some references to all the messages there:

imap = IMAP4_SSL(hostname)
imap.login(user, password)
imap.select()
typ, data = imap.search(None, 'ALL')

That code is straight out of the docs.

If I have some messages I want to be able to parse them. The email library does that just fine. When you fetch a specific message from the IMAP server it can be parsed from a string into a message object. For the sake of simplicity I’ve tossed aside any message that is MIME multipart:

messages = []
for num in data[0].split():
    typ, data = imap.fetch(num, '(RFC822)')
    message = message_from_string(data[0][1])
    if not message.is_multipart():
        messages.append(message)

Now it’s possible to do things like

print 'Subject: %s' % messages[1]['subject']

Okay, satisfactory. Now we know how to get at stuff on the IMAP server. Time to start making a store. We can use the same file. A store in TiddlyWeb is a subclass of StorageInterface. There is a set of methods it may support. No method is required. It’s really up to you what you want to handle. A store gets, puts and deletes various entities and lists collections thereof.

To get things rolling I decided my store was going to be readonly and work for just one user, at least at first. Making messages map to tiddlers, and folders/mailboxes map to bags seems to make sense. Recipes? Dunno. Maybe someday that would be a mechanism for handling multiple user accounts on the server.

The store methods we need to implement, then, are list_bags, bag_get and tiddler_get. List all the folders on the server, list all the messages in a folder, retrieve an individual message and map some fields to tiddler fields. Mostly straightforward, with some blips:

• I don’t know about you, but I have several hundred folders in my imap account, some with an astounding number of messages in them. Anything with “list all” in the description is kind of bad news. So the first hack was to only list folders beginning with ‘a’ (in IMAP-style pattern matching this is ‘/a%’).
• Any bag may only contain one tiddler with any one title. This is not true in an email folder, so when listing the contents of a folder/bag and creating a list of messages/tiddlers it is necessary to create a unique ID. I chose the union of message-id and subject. The message-id should be unique but is not particularly meaningful in lists. I thought about using the IMAP UID but these are not guaranteed to be the same between sessions.

From there it is pretty cool with the tiddlers: the body of the message is the tiddler text (we are only accepting non-multipart messages), the date of the message is the tiddler modified time, the email address part of the from is the tiddler modifier.

And so it all pretty much just works. If you go to /bags you get a list of all the folders starting with ‘a’. If you follow links, you eventually get a list of all the messages in a folder. If you click on one, you can read it. If you click on a tiddlywiki link, all those mail messages show up in a tiddlywiki, which I must say is quite wacky.

Things that would be useful:

• Some form of caching. Lord can it be slow. Even since some core performance changes to TiddlyWeb in the last few days.
• The same date used to set modified should set created too.
• Other message headers in the tiddler as extended fields.
• A TiddlyWiki plugin plus a TiddlyWeb plugin that lets you send mail from TiddlyWiki in response to messages.
• tiddler_put and tiddler_get for moving and removing messages.
• Some kind of role for recipes.
• A something, maybe called polystore, that lets one TiddlyWeb server access multiple different stores.

What else? Give it try if you are curious and hardy. Contact me if you want some help.

The code is in the TiddlyWiki subversion repository, in my tiddlyweb-plugins dir. I’ve also started keeping track of such things on github.

Writing documentation is spectacularly difficult. At least for me. While I’m capable of being quite verbose (perhaps too verbose) in response to a question (perceived or actual) I’m crap at predicting what people will want or need before they get to the point of asking questions.

This is having an unfortunate effect in the land of TiddlyWeb: documentation is lagging quite far behind the functionality of the tool and while I’m generating new documentation on a fairly regular basis I have no way of knowing if it is the right documentation.

Any software project, especially one which is both an application, a library and an HTTP API, has several different types of documentation. There’s the documentation in the code explaining functions, classes, methods, modules and such. With Python there are tools like pylint and pep8.py which will tell you when your code is missing docstrings so TiddlyWeb has most of that covered with at least basic documentation on stuff that is public.

More complicated is the narrative that explains what TiddlyWeb can do and the documentation that explains how to do it. Making it go, getting the best out of it. That’s the stuff that’s hard to do alone; since narrative is a story it only emerges through human communication: feedback loops, a dialog of participants.

I made raamable.

My friend Stan is going to be part of the support crew of Team Hoosiers when they once again participate in the Race Across America: a west to east cross-continental traverse of the US. On bikes.

Stan wondered about the possibility of some web based resources to help make things go. When heading across the country where every minute counts it’s good to know where you are going; what’s coming up next; weather conditions; the nearest grocery, gas, laundry, bike store.

When I was working for Socialtext most of the developers worked from home, scattered around the world. When we did get together we’d sometimes experience bursts of productivity that were very pleasant: a great sense of being in the zone. Bourbon was often involved. Intense work and intense play.

Seemed like Stan’s project was a good reason to do some good work for a good cause.

So I hunted out Matt and Zac, also former Socialtexters, and we planned a hankathon (hack and drink a thon) in Bloomington, the font of much goodness. The idea was we’d gather, mess about with some Google mashups, see what we ended up with, and see what we learned about how to have effective similar things in the future. These days it should only take a few days to establish a startup.

We made plans. Zac couldn’t come: teething baby. Down to just Matt and me. I was sick, Matt had to work, and hanging out with Stan is too rare and too fun to be wasted on staring at computers so we didn’t have the focus and concentration we originally desired. We still managed to get some things done and learn some things.

On the learning side we had some thoughts for next time:

• Clear the decks. If you’re going to a multi-day hackathon, don’t bring other work.
• Have a least three people. If you know exactly what you are doing, pairing or sharing work can work out very well, but if you are exploring as much as executing, an odd number keeps thinking unstable and diverse.
• Spend the first day, half day, some unit of time brainstorming and finding the point of gravity. The group needs an articulated shared goal to have shared momentum.
• Have a tool for keeping track of what’s up. We used a TiddlyWeb hosted tiddlywiki but didn’t initially use it all that well. Since we were in person I suspect something more tactile like a whiteboard with stickies on it might have worked better.
• Get necessary infrastructure in place beforehand.

Despite being a bit discombobulated we did manage to create something.

Uncle Ingy says every good project starts with a good name, so I baked the name “raamable” into a git repo on github and started gathering the necessary pieces.

Since we knew we’d be working on this project separately before and after the hankathon, we’d need a convenient way to work in a test and deployed environment. Google App Engine makes that easy: do local development until you get it right and then deploy to the servers.

We had some known data sources: gps waypoints for the race, locations and frequencies of NOAA weather radio towers, a web service that gives elevation for longitude and latitude in the US.

First pass plan: put all that stuff on a google map and see what happens. There were some challenges.

It’s easy to put a bunch of markers on a map. The code shows that well enough. What’s harder, when there are several hundred markers, is plotting the line of the route on roads. Plotting a route requires using GDirections, which will only take 25 waypoints per use. Our solution was to loop over the markers in chunks of 25. The results in the map shown on the front page. The animation is a side effect of the process being somewhat slow. The darker blue spots happen where one block of 25 stops and another starts.

On the interactive map yellow markers are time stations, orange are turns on the route, and the towers mark nearby weather radio towers. Each GMarker that represents a turn has an added attribute altitude containing meters of elevation at that location. Each GMarker representing a radio has a frequency attribute.

The race route turns information did not come with elevation data. I initially planned to make the map queryable for elevation at any point (thus the /proxy URL in the app engine code) but then we decided an elevation profile image would be nice. Dynamic querying for that would be much too time consuming and barring catastrophe elevation doesn’t change much so storing it on the map seemed the way to go. For kicks I processed the route turn CSV file through a shell one liner using curl to get the elevation data and add it back into the file.

The elevation profile is a google chart that is redrawn each time the map is moved or zoomed. The crux here was to make sure that we only draw information for those markers which are within the bounds of the current view. updateProfile in the code shows how this is done.

There are many hundreds of radio towers in the continental US. We only want to show those which are near the route. There’s doesn’t seem to be a straightforward efficient way to say “plot only those waypoints which are near to this other set of waypoints” so we constructed our own. The entire route describes a rectangle that has northern and souther latitudes that can exclude most of the towers. So first we do that. The slope of the route (southwest to northeast), however, is too steep for this to be good enough: a larger number of towers are plotted which are too far away to be tuned in and make the map uselessly noisy. To deal with this an array of bounds, one containing each ten turns (plus and minus 1 degree of longitude and latitude) along the map, is created. The remaining radio tower locations are compared against the bounds. See _establishRouteCallback and _addTowersCallback.

We hope to do quite a bit more. We’re tracking our plans on the wiki. What’s running now is a useful tool but suffers one rather extreme flaw: it requires network access. We hope to modify things so that useful printouts can be made from the system and carried on the route.

This is an open source project, you are welcome to help out, use this for your own bike ride, or whatever. If you do, we’d love to hear from you.

I was talking this afternoon with Fred. Over the past few days, he and Jon have been creating a Twitter archiver called TiddlyTweets using TiddlyWiki plugins. It’s cool: you give it a twitter username and it retrieves all the tweets and archives them in a useful fashion, preserving a link to their original URI.

I’ve been hassling Fred about this, as I like to do, because one of the reasons I have liked Twitter is because tweets have, or at least had, a sense of being ephemera. In my mind, if you feel like you need to archive your tweets, then you are doing it completely wrong.

But that’s just my opinion and Fred’s is different. He says there’s good stuff in those tweets which you might want later. This is true enough.

Today, in IRC, he said:

without starting the whole discussion again, after
yesterday's(?) archiving exchange, it struck me as
odd that you're so indifferent - after all, you're a
proponent of cool URIs, which kinda includes permanence

My response went something like this:

Every tweet has its own URI already, the “statuses” URL, so the coolness is basically there already. Fred responds that Twitter is not reliable. This is true but I suggested that archiving was a short term solution to the problem that allows the long term problem to persist. The long term problem is that ones tweets live at Twitter, not in ones own persistent store: Twitter should be a presentation engine which has access to your own microcontent.

This reminded me that in some ways what I’ve been working on for the last ten years or so with Purple Numbers, wikis in general, TiddlyWeb, persistent identifiers etc. are tools to make such things possible. What I’d like to see is an inversion of the current content relationship that is present on the web.

Today I go to a web site and give it some output of my brain. The web site takes my content and in exchange it gives me a URL so I and other people can get to the content again. What should happen instead is that I make some content and give that URL to some websites so they can get to the content.