Peer Pressure

A common question associated with TiddlyWeb is “What can I do with it?” or “Why would I want to use it instead of Django or something?” The answer is not simple.

Because TiddlyWeb has tools for storing data and presenting that data in various forms over HTTP, it has some of the core features of a web application framework. However, since it (intentionally) is also missing some of the core features of a web application framework, a fair piece of cognitive dissonance can happen when the tool is approached as a web application framework.

So it seems best to stay away from saying that TiddlyWeb is a web application framework, so as to manage expectations. The question remains, “What is it?”

One pathway to answering that question is to create tools with it and see what the commonalities are between those tools. A derivative of this approach is to create a killer app with TiddlyWeb: a something which demonstrates all the salient features and how to use them, both as a user and as a developer.

This approach seems to falter: The concepts in TiddlyWeb are sufficiently strange (I won’t say unique, because they aren’t that, they are common concepts, with their own spin) that in order to create a tool that uses them well you first need to either understand them in the abstract, or see some examples. Catch 22, especially if you are an examples oriented person.

FND and I tend to converse on this topic about daily, trying to come up with different strategies for answering it. Today I suggested that instead of creating a new application, from scratch, that is cool and interesting because it uses TiddlyWeb, it might be better to find some application that could be made better through the addition or inclusion of TiddlyWeb.

It turns out this process has already started and by analyzing that process we might understand TiddlyWeb better.

The process has started with TiddlyWiki. TiddlyWiki is made better for a particular set of use cases by taking advantage of TiddlyWeb features. The fundamental experience of TiddlyWiki does not change: you’ve still got a wiki in an HTML file, with these things called tiddlers, some of which are special plugins that change behaviors.

What TiddlyWeb adds when used with TiddlyWiki is:

• Centralization of content storage, storing at the level of the tiddler, with revisions, protection against editing conflicts and access control to read, write, create and delete; accessible from multiple locations on the network.
• A (hopefully) straightforward system of managing access control of tiddlers and making other kinds of groupings of tiddlers through the use of bags. Bags are collections of uniquely named Tiddlers.
• A system for composing functional collections of tiddlers selected from multiple bags using recipes. Recipes are list of bags paired with filters. Filters are rules which limit the tiddlers pulled from a bag when processing the recipe.
• Tools for viewing and manipulating tiddlers in different forms than those used in the wiki (e.g. JSON, Atom).

These four things together provide a multi-user environment for TiddlyWiki that allows multiple custom views of the same or similar content, depending on some piece of context. Different recipes can provide a different view on the same stuff, providing a different look and feel, different security handling, etc.

TiddlyWeb has many other features, but the above four are probably the key aspects which define its special sauce. When I evaluate a nascent TiddlyWeb tool for its tiddly-fitness the evaluation is done based on the extent to which the system takes advantage of and understands the above features.

These ideas should allow us to create some mental heuristics to use when approaching a TiddlyWeb project. If we have an existing project or use case that we intuit will be helped by using TiddlyWeb we can ask ourselves: 1) What aspects of this project will benefit from bags, recipes and filters? 2) How can we take advantage of different representations?

If, for some reason, we are planning to make a project from scratch, and have decided to use TiddlyWeb prior to really evaluating our use cases, then we can ask ourselves: 1) How can we best structure our data and UI to use recipes, bags and filters to provide flexible user experiences? 2) What representations can we present to allow other people to provide flexible user experiences?

In Friday’s TiddlyWeb Dev/Deploy Workshop posting I said that mature plugins need to be packaged so they can be indexed by PyPI and installed via easy_install and pip. At base this is relatively straightforward, put some stuff in a directory, make a setup.py, register the package, package up a source distribution, tell PyPI about it.

This is okay if you have a good name for your package, but what about packages, like plugins for TiddlyWeb, which have names which are only meaningful in their use context? You can’t just use the name of the module, otherwise you end up with namespace collisions, trouble finding stuff, and associated beasts of chaos.

Thankfully distutils, setuptools, pip, etc conspire to support a notion called namespace_packages which can solve this issue. Unfortunately using the feature is not exceptionally well documented. I found getting started a bit frustrating, until I sort of cracked the nut. Here’s some info for reference.

First some prerequisites:

1. You need setuptools
2. You need a username and password on PyPi.
3. Some understanding of how to make a python package distribution. Here’s one tutorial: Use setup.py to Deploy Your Python App with Style.
4. Some understanding of Python packages.

That tutorial includes a section about using namespace_packages under the heading “Multiple Distributions, One (Virtual) Package”. That “Virtual” is key: if you wish to use namespace packages there much be no real distribution which occupies the namespace. In the example of tiddlywebplugins, you can have tiddlywebplugins.static and tiddlywebplugins.utils distributions which are members of the tiddlywebplugins namespace, but you must not have a tiddlywebplugins distribution. If you do the packages which are supposed to occupy the virtual namespace will not be found. The upshot of this is that if you already have a package out there using a name that you want to use as a namespace, you will need to rename the existing package (this is why the old tiddlywebplugins package is now tiddlywebplugins.utils

So say you have a TiddlyWeb plugin called foobar.py sitting in a directory somewhere. You’ve determined that it is a happy little plugin and the world would benefit if it could be installed easily. You’ve heard of the tiddlywebplugins namespace and you’d like to join the party. Here’s what you do.

• In that directory make a tiddlywebplugins directory.
• Edit tiddlywebplugins/__init__.py to include just this line: __import__("pkg_resources").declare_namespace(__name__)
• Move foobar.py into the tiddlywebplugins directory.
• Create a setup.py (in the original directory) that includes at least:

     from setuptools import setup, find_packages

     setup(
         version = '0.1',
         namespace_packages = ['tiddlywebplugins'],
         name = 'tiddlywebplugins.foobar',
         description = 'A TiddlyWeb plugin for foobaring the fritz.',
         install_requires = ['setuptools', 'tiddlyweb'],
         )

• Do not import from the tiddlywebplugins package in setup.py. This will make installs struggle or fail later.
• Learn enough about distribution to register the package and upload it. The links above point to enough documentation to figure that part out. If you can’t be bothered to read that documentation then you shouldn’t be distributing packages. We wouldn’t want you aoling all over PyPi.
• When you want to use your foobar plugin in a TiddlyWeb instance or application refer to it as tiddlywebplugins.foobar.

Yesterday I gathered with the Osmosoft gang to discuss developing and deploying applications that are built on top of TiddlyWeb. Since I mostly develop TiddlyWeb itself and not things built upon it, I tend to be fairly distanced from the issues so getting together and having a chat and messing about with some code was a good thing.

Mike has written up his summary of the things to do next. I took some notes as well, which I’ll attempt to summarize here.

To get a sense of the issues we went around the room and people reported on how they’ve been managing development and deployment and what’s working well and what’s not. There were some but not many common threads.

One common issue is that most of the people have a tendency to do a code a bit, reload a web page cycle of development and test. When using twanager server this does not work well as the server needs to be restarted with each code change. Restarting CherryPy’s wsgiserver is less easy than some desire because it can be some time before the main thread will come round to hearing an interrupt signal. The initial solution to this problem, the reloader plugin, helps but gets a bit weird when syntax errors exist in the watched Python files (which happens a lot if you have a esc:w trigger finger): The entire server will crash out and you have to start the process again by hand. This mostly makes sense from a logical standpoint but is not that helpful.

One proposed solution to this problem is to host TiddlyWeb under a simple CGI setting. This will reload and recompile the code with every request. This can be slow for production settings but plenty fast for development. The TiddlyWeb source package includes an index.cgi script for this. Ben Gillies has written some docs on how to use it.

Of course to use the CGI option you have to have some kind of web server around. Many systems come with Apache already on them or easy to install. If you prefer not to run Apache (like me) another option is Spawning a fast multi-process, multi-threaded, Python web server. There’s a TiddlyWeb “factory” for Spawning called spawner.

Paul wisely pointed out that much of this reload fiddling about can be avoided by writing and testing Python modules that perform most of the activity for you application, minus the web handling, and then using these modules in small layers of Python code that integrate with TiddlyWeb.

There was a fair amount of resistance in the group to formal and rigorous testing, which I think is extremely unfortunate, but Simon rightly points out that lots of people are like that, and TiddlyWeb is going to limit its audience a great deal if it requires developers to be totally test driven.

Another point of difficulty and contention is lack of familiarity with Python, Unix and TCP fundamentals. I’m not sure how to respond to that sort of thing other than to say that if you want to develop networked applications in Python, on Unix, then it’s probably pretty good to be familiar with such stuff.

By the far the biggest issue with development is managing the inclusion of necessary stuff in the development environment. This includes auxiliary plugins that are required for the system, TiddlyWiki content and plugins, and static content that the web app needs. Mike’s blog post has most of the plans in this area. The gist is two pieces of development:

• Improving the existing devtext store so it more effectively (and correctly) supports during development manipulation of the store and inclusion of TiddlyWiki content. We’ll be talking about the details of this in IRC this afternoon.
• Creating a suite of shell-based tools that provide Make like functionality for establishing and maintaining a development environment and easing deployment. The long term plan is to migrate these tools to Python as time and knowledge allows.

For myself the things that have become important are:

• Making existing plugins more visible to developers so they know what tools are available.
• Packaging so-called mature plugins as Python packages that are installable via easy_install or pip.
• Enhancing documentation of configuration options so they are more visible.
• Publishing the quick hacks and tricks I do to make my own development life a bit more zipless.