Peer Pressure

The Synthetic Web

In a recent thread on the TiddlyWeb Google group, I stated:

My opinion is that stuff, i.e. information, should be addressable and reusable. That’s my fundamental number one design goal with TiddlyWeb.

This was while making the assertion that the URIs used for entities in TiddlyWeb should have names and structure that is most meaningful and most useful to the general user who stumbles across the entities while browsing the web, not names and structure that is most meaningful to the developers of the system. Furthermore the names and addresses and reusability are for humans.

Ideally the names should be useful to both sets but when there is a conflict I think the unknown user should get the benefit or assist from the system, not the developer. This is my personal opinion, but because I’ve written most of TiddlyWeb, it shows throughout its design.

The opinion has been active throughout what might be called the second half of my career. The first half could be described as managing information systems, the second half could be described as creating them. The opinion goes something like this:

• Information exists. However it is represented, it is in a system and can be used.
• Some information systems exist to confirm what we think we already know or for which we can phrase a fairly concrete question (e.g. “Where is London?” or “What is the definition of ‘information’?”). In these cases, searching or similar interaction models are excellent tools: our inquiry is algorithmically transformed into a query against an index (note an index can take many forms). Think dictionary. Or google.
• Other information systems are storehouses (real or otherwise) which provide affordances for navigation, browsing and discovery. These allow for that extraordinary thing: learning new stuff. We’re wandering along, soaking up new information or new ways of stating old information and BAM, we make a connection, disparate paths are linked and now we know something. We have made a synthesis of A and B et voila new thesis C. Think narrative book. Or the web at large.
• These systems that enable synthesis are fundamentally more important than others because synthesis is the source of learning, innovation and change. They drive intuition, inspiration and insight.
• While data is fundamentally useful to provide authority to new insights, human narrative, human summarization (nee synthesis) is the part that makes you go “hmmm”. In an academic paper the part that thrusts us into the future is the frame: the introduction and conclusion, the setting of the stage and the synthesis. The parts in the middle make the rest believable.
• Some data can representable with rich syntax that encodes semantics that make it automatically reusable. That is, you can package up data in a structured way such that computation can be performed on it. This is the fundamental premise of the semantic web. Such computation can deduce (deduction: conclusion drawn from examination of facts) true statements such as “Chris Dent is a male living in England working on TiddlyWeb”.
• Narrative is not so clean cut. It is full of connotation and contextual dependence. What it means must be inferred and that inference is inescapably done in the personal headspace of the reader/listener/participant.
• Opportunities for innovative synthesis from narrative are augmented by arraying (representing) the narrative within the context which created it in the first place and things which are similar or can be identified as being related in time, place, group.
• Computers have achieved a special place in the history of information systems because they are the most capable tool (thus far) for easing the task of navigating these multi-dimensional representations of narrative.
• Critically, thus far only humans are able to perform the synthesis that results in new knowledge and from which new narrative might be created.
• Often it is but a small piece (microcontent) of a larger whole which strikes the chord that plays eureka.
• Therefore:
  • Information, especially narrative, should be published in a way that humans can easily make reference to it and use of it as they do the re-representing that is required to engender synthesis. It should have good names and addresses.
  • Tools which enable this publishing should make as few assumptions about the people and tools accessing them as possible, so as to enable as yet unknown ways of (re-)representing the information.
  • Ideally the tools should allow addressing down to small pieces, for clear reference and reuse.

There’s nothing particularly new about any of this. You can distill versions of the same reasoning out of Vannevar Bush, Doug Engelbart and Ted Nelson. Hypertext does a pretty good job of implementing a lot (but not all) of this stuff. The HTTP web does a pretty good (but not perfect) job of implementing hypertext. Wikis pick up some (but not all) of the gaps.

What I’ve been exploring for the last ten years or so are tools which help enable the multi-dimensional contextualizing of information, so that people can blink a bit, think a bit and then nod knowingly. Granular addressability and transclusion with Purplewiki and Purple Numbers. Various tools in Socialtext to get information into or out of the wikis or get some sense of how things fit together (feeds, backlinks, page or resource inclusion, the REST API).

Which leads to TiddlyWeb. Because TiddlyWeb started out as a server-side for TiddlyWiki, and TiddlyWiki is a) already a wiki, b) already oriented towards microcontent, c) a curious hybridization of narrative, code and data, TiddlyWeb starts out naturally in tune with my inclinations. Where my opinions have impacted it are in the efforts to insure diverse storage and representation types, sensible granularity, and strong adherence to being webby, in the open web sense.

Thankfully, many people seem to have the same idea. The open web has always been about people doing the stuff that people want to do with information, with as few restrictions as possible.

Things should have a name, so in contrast to the semantic web, this thing for human learning could maybe be called the synthetic web. Not because it is fake, but because it has been made and is making all the time.

TiddlyWeb is Tools (not framework)

A couple of fairly extensive email threads on the TiddlyWeb google group have resulted in some noodling on the nature of TiddlyWeb, tiddlyweb-based applications and how those things differ from other tools associated with web based development. The threads have made it clear that those differences are less clear but more important than was previously obvious so I’m restating them here for future reference.

While it is the case that TiddlyWeb can be used as the basis for developing web applications, it was not designed or developed to be a general web framework. Yes it does have facilities that allow one to create what amount to views and controllers that operate with a model that is stored in and represented by TiddlyWeb. But no, it does not (yet) have facilities which ease the creation, construction and deployment of a project which could be identified as a TiddlyWeb project. Such things might be created at some time, but if they are they will be pieces that are external to TiddlyWeb itself. Add ons.

This is because TiddlyWeb is not and has no intention of being a full-stack, all in one, web application rapid application development platform. It adheres to a philosophy of componentization, modularization and composition. This philosophy is most apparent in the TiddlyWeb approach to installation and deployment of applications or projects. Those were the topics at hand in the email threads referenced above.

TiddlyWeb makes a big distinction between deployment and installation. Installation is getting the software and related common components installed on the server. This involves packaging and is closely tied to what is found in the setup.py file associated with a python package (and thus a TiddlyWeb application, because that’s what a TidldyWeb application is: a Python package).

Deployment is getting a version of the application, customized for my specific needs, running on my target machine(s), with all the expected initial data. In the TiddlyWeb universe, when you deploy, what you are doing is handling the special cases of a particular instance. All the general cases should be handled by simply installing (e.g. pip install) the package which contains the general application.

This differs quite a lot from “proper modern” frameworks like Rails. In Rails you install Rails, you generate a project, you install (in the project) the various plugins you might need. You then edit, in the project, your views and controllers and associated files. Throughout this process you are likely committing the entire project, or at least significant portions of it to a code repository. Other developers, on their machines, have Rails installed too, they can hack on your project too by doing a checkout. On the production server(s) you’ve got Rails installed and something like Capistrano which takes your project and makes it go.

That model could be developed for some TiddlyWeb applications if an enterprising person were to aggregate the necessary parts into a TiddlyWeb-as-a-framework toolset. Using TiddlyWeb as a framework is a relatively new thing so there hasn’t yet been time to identify and extract the kinds of patterns that would lead to the application development systems. Django and Rails, the two most buzzworthy full-stack frameworks, emerged out of existing applications and application development processes.

I personally have not spearheaded development along those lines because it is not the way I, personally, prefer to work. I work via a process of decomposition to comprehensible and distinguishable parts. I then make, as independently as possible, those parts. Later I compose those parts, plus parts from other people, as required, in different and exciting configurations. Those “parts” are usefully black-boxed for me but at the same time extremely flexible in how they can be used.

I work like this because I come out of the Unix tradition. 20 happy years of me working with small tools doing small, fairly specific, jobs. All these tools composed in fun and flexible and ultimately extremely powerful ways.

So the TiddlyWeb application development strategy (for me) is:

• Think.
• Decompose.
• Make some bits (i.e. tools).
• Make a bit which is the bit which assembles the bits.
• Install the bits somewhere.
• Tweak them as necessary for special cases in any instance.

If you are the sort of person who can’t think without coding at the same time (I’m often like this) then fine: Think in code then extract the decomposable bits. Use code as a scratchpad, with the results eventually being something you can reuse later.

There are two critical parts to this model: One is that it is essentially reentrant at any step. Your environment may already include some bits (i.e. tools) which you are using as fundamental building blocks to make your building blocks (which you’ll use later to build other things). They are “just there”. TiddlyWeb itself is such a thing for TiddlyWebWiki.

The other is the bit which is the bit which assembles the bits. This must exist, but because it addresses granular bits, the assembly is both easy to comprehend and to adjust. And because it is itself just a bit, it can be addressed by other assemblers.

It is clear there are plenty of people who do not work this way. In my efforts to understand the differences I’ve looked for parallels or patterns. What I’ve found is that the people who seem to get my method easily come out of a Unix administration or systems programming background or are just Unix naturals, whereas the people that struggle are traditional OO-based developers (with a preference for inheritance and identification by class rather than composition and identification by behavior), developers raised on rails/php or other start-you-from scratch frameworks, or just plain non-Unix people.

It takes all types to make the world go round, so that there is difference is good but what’s needed are some ways (I’d say tools) to bridge the gap. What this comes down to is a need for greater engagement from the non-Unixy side. Any takers?

TiddlyWeb and ccTiddly

In the comments to TiddlyWeb Plugin Tutorial Part 1 Eric asked:

Has ccTiddly been discarded in favor of TiddlyWeb? What are their essential differences?

I wrote a long answer as another comment which I’ve now copied (with some edits) up as a proper entry to give it a bit more visibility. It helps explains a bit more about the whys of TiddlyWeb.

Short answer is: No ccTiddly has not been discarded. ccTiddly has a large and still growing user base. TiddlyWeb exists as an alternative, with some benefits and some disadvantages. As the author of TiddlyWeb, I’m biased in favor of it.

The general design of TiddlyWeb comes from conversations between Jeremy Ruston (the original author of TiddlyWiki) and me. The basic idea was “If we were to create a TiddlyWiki server side from scratch, what would it look like?”

From my perspective the essential difference between TiddlyWeb and ccTiddly is that TiddlyWeb was designed from the ground up as a fully HTTP compliant web service for the storage and management of tiddlers. That is, in TiddlyWeb, tiddlers are first-class, primary entities: the basic thing with which you want to work. Not TiddlyWikis.

Tiddlers as first class entities leads to bags and recipes which are containers and builders for collections of tiddlers that are subsets of the entire body of tiddlers stored by the TiddlyWeb.

While TiddlyWeb was being developed ccTiddly has evolved as well, so it now has greater access to individual tiddlers and combinations of tiddlers. I don’t know the details of that, but as far as I can tell it is not as flexible as bags and recipes nor as accessible via the HTTP API.

The “I am a web server for tiddlers” nature of TiddlyWeb also leads to other things:

• A simple system for serializing tiddlers (and bags and recipes) into a whole load of different representations (wikis, HTML, text, JSON, Atom, etc) that is easy to extend by plugins.
• A straightforward system for storing tiddlers (and bags and recipes) in arbitrary storage systems (text on disk, on a remote webserver, in an RDBMS, in IMAP, etc) that is easy to extend by plugins.
• A simple plugin system for changing just about anything else on the system.

TiddlyWeb is built with Python as a WSGI application and relies upon Python packaging and distribution solutions for installation and configuration. This means it can be run with just about any web server (including the one that is built in) on just about any computer that supports Python (it’s been made to work on an iPhone). However it also means that if your computer does not have Python or does not provide a way for you to install Python or Python modules you are out of luck.

This last problem is common on some shared hosting services, where Python can be considered a bit special. In these situations ccTiddly has an advantage because it uses PHP and PHP is commonly available on shared hosting services. Often ccTiddly can just be ftp’d or scp’d to the shared host into a web directory and it will work.

From my perspective the good long bet is to go with TiddlyWeb. It’s very flexible, very powerful, and very much designed for and by the web: it is built with every hope that people will use it in unexpected ways, building new stuff on top of it.

From http://tiddlyweb.com/ lots of additional stuff can be found.

Indexed TiddlyWeb Filters

One of the core features of TiddlyWeb is its ability to use filters to constrain the tiddlers that are selected from any collection of tiddlers (bag, recipe, search results, etc.). In the early design discussions that led to the creation of TiddlyWeb filters were conceived as the mechanism a recipe would use to choose only some tiddlers from a bag. Bags are containers for tiddlers that have been grouped together for some reason. Recipes are lists of bags that lead to the creation of some useful set of tiddlers. When using TiddlyWeb and TiddlyWiki together, a recipe can create a particular application or vertical of TiddlyWiki. In that context one would use a filter to select some tiddler tagged “systemConfig” from one bag, others tagged “faq” from another, and others with modifier “cdent” from another.

When TiddlyWeb is used generally as a data store, filters are just as useful. When requesting tiddlers from any bag you can select and sort by attributes, and limit the number of tiddlers. Application developers can also make new filters as plugins (see mselect for an example).

This is all quite grand and useful but recent explorations by Mike Mahemoff while developing Scrumptious have revealed some (fairly expected) problems. Imagine a bag called “comments” containing some 10,000 or more tiddlers which are comments on URLs. Now imagine you’d like to get those tiddlers which have the field ‘url’ set to http://cdent.tumblr.com/.

The naive way to do this is to look at each one of those tiddlers, one at a time, and say “Hello tiddler, have you got your url field set to http://cdent.tumblr.com/? Oh you do? Well then I’ll have you, thanks!” This is time consuming and resource intensive.

It is also the way TiddlyWeb does filters. It’s like this for a few different reasons:

• The original design imagined many bags, not few bags with large numbers of tiddlers.
• It preserves the strict separation between the filter system and the storage system, meaning that the storage system can be simple and very adaptable: any filter can work with any store.
• It makes the filter system fairly transparent: There’s no magic going on; a filter works by looking at tiddlers and making a decision.
• It makes the filter system easy to extend: The contract between a filter and the rest of the system is “look at some tiddlers, return some tiddlers”. What the filter does when looking is arbitrary.

The 0.9.74 release of TiddlyWeb includes support for querying an index when doing select style filters. The support only kicks in in special circumstances (explained below) but when it does it can speed some filters up immensely. In a test (using profile/list_tiddlers.py) of 10000 tiddlers a filter that took 13.96 seconds without an index took .30 seconds with an index.

That’s great news. Here’s the bad news: In at least this initial implementation the prerequisites for the indexing system to be used (and be useful) are quite complex. Here’s the list:

• The filter being performed must be a select filter (sort and limit do not).
• If there are multiple filters being performed on the collection of tiddlers, the select filter must be first in the stack and only that one filter will use the index.
• The collection of tiddlers must be what’s called a “natural” bag. That is, the thing being filtered is a bag that exists in the store and the entire contents of the bag is what’s desired to be filtered.
• That bag should be skinny, meaning when it was loaded from the store its tiddlers contents was not determined. If you are processing recipes or working from URLs this is handled for you by the code. It’s only a concern if you are writing your own handlers.
• tiddlyweb.config[‘indexer’] is set to a string which is the name of a module which provides an index_query(environ, **kwargs) callable that returns tiddlers which have been loaded from the store. tiddlywebplugins.whoosher has been updated to provide this and the next item.
• Something must provide an index for index_query to query. That index needs to be kept up to date as tiddlers are changed. tiddlywebplugins.whoosher has this functionality. The sql and mappingsql have the guts to make the functionality possible, but have not yet been extended with an index_query method.

The TiddlyWeb at http://tiddlyweb.peermore.com/ has been updated to use the filter indexes. The relevant changes to the tiddlywebconfig.py are:

• Add tiddlywebplugins.whoosher to twanager_plugins.
• Add tiddlywebplugins.whoosher to system_plugins.
• Set indexer to tiddlywebplugins.whoosher.

Then twanager wreindex is run to build the initial Whoosh index.

You can get some sense of the effects of the index by comparing the following to URLs (this is not an exact test, but gives the sense of things):

• Not indexed: http://tiddlyweb.peermore.com/wiki/bags/docs/tiddlers?limit=5000;select=tag:systemConfig
• Indexed: http://tiddlyweb.peermore.com/wiki/bags/docs/tiddlers?select=tag:systemConfig;limit=5000

The docs bag only has a couple hundred tiddlers in it and memcached is involved, so the effect is not huge, but if you imagine bags orders of magnitude bigger…

Astute observers will note that what’s going on here is not particularly innovative: It’s simply the addition of an index to a query system. One can imagine future improvements a la SQL query optimization, wherein the order of the filters are adjusted to allow most effective use of the index, and the index is used for more queries than just those against “natural” bags. Constant evolution, constantly building on the shoulders of that which has come before.

For more details, a browse of the code will be instructive. tiddlyweb.control:filter_tiddlers_from_bag is a good entry point.

TiddlyWeb Value Proposition

A common question associated with TiddlyWeb is “What can I do with it?” or “Why would I want to use it instead of Django or something?” The answer is not simple.

Because TiddlyWeb has tools for storing data and presenting that data in various forms over HTTP, it has some of the core features of a web application framework. However, since it (intentionally) is also missing some of the core features of a web application framework, a fair piece of cognitive dissonance can happen when the tool is approached as a web application framework.

So it seems best to stay away from saying that TiddlyWeb is a web application framework, so as to manage expectations. The question remains, “What is it?”

One pathway to answering that question is to create tools with it and see what the commonalities are between those tools. A derivative of this approach is to create a killer app with TiddlyWeb: a something which demonstrates all the salient features and how to use them, both as a user and as a developer.

This approach seems to falter: The concepts in TiddlyWeb are sufficiently strange (I won’t say unique, because they aren’t that, they are common concepts, with their own spin) that in order to create a tool that uses them well you first need to either understand them in the abstract, or see some examples. Catch 22, especially if you are an examples oriented person.

FND and I tend to converse on this topic about daily, trying to come up with different strategies for answering it. Today I suggested that instead of creating a new application, from scratch, that is cool and interesting because it uses TiddlyWeb, it might be better to find some application that could be made better through the addition or inclusion of TiddlyWeb.

It turns out this process has already started and by analyzing that process we might understand TiddlyWeb better.

The process has started with TiddlyWiki. TiddlyWiki is made better for a particular set of use cases by taking advantage of TiddlyWeb features. The fundamental experience of TiddlyWiki does not change: you’ve still got a wiki in an HTML file, with these things called tiddlers, some of which are special plugins that change behaviors.

What TiddlyWeb adds when used with TiddlyWiki is:

• Centralization of content storage, storing at the level of the tiddler, with revisions, protection against editing conflicts and access control to read, write, create and delete; accessible from multiple locations on the network.
• A (hopefully) straightforward system of managing access control of tiddlers and making other kinds of groupings of tiddlers through the use of bags. Bags are collections of uniquely named Tiddlers.
• A system for composing functional collections of tiddlers selected from multiple bags using recipes. Recipes are list of bags paired with filters. Filters are rules which limit the tiddlers pulled from a bag when processing the recipe.
• Tools for viewing and manipulating tiddlers in different forms than those used in the wiki (e.g. JSON, Atom).

These four things together provide a multi-user environment for TiddlyWiki that allows multiple custom views of the same or similar content, depending on some piece of context. Different recipes can provide a different view on the same stuff, providing a different look and feel, different security handling, etc.

TiddlyWeb has many other features, but the above four are probably the key aspects which define its special sauce. When I evaluate a nascent TiddlyWeb tool for its tiddly-fitness the evaluation is done based on the extent to which the system takes advantage of and understands the above features.

These ideas should allow us to create some mental heuristics to use when approaching a TiddlyWeb project. If we have an existing project or use case that we intuit will be helped by using TiddlyWeb we can ask ourselves: 1) What aspects of this project will benefit from bags, recipes and filters? 2) How can we take advantage of different representations?

If, for some reason, we are planning to make a project from scratch, and have decided to use TiddlyWeb prior to really evaluating our use cases, then we can ask ourselves: 1) How can we best structure our data and UI to use recipes, bags and filters to provide flexible user experiences? 2) What representations can we present to allow other people to provide flexible user experiences?