Peer Pressure

The Synthetic Web

In a recent thread on the TiddlyWeb Google group, I stated:

My opinion is that stuff, i.e. information, should be addressable and reusable. That’s my fundamental number one design goal with TiddlyWeb.

This was while making the assertion that the URIs used for entities in TiddlyWeb should have names and structure that is most meaningful and most useful to the general user who stumbles across the entities while browsing the web, not names and structure that is most meaningful to the developers of the system. Furthermore the names and addresses and reusability are for humans.

Ideally the names should be useful to both sets but when there is a conflict I think the unknown user should get the benefit or assist from the system, not the developer. This is my personal opinion, but because I’ve written most of TiddlyWeb, it shows throughout its design.

The opinion has been active throughout what might be called the second half of my career. The first half could be described as managing information systems, the second half could be described as creating them. The opinion goes something like this:

• Information exists. However it is represented, it is in a system and can be used.
• Some information systems exist to confirm what we think we already know or for which we can phrase a fairly concrete question (e.g. “Where is London?” or “What is the definition of ‘information’?”). In these cases, searching or similar interaction models are excellent tools: our inquiry is algorithmically transformed into a query against an index (note an index can take many forms). Think dictionary. Or google.
• Other information systems are storehouses (real or otherwise) which provide affordances for navigation, browsing and discovery. These allow for that extraordinary thing: learning new stuff. We’re wandering along, soaking up new information or new ways of stating old information and BAM, we make a connection, disparate paths are linked and now we know something. We have made a synthesis of A and B et voila new thesis C. Think narrative book. Or the web at large.
• These systems that enable synthesis are fundamentally more important than others because synthesis is the source of learning, innovation and change. They drive intuition, inspiration and insight.
• While data is fundamentally useful to provide authority to new insights, human narrative, human summarization (nee synthesis) is the part that makes you go “hmmm”. In an academic paper the part that thrusts us into the future is the frame: the introduction and conclusion, the setting of the stage and the synthesis. The parts in the middle make the rest believable.
• Some data can representable with rich syntax that encodes semantics that make it automatically reusable. That is, you can package up data in a structured way such that computation can be performed on it. This is the fundamental premise of the semantic web. Such computation can deduce (deduction: conclusion drawn from examination of facts) true statements such as “Chris Dent is a male living in England working on TiddlyWeb”.
• Narrative is not so clean cut. It is full of connotation and contextual dependence. What it means must be inferred and that inference is inescapably done in the personal headspace of the reader/listener/participant.
• Opportunities for innovative synthesis from narrative are augmented by arraying (representing) the narrative within the context which created it in the first place and things which are similar or can be identified as being related in time, place, group.
• Computers have achieved a special place in the history of information systems because they are the most capable tool (thus far) for easing the task of navigating these multi-dimensional representations of narrative.
• Critically, thus far only humans are able to perform the synthesis that results in new knowledge and from which new narrative might be created.
• Often it is but a small piece (microcontent) of a larger whole which strikes the chord that plays eureka.
• Therefore: ** Information, especially narrative, should be published in a way that humans can easily make reference to it and use of it as they do the re-representing that is required to engender synthesis. It should have good names and addresses. ** Tools which enable this publishing should make as few assumptions about the people and tools accessing them as possible, so as to enable as yet unknown ways of (re-)representing the information. ** Ideally the tools should allow addressing down to small pieces, for clear reference and reuse.

There’s nothing particularly new about any of this. You can distill versions of the same reasoning out of Vannevar Bush, Doug Engelbart and Ted Nelson. Hypertext does a pretty good job of implementing a lot (but not all) of this stuff. The HTTP web does a pretty good (but not perfect) job of implementing hypertext. Wikis pick up some (but not all) of the gaps.

What I’ve been exploring for the last ten years or so are tools which help enable the multi-dimensional contextualizing of information, so that people can blink a bit, think a bit and then nod knowingly. Granular addressability and transclusion with Purplewiki and Purple Numbers. Various tools in Socialtext to get information into or out of the wikis or get some sense of how things fit together (feeds, backlinks, page or resource inclusion, the REST API).

Which leads to TiddlyWeb. Because TiddlyWeb started out as a server-side for TiddlyWiki, and TiddlyWiki is a) already a wiki, b) already oriented towards microcontent, c) a curious hybridization of narrative, code and data, TiddlyWeb starts out naturally in tune with my inclinations. Where my opinions have impacted it are in the efforts to insure diverse storage and representation types, sensible granularity, and strong adherence to being webby, in the open web sense.

Thankfully, many people seem to have the same idea. The open web has always been about people doing the stuff that people want to do with information, with as few restrictions as possible.

Things should have a name, so in contrast to the semantic web, this thing for human learning could maybe be called the synthetic web. Not because it is fake, but because it has been made and is making all the time.

Info Condom

The recent suffering of Twitter and Facebook (and others apparently) at the hands of a huge DDOS is just one more reason to disapprove of centralized, single-source, non-federated services: They are vulnerable to just this sort of thing.

A system of federated presentation services and many Banks of Content would be far less vulnerable. Oddly, this is a well known lesson and a fundamental piece of the architecture that underlies the internet. However it is a problem that is far easier to solve at the level of technological interaction than at the level of social interaction.

One of the appeals of Twitter and the like is that you can experience moments like this: “I’m on Twitter! ZOMG are you on Twitter too!!?? You totally have to get on twitter and then we can be on twitter together!”.

That’s far more fun and engaging than: “I propagate my microblog content by URI from my robust distributed content store to multiple presentation services, it would be great if your microcontent could co-mingle with mine on the open internet and together stimulate the generation of new knowledge.”

(Actually that sounds really sexy to me, but I’ve been told I’m a bit strange.)

Tools Together

Brain Dump

As TiddlyWeb has matured, one of the things I have noticed is that people are often wanting it to do more than it does. The generic form of the query is something along the lines of “Wouldn’t it be more complete if it did X”.

This is interesting to me because in a way it is contrary to the TiddlyWeb point. At base TiddlyWeb is a tool for manipulating chunks of stuff called tiddlers on the web. It is one of many tools in the toolbox for doing web stuff.

An analogy may help. There are lots of tools out there that spin and accept attachments: screwdrivers, socket wrenches, drills, etc. You can make a drill be a screwdriver if you want, but sometimes it is overkill. TiddlyWeb is an extensible web service: with extensions it can serve up static content, use templates, do all the usual web framework stuff. But that doesn’t mean it should be used for all the usual web framework stuff. Maybe, maybe not.

TiddlyWeb comes out of the Unix tradition: small tools that do one job well and can be integrated with other small tools through pipelines. This is true in how TiddlyWeb can interact with other tools and also internal to the architecture of TiddlyWeb: separated layers with strict contracts for interaction. This means it is easy to extend and it interoperates nicely with other stuff.

The most performant and scalable implementation of a large scale service that involves TiddlyWeb likely involves TiddlyWeb; a separate web server (or two or three or four) serving static content, doing some URL rewriting, some caching; a memcached server; a database server; and maybe another framework (django, turbogears, rails, etc) presenting UIs that might use the TiddlyWeb server as a datastore.

This last thing (presenting UIs) is the main area in which TiddlyWeb intentionally makes no provision (to concentrate on effective backend handling). Good web frontend presentation is strenuous, there are existing tools which make it easier. A tool approach means that TiddlyWeb can play well with those guys.

Leaping Forward

I’ve been having a bit of a debate with @Casablanca about the need or merits of designing and developing for people who use Outlook or IE. It started with:

@cdent: This #fixoutlook thing is bullshit. Just don’t use outlook!

@Casablanca: @cdent it’s not about whether you use outlook. It’s about having to design for people who do (e.g. our client’s customers)

@cdent: @Casablanca don’t design for those people! Design well enough that they’ll want to use what you’re using. If U cowtow, they’ll never learn.

@Casablanca: @cdent you mean we should tell our clients to tell their customers to change their email clients? A bit impractical, no?

@cdent: @Casablanca Just because it is impractical doesn’t make it wrong.

@Casablanca: @cdent does that mean you’ve never tested code in ie, because you don’t want to design for those people either?

@cdent: @Casablanca I haven’t personally tested code in IE for several years, if ever. Nor do I test other code in non-posix envs.

My basic assertion here is that people or organizations won’t change to things that are better if they can get the cool stuff on what they already have.

A secondary assertion is that by requesting Microsoft fix Outlook, Microsoft is not getting the punishment they deserve for causing the problems they have caused over the last…well the last very long time. There shouldn’t be a campaign to fix Outlook, there should be a campaign to boycott Outlook and boycott developing for Outlook.

I understand the economic considerations involved in the calculus used to determine “shall we do the work to make this work in IE”. For people encumbered by relationships with the corporate world, it’s kind of inevitable that someone along the chain will declare an operating environment where things must work. Likely that is an environment where openness, consensual standards and interoperability are meaningless tokens, not thriving actors in the system.

I try to avoid those kinds of relationships, and think other people should as well. I’m well aware that some people can’t and some people don’t, and more power to them. In fact it works out quite well for me: Because I work on open code in open environments, if there are people who are interested in making the stuff I do work in non-standard environments, they have easy access to do so. The pain stays with the people who need or want to suffer it.

Yes, I’m being impractical. Yes, I’m being unrealistic. But I can see little to suggest that the practical realities of the world are the best ways of doing things, so I’ll continue.

Don't Shape My Net Bro

I have always been, and plan to remain, fiercely of the opinion that ISPs should never be involved in traffic shaping at the application/layer 7 level. It is completely antithetical to the spirit of the network (stupid network and all that) and I think in the long run it is very bad business for ISPs.

If a customer wishes to do their own shaping to maintain quality of service (e.g. prioritizing interactive ssh sessions over retrieving web pages) more power to them. Such tools are available.

If an ISP wants to do shaping of the the total bandwidth any single endpoint gets across a switch, that makes pretty good sense.

But: If the ISP starts digging into packet headers and even packet bodies to do shaping they are introducing a huge cost for themselves. On the one hand their switches much do greater work. On the other hand there is now a presumption that the ISP knows or can know the content that is traversing their network and can be held liable for producing data about that content or controlling that content.

I get the impression that this latter point about liability is already a lost cause in the UK, which is a damn shame.

If BT or any other ISP is selling 8mbps connections to customers, then the customer should be able to expect that level of service for any type of content they get across the wire. If BT insists on being so oversubscribed that they can’t support that level of service at all times, they should shape total usage, not specific protocols or applications.

It’s interesting to me that BT would want to perform any kind of shaping. BT has an effective monopoly on the physical infrastructure that provides these services. By encouraging more and more intensive use of the internets they increase the demand for the infrastructure they provide.

Disclaimer: I do contract work with Osmosoft which is part of BT. These are my opinions. I’m sure there are lots of people within and without BT with lots of different opinions on this matter.