Peer Pressure

Internet Bank of Content

This is a followup to Invert the Web where I wrote:

Today I go to a web site and give it some output of my brain. The web site takes my content and in exchange it gives me a URL so I and other people can get to the content again. What should happen instead is that I make some content and give that URL to some websites so they can get to the content.

In typical me fashion that wasn’t very clear. Here I shall try to make parts of it more clear and other parts more muddy. Let’s expand the above.

Today when I publish a photo or make a tweet the mechanism goes something like this:

• I suck the photo off my camera, do whatever tweaking I like to it, and then upload a copy of it to some photo site (e.g. Flickr). My photo then gets a URL somewhere in Flickr’s URLspace and I can tell people about that URL, or people can browse Flickr and stumble upon my photo. If things go my way and I play the game right, the photo might even show up on Explore.
• I go to http://twitter.com/ or use an app like Twitterrific and type up my tweet. The content is transmitted to and stored on Twitter’s servers and gets a permalink in their URLspace.

What I would prefer is something like this, where the URL becomes a more Universal Resource Locator:

• I suck the photo off the camera and store it in my own personal URLspace, getting a URL that links to the content. I then give that URL to Flickr and whatever other services provide a nice URI for browsing photos. People can still browser Flickr and stumble upon my photo and the photo may still show up in Explore. And lots of other places too.
• I write my tweet and store it in my own personal URLspace. I use the returned URL and give it to Twitter, Pownce, Jaiku, Facebook, MySpace, iChat and a variety of other “status” related services.

At this stage of the conversation the first objection is usually, “Not everyone can host their own content.” Indeed this is true. It’s also true, though, that not everyone keeps all their money in their wallet, under the mattress or in a mason jar in a hole in the back yard. They use banks. And those banks provide a variety of services to provide incentives for people to deposit their money with them. So we could have banks for content.

Another objection is “I don’t want to do that two step publishing dance just to get something on Flickr.” Me neither, that’s why we have computers to automate the parts of processes which are fully describable. POSTting something in the content bank and then PUTting the returned URL to Flickr is fully describable.

Why would you want a bank for content? What would the benefits be?

• All content remains yours, held in trust by the bank, forever.
• If a presentation service goes out of business, your risk or loss is more manageable.
• Any publication of content can be immediately rescinded by changing access policies on the URL (modulo whatever caching policies you’ve chosen).
• You can (in theory) have access policies or capabilities statements on all your URLs.
• Anything which can be captured by a MIME type can be stored in the bank.
• You can host your own bank if you want: It’s just HTTP.
• You can publish your content to multiple places by reference instead of copy. When you make changes, you don’t need to worry about going round to several different presentation services.
• You have the option of maintaining several content accounts for all your various internet personas.
• Banks could differentiate themselves by providing different levels of service: different ways of managing access control, service level agreements, data sharding and redundancy. Some people might opt for free content accounts in exchange for allowing some or all of their content to be anonymously mined for targeted advertising purposes or market research.
• A suitably well featured bank could provide persistent and versioned storage for all your stuff, ever. Kind of like a github for everything, not just code.

And what about places like Flickr or Twitter, what we might call presentation services? What could they gain?

• They don’t need to worry about storage quite as much.
• They can concentrate specifically on the UI of their service, distinguishing themselves through innovations in that area.

These ideas are the latest synthesis of a variety of things which have proven to be important on the internet:

• Separation of concerns. The presentation services worry about presentation and banks worry about storage.
• Cool URIs.
• Data ownership and portability. It’s always yours.
• MIME. Lots of different types of content.
• Stupid Network. Put the smarts where the smarts need to be and nowhere else.
• The awesomeness of REST, or more accurately HTTP, and the open web it/they allow.
• Using content by reference rather than copy (c.f. Xanadu, Transclusion, Purple Numbers).

It would probably be possible to prototype a content bank in TiddlyWeb or CouchDB or some kind of interface to S3. My own first work related to such things was closet from which TiddlyWeb got a few ideas.

Invert the Web

I was talking this afternoon with Fred. Over the past few days, he and Jon have been creating a Twitter archiver called TiddlyTweets using TiddlyWiki plugins. It’s cool: you give it a twitter username and it retrieves all the tweets and archives them in a useful fashion, preserving a link to their original URI.

I’ve been hassling Fred about this, as I like to do, because one of the reasons I have liked Twitter is because tweets have, or at least had, a sense of being ephemera. In my mind, if you feel like you need to archive your tweets, then you are doing it completely wrong.

But that’s just my opinion and Fred’s is different. He says there’s good stuff in those tweets which you might want later. This is true enough.

Today, in IRC, he said:

without starting the whole discussion again, after
yesterday's(?) archiving exchange, it struck me as
odd that you're so indifferent - after all, you're a
proponent of cool URIs, which kinda includes permanence

My response went something like this:

Every tweet has its own URI already, the “statuses” URL, so the coolness is basically there already. Fred responds that Twitter is not reliable. This is true but I suggested that archiving was a short term solution to the problem that allows the long term problem to persist. The long term problem is that ones tweets live at Twitter, not in ones own persistent store: Twitter should be a presentation engine which has access to your own microcontent.

This reminded me that in some ways what I’ve been working on for the last ten years or so with Purple Numbers, wikis in general, TiddlyWeb, persistent identifiers etc. are tools to make such things possible. What I’d like to see is an inversion of the current content relationship that is present on the web.

Today I go to a web site and give it some output of my brain. The web site takes my content and in exchange it gives me a URL so I and other people can get to the content again. What should happen instead is that I make some content and give that URL to some websites so they can get to the content.

Resources Before Actions

When doing open systems design why should we think about resources before actions?

It’s just polite.

When you design a system that provides information to other people, you can’t really know with true confidence what those other people might want to do with it. What you can know is what you have: What you are are resources.

And while you can’t predict the details of what might be done with your resources, you can predict or assume that people will want to do some very basic things with them: get at them, put to them, maybe even delete them.

Getting those basics in place first is a good start.

REST, I just want it

Yesterday I was in London for an Osmosoft TiddlyWiki hackathon. I didn’t get much hacking done but had several very valuable conversations and met some good folk. Running throughout these conversations were threads of REST, HTTP, web and software architecture. Common to these threads was a lack of consensus about what counts as “good” in any of those contexts and what concepts shared in those contexts could be be considered good in a more universal sense.

So it was with some pleasure that I woke up this morning to see that Simon Willison had linked to Damien Katz (of CouchDB fame) saying REST, I just don’t get it. The comments are instructive in two ways:

• They provide a lot of good and varied input on why thinking about software systems in a REST kind of way is useful.
• They show that the rest of the world, not just some guys at a TiddlyWiki hackathon, are also lacking in consensus and shared understanding when it comes to web architecture.

In the end it is all really quite a subjective matter of opinion and in such cases, a diversity of opinion is a great way to learn, explore, find the edge cases; to gain new perspectives. So with opinion being the operative word, here’s where I think recent commentary gets it right and wrong.

While talking with Martin and Fred yesterday about the TiddlyWeb API Martin made the statement that it is generally accepted wisdom that minimizing the surface area of an API is good practice because it allows the most stability at the surface while allowing internal change. I agree with this.

chu’s comment:

I think only allowing up to 4 methods is a key benefit. Having to express an app through nouns rather than verbs leads to a much cleaner, more robust and encapsulated design.

says sort of the same thing, from a different angle. It also introduces my favorite bit about systems that try to have a REST style: nouns; the parts of the system as resources on which there are a very small number of operations that can be performed. Building blocks, lego, in a system with a simple grammar but complex expressivity.

Damien says:

I guess what I mean to say is just because SOAP is a disaster, doesn’t somehow make REST the answer. Simpler is better, and REST is generally simpler than SOAP. But there is nothing wrong with a plain old POST as an RPC call. If its easy to make all your calls conform to the RESTful verb architecture, then that’s good, I guess. But if not, then just use a POST as an RPC call, keep it as simple as possible and be done with it. And don’t spend another minute worrying about being RESTful or not.

I think there’s right and wrong in here and one of the comments nails the problem. Avdi says (my emphasis):

One of the big (but non-immediate) wins is explorability. If you don’t care about emergent properties, if you don’t care about what other people might build on it, then sure, use nothing but POST; use SOAP; use CORBA, for that matter. The magic of REST is that no WSDL or IDL can replicate the experience of someone clicking around a RESTful API in their browser like a kid opening up a box of Tinker Toys and saying “yeah, I get how this fits together! I could build on this!”.

RESTful services are easier to build on, especially when the things being built are things the providers of the service couldn’t or didn’t predict. I use the term service intentionally. If you’ve got yourself an application, where you want to control the user experience, then whether you use a REST style of architecture or an “RPC call” might not matter.

Unless, that is, you are concerned about scalability and availing yourself of the caching capabilities of the web. The comment from Colin Percival hits the REST association with caches.

Not mentioned in Damien’s discussion, but present, at least tangentially in yesterday’s hackathon is what I think is perhaps the most interesting part of having a web oriented frame of mind when thinking about software system design: pipelines, chainability, stackableness, composability. The Lego Factor. For people who grew up on a unix-like command line this is the feature that separates the good from the bad.

A web service that is built upon RESTful principles is easy to pipeline because it has a small and well known API (some or all of GET, PUT, POST, DELETE) that makes it possible to GET a representation from one service, transform it in some fashion and send it on to another service.

The WSGI specification for Python helps extend this model down into code with simple middleware that transforms either the web request or response. This is awesome because it helps encourage code modules which are encapsulted and decoupled while not requiring a adherence to a bunch of framework. These modules are just like web services but running in the same process space.

I could go on endlessly here. To sum: RESTful principles are for the most part an expression, in a distributed space, of a lot of time worn wisdom about how to make good software systems that are reusable:

• minimize complexity
• maximize discoverability
• minimize exposure of special knowledge and methods
• maximize opportunities for reuse and chaining

Damien’s right that some of the time the principles aren’t going to be right for a particular use, and there’s no point getting in a twist about them. But it is obviously true, that on the web, designing and building with respect to what works on the web is good. Tim Bray gets the nod for that. His comment is a good ending:

Well, a REST architecture serves as the basis for the most biggest and most successful information system the world has ever seen, that effortlessly deals with heterogeneous hardware and software, and provides a high-quality user experience while scaling towards a billion users. So think of it, as nothing else, as a good benchmark that you have to be better than.

What about REST makes you want it?

Edification Without Edifices

My flatmate sees me walking round with a tshirt that says “Wiki Way” on it, he knows I used to work for a company called Socialtext and that they make wikis. He knows I work with a thing called TiddlyWiki and once helped make a thing called Purplewiki. So it is only natural that when he finds an article in the New York Times about Wikipedia folk meeting up in Egypt for Wikimania that he set it aside for me.

What isn’t natural is that this makes me grumpy. Having my vocation or what have you associated with Wikipedia gives me the scowly face. This is really unfortunate because Wikipedia is obviously a wondrous step forward, a great resource for people all around the world, a truly great accomplishment by huge groups of people. I use it nearly every day. Yet it makes me a grrr a bit. This is obviously my problem, not Wikipedia’s problem.

Since writing is a good way to get some thinking going, here’s something.

I guess the basic reason I have issues with wikipedia is because it has absconded with the word “wiki” and taken it in a direction that is lovely and exciting but not the one I wanted. Wah, they took my toys! The wikipedia direction involves an enormous number of readers and a still very large number of writers/editors but a ratio of readers to writers that is very large. An effort to aggregate what is already known. An information resource.

Another style of thing that also has the name wiki has a much closer ratio of readers and writers, a more inclusive sense of participation and culture of engagement (say that with a French accent). They get this, in part, by being small and associated with a community of fairly specific practice. A learning orientation. An effort to create knowledge. An action resource.

In those communities the facilities of the wiki software that encourage a synthetic and shared learning process are highlighted and relevant: LinkAsYouThink (I really like that Eugene says, “What makes this feature beautiful is that you can link to a page that does not already exist”), recent changes, back links, and of course easy access to editing.

Then again, it isn’t the software that matters: it is the community. At the scale of Wikipedia you simply can’t have the sense of homegrown localness that you get with little wikis. The sense of earnest participation.

And, at scale, you get the apparatiks. Them what’s got the power tell them what don’t how they should be getting on with things. Information and power collects to a center.

Which leads to the final winge: If the things about wikipedia that make it a wiki (LinkAsYouThink etc) are de-emphasized by design or necessity, what’s left? What’s left is a big collection of editable hypertext. Which means it is really a centralized, editable library located as an edifice in the rest of the web. As a library it is a huge step forward, but the ideal vision is that the entire web is the editable library.

Wikipedia gets things more right than loads of other edifices and it deserves major props for that. Compare it to other gardens, with much higher walls. Like Facebook. Facebook takes the web, puts a nice invitation on it and gets enough of your friends involved that you can’t help but participate. But is there anything happening there that couldn’t be done with diverse and open services on the web that can be independently aggregated and federated?

Before the web it was often the case that in order to get a large change done it required something like a national government to amass the resources. Now that most of the required resources are information, what’s left to overcome before we no longer need edifices?