Peer Pressure

Update 20130122: So it turns out that my predictions about the expected uses of TiddlyWeb, especially with regard to the management of HTML presentation, were not correct. In the 1.4.x version of TiddlyWeb, HtmlPresenter was removed in favor of the HTML serialization providing the full HTML document. There’s a related google group thread.

I read somewhere, long since lost, someone talking about what happens when you start integrating web, REST and WSGI styles of thinking, designing, and developing into your brain. The result is much the same as what you get if you came of technological age during the ascendency of the Unix Pipeline.

This thinking is variously described as right tool for the right job, one tool that does one thing well, pipelines, one liners, compartmentalization, separation of concerns, lego, simple grammars leading to complex behaviors, constraints will set you free…

WSGI makes it possible to stack little web services on top of one another. On the way in you can modify the request environment and context, making choices and adjustments as required. On the way out you can modify the response, headers and content in as simple or complex a manner as required. Each pass through the service stack is in the same process space but the specification encourages the use of only three known entities: the request environment, the response headers and the content being delivered.

It’s very easy to share data globally if you like, or with objects passed around here and there, but if you imagine each WSGI app as its own server a sort of radical simplicity and separation can emerge. Each service is a world unto itself, a world so seemingly complete and perfectly contained that when someone wants a change to the world it kind of goes like this:

them: How about you make it do X?
  us: No.
them: Why not, all you need to do is Y?
  us: It doesn't do that.
them: So. Change it?
  us: No, see, it doesn't _do_ that.
them: That's kind of lame. We need X.
  us: I'll make something else. Something that does _do_ X.

Now it may seem like us is kind of a jerk here, but us, who is really me, has discovered that the sort of architecture that leads to this thinking is very easy to test, to build, to deploy, to maintain, to extend. It’s got good lego characteristics.

This scenario played out, a bit, in real life today. TiddlyWeb is primarily designed to be a data store and provider for the generation of dynamically created aggregations of Tiddlers. You can store tiddlers and get them back in a variety of representations.

TiddlyWeb is at core a WSGI application using selector for dispatch. Surrounding that core are about ten other WSGI apps handling tasks like content negotiation, credentials extraction, exception handling. When a GET request is made the request entity is retrieved from a Store and Serialized into a representation.

From quite early on in its development TiddlyWeb presented HTML representations of its various resources and collections thereof. In addition to being a useful representation for other tools, HTML makes it possible to browse around the TiddlyWeb service and see what’s there.

To make this work in a hurry the HTML serialization system created full HTML pages as representations. As people have played with TiddlyWeb they’ve found the pages useful and wanted more on them: they’ve wanted TiddlyWeb to act like a web site.

My response to that is the hopefully predictable by now “no way, man, TiddlyWeb doesn’t do that”. TiddlyWeb is supposed to be a web-service not a web-application. My dogma, however, doesn’t change the fact that a clean and usable presentation of TiddlyWeb content is useful nor that people are used to and gaining benefit from the HTML presentation TiddlyWeb already does. So something needed to be done.

One method already exists: TiddlyWiki presents TiddlyWeb content very well. This is not surprising as that’s what TiddlyWeb was originally built to do.

The method I prefer is some other web application that wraps TiddlyWeb in a luxurious and usable HTML embrace. This hasn’t happened yet, but I’d like to try it with django soon.

The other server method is not great for some: it presents administrative and technical challenges dealing with installation and authentication complexity. So a third option is to build another piece of TiddlyWeb, separate from the core, which can be used for beautifying the HTML output.

I such a tool today and I think it is going to work out pretty well. It puts one job in one place (an HTMLPresenter) and make the job of the HTML serialization tighter, more in alignment with its real purpose: Turning a resource into an HTML representation.

The HTMLPresenter is an optional and replaceable wsgi app that watches outgoing responses. If it detects that they are HTML serializations of resources, the representation is wrapped in warm loving HTML provided by the HTMLPresenter.

Because the HTMLPresenter is a replaceable piece of code (by changing the config file) and because it is in TiddlyWeb’s wsgi stack, an installation can make the presenter be whatever is needed. The default one sticks a header and footer around the provided HTML. One that was more interesting could use a templating engine, suck in content from elsewhere, present user dependent data, whatever.

Or, if you want, you can get rid of the HTMLPresenter entirely and let something else worry about it.

Mark Bernstein says

Software designers — especially those whose starting point is UI/User Experience — often argue that the user’s needs are mostly simple. It’s just not true.

with no reference. I’m not sure who he’s been talking to in the design field. The world is a pretty complicated place and each one of us bring our own complicated interpretations to every moment. We can choose to use tools to help us clarify and order our information surroundings but our choices are chaotic in the richest sense of the word.

Adam Greenfield’s extended bit on the context users bring is a good kick to the head that illuminates how much depth there can in a moment of interaction with a piece of technology.

I reckon Adam and Mark agree on that point. Who are these designers that disagree?

Recently Joe Gregorio posted about RESTful JSON. It generated such lively comments that a mailing list/google group was created. Discussion has been flying along at a brisk pace, a fair number of people trying to locate the common ground.

This is great, clearly there is a need for something, otherwise the audience wouldn’t gather so quickly; but, you know, I can’t figure out what the hell these guys are talking about. I’ve been through this sort of thing enough of the time to know that the problem is that the context I’m bringing to the conversation is different from the context others are bringing.

Joe says

What we seem to need is a data-oriented REST protocol. We already have document-oriented REST protocols covered with the Atom Publishing Protocol, but what if the information you want to convey is data, i.e. doesn’t have the minimum meta-data to qualify as a document, such as an author, title, published data, and id. If you’re going to be slinging data around these days the best thing is probably JSON, so what would RESTful JSON look like?

and then proceeds to stub out some ideas on how to represent collections of stuff in a JSON representation that allows efficient access and management over HTTP. On the mailing list these ideas have been extended to include issues surrounding partial updates of resources. There appears to be confusion about what the resource is. Is a resource a thing and its attributes? Are the attributes yet more resources? Is a collection a resource itself? There appears to be confusion about the distinction between a representation of a resource (the JSON itself) and the resource (the stuff that the representation represents). This confusion seem to be leading to a desire to create a universally applicable URL structure that will assist access to “objects” that exist somewhere (e.g. some people want to model things to be helpfully aligned with ActiveRecord, Jester, and associates).

This doesn’t sound like REST to me. At least not the REST that I imagine in my head. I’m comfortable with imagining my own version with REST because as far as I can tell that’s what everybody does: They take the parts they like and create a world view which makes them comfy, and they argue about it for fun, or at least a diversion.

In a comment on Joe’s posting I said:

At core here I guess my confusion is that I don’t understand the phrase “RESTful JSON specification”. JSON is a content-type, something we might like to see some of our resources represented as as we transfer them about. Can you be more explicit about why the specification should exist? You’ve said to me that most RESTful APIs you’ve seen end up being RPC, but that doesn’t strike me as an answer to the question so much as an observation about a sad (but true) state of affairs.

in his response Joe said:

“Just use HTTP” doesn’t seemed to have worked out so far as the vast majority of JSON based APIs are RPC.

We seem to have missed one another entirely there.

Especially as on the mailing list people seem intent on formalizing a way to do gets and sets on “objects” on the server. Feels like RPC to me.

One of the REST essences that I hadn’t made concrete in my own brain until Dare decided to explain REST to Damien Katz is the idea that by introducing some complexity in the client, services can more effectively scale and be more robust in the face of weirdness. I was already establishing that expectation in services I was making (i.e. this here service, it’s got some resources, if you want to do some things with those resources, well then, how about you GET them, do some stuff with them, and then PUT them back, thank you very much) but hadn’t realized I was doing something sanctioned. I had realized that the choice made things easy to build, test, use, deploy, improve, explore, etc.

What I’m seeing on the restful-json mailing list is people wanting to lessen bytes on the wire and explicitness of client requests at the cost of increased complexity in both the server and the so-called protocol (AtomPub, for example) involved. They hope to alleviate the cost of managing moving the complexity around (or even expanding complexity on all dimensions) by creating a layer of abstraction (a protocol) to contain the complexity.

This leads me to conclude that perhaps the people who are interested in RESTful-JSON are primarily concerned with internally deployed systems where they can be in charge of both ends of a request and handle requirements for supporting the protocol. Managing such a dependency in closed systems is easy to do and warranted where there are special requirements.

However, on the wide open web I can’t see it playing. It may be counterintuitive to many developers, but making additional protocol requirements is a turn off for exploration, learning, experimentation and innovation. Protocol abstractions are certainly useful, but they also have their costs. ws-* is too much like hard work and I’m beginning to feel the same way about AtomPub, let’s keep JSON fun. We want a heterogenous web, a diverse web, a web of learning and exploration with resources at cool URIs doing and/or being cool stuff.

For my own purposes, out on the open web, JSON is a content-type, a possible serialization for some resources that I want to put on the web. The protocol is HTTP.

TiddlyWeb has support for arbitrary storage mechanisms. For the sake of maximum flexibility a storage system does not have to support all the methods in the StorageInterface. Today I made it so when a method is not implemented a StoreMethodNotImplemented exception is raised. This is nice because it means we don’t need to check or create return codes from the store. Exceptions are a happy making nice thing, at least in this context.

However, in the web handlers, that StoreMethodNotImplemented exception needs to be translated into some kind of HTTP response code to inform the calling client that what they wanted, they can’t do, not on this server. I ran into some mental confusions about which one was the best. Anybody know?

Because TiddlyWeb is supposed to present a fairly uniform API for clients the response code should indicate “What you did there, that is sometimes a normal thing, but on this server, this particular implementation, you can’t do that.”

To make this concrete let’s say that the store doesn’t support recipe_delete so we need a reasonable response to:

DELETE /recipes/recipe_name

404 Not Found makes some sense:

The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable. spec

The server can find something matching the Request-URI. In fact it’s running code associated with it right now. Also, this is a sometimes valid TiddlyWeb entry point so do we want to give the impression that it is not? And the resource we’re talking to, the recipe, it does exist, you just can’t delete it given the current constraints of the server. Also what about /search. An empty results set returns a 404. If lack of support for /search also returns a 404 that’s a confusing bit of ambiguity.

405 Method Not Allowed makes some sense too:

The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource. spec

For our recipe example it is very much the case that we don’t support the DELETE method for this particular resource. But for list_tiddler_revisions it makes less sense. If there is no support for listing revisions, a 404 makes more sense: there are no methods at all on the revisions Request-URI. Neither gives the sense that the situation which obtains in this context may not be true in another. It would be nice to have a generic response we can throw here, not something specific for each URI.

501 Not Implemented seems an option:

The server does not support the functionality required to fulfill the request. This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource. spec

The server does not support the functionality required. However it does support the DELETE request method for some resources. Just not this one. And 500 and above is often interpreted to mean that something hit the fan, and that’s not really the case here.

400 Bad Request covers a lot bases:

The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.spec

This is close, except that the “malformed syntax” is specific to this server rather than to the TiddlyWeb class of servers.

For the time being, after some chatting in #tiddlywiki, a 400 is being returned, but I suspect the code will need to be changed to be more specific; more aligned with the specific Request-URI. I suspect 405 when one of the *_{get,put,delete} is not implemented and 404 when one of the list_* or search methods are used. Hmmm. Or maybe just 405 all round.

Input appreciated. Lazyweb?

I have tendency to want to tie together threads that don’t appear to have much to do with one another. A few days ago JP made a posting …musing about leadership… in which he suggested:

Leadership is about taking the risk of managing meaning

Perhaps true in hierarchies, but I prefer to think that a more critical action of leadership is feedback. So, to me, it’s not about accepting risk, it’s about helping to shape meaning, and the process of shaping meaning leads to understandings which lead to goals. Leadership is spread around among all the communicating participants. Leadership is something anyone, in the right environment, can do.

What this can mean is that sometimes the person who gives the most feedback in any situation can exhibit significant power. In conversation with Fred (one of the Osmosoft guys) I suggested that he, by asking the most questions and making the most comments about TiddlyWeb (i.e. providing feedback), was leading the direction of TiddlyWeb.

So asked him what he wanted next. And he wanted fudge cake. So I set about making that possible. Cake from TiddlyWeb.

In Web of TiddlyWebs with TiddlyWebWeb I gave an intro to the StorageInterface system in TiddlyWeb. It makes it relatively easy to provide different persistence engines.

There is a similar mechanism for handling multiple content-types for incoming and outgoing representations of resources. There is a SerializationInterface. Implementors of that interface are responsible for taking the internal object representation of some entity (Recipe, Bag, Tiddler) and turning it into a string that represents a particular type (HTML, JSON, a raw text format, Atom, etc). They also take a string of a particular type and turn it (if possible) into an object representation. The relevant methods are:

recipe_as(self, recipe):
as_recipe(self, recipe, input_string):
bag_as(self, bag):
as_bag(self, bag, input_string):
tiddler_as(self, tiddler):
as_tiddler(self, tiddler, input_string):
list_tiddlers(self, bag):
list_recipes(self, recipes):
list_bags(self, bags):

The default TiddlyWeb installation comes with four serializations: html, json, text and wiki. What serializations are available is controlled by entries in tiddlyweb.config.

Which serialization is used is determined by a simplified form of content negotation, handled by code in tiddlyweb.web.negotiate. For a GET request we look at the Accept header (or an extension on the URL) and eventually use a *_as or list_* method on the serialization to turn an object or objects into the requested format representation. For a PUT or POST request we look at the Content-Type header and eventually use a as_* to turn the provided representation into an object.

Calling code does not directly call recipe_as or similar. Instead it asks the Serializer to do a to_string() or from_string() on a provided object and optional string. A Serialization doesn’t have to provide support for transforming all objects, just stuff it cares about.

All these things make it quite easy to add support for other serializations. The first optional one I made added Atom support as a content oriented web service without Atom is very sad making. There’s a README beyond that link which explains how to add Atom to your own TiddlyWeb installation.

So back to Fred. Fred wants TiddlyWeb to produce cake. In my mind a TiddlyWeb that produces cake is producing cake with Tiddler information in/on/around/with the cake. To keep things simple I decided all we need to do to make TiddlyCake was to write tiddler content onto a picture of a cake and send it out. This means that we only need to write a tiddler_as method and choose a content type for it. cake/x-fudge is what we’ll use.

First we customize the tiddlyweb.config by making our own tiddlywebconfig.py:

config = {
        'extension_types': {
            'cake': 'cake/x-fudge',
            },
        'serializers': {
            'cake/x-fudge': ['cake.cake', 'image/jpeg'],
            },
        }

This says that when we put .cake on the end of a resource think of that as an Accept or Content-Type header being set to cake/x-fudge. When we have that content-type for requests, use the Serializer in the module cake.cake and output the results as image/jpeg.

Now we need to write cake/cake.py. We need a class called Serializer that inherits from SerializationInterface and implements tiddler_as().

In tiddler_as we gather up tiddler.text and do a bit of math to write it over the top of a picture of cake with some really lame line wrapping and then return a string in JPEG form.

Python has the very useful Python Imaging Library that makes most of the image manipulation easy (easy if you are willing to accept ugly).

I searched Creative Commons images on Flickr and found a nice looking cake from Just American Desserts in Spokane.

When you gather all the necessary pieces you can start up a server that will provide TiddlerCake. You give a URL like this:

http://0.0.0.0:8080/recipes/TiddlyWeb/tiddlers/TiddlyWeb.cake

and the browser will show something a little bit like this:

Sadly, I wanted to make this bit of silliness show up on the live TiddlyWeb server at peermore.com but it does not have the necessary graphic library support on it right now, and updating it is a task I don’t want to do today. So images of images will have to do for now. You can, of course, try it out for yourself on your own TiddlyWeb server.

In the previous posting I said creating a new storage mechanism for TiddlyWeb was a simple matter of creating a new module that supported an interface. It occurs to me that I can kill a few birds by writing about the process of creating such a module. One of the birds is explaining some of the facets of TiddlyWeb’s architecture. Another is explaining why, despite Damien Katz getting all up in people’s business and calling bullshit, REST is teh awesome and I want it.

What we’ll do here, just to be a bit perverse, is a create a store for TiddlyWeb that uses another TiddlyWeb as the place where resources are stored. In the process we should see why TiddlyWeb’s API being “RESTful” is useful and also expose a few bugs that need to be fixed to have better behavior.

I started this a while ago and called it tiddlywebweb, so we’ll carry on with that.

In TiddlyWeb a store is a module with a known name, containing a class with the name Store that is a subclass of StorageInterface. The interface is defined in tiddlyweb/stores/init.py. The default store for TiddlyWeb is called text and is in tiddlyweb/stores/text.py. text uses the filesystem and easy to read (by humans) text files for storing data. It’s not terribly speedy, but it is easy to grok.

In usual use a particular implementation of a StorageInterface is not directly instantiated. Instead the calling code creates a Store object of the class defined in tiddlyweb/store.py, choosing the type of store by name:

from tiddlyweb.store import Store
store = Store('text')

Methods are then called on the store object to access data:

from tiddlyweb.tiddler import Tiddler
tiddler = Tiddler('MyTiddler', bag='foo')
store.get(tiddler)
print tiddler.text

(The unique id of a tiddler is it’s title and the name of the bag in which it lives, so we need both in order to be able to retrieve a tiddler from the store.)

When a Store object is being created the system first looks in the tiddlyweb.stores package for a module with the given name. If it finds that, it is imported and the Store class within is used. If that import does not happen, then the system tries to import the name directly (searching sys.path). Our new tiddlywebweb code is located in tiddlywebweb/tiddlywebstore.py so we could load it like this:

from tiddlyweb.store import Store
store = Store('tiddlywebweb.tiddlywebstore')

Except that in most cases we would not. Each TiddlyWeb instance is assumed to have just one store (for now) so the storage system can be defined in the system configuration. The default system configuration lives in tiddlyweb/config.py and can be overridden by a Python file called tiddlywebconfig.py kept in the working directory of the TiddlyWeb server. The config dict in tiddlywebconfig.py is merged over the top of the config dict in tiddlyweb/config.py.

The server_store key in that dict holds the name of the store that is to be used, and a dict of any configuration information that it needs (path information, username and password handling, etc.) For the default text store it looks like this:

'server_store': ['text', {'store_root': 'store'}],

text is the name of the store. store_root is the path to the directory where data is to be stored.

To use tiddlywebweb as the store for our server A, we know we need the the base URL of the other TiddlyWeb server (server B) so to start out our config entry will look a bit like this (assuming the other server is running on localhost:8000):

config = {
    'server_store': ['tiddlywebweb.tiddlywebstore',
        {'server_base': 'http://localhost:8000'} ],
}

That will go in the tiddlywebconfig.py of server A (more detail on that in a bit). Server B needs no particular modification (for the purposes of this exercise we’re not going to worry about access control, maybe next time), it just needs to be running.

Okay, so now we know how to use a different store, but what does a store do? As said before a Store needs to implement (some of) the StorageInterface. This is a collection of methods that get, put, delete and list a variety of entities used by TiddlyWeb. Here’s the complete list:

recipe_get(self, recipe):
recipe_put(self, recipe):
bag_get(self, bag):
bag_put(self, recipe):
tiddler_delete(self, tiddler):
tiddler_get(self, tiddler):
tiddler_put(self, tiddler):
user_get(self, user):
user_put(self, user):
list_recipes(self):
list_bags(self):
list_tiddler_revisions(self, tiddler):
tiddler_written(self, tiddler):
search(self, search_query):

Some caveats:

• Support for deleting recipes and bags is planned but not yet supported.
• Support for users is optional (for example the GoogleAppEngine version of TiddlyWeb uses Google Users, so doesn’t need to store them).
• Support for listing tiddler revisions is optional. If a store doesn’t support revisions, then a tiddler is always revision one.
• tiddler_written is by default a pass. It is called by tiddler_put. If overriden it can be used to update an index if one is used by search.

Alright, let’s think about this a minute. If our server A is using server B for storage, then when a user asks server A “GET /recipes/foo” what needs to happen is that server A asks server B “GET /recipes/foo”. And when a user tells server A “PUT /recipes/foo”, server A needs to tell server B “PUT /recipes/foo”. Similar things for a bag or a tiddler or collections thereof. So what we need to do is proxy web requests to server A through to server B, right?

Wrong. In order for TiddlyWeb to be able to support multiple storage types it has to have fairly disciplined separation of concerns amongst all the bits of code. When a user asks server A for a recipe the request is processed something like this:

1. The selector map dispatches to the right piece of code (tiddlyweb.web.recipe:get) based on urls.map.
2. That piece of code uses the name provided by the URL to instantiate an empty Recipe object, recipe.
3. The configured store is then asked store.get(recipe) to populate the recipe.
4. Based on content negotiation, that recipe is then serialized to a particular string and sent out in the response.

Step 2 is required in order for step 3 to be most flexible. And Step 3 has no clue what the configured store is, just that it calls store.get().

So proxying is right out. Which means our tiddlywebweb store needs to be an HTTP client that constructs proper URLs and makes requests to a remote server. We’ll only pass content that is application/json to be simple (and because it is the most robust of the default TiddlyWeb serializations). And to keep things simple we’ll just get the bare bones in. Later we’ll add the fancy.

We know about recipe_get. Let’s describe what it needs to do:

• Accept a recipe.
• Use its name to construct a URL.
• Make a GET request to that URL with an Accept header of application/json.
• Transform a successful response from JSON into a proper Recipe object.
• Get the object back up the stack.

recipe_put does something quite similar:

• Accept a recipe.
• Transform the recipe into JSON structure.
• Use the name of the recipe to construct a URL.
• Make a PUT request to that URL with a Content-Type header of application/json and a body of the JSON.
• Promulgate success up the stack.

The rest of the methods are in some very fundamental ways basically the same (this is why mnot is basically right for many cases) so we’ll not bother to go into too much detail.

In tiddlywebweb.tiddlywebstore recipe_get looks like this:

def recipe_get(self, recipe):
    url = self.recipe_url % urllib.quote(recipe.name)
    self.doit(url, recipe, self._any_get, NoRecipeError)

Wait, what? That gets the first step (url construction) but what’s going on with the rest of it. Well, like I said, all the methods are basically the same so we can abstract that out. doit() looks like this:

def doit(self, url, object, method, exception):
    try:
        method(url, object)
    except TiddlyWebWebError, e:
        raise exception, e

Which says “try to do method method, if it works, we’re golden, otherwise raised the exception named by exception”. doit() wraps either _any_get() or _any_put(). Here’s _any_get():

def _any_get(self, url, target_object):
    response, content = self._request('GET', url)
    if self._is_success(response):
        self.serializer.object = target_object
        self.serializer.from_string(content)
    else:
        raise TiddlyWebWebError, '%s: %s' % (response['status'], content)

1. Make the request.
2. If it was good, transform the JSON response to the object form.
3. If it was bad, throw an error.

So I’ve done that. You can see the store module and some extra bits for making it go.

This covers the basics:

• You can lists tiddlers, bags, recipes.
• GET representations of each.
• PUT some tiddlers.

So the basic concept of having the content hosted on a remote server is doable. But there are problems:

• Listing a bag or recipe with a lot of tiddlers in it results in a lot of requests to server B.
• We’ve got not delete yet.
• There’s no auth handling.
• It’s kind of slow.

Some other time, sooner than later if there is interest, I’ll improve the code to show how some solid use of good HTTP principles will make the code better and make the system operate more efficiently.

Buried in the “Moonbeams from Ponies of the Future” section of the TODO file on TiddlyWeb is a link to Mark Nottingham’s post on Moving Beyond Methods in REST with the suggestion that perhaps TiddlyWeb should align itself with the ideas Mark’s selling.

I haven’t done this yet, and I’m pretty sure I no longer want to. Mark’s ideas are smooth and clean and lovely but do a thing I just can’t get behind: persistence is handled in the class that represents the resource. Here’s his pseudocode:

 # tell the Resource what implements GET, PUT and DELETE
@store_type("mysql")
 # tell who / when access is allowed, per-method and finer-grained
@acl("choose your ACL poision")
class Person (Resource):
    store_format = PersonML
    def POST(self, representation):
        # operate on the store...
        return representation
    def PUT_effect(self, representation):
        # called IFF the presented representation is storable, 
        # but before it is available;
        # raising an exception will back it out
        return status_representation

The idea is that it is possible to abstract away some of the GET, DELETE and PUT handling (read the posting for details) and concentrate on the specific processing that needs to happen with POST and some PUTs.

I’m not sure why but I don’t like including storage handling methods (or really many methods at all) on my resource classes. In TiddlyWeb, the Bag, Recipe and Tiddler classes have attributes and a small number of methods for manipulating those attributes. If you want to GET a resource, or store a resource there’s controller code for that. This keeps the number of modules that are collaborators for the resource modules very low and contains conceptual behaviors in particular modules (this module is a resource, this module is web handler, this module performs persistence, this module transforms a resource into a particular representation).

I’ve found this works well for my purposes. It makes the modules easy to test. It also makes it easy to build in extensibility. In TiddlyWeb adding a different store or serialization (a tool for transforming a resource to or from a particular representation) is just a matter of adding a module that supports an interface. The resources remain stable.

So, yeah, there’s lots of ways to do things, depending on what’s going on.

Yesterday I was in London for an Osmosoft TiddlyWiki hackathon. I didn’t get much hacking done but had several very valuable conversations and met some good folk. Running throughout these conversations were threads of REST, HTTP, web and software architecture. Common to these threads was a lack of consensus about what counts as “good” in any of those contexts and what concepts shared in those contexts could be be considered good in a more universal sense.

So it was with some pleasure that I woke up this morning to see that Simon Willison had linked to Damien Katz (of CouchDB fame) saying REST, I just don’t get it. The comments are instructive in two ways:

• They provide a lot of good and varied input on why thinking about software systems in a REST kind of way is useful.
• They show that the rest of the world, not just some guys at a TiddlyWiki hackathon, are also lacking in consensus and shared understanding when it comes to web architecture.

In the end it is all really quite a subjective matter of opinion and in such cases, a diversity of opinion is a great way to learn, explore, find the edge cases; to gain new perspectives. So with opinion being the operative word, here’s where I think recent commentary gets it right and wrong.

While talking with Martin and Fred yesterday about the TiddlyWeb API Martin made the statement that it is generally accepted wisdom that minimizing the surface area of an API is good practice because it allows the most stability at the surface while allowing internal change. I agree with this.

chu’s comment:

I think only allowing up to 4 methods is a key benefit. Having to express an app through nouns rather than verbs leads to a much cleaner, more robust and encapsulated design.

says sort of the same thing, from a different angle. It also introduces my favorite bit about systems that try to have a REST style: nouns; the parts of the system as resources on which there are a very small number of operations that can be performed. Building blocks, lego, in a system with a simple grammar but complex expressivity.

Damien says:

I guess what I mean to say is just because SOAP is a disaster, doesn’t somehow make REST the answer. Simpler is better, and REST is generally simpler than SOAP. But there is nothing wrong with a plain old POST as an RPC call. If its easy to make all your calls conform to the RESTful verb architecture, then that’s good, I guess. But if not, then just use a POST as an RPC call, keep it as simple as possible and be done with it. And don’t spend another minute worrying about being RESTful or not.

I think there’s right and wrong in here and one of the comments nails the problem. Avdi says (my emphasis):

One of the big (but non-immediate) wins is explorability. If you don’t care about emergent properties, if you don’t care about what other people might build on it, then sure, use nothing but POST; use SOAP; use CORBA, for that matter. The magic of REST is that no WSDL or IDL can replicate the experience of someone clicking around a RESTful API in their browser like a kid opening up a box of Tinker Toys and saying “yeah, I get how this fits together! I could build on this!”.

RESTful services are easier to build on, especially when the things being built are things the providers of the service couldn’t or didn’t predict. I use the term service intentionally. If you’ve got yourself an application, where you want to control the user experience, then whether you use a REST style of architecture or an “RPC call” might not matter.

Unless, that is, you are concerned about scalability and availing yourself of the caching capabilities of the web. The comment from Colin Percival hits the REST association with caches.

Not mentioned in Damien’s discussion, but present, at least tangentially in yesterday’s hackathon is what I think is perhaps the most interesting part of having a web oriented frame of mind when thinking about software system design: pipelines, chainability, stackableness, composability. The Lego Factor. For people who grew up on a unix-like command line this is the feature that separates the good from the bad.

A web service that is built upon RESTful principles is easy to pipeline because it has a small and well known API (some or all of GET, PUT, POST, DELETE) that makes it possible to GET a representation from one service, transform it in some fashion and send it on to another service.

The WSGI specification for Python helps extend this model down into code with simple middleware that transforms either the web request or response. This is awesome because it helps encourage code modules which are encapsulted and decoupled while not requiring a adherence to a bunch of framework. These modules are just like web services but running in the same process space.

I could go on endlessly here. To sum: RESTful principles are for the most part an expression, in a distributed space, of a lot of time worn wisdom about how to make good software systems that are reusable:

• minimize complexity
• maximize discoverability
• minimize exposure of special knowledge and methods
• maximize opportunities for reuse and chaining

Damien’s right that some of the time the principles aren’t going to be right for a particular use, and there’s no point getting in a twist about them. But it is obviously true, that on the web, designing and building with respect to what works on the web is good. Tim Bray gets the nod for that. His comment is a good ending:

Well, a REST architecture serves as the basis for the most biggest and most successful information system the world has ever seen, that effortlessly deals with heterogeneous hardware and software, and provides a high-quality user experience while scaling towards a billion users. So think of it, as nothing else, as a good benchmark that you have to be better than.

What about REST makes you want it?

My flatmate sees me walking round with a tshirt that says “Wiki Way” on it, he knows I used to work for a company called Socialtext and that they make wikis. He knows I work with a thing called TiddlyWiki and once helped make a thing called Purplewiki. So it is only natural that when he finds an article in the New York Times about Wikipedia folk meeting up in Egypt for Wikimania that he set it aside for me.

What isn’t natural is that this makes me grumpy. Having my vocation or what have you associated with Wikipedia gives me the scowly face. This is really unfortunate because Wikipedia is obviously a wondrous step forward, a great resource for people all around the world, a truly great accomplishment by huge groups of people. I use it nearly every day. Yet it makes me a grrr a bit. This is obviously my problem, not Wikipedia’s problem.

Since writing is a good way to get some thinking going, here’s something.

I guess the basic reason I have issues with wikipedia is because it has absconded with the word “wiki” and taken it in a direction that is lovely and exciting but not the one I wanted. Wah, they took my toys! The wikipedia direction involves an enormous number of readers and a still very large number of writers/editors but a ratio of readers to writers that is very large. An effort to aggregate what is already known. An information resource.

Another style of thing that also has the name wiki has a much closer ratio of readers and writers, a more inclusive sense of participation and culture of engagement (say that with a French accent). They get this, in part, by being small and associated with a community of fairly specific practice. A learning orientation. An effort to create knowledge. An action resource.

In those communities the facilities of the wiki software that encourage a synthetic and shared learning process are highlighted and relevant: LinkAsYouThink (I really like that Eugene says, “What makes this feature beautiful is that you can link to a page that does not already exist”), recent changes, back links, and of course easy access to editing.

Then again, it isn’t the software that matters: it is the community. At the scale of Wikipedia you simply can’t have the sense of homegrown localness that you get with little wikis. The sense of earnest participation.

And, at scale, you get the apparatiks. Them what’s got the power tell them what don’t how they should be getting on with things. Information and power collects to a center.

Which leads to the final winge: If the things about wikipedia that make it a wiki (LinkAsYouThink etc) are de-emphasized by design or necessity, what’s left? What’s left is a big collection of editable hypertext. Which means it is really a centralized, editable library located as an edifice in the rest of the web. As a library it is a huge step forward, but the ideal vision is that the entire web is the editable library.

Wikipedia gets things more right than loads of other edifices and it deserves major props for that. Compare it to other gardens, with much higher walls. Like Facebook. Facebook takes the web, puts a nice invitation on it and gets enough of your friends involved that you can’t help but participate. But is there anything happening there that couldn’t be done with diverse and open services on the web that can be independently aggregated and federated?

Before the web it was often the case that in order to get a large change done it required something like a national government to amass the resources. Now that most of the required resources are information, what’s left to overcome before we no longer need edifices?

A while back I got into an email conversation with some colleagues that trended, somewhat unintentionally, in the direction of innovation. The original topic, very generally, was what obligation a product producing group has to respond and react to user/customer/community input and how best to filter, understand and use that input.

Someone suggested that we needed to add a lot of salt to input from users and use it as an influence (effectively, seasoning) rather than a guide. What should guide is an independent view based on logic and reason. I agreed with the first (seasoning) but demurred on the second (logic) and went off on a tangent (as I do) about innovation and real change. That conversation informs what follows.

Many people who are working on product development, especially products resulting from open source software development, characterize the stuff they are making in terms of how it is different from other stuff of the same sort (better, faster, stronger), or in some cases so different that it is in a class of stuff all its own. Many people are striving to enhance the level of productivity people experience when performing some task.

It’s clear that input from users and user testing is vital in understanding a task or process and incrementally improving the pleasure and performance of doing the task. It is less clear, however, that such things are useful if the goal is real and significant change in how a task is performed. If the goal innovative change.

Evidence in the user experience and design fields suggests that if innovative designs are the goal none of the following are particularly successful as the first step in or highest priority part of the design process:

• user testing
• user surveys
• heuristic walkthroughs
• focus groups
• analytic logic and reason

What has the most impact is informed strokes of genius by people who listen well (with more than just their ears), who are paying attention to the large context surrounding an activity. Strokes of genius have very little to do with logic or deductive reason. Certainly inference is involved, but the inferential leaps are across chasms that to the casual observer are either too large to leap or not even in the same dimension.

This all makes some form of sense if we think about what innovation is. Does “a new way of doing things” work as a definition? If it does then a truly new way of doing things is not going to be identifiably derivative of an existing way of doing things, so the new way can’t really be deduced from existing principles. And if it is truly new, then we can’t test it, survey it or walkthrough it to see if it is any good, because it doesn’t exist yet, except as an idea that someone just had.

People do like to toss the term “innovation” around an awful lot. It’s easy to claim that using fisher price colors and buttons on a to do list web app is innovative but it is much harder for the claim to be true. Page furniture do not a new process make.

Real innovation, though, is presumably a good thing. It would be nice to foster it. I found a more formal sounding (and rather curiously situated) definition of innovation at the Australian Research Council:

Innovation is the process that translates knowledge into economic growth and social well-being.

Translates knowledge. So knowledge is a prerequisite. In traditional information science terms, knowledge is solely in an individual’s head. It is information which has been learned; placed into the context of an individual’s experience and made useful for dealing with and understanding future experiences. So information and learning is a prerequisite.

The open learning opportunities offered by the web have proven to be quite the crucible for innovation.

(I think I’ve found a theme for this blog. All I seem to be writing about is open access to learnable bits of info.)

For an organization or community to foster innovation it must enable open access to information by its constituency and it must foster the creation of new information that contributes to a synthetic learning process. The highly contextualized active learning increases the opportunities for some bright bulb to have that eureka moment which brings disparate islands together to form an innovation.

This wouldn’t be necessary if I were less lazy and ran my own server but: I’ve switched things about so the content here is being fed by feedburner so I can have some insight into the adverbs of the feed. If you’re already subscribed to the tumblr feed, that will continue to work, but I won’t know about you and that will crush my fragile ego. If you don’t want to crush my fragile ego, please update to the new feed.

When I left Socialtext I declared I was starting work on my own blog engine, called simper, but it slipped away subject to the ennui described in Work Satisfaction. Requests that I publish now led to this tumblelog.