Peer Pressure

Matt Liggett and I were chatting about finding satisfying work, trying to suss out why we do what we do and what we might like to do under ideal conditions and the like. Matt asked me what I would do if I got a MacArthur Fellowship. Not that I’m likely to get one, but it’s a good question to ask to feel out what the perfect vocation would be, given the chance.

What follows is a rewrite of how I responded to him.

I’ve thought about my work habits and where I experience satisfaction in work and I’ve been forced to make the conclusion that at least in the technology world I am not self motivated. If I’m presented with a person who has a problem that I find interesting I can go full bore and really get something done (such as “inventing” transclusion in Purplewiki).

A consistent pattern has emerged over the years. In those situations where I take off a ball is rolled by someone else that starts an information feed into which I can inject myself and experience a high bandwidth, low latency feedback. If the feedback loop is not there I will struggle to make it exist by techniques that are sometimes friendly and sometimes less so.

This TiddlyWeb work I’ve been doing is quality stuff, but it is based very directly on request from someone else who expressed a fairly specific problem and provided a dense context.

Projects of my own, such as simper and closet, have been interesting as learning exercises but I don’t get all that jazzed about them as they are mostly context free. Sometimes this makes me think I’m in the wrong field.

Maybe I’m in the wrong field if “right” is defined as perfect. I’m working in the field that pays the bills and satisfies a critical and driving addiction to fast gratification learning.

If I recieved a MacArthur Fellowship what I might do is find someone I thought was smart, interesting, good at communication and say, “here’s half, what’s a problem for you?”

I can conclude a few things from this with regard to future work environments. An ideal job would include:

• An organization that identifies itself as improving some situation.
• A problem seeking and solving approach to achieving organizational goals.
• A highly interacting team. One that actively engages in robust debate and critique.
• A fat communication pipe to the users or customers, whoever they may be, so that feedback is real, diverse, and unfiltered.
• A willingness to experiment and takes risks when attempting to solve problems, including willingness to send individuals or groups off on wild goose chases.
• Constant learning.

What else?

One of the intriguing aspects of working on TiddlyWeb is the sort of behaviors (in people) its architecture encourages. The architecture is setup so that you can extend or modify TiddlyWeb’s behavior by writing some more code. In some cases this means a straight up new module that implements an interface, in others it means subclassing something else that already implements the interface.

Some folk respond to this kind of architecture by wanting to make some very general classes that will cover most bases, are abstracted out the wazoo, and birth a deep hierarchy of subclasses. In the name of DRYness and the like.

Other folk just write some code and see how things turn out. In a language like Python with an architecture like TiddlyWeb’s this is the cheapest and most effective way to go. Cheap because code is easy to write and test, effective because the distance to new understanding and use is short in time and space.

Just writing some code is how TiddlyWeb got support for Atom feeds and was made to work in google app engine (remarkably easy by the way).

And if in the end some opportunities for abstraction are found in the code, there’s a good time for some refactoring: a better abstraction usually happens from extraction rather than prediction.

Fred from Osmosoft is using TiddlyWeb to help make a plugin library for TiddlyWiki. Because TiddyWeb was designed in the world of abstractions and the plugin library is in the real and concrete world, Fred is often wanting additional features in TiddlyWeb or modifications to existing features.

TiddlyWeb is written to be easy to extend (there are interfaces for Storage, Serialization, auth Challengers, and credentials Extractors) but I frequently find myself encouraging Fred to implement the feature in some other application or service that uses TiddlyWeb. From a recent email:

I’ve been trying to work out how best to address these comments. There’s the half which is just trying to get you able to do what you want to be able to, but there’s also the other half which is trying to get you thinking in terms of good development behaviors and attitudes for the future. I tend to be walking back and forth across that dividing line. So take the following with a grain of salt: I’ve already said I think the idea of making an extension to the HTML serialization is an okay idea; I’m telling you this stuff in a more general way.

There’s inefficiency for the one use case you have in mind and then there’s efficiency in terms of making a system that is most flexible for lots of uses.

Consider an RDBMS. When you use it, you have to write SQL statements and framework code that allows you to access that code. If an RDBMS were efficient by the definition you offer it would include ways to make it do exactly what you want. But it doesn’t do that. What it does is make a promise to you that if you give it data in a particular form it will give you that data back later when you ask for it again. This means that you can use the RDBMS for lots and lots of different things. Importantly it can be used, efficiently, for things which you haven’t thought of yet.

The promise made by TiddlyWeb is that if you put a Tiddler in it, you can get it back out later. The canonical form of that Tiddler, as far as TiddlyWeb is concerned, is the JSON representation. That is the most simple and flexible representation available from the system. Any of the other representations make assumptions about the destination environment, assumptions that constrain the possibility of how things are presented. For example, the HTML representation of a Tiddler makes links between Tiddlers. Those links are structured to work when the tidderl HTML Is presented in the URI space of the TiddlyWeb server. That’s kind of a pain if you want to use that HTML elsewhere. The text representation has to be parsed to be used as a data structure.

So from the TiddlyWeb’s side of the picture, if its measure of efficiency is “tiddlers can be used in lots of different ways, without using a bunch of CPU on me” then success is when the JSON rep is pulled and used in lots of exciting ways.

Or look at it from the viewpoint of the application that is created to use this JSON: If it gets already processed data from the TiddlyWeb server (HTML) its options for doing things with it are quite limited. Mostly it can just style it and show it. Any processing it does do will be very dependent on the apparent “structure” of the HTML (square quote to indicate that the structure is not real or useful). If, however, the application is willing to accept some data that is fairly rigorous (like JSON) the options grow for what can be done with the data and, most importantly, it becomes the case that the application can more easily work with data from sources other than TiddlyWeb.

It’s odd because in practical terms the right thing to do is make it work, then make it right, but very often the definition of “work” and “right” are very much dependent on point of view, experience, and expectations.

Brad Neuberg asks:

Do you toss the term Open Web around? Ever wonder what the heck it means? Me too.

One sentence version: The Open Web is a networked system of people and processes that lets those people and processes create, manage, own, give and learn from socially accessible information without requiring allegiance to one or more central controls.

That’s much too abstract. I think the crux of the biscuit is twofold:

• What’s on the Open Web are information artifacts.
• What makes it open is that the information is accessible in a social fashion: people can and do “play” with it.

Brad’s post adds three questions for bonus points. Here are my efforts:

If Adobe and Microsoft were to open Flex/Flash and Silverlight, that would not be the open web (in and of itself) because those systems have a tendency to encapsulate information artifacts, limiting diverse social access. On the open web, in the ideal, the information is just out there without requiring a specific set of rules to be followed in order to make some use of it. I can use wget, or a browser, or a desktop app or whatever to get at a picture on flickr because that picture is itself loose on the open web with its own URI.

So an open web needs things like URIs, and granularity on those URIs is good.

Developers should care about the Open Web because developers make tools to process (i.e. play with) information. The more information there is to process the more interesting development work that can be done. The more that information is accessible in a straightforward fashion, the more creative the things developed can be.

Users should care about an Open Web because the Open Web is a fabulous crucible of learning, where learning includes both formal learning and the growth that comes from play, interacting with people, and that fun thing that happens when you go to Google to find something or Wikipedia to read about something and take a left turn into a universe of loose goosey browsey discovery.

The current web is and can be the Open Web, with care.

One of the reasons I’m keen on using lots of little web servers is because it gets one in a mindset of how the web could work: small simple systems interacting complexly rather than complicated systems interacting simply.

Screw communities where membership requires obeisance at the feet of a centralized authority. So screw facebook. And screw twitter too. And screw nation states. Mostly screw nation states.

When we talk about cloud computing the cloud is important and interesting but not as important and interesting as the nascent my cloud. My cloud is the growing collection of my information artifacts (by, of, and/or about me) that are on the web in some form or other. If the names of those artifacts begin with someone.else.com it’s not really my cloud, is it?

I’ve started thinking that except in special circumstances travel for work, when you’re a so-called knowledge worker is sort of immoral (from a use of resources standpoint). This is having significant impact on my decision making related to future work, conference attendance and such things.

I need to think about this more as I could very well be wrong. Obviously collaboration in person has big wins over collaboration in other situations but it is also the case that collaboration in person that is time boxed is often more effective than open ended time spent in person.

This suggests, to me, that commuting is regressive and scheduled co-working events are progressive. That’s probably obvious to people who have already bought into the co-working ideas.

This came up because I’m trying to decide if I will go to Wikisym. Early registration deadline is soon.

When I implemented an OpenID challenger for TiddlyWeb, I found the process rather frustrating. Most of the examples and libraries I could find make a lot of a assumptions about the features of the codebase that wants to use OpenID and say things akin to “insert the call to library method foo at stage X in your app, this will set Y in your session store and provide a URL to send as a redirect”.

OpenID is a system on the web. To show me how it works, show me the HTTP. The best resource I found was the introduction on the OpenID wiki.

TiddlyWeb has an intentional simple challenge system. If there’s a permissions exception that’s eventually trapped and the request gets a 401 on a particular URL which can present options for resolving a challenge. Any challenge is just a bit of WSGI for which the end result is the injection of some credentials in the response headers which later requests can include. Those later requests pass through a credentials extractor. Valid credentials provide the system with a valid user used for authorization. No user or wrong user causes a permissions exception and the cycle starts again. TiddlyWeb doesn’t have a session store. TiddlyWeb doesn’t require a user store.

So what I wanted to create was the bare minimum:

1. Present a challenge form to a user.
2. Get an open id from them.
3. Follow that open id to get permission to “use” the identity.
4. Put the identity into the response for later extraction.

I wanted this in a short amount of code that is as decoupled from the rest of TiddlyWeb system as possible. The result is a piece of WSGI middleware that does the trick. That code is straightforward to me, but I reckon for anyone else it is not.

The flow of HTTP goes like this:

1. On an intial GET of the challenger, send a form with two input fields. One asks for the OpenID, the other, hidden, holds the path we will eventually redirect to upon success.
2. The form sends a POST back to the challenger.
3. If the openid field of the form is of the right form (after canonicalization to a proper URL), make an HTTP request to fetch that page and parse it for a required link with rel of openid.server. Also parse for an optional link with rel of openid.delegate.
4. Construct a URL to the href in the openid.server link with a messy query string. See _find_speak_to_server() in the code. Send a 303 to the client with a Location of that URL.
5. The client will go off to that URL where the server will do whatever it wants to validate the user and ask if the user wants your server to have access to your identity.
6. The OpenID server will redirect the client back to your server, to the value of the openid.return_to paremeter in the query string generated above. The redirect will include a long and complicated query string on a GET request, including a cryptographic signature.
7. The challenger sees this GET request and looks for an openid.mode parameter. If it is id_res it is time to handle the server response. _handle_server_response in the code sample.
8. The challenger is now required to validate the information it has received from the server by sending it to the server again. A swath of parameters are built up, including the signature, and they are sent in a POST back to the OpenID server. Note this POST is done by the server, not by the client.
9. If the OpenID server responds with is_valid:true, we’re golden, the user is who they say they are (assuming we trust the OpenID server). Otherwise go back to initial form with an error.
10. If the OpenID has been accepted TiddlyWeb sets a cookie, and redirects the user back to their original location (or / if unspecified).

Bollocks. This isn’t any more useful than any of the other explanations. And now I’m sleepy. Look at the code? I’ll try again soon.

This is a posting to be used an about page.

Stuff to look at about looking at stuff. What?

What we do on the web is look at stuff and when we look at it we think some stuff, and sometimes we make some more stuff. A synthetic learning process. The web is great for this. It’s fast and relatively easy.

Why Peer Pressure?

Peer pressure in the normal sense of the phrase is what got me to create this tumblelog. “You should have a blog” is something I’ve been hearing a lot since I stopped the last two.

Peering is what we do when we look at stuff. Creating peers is what we do when we establish relationship with other people based on looking at their stuff, learning about them, and forming links.

Peermore is what my company is called. A systems design and development consultancy focused on social information access, use and creation. The name comes from wanting to enable people and information both to look more, see more and connect more. I’m always looking for interesting work.

Who are you?

I’m Chris Dent. I’m currently living in Oxford, England. Before that I was in Seattle for a while. I like to think about the things we use to help us think. Information things. Which is pretty much everything. I’ve started calling myself and Information Theologist.

I write the stuff on here in an intentionally informal style often making no sense. If I stopped to try and make things clean, clear and formal I’d decide not to write. If you find what you think are some egregious errors of style, structure, grammar spelling or thought, I’d love to hear about it so I can clean things up.

I’ve been working on this thing called TiddlyWeb. It’s elevator pitch (from Jeremy Ruston, author of TiddlyWiki) goes like this:

TiddlyWeb is a practical, readable, reference implementation of a rigorously designed headless wiki server that may use TiddlyWiki as a user interface. The combination of TiddlyWeb and TiddlyWiki allows a complete separation of server and UI concerns.

One of the important things to keep clear in working on this, and perhaps more importantly when talking about this with others is the difference between a web-application and a web-service.

We can come up with many different definitions for these terms, but for the sake of this discussion let’s try:

In a web-application the end user interface is constructed by and presented with the same logical web server system. That is, the web-application controls the UI.

In a web-service the end user interface is created outside the web-services’s logical web server. This could be done by a web-application (logically elsewhere), a desktop application or perhaps some javascript in a browser building the UI from one or more services.

TiddlyWeb is being built to act as a web-service for TiddlyWiki. TiddlyWiki gains its awesomneness by being a UI that is customizable on the edge of the network (or even without the network). UI customization in the hands of individuals.

TiddlyWeb extends the data of TiddlyWiki out to a collaborative opportunity for multiple people (by getting content onto the internet) while leaving UI on the edge.

(This reminds a lot of Fluency. There’s also a lot to say here about REST and decoupling of concerns. Some other time.)

The implication of all this is that if TiddlyWeb wants to excel as a web service it should eschew, as possible, those features which require a complex UI on the server; such things cloud the server side system, limiting its applicability to unexpected or unplanned situations. I have no proof of this assertion other than experience and intuition. Take my word for it?

A well designed web-service supports a diversity of UIs rather than, as applications do, enforcing one or more than one UIs. This is especially true of REST-based services. These only say “here is some stuff, have at!”

Note: The term end user interface is important here because of course any system as an interface of some kind. TiddlyWeb’s REST API has an interface, has users, and needs to be usable. What’s different is the users of the API are a step in a chain between the resources and their final presentation for some purposes. That final presentation is the end user interface and is outside the circle of control of the web-service.

brier:

cdent:

Brier demanded I start a tumblr thingie.

“Now I have a place to put my hand.”

And, incidentally, this is a comment, you weiner.

And this and Disqus both require that the person doing the comment establish an identity association with some service: it seems you gotta sign up. That’s adequate but not great.

Adrian asked what’s the real difference between PUT and POST. I responded with:

The basic idea with PUT is that if you put the same thing to the same URI a second or more time you’ll (mostly) continue to get the same thing back. The same is not true with POST. This is the idempotency thing, although its not actually that strictly followed.

In practice these concepts are diddled to mean that you PUT when you know the name of something and you are creating it or editing it by PUTting to a URI which has the name in the URI.

You POST when you are adding something to a collection and may not know the name or are performing an injection of information that may result in the creation (or updating) of a resource with a name that is different from what you are POSTing to.

So for example you would PUT a wiki page if you know the name:

 req:
 PUT /wiki/pages/foo_bar

 res:
 HTTP 204 OK
 Location: http://example.com/wiki/pages/foo_bar

But you might POST a blog entry if the title gets translated:

 req:
 POST /blog/entries

 <body of entry, including a title filed: Mom Has Herpes>
 res:
 HTTP 204 OK
 Location: http://example.com/blog/entries/2008/07/15/mom_has_herpes.html

In my personal efforts at REST a POST is sent to a URI that results in a different URI being the location I want to look at for the new or changed thing. A PUT is sent to a URI that results in the URI being the same place I PUT to. This usually ends up meaning I POST to collections and I PUT to entities.

Make sense?

See also the rfc. Where the main difference is that you POST to something that handles a resource (in the request) and you PUT a resource to itself.

Brier demanded I start a tumblr thingie.