Peer Pressure

TiddlyWeb Dev/Deploy Workshop

Yesterday I gathered with the Osmosoft gang to discuss developing and deploying applications that are built on top of TiddlyWeb. Since I mostly develop TiddlyWeb itself and not things built upon it, I tend to be fairly distanced from the issues so getting together and having a chat and messing about with some code was a good thing.

Mike has written up his summary of the things to do next. I took some notes as well, which I’ll attempt to summarize here.

To get a sense of the issues we went around the room and people reported on how they’ve been managing development and deployment and what’s working well and what’s not. There were some but not many common threads.

One common issue is that most of the people have a tendency to do a code a bit, reload a web page cycle of development and test. When using twanager server this does not work well as the server needs to be restarted with each code change. Restarting CherryPy’s wsgiserver is less easy than some desire because it can be some time before the main thread will come round to hearing an interrupt signal. The initial solution to this problem, the reloader plugin, helps but gets a bit weird when syntax errors exist in the watched Python files (which happens a lot if you have a esc:w trigger finger): The entire server will crash out and you have to start the process again by hand. This mostly makes sense from a logical standpoint but is not that helpful.

One proposed solution to this problem is to host TiddlyWeb under a simple CGI setting. This will reload and recompile the code with every request. This can be slow for production settings but plenty fast for development. The TiddlyWeb source package includes an index.cgi script for this. Ben Gillies has written some docs on how to use it.

Of course to use the CGI option you have to have some kind of web server around. Many systems come with Apache already on them or easy to install. If you prefer not to run Apache (like me) another option is Spawning a fast multi-process, multi-threaded, Python web server. There’s a TiddlyWeb “factory” for Spawning called spawner.

Paul wisely pointed out that much of this reload fiddling about can be avoided by writing and testing Python modules that perform most of the activity for you application, minus the web handling, and then using these modules in small layers of Python code that integrate with TiddlyWeb.

There was a fair amount of resistance in the group to formal and rigorous testing, which I think is extremely unfortunate, but Simon rightly points out that lots of people are like that, and TiddlyWeb is going to limit its audience a great deal if it requires developers to be totally test driven.

Another point of difficulty and contention is lack of familiarity with Python, Unix and TCP fundamentals. I’m not sure how to respond to that sort of thing other than to say that if you want to develop networked applications in Python, on Unix, then it’s probably pretty good to be familiar with such stuff.

By the far the biggest issue with development is managing the inclusion of necessary stuff in the development environment. This includes auxiliary plugins that are required for the system, TiddlyWiki content and plugins, and static content that the web app needs. Mike’s blog post has most of the plans in this area. The gist is two pieces of development:

• Improving the existing devtext store so it more effectively (and correctly) supports during development manipulation of the store and inclusion of TiddlyWiki content. We’ll be talking about the details of this in IRC this afternoon.
• Creating a suite of shell-based tools that provide Make like functionality for establishing and maintaining a development environment and easing deployment. The long term plan is to migrate these tools to Python as time and knowledge allows.

For myself the things that have become important are:

• Making existing plugins more visible to developers so they know what tools are available.
• Packaging so-called mature plugins as Python packages that are installable via easy_install or pip.
• Enhancing documentation of configuration options so they are more visible.
• Publishing the quick hacks and tricks I do to make my own development life a bit more zipless.