May 18, 2013

In Kim Stanley Robinson’s Antarctica, there is a character, Ta Shu, who is a famous Chinese feng shui expert. Ta Shu is travelling around Antarctica with an adventure tour group, relaying his experiences back to an audience at home. His descriptions of the landscape are detailed and insightful and extremely poetic. In Chinese, at least. His English is more limited, which leads to some of the other characters in the book underestimating his intelligence, particular as he responds to most of their queries about what he thinks of places with the phrase “This is a good place” as a substitute for the detailed explanation he would give in Chinese.

Anyway, yesterday we went to a place that Ta Shu would have described as a “good place”. We were spending the day with Rita’s parents, her brother and his girlfriend, and her parents. They were all visiting our part of the country for a “parents meet the parents” holiday. We first drove from Igls to Achensee, a long lake in the mountains about 50 km east of Innsbruck. While the others went on a boat trip, Rita’s parents, Rita, me and Winnie went up on a cable car to do a bit of walking. The weather wasn’t great and there was still a lot of snow around, so we didn’t get all that far (although Winnie did a bit of marmoset-chasing, which was more fun for her than for me, as I had to then chase her…). After coffee and schnapps in the hut at the top of the cable car, we headed back down to meet the others.

The plan then was to drive to Perttisau on the other side of the lake, then up to Gramaialm where there is a Gasthaus, for everyone to have lunch. The Gasthaus was very traditional, so nothing much for vegans, so we decided to go for a walk with Winnie while the others ate. Rita wasn’t keen at first because it was raining, but I pulled out the classic “Winnie needs some exercise” line and she was persuaded. (The rain stopped about a quarter of an hour into our walk anyway.)

The Gasthaus is close to the top of the valley, but we headed back the way we’d come a little. The floor of the valley is very green and there are dozens of different kinds of alpine flowers everywhere, along with lots of stunted pine trees. It’s very beautiful, and the setting between the high mountains hedging the valley is very impressive.

After a little while, we came to what looked like a big bank of gravel. At first we thought that it was artificial, since it was flat-topped and twinned with another, parallel, bank about 50 metres further on, with a flat channel in between. Not artificial at all though! We followed the channel up towards the wall of the valley, crossing a small stream that we eventually saw issued from a narrow gorge twisting back into the mountains. We scrambled up the gorge as far as was safe for Winnie – she’s a super little mountain dog, but she doesn’t have thumbs and she is only small, so she has her limits! It was really beautiful, with crystal clear water splashing over the rocks, many of which were smoothed by the flood waters that have obviously carved out the channel we’d seen.

And as we were walking around there, I had the greatest feeling of happiness and peace that I’ve had for a long time. Here I was, with my most beloved and our little canine friend, exploring a wonderful place, somewhere we found purely by serendipity. Truly “a good place”.

May 17, 2013

TL; DR: use package version number bounds in the build-depends clauses of your Cabal files. You may save yourself some time and frustration.

Most Haskell programmers seem to have Cabal war stories to tell. Library version management in Haskell is complicated by the aggressive inter-module optimisation that GHC tries to do, which makes Cabal’s life tough and sometimes necessitates blowing away both Cabal and GHC’s knowledge of installed packages to start over. It’s also frustratingly easy to inadvertently get into a state where you have versions of packages installed that prevent Cabal from finding a consistent set of packages to install to satisfy the constraints for a new package you’re trying to install.

I thought I’d cleverly avoided most of these problems by using the hsenv sandboxing tool, which allows you to isolate the set of packages needed for a particular project, and even to maintain projects that use different versions of GHC, and cabal-meta, which allows you to build multiple packages from source together so that Cabal can find a set of package versions that are acceptable for all of the packages at once. I often found myself succumbing to a smug sense of self-satisfaction as I read about other peoples’ Cabal problems on haskell-cafe.

Of course, pride goeth before a fall. My hubristic little bubble was burst the other day when I decided to reset my sandbox for a project I’m currently working on. For some reason, typing cabal-meta install just wasn’t working, and I was being told that there was no set of package versions that could consistently satisfy the requirements of the packages I was trying to build from source.

Off down the rabbit hole I went. How could this have happened? Before I reset my sandbox, everything was building fine. That meant that a new version of some package must have appeared on Hackage with different version dependencies than the version I had in the sandbox before reset. Before reset, the old package was fine and everything worked. After reset, Cabal downloaded the most recent package, which broke the dependencies.

OK, so which package was the culprit? And, more fundamentally, why was everything so brittle that a single update of a dependent package (that’s all it turned out to be!) broke everything?

The first part involved some fun. First of all, I wanted to find the full set of packages that all of our code depended on, which is essentially just the transitive closure of the build-depends relation in the relevant Cabal files. I wasn’t thinking very clearly at this point (having one’s hubristic little bubble popped will do that), so I started writing a Perl (!) script to parse Cabal files. That plan ran aground very quickly once I realised what an idiot I was being. I switched over to using the Haskell Cabal library, which made parsing and processing Cabal files a real breeze. Right tool for the job.

I thus ended up with a (long) list of packages. Did I really want to look at all of those to figure out what was going on? At this point, I made the critical observation, which after the fact, feels a bit obvious: very few of the packages listed in the build-depends clauses of our Cabal files had version number bounds. There was no excuse for this – it was plain laziness. Whenever I needed to add a new package dependency, I just stuck the package name in the build-depends list and went on about my business.

The problem with this approach, of course, is that if your package A depends on Hackage packages B and C and both B and C further depend on package D (all without any version bounds specified), it’s very easy to transition from a state where everything works (B and C have compatible version requirements for D) to a state where nothing works (C has been upgraded and now depends on a version of D that is incompatible with the version required by B).

So, I learnt a valuable lesson – always provide version bounds for packages listed in build-depends. And I paid the price for my lesson: half a day of frustration trying to figure out what was going on, then a couple of hours of laboriously tracking down appropriate version bounds for all of the packages we refer to. And now everything works and I can retreat back into my hubristic little bubble…

May 11, 2013

In the last three days, I’ve averaged 10 hours of billable time per day. That’s ten hours per day actually standing at my desk coding, perhaps 10-20% more than that “on the job”. I’ve been working on social and search aspects of the BayesHive web app: we have a pile of different kinds of items (data sets, documents, models, and some others) that are all searchable and shareable in slightly different ways, and this all needs to be managed in a sensible way.

After a frenzy of hacking, I merged the branch I’d been working on just before dinnertime. It all works pretty well, I think, although there are probably some holes. It’s pretty inefficient, but we’re planning to move to a different storage system for documents and data soon, and that will make full-text indexing much easier, streamlining all of this stuff, so I was happy to produce a clean design and not worry too much about optimisation for now.

I’ve been really enjoying this work. I think what we have now may be one of the more complex Haskell web apps out there, and we’re gradually converging on a design that looks really pretty nice. The combination of Haskell, Yesod and AngularJS is very effective, and there’s enough Haskell work to balance out the horror that is JavaScript.

Tom should be inviting some people to start playing with BayesHive some time in the next week or so. It will be good to get some other eyes on what we’re doing. I’m going to spend some of tomorrow trying to do some example analyses, but I’m too familiar with the foibles of the web app to be a really good tester.

The only question now is whether, after three glasses of wine, I’ll still wake up at dawn (about 5:30 here right now) tomorrow morning, as I’ve done for the last few days…

May 6, 2013

I’m a good dog, I am!

Our dog Winnie is a great little dog. Of course, we would say that, but it really is true. The more I read about dog development and behaviour, the more pleased I feel about how far she’s come since we adopted her.

Sweet as she is though, there’s one thing she does that drives both me and Rita spare, something that can make a nice outing into a big pile of stress. Winnie is a hunting dog, and oh, does she love to hunt. She never catches anything, but she loves to chase things. Birds and squirrels mostly, but deer, hares, cats, anything that runs away. (She will stand and watch deer that aren’t moving or that haven’t seen her, but if they run, she’s off.) She has in the past thought about chasing Gämse in the high mountains, but I managed to put her off that – she had no chance of getting anywhere near them and she would probably have killed herself falling off a cliff trying to keep up with them…

When she runs off, she can disappear for a while, and she just doesn’t listen to us. (In fact, from what we know, she probably can’t even hear us once she switches into hunting mode.) The longest she’s disappeared for was only about half an hour, a couple of weeks ago, which by beagle standards, is nothing. It’s not unusual for them to disappear for days. But even a short time out of sight crazily chasing whatever (often it’s just a smell she’s after) is dangerous. There are hunters here who will happily kill dogs that are running free, and there are other more mundane dangers too.

So, what do we do? I’ve been reading a book by David Ryan, an ex-police dog trainer and instructor. It’s giving me some good insight into why Winnie behaves the way she does, and it and Ryan’s other book give me some hope that we might be able to do something about it.

The basic problem, as Ryan describes it, is that this sort of predatory behaviour is, first of all, innate and instinctual, and second, like puppy crack. Exercising the various aspects of this behaviour is so exciting for dogs that they fixate completely on the hunting to the exclusion of everything else, and the endorphin rush they get is supremely addictive. They are never going to interrupt that to come back to you for a biscuit.

Let me at him!

Ryan describes the different phases of the natural canine predatory behaviour as: track, scent, freeze, stalk, chase, grab and kill-bite. Different dog breeds have been bred over time to emphasise different phases of this behaviour more or less. We’re pretty sure now that Winnie is a cross between a beagle (or something like a beagle, a harrier or an Anglo-Français Tricolore maybe) and some sort of terrier. You can see this in the way that she concentrates on tracking (sniffing on the ground), chasing (self-explanatory), grabbing (she loves pouncing on toys) and kill-biting (typical terrier behaviour of shaking things, mostly toys, to “kill” them).

From what I can tell so far, Ryan’s scheme for helping with this problem has three steps.

First, make sure that the emotional environment of your dog is as stress-free and positive as possible. Many dogs engage in predatory behaviour for stress relief. For Winnie, life is kind of stressful in general, because of her lack of socialisation and exposure to “the human world” when she was a puppy. We do what we can to make life peaceful and fun for her, but she still gets scared by frightening things like plastic bags, people, boxes, and so on.

Second, you need to remove the opportunity for your dog to exercise its predatory behaviour. That doesn’t mean no more off-leash walks, but it does mean you need to be a bit careful. Winnie mostly does her crazy chasing in the woods, not so much on the open fields. There are only rarely deer out in the open, and I’ve not seem any hares. She will track mice in the fields and do some funny pouncing, but she never catches anything. (The only taste of mouse she ever gets is when a bird of prey leaves some bits…) Unfortunately, dogs have to be on leash on agricultural land here between March and October, so we’re going to have to be a bit clever about what we do.

The third part of Ryan’s “chase no more” plan is extensive training to condition your dog to displace their chasing behaviour to a toy that you control. I’m hoping to get some more details from Ryan’s book “Stop!”. It looks like it’s going to take quite a bit of work, but I think it will be worthwhile. At the moment, we can’t really trust Winnie out in the woods – the smells of spring have led to an increase in chasing of birds and squirrels and it’s impossible to get her to listen when she’s in psycho-squirrel-chaser mode, so we have to do something!

April 28, 2013

Copyright 2010 Raymond Long

When I started doing yoga, it looked simple. Twist yourself into a little pretzel, sit there for a little while, then choose a different pretzel shape and repeat. It turns out that things aren’t quite so simple. You have to engage muscles to remain in these twisty positions and even to pull yourself further into the postures.

Knowing which muscles to engage and where to focus your attention in each posture is something that a good teacher would help with, of course, but I’m a bit teacherless. There is some advice available online, much of it expressed in terms of analogies (“embrace your thigh”, and so on). That’s OK, but it’s not a substitute for a precise anatomical description of what’s going on.

Raymond Long’s two series of books, “Scientific Keys” and “The Mat Companion”, both available from his Bandha Yoga site, are an amazing and unique resource to help with this. There are two “Scientific Keys” books and four “Mat Companion” books, all of which are full of high quality anatomical illustrations and detailed descriptions of how to make a whole range of yoga postures work. Some of the descriptions are at the level of “engage the lower third of the trapezius here”, but there are also a lot of very simple and clear cues for engaging and getting to know about muscles that you might not otherwise think about.

There are four or five cues for each posture, plus a lot of general advice about flexibility and joint health. I’ve not seen anything else anything like Long’s books out there, and I’d really recommend them to anyone who’s either studying yoga on their own, or who just wants to improve their practise.

April 24, 2013

Courtesy of Hideharu Sakai, this article is also available in Japanese.

One of the things we needed to be able to do for the BayesHive software I’m working on with Tom Nielsen of OpenBrain is easily produce plots within our front-end web app. There are lots of JavaScript libraries for doing this: Tom started off using Flot, which is very quick and easy to use, but wasn’t really flexible enough for what we wanted.

I’d just been working on another part of the software, a tool for constructing algebraic representations of dynamical systems (defined by systems of ordinary or stochastic differential equations), and I’d been using the AngularJS framework. It took a while to get used to, but it seemed as though it might, in conjunction with D3.js, provide a way to produce a declarative plotting API that would be really easy to use and could take advantage of the clever data binding features of Angular.

Thus was Radian born: a declarative extension to HTML for rendering data and functional plots as SVG graphics. You can go straight to a gallery of examples here. Details and some more examples below the fold. (Note that, since Radian uses SVG graphics, you’ll need a fairly recent browser to see anything…)

The basic idea

Angular allows you to define directives, custom HTML elements that can use template replacement or arbitrary JavaScript code that is called at various points during Angular’s “HTML compilation” process. Plots in Radian are specified using custom directives: the main wrapper around a set of plots on the same axes is a <plot> directive. Within a <plot> directive, you can place multiple directives specifying different kinds of plots. For example:

<plot height=300 aspect=3 stroke-width=2 x="[[seq(0,4*PI,101)]]"
      axis-x-label="Time" axis-y-label="sin(x) / cos(x)">
  <lines y="[[sin(x)]]" stroke="red"/>
  <lines y="[[cos(x)]]" stroke="blue"/>
</plot>

The attributes with the double square brackets are Radian expressions. These are very much like Angular expressions, but the language they support is a strict superset of the JavaScript rather than a restricted subset as for Angular expressions (I use a customised version of the very nice Acorn JavaScript parser to deal with these things). Here, the seq function produces an array of regularly spaced values, and the sin and cos functions are implicitly vectorised over arrays so that the y values are calculated as you might expect. (The Radian plotting library includes a bunch of useful functions, all of which exhibit this useful vectorised behaviour, as do arithmetic operators in Radian expressions.) The above code results in the following plot:

(All the examples here are produced by just cutting and pasting the appropriate HTML into the source of this blog article. Apart from making sure that the appropriate JavaScript libraries are loaded and activating Angular on the page, nothing else is needed.)

Handling data

Plotting functions is all very well, but we more often need to generate plots based on data – either data uploaded by users or the results of statistical analyses. Radian makes this easy with a directive called <plot-data>. We can specify data in-line within our HTML page or cause it to be loaded from a URL as required. Here’s an example:

<plot-data name="fam" src="familyr.csv"
           format="csv" cols="name,age,height,sex,salaryhr">
</plot-data>

We can then use this data in a plot as:

<palette name="mfpal" type="discrete">
  female #FF7F7F; male #7F7FFF
</palette>

<plot height=600 aspect=1 stroke="none" marker="circle"
      axis-x-label="Age" axis-y-label="Height">
  <points x="[[fam.age]]" y="[[fam.height]]"
          fill="[[mfpal(fam.sex)]]"
          marker-size="[[30*sqrt(fam.salaryhr)]]">
  </points>
</plot>

yielding the following figure:

female #FF7F7F; male #7F7FFF

Note how we’ve used a palette here to specify the mapping from gender values to colours – the <palette> directive produces a JavaScript function that can then be applied to a data array. In general, all paint attributes (fill, stroke, marker size, stroke width, etc.) can be calculated as functions of data. This orthogonality, along with the power of the Radian expression formulation, makes it possible (and easy) to produce bubble charts and other data-dependent graphics.

Binding to Angular

The relationship between the variable names used in Radian directives and Angular scope variables is designed to make using Angular’s data binding capabilities easy. All attribute names in Radian plotting directives generate Angular scope variables (with nested scopes being created as appropriate), and all free variables in Radian expressions are treated as referring to Angular variables in the enclosing scope. (It helps to understand how Angular scopes work here, of course…)

The upshot of all this is that you can do things like this:

<div class="form-inline">
  <label>Mean</label>
  <input type="range" min=0 max=10 step=0.01 ng-model="mu" ng-init="mu=5">
  <label>&nbsp;&nbsp;Standard deviation</label>
  <input type="range" min=0 max=10 step=0.01 ng-model="sigma" ng-init="sigma=1">
</div>
<br>

<plot height=300 aspect=3 stroke-width=2 stroke="red">
  <lines x="[[seq(0,10,200)]]" y="[[normal(x,mu,sigma)]]"/>
</plot>

to get the following:

Manipulating the UI elements (bound to Angular variables using ng-model) leads to immediate changes in the plot.

Summary

Radian is very much a work-in-progress. I’m adding new functionality (and fixing bugs) nearly every day. I’m learning a lot as Tom and I use Radian in the BayesHive web app: the examples here and in the gallery are standalone, and don’t really get into the gritty corners of the interactions between Radian and other Angular code. Those gritty corners are being exercised by the integration of Radian into BayesHive, which is essentially a document authoring system oriented towards Bayesian data analysis. The functionality of the system is completely open-ended and every data analysis example we come up with stresses Radian in a different way.

Radian is currently proprietary. There are some contractual encumbrances that prevent us from open-sourcing it right now, but that may change at some point in the future. In the meantime, if you’re interested in learning more, contact me or Tom.

April 22, 2013

When we lived in Canada, we exercised a lot: mountain and road biking, running, sea kayaking, CrossFit. That dropped off in France, but I managed to do some gymnastics and basic conditioning, which continued for the first summer here in Austria (lots of gymnastics: I even managed to do a few muscle-ups on my rings!).

Over the winter in Igls, I’ve not been doing a whole lot, partially for lack of time (I keep telling myself that three hours of dog-walking a day has to be worth something), partially for lack of inclination, and partially because the pull-up bar is too cold to hold onto in the mornings when it’s -10°C outside (that sounds like a better excuse than “lack of inclination”!). I’d been thinking for a while that I needed to start doing something again, and Rita had a yoga book lying around, so I thought I’d give that a go, if only for a while.

I spent a few weeks doing a mixture of things out of this little book and it was kind of fun, but I didn’t really know where I was going with it. I was looking around for videos and websites one day, and I found Kino MacGregor’s site. “Heh, that’s funny,” I thought. “She looks more like a CrossFitter than anything else.”

Kino has been doing Ashtanga yoga for fourteen years. Ashtanga is usually described as a “vigorous” style of yoga, which seems to mean that it’s excruciatingly hard, you sweat like a pig (even without that wacky hot room that Bikram yoga uses…), and it will take you years to be able to do some of the postures in the “Primary Series”. Sounds like fun!

Convenience dictates that this is mostly going to be self-guided: the likelihood of practising regularly is much higher if all that’s needed is to unroll your mat and start; getting on a bus into town to a class is a bit of a barrier. So, I ordered some books and made a plan.

So far, it’s going pretty well. I’ve been practising nearly every day for three weeks and have built up to doing a bunch of Sun Salutations, all of the standing postures and about half of the closing sequence. The Ashtanga practice is made of four parts: two kinds of Sun Salutations, a series of fundamental standing postures, either the “Primary” or “Intermediate” series of seated postures, then a closing sequence involving a mixture of backbends and inversions. The Intermediate series isn’t something to be tackled until after a couple of years of work on the Primary series: the Primary series already has some postures that look nearly impossible…

I’m planning to do 30 practises with my current setup (I’m up to 13 so far), then gradually to add the Primary series postures and the rest of the closing sequence to build up to the full sequence. The two books I’m following are pretty good, as they show a lot of variations for the more difficult postures. And as a beginner, you really do need the variations. For example, there are a number of postures that require you to “bind” a foot, which means putting it in half-lotus position then reaching around your back with the opposite hand to grab the foot. I don’t quite know whether it’s hip flexibility or shoulder flexibility that’s needed for this, but I don’t have enough of either yet!

April 21, 2013

So, I lied a little in my last post. I didn’t quite get to go off right away and update my Beeminder totals, because I had a bit a problem with my blogging software. I use Jasper Van der Jeugt’s Hakyll, a Haskell static site generator. Jasper recently (well, not that recently…) released a new major version of Hakyll, Hakyll 4. I had been putting off upgrading because I have so much custom code that it’s kind of a big job – my blog is a little different to most Hakyll sites, so I had to write quite a bit of extra stuff to work around that.

When I tried to rebuild the blog with the Beeminder article though, I started off down an amusing little path. I’d upgraded my version of GHC, so needed to reinstall things. No problem. Well, maybe. The version of Pandoc that Hakyll 3 uses no longer seems to build, even in a completely clean sandbox. I’d guess that there’s some inconsistent version bound resulting from one of Pandoc’s many dependencies in that version (1.9.4.5, if anyone cares).

So, I had a choice. I could try to track down the dependency problem in this old version of Pandoc (it’s now at version 1.11.1, so that would be an exercise of purely historical interest for most people), or I could upgrade to Hakyll 4. Cue much rolling up of sleeves and girding of loins. I jumped in. It wasn’t so bad, though it took me all day plus a little bit.

In the end, I had far less custom code than before: the new Hakyll is quite a bit more flexible than the old one and Jasper has made some really good design choices this time around. Two things stand out:

Arrows be gone!

First, Hakyll 3 used arrows for its main sequencing mechanism for the steps required to turn a Markdown (or whatever) blog post into its final HTML incarnation. Arrows are great, and can be really neat if you learn and use the special proc syntax for them. I’ve never got around to learning that though, so I had a lot of code that looked a little like this:

processTikZs :: Compiler (Page String) (Page String)
processTikZs = (id &&& (arr pageBody
                       >>> (id &&& unsafeCompiler generateTikZs)
                       >>> arr (uncurry xformTikZs)))
               >>> (arr (\(p, pbnew) -> p{pageBody = pbnew}))

Urgh. I felt a bit like this guy writing this sort of thing. In Hakyll 4, the arrows are gone, and there’s a Compiler monad for sequencing. That processTikZs function now looks like this:

processTikZs :: Compiler (Item String)
processTikZs = do
  b <- fmap itemBody getResourceBody
  ts <- unsafeCompiler (generateTikZs b)
  makeItem (xformTikZs b ts)

No more arrowised tuple juggling. You can actually see what’s going on. For sure, you can do exactly the same thing using proc with GHC’s Arrows extension enabled, but that’s nowhere near as familiar to most Haskell programmers as a monad. It’s also pretty clear that the additional flexibility of arrows compared to monads just isn’t needed in this case.

Contexts

The second major change in Hakyll 4 is the introduction of “contexts”, which are a monoidal interface for packaging up information about posts, either as constant values, or as functions for extracting values from page metadata or whatever you like. This replaces the old system of carrying this stuff around with the Page type, which led to all sorts of one-hand-behind-the-back arrow tricks.

Everything is much clearer and cleaner now, and it’s easy to write extra context field functions to pick bits out of pages (for example, I have a thing called teaserField which pulls out the text from the body of a post up to the “Read more” link and puts it into a named field in the context that can be used for rendering index page entries). It’s much easier than the old way of doing this stuff.

Summary

There are other things that are good (snapshots are cool, for instance), but those are the two that struck me most as I was upgrading my blog.

All in all, I think Jasper has done a great job with this. Hakyll is now much easier for new users to get into, and it’s far easier to customise the behaviour of Hakyll now.

I’m actually kind of disappointed with myself now that I didn’t do the upgrade earlier! You can see my new code on GitHub.

April 20, 2013

I learnt a new word yesterday: akrasia (rhymes with “aphasia” and “Malaysia”). It means not doing the things you want to do and know that you want to do; acting against your better judgement. The classic example of akratic behaviour is weight loss (or weight control). That extra slice of pie always looks like such a good idea at the time and there’s no short-term consequence. It’s only later that you look back with a queasy feeling of regret at the empty pie dish and think of how your rolls of kidney flab are just not fashionable any more (and probably haven’t been since the 1830s).

Everyone is akratic to some extent. I know that I am. Some days I just can’t face the work I have to do (even though it’s usually fun once I get started) and just doodle around on the internet. I’ve tried various methods for organising myself (I use org-mode in Emacs for pretty much everything: TODO lists, scheduling, time tracking for invoicing), I’ve tried structured procrastination. Sometimes these things work and sometimes they don’t. The main problem is that, while these methods can help you to organise yourself, they don’t impose any short-term consequences on you to make you do things.

A good example for me is blogging. There are no short-term consequences at all for me if I fail (for weeks and weeks) to write any blog articles. In the long run though, there probably are consequences. One of my current contracts came about because Tom Nielsen at OpenBrain had been reading some stuff on my blog and as a direct result contacted me to ask if I’d be interested in some Haskell contracting work. No blogging ⇒ no contract.

So how do you get some control on this? Enter Beeminder. I first read about this on Brent Yorgey’s blog a while ago, but Brent just posted a follow-up to his experiences with Beeminder. He had such positive outcomes from using the thing that I decided I ought to give it a try too.

The basic idea of Beeminder is pretty simple. You set yourself a goal which has some numerical measurement associated with it (it might be body weight, it might be number of blog articles written per week, it might be hours per week spent on a particular project, whatever you like). Beeminder then shows you a “yellow brick road” which you need to follow – if you fall off your target a little bit, you have a day or so to recover, if you get ahead of your target you can relax a little. If you fall off the wagon again though, there are consequences. The consequences, as well as the natural embarrassment that anyone would feel, are monetary. There is an exponential scale of payments you have to make each time you stray from the yellow brick road – small payments at first, but they get bigger quickly. The idea is that you need to reach a level of payment that really motivates you to stick to your goals. And because there is continuous monitoring (if you let it, Beeminder will remind you every day how much of a safety buffer you have before you end up in the hole) and short-term consequences, there really is a strong chance that you will stick to the plan.

There is quite a bit more clever stuff involved (there are nice graphics, there’s flexibility about changing your goals as you go along, but done in a very smart way that prevents you from just weaseling out of the goal you’ve set yourself, and there’s some good stuff for data smoothing for weight measurements), but that’s the essential idea.

For me, it’s already working! I’ve set up goals for German study, blogging, a work project (Beeminder has GitHub integration, which means you can set yourself a target of a certain number of commits per week to a certain repository) and a bunch of other things. And as soon as I’ve published this blog article, I’ll go and update my stats…

March 20, 2013

So, I’ve not done any blogging since the ill-fated Midwinter Polyphasic Sleep Experiment. I have a backlog of articles I want to write, but most of those involve some significant investment of time and effort. So you get this instead…

Context

I’ve been doing a lot of JavaScript programming recently. One of the two contracts I’m working on at the moment is for a UK university spin-off called OpenBrain, who are trying to bring Bayesian statistics to the masses by making tools that allow scientists to build complex (or simple!) Bayesian models and have them analysed using Markov chain Monte Carlo methods. All the really clever stuff is in the server-side MCMC inference code, but the user-facing side of things is a web app with a Yesod backend. My mission is to prettify this and make it ready for prime-time.

It’s a fairly complicated application that needs to manage literate Markdown documents describing statistical models, data sets for analysis and a bunch of other stuff. The first thing I looked at was a tool for building statistical models based on dynamical systems represented as systems of ordinary or stochastic differential equations. This tool allows you to give algebraic representations of ODEs or SDEs, which are then rendered as MathML (for prettiness and familiarity), you can do simulations of the systems and visualise the results, you can specify data sets that correspond to observations of system variables, you can set up prior distributions for system parameters, and you can then get some code (in a proprietary Haskell-like language specialised for statistical modelling) that can be used to drive MCMC inference for the parameter values.

All that stuff happens on the client side, except for the listing of the available data sets and the rendering of the model into code. This means that you need client side code for expression parsing, analysis of coupled systems of ODEs and SDEs (to turn them into a canonical form that you can simulate from), numerical integration of ODEs and SDEs, graphing, and a framework to tie it all together. That’s a lot of client side code!

Choices, choices…

My original inclination was to use Fay, both because it’s fun and because the server side is all Haskell, but I was a little anxious that it might be too much of a moving target. There’s a lot of work being done at the moment to make Fay better and more robust, which is great, but I don’t have the resources to contribute to that as well as trying to write something fairly substantial using Fay, with the expectation that my code will rot quite quickly as Fay development proceeds.

None of the other Haskelly options appealed very much, which meant I was left with the prospect of writing a big old pile of JavaScript. The expression parsing stuff wasn’t too painful, mostly because of Jison, a JavaScript almost-clone of Bison that made it easy to write a JavaScript equivalent to my Haskell Parsec parser. And the numerical stuff for simulations wasn’t hard either. But the code for taking a set of ODEs and transforming them into a canonical form that can be numerically integrated? That looked super-simple in Haskell and really horrible in JavaScript¹. Case analysis by pattern matching really does work better when you have types to match on…

However, the really tricky part was the user interface side of things. I wanted something responsive: if you’re typing an expression for a differential equation and you introduce a new parameter ($\sigma$, say), I want a UI element for that parameter to appear as you type (and to disappear if you erase all references to $\sigma$). In the simulation panel, I wanted to be able to drag over parameter values to change them and have all the plots update immediately. That sort of responsiveness, using the normal JavaScript event-driven approach to interaction, is going to lead to a big bowl of spaghetti.

Angular it is then

Shortly before this, Michael Snoyman had written a blog article about using Fay with Yesod. He also talked about integrating the AngularJS JavaScript framework with Yesod. As we’d ruled out Fay for the moment, I took a look at Angular. It turns out to be really quite neat. It’s a little hard to explain briefly exactly all that Angular is, but there are two factors that stuck out for me:

1. Data binding: Angular allows you to associate JavaScript variables with DOM elements (these variables live in nested “scopes”, which are also associated with DOM elements). The binding is bidirectional so that if you have a piece of text bound to a variable and you change the variable, the text changes; if you have an input element bound to a variable and the user changes the contents of the input element, the variable changes. There are various ways of writing code that interacts with this data binding and it’s quite a flexible and sophisticated system.

2. Directives: you can “extend” the HTML language by defining new attributes, classes or element types. These can have HTML templates associated with them, as well as bits of code that can run at different points during Angular’s “HTML compilation” process. Directives are flexible enough that you can (with a little patience) do more or less anything.

A couple of examples of how these things really help: first, the equation editing input elements are now <equation> tags, where equation is an Angular directive that encapsulates a normal input element (active when the element has the focus), error reporting elements and a MathML rendering of the equation content. All of the equation parsing and event handling associated with the interactions between these three elements is encapsulated within the equation directive. The second example is a <scrubbable> element directive. This is a new kind of input element that shows a MathML rendered equation for a parameter (e.g. $\sigma =28$). Clicking on the element allows the user to edit it, control-clicking resets the original value, and click-dragging continuously (or discretely, if an attribute on the element is set appropriately) varies the value of the parameter. Using data binding, the changing value of the “scrubbed” parameter can be directly communicated to other code: in the dynamical systems simulation page, this is used to trigger re-running of simulations and re-rendering of plots.

All in all, I’m pretty happy with Angular. The documentation is a little patchy, there’s a bit of a learning curve, and doing complicated things can require a bit of experimentation, but it’s a very good system for writing complicated interactive client side JavaScript.

Yesod integration

Furthermore, the integration with Yesod is really great. Michael Snoyman wrote a little yesod-angular module as part of his Yesod client-side experimentation, and it turns out to be really rather nice. I’ve hacked my version around quite a bit to help with managing Angular’s dependency injection mechanism, but that’s about all I’ve had to change. Perhaps the nicest feature is the idea of “commands”. Here, in my Yesod handler, in the bit of code that runs in the Angular monad, I can define commands as, for example

    cmdContents <- addCommand treeGetContents

    ...

treeGetContents :: [Text] -> Handler [TreeItem]
treeGetContents path = do
  ...

Here, TreeItem is some type with a ToJSON instance so that it can be converted to JSON for transmission back to the client. On the client side, you can call this command as

      $http.post("#{cmdContents}", [node.data.fullPath]).
        success(function(data) {
          ...
        });

All the data marshalling is handled by ToJSON and FromJSON instances. On the server side, the code is in Yesod’s Handler monad, so you can do all the kinds of things you might normally do in response to client requests, and everything is nicely encapsulated. It’s a really good system.

Conclusion

Writing a Haskell web app? Dithering about the client side? If you can take the uncertainty of programming against a dynamic and developing interface, go with Fay. A lot of people are investing a lot of time in it, it’s already really good, and it’s going to be great. Can’t stand the uncertainty? Look at Yesod.Angular. It’s in Michael’s yesod-js repository.