I’ve been meaning to start a blog for some time, but couldn’t settle on a platform. I’d used Wordpress before for a little blog we set up to let our friends and family know what we were up to when we moved to Montpellier, and I liked it (easy to install, easy to use). For a personal blog though, I wanted something a bit more hackable (yeah, I know, you can hack on Wordpress too, but if it’s going to be for fun, I want to be using something other than PHP!).

I spent a bit of time writing the beginnings of a blog platform of my own in Haskell, using the Happstack framework. I got some way with that, but it was never really ready for primetime. Recently though, I heard about Hakyll, thought “What a great idea!” and jumped back in to getting something in shape.

Hakyll is a static website generator, which makes it great for situations where server resources are limited. I’m serving my site from a Linode VPS using the Cherokee web server, and I just don’t need the full flexibility of a general webapp for what I want to do. Hakyll seemed like an ideal solution, and it’s Haskell, so it would be fun!

I’ll write later about how I set Hakyll up for what I wanted, because there were a number of things I wanted to do differently to the way most Hakyll-based sites are set up, but you can see what I did here. The basic requirements were:

• Markdown syntax for posts;
• Wordpress-style YYYY/MM/DD/TITLE post addresses;
• Easy to organise posts with ancillary material (images, code, etc.);
• Good LaTeX math support.

In addition, I wanted a feature I’d implemented in my not-quite-ever-finished blog framework. This allows me to embed TikZ code representing an image in a blog post and have it rendered as an inline SVG image in the resulting HTML. Useful? Sometimes. Cute? Well, I think so. I’ll show how I did that in a later article.

As for what else I’m going to write about here, it’ll be pretty eclectic. Basically, anything I find interesting. Which might range from Haskell and Common Lisp stuff, to ecology, remote sensing and vegetation modelling (my day job), book reviews, plus whatever else takes my fancy.

I’m a relative beginner with Haskell, and like many people, to start with I was a little perplexed by the Haskell approach to I/O. A small worked example helped a lot. I was curious to see how easy it would be to do something like the webcomic scraper application implemented in Clojure here and here. This is a simple application, but it does do realistic I/O, downloading files from the web, writing them to disk, and also doing some computations on the file contents. Over the course of two articles, I’m going to build something comparable in Haskell. It turns out to be pretty easy!

The I/O problem

Before we get started, just why is I/O a problem in Haskell? (A problem for beginners anyway.) Haskell places great weight on purity and referential transparency, and this makes dealing with the outside world more interesting than in imperative languages. Haskell’s approach to dealing safely with impure actions with side effects is to use a monad (the IO monad) to segregate I/O “actions” from pure functional code. This takes a bit of getting used to.

The other common approach to handling I/O in pure code is exemplified by the uniqueness types of Clean and Mercury, where some “world” object is threaded through purely functional code — the old state of the world is passed into a function that does I/O, the new state of the world is passed back out, and the type system of the language enforces “single-threaded” access to the state of the world (Mercury calls the input state a destructive input argument, and the output state a unique output). The same sort of threading of I/O state has to happen in the run-time system of any functional language, and Haskell’s monadic I/O syntax can be thought of as a way to hide this plumbing. Of course, monads can be used for much more than this, but I’m not going to talk about that at all. Here, I’m taking a beginner’s eye view of things, and I’m going to talk only about the I/O monad.

Getting started

First, let’s define the problem we want to solve. We want to start with definitions of some webcomics (base page, a regular expression to use to pick out the address of the image file for the latest episode, etc.) and download the image files for the latest episode of each comic. To do this, we’ll get the base page of the webcomic, use the contents of that page to figure out the URL of the latest episode image, download that image and save it to a file. We’ll get a little more fancy about this later (saving images to directories based on today’s date and reading the comic definitions from an XML configuration file), but let’s start with this simple goal in mind. The code for this stage of the example can be found here.

We’re going to need a couple of Haskell packages that aren’t installed by default, in particular Network.HTTP and the Posix regular expression package Text.Regex.Posix. If you have Cabal set up (which you most definitely should), installing these is as simple as typing

cabal install HTTP
cabal install regex-posix

Let’s start by playing a little with a function to download the contents of a URL. If we put the following into a Haskell script file, tst.hs, say:

import Network.HTTP
fetchURL :: String -> IO String
fetchURL url = do
  resp <- simpleHTTP (getRequest url)
  return (getResponseBody resp)

then we can do the following in ghci

Prelude> :load tst.hs
*Main> fetchURL "http://www.xkcd.com/"

and we get back a string containing the HTML contents of the XKCD front page. So, how does this work? First, let’s think about the type of fetchURL, which is String -> IO String: it takes a single string as input (the URL we want to fetch) and returns a value of type IO String. This return type represents an I/O action that yields a String when we execute it.

If we think about a typical I/O action, reading a character from a file say, it’s clear why these need to be treated differently from pure functions. If we call a putative getCharFromFile function twice in a row, passing the same file handle as an argument, we don’t expect to get the same result from each call — the results that we get depend on the contents of the file, and in general, each call to getCharFromFile will yield a different result. This is quite different to the behaviour of pure functions: if we evaluate head [1, 2, 3], we always get 1 as the result, independent of the state of the outside world.

These considerations mean that we need some framework for sequencing and executing I/O actions. In Haskell, this is provided by the IO monad, which the Haskell run-time system treats specially to enable this non-pure interaction with the outside world.

I/O actions are sequenced using Haskell’s do syntax, which we will examine in more detail below. For the moment, take the details of fetchURL on trust, and concentrate more on the similarity to a more familiar imperative way of programming displayed in the do block.

Data structures

We’re going to represent the comics we want to download as simple structures, with string entries for the name of the comic, the URL of the base page, a regular expression to allow us to extract the URL of the latest story image from the base page, and a prefix to put on the front of that URL (which will probably be a relative URL) to make an absolute URL we can use to download the story image:

data Comic = Comic { coName :: String,
                     coURL :: String,
                     coRegex :: String,
                     coPrefix :: String }
             deriving Show

(The Comic type is defined using Haskell’s record syntax.) Eventually, we’ll read a list of comics to download from a configuration file, but for now, let’s define a few comics for testing:

comics :: [Comic]
comics = [ Comic "Girl Genius"
                 "http://www.girlgeniusonline.com/comic.php"
                 "ggmain/strips/ggmain[^']+"
                 "http://www.girlgeniusonline.com/",
           Comic "XKCD"
                 "http://www.xkcd.com/"
                 "comics/.+png"
                 "http://imgs.xkcd.com/",
           Comic "Girls With Slingshots"
                 "http://www.daniellecorsetto.com/gws.html"
                 "images/gws/GWS[^\";]+"
                 "http://www.daniellecorsetto.com/" ]

Regular expression matching

Given a Comic definition, the first thing we want to do is download the base page and extract the URL of the latest image file. We’ll write a function imageURL to do this. It will have type Comic -> IO String, returning an I/O action with a String result (the URL we want). The result here is an I/O action because we’re interacting with the outside world: we certainly don’t expect to get the same result from imageURL every time we call it with the same arguments. It wouldn’t be very useful if we did… This is a good general guide for when you need to think about using actions: if a piece of code should return the same answer whenever it’s called with a given set of arguments, then it can be implemented as a pure function. Otherwise, the I/O monad is going to be involved.

Here’s imageURL:

imageURL :: Comic -> IO String
imageURL c = do
  pg <- fetchURL (coURL c)
  let re = pg =~ (coRegex c) :: String
  case re of
    "" -> return (error $ "no match for regular expression for " ++ coName c)
    otherwise -> return (coPrefix c ++ re)

One feature of I/O actions that distinguishes them from pure calculations is the need to impose an ordering on the execution of actions. In a pure computation, order of evaluation does not affect the result of the computation: the expression a + b yields the same result, regardless of which of a or b is evaluated first. Here, we need to fetch the base page for the comic, and only then can we match the regular expression that will give us the URL to get the latest story image. This sequencing is expressed using Haskell’s do notation¹. Three kinds of expressions can appear within a do block:

1. Regular Haskell expressions;

2. let forms, which bind names to pure functional expressions;

3. Expressions of the form name <- action, which execute an I/O action and bind the result of the action to the given name.

Our imageURL function contains expressions of each of these three types. First, we use the fetchURL function (which returns an I/O action of type IO String), executing the returned I/O action to retrieve the comic base page, and binding the result to the name pg. Subsequent lines in the do block can then refer to this value. This binding of names permits only a single assignment to a name: once pg is bound here, it may not be rebound. Next, we use a let expression to do regular expression pattern matching on the comic base page. The =~ operator from the Text.Regex.Posix module is used (we have to give an explicit type annotation to resolve the ambiguity of the return type here). Finally, we use a simple Haskell case expression to decide what to return from imageURL on the basis of the pattern match. Note that both branches of the case expression contain uses of the return function, which has type a -> IO a and is used to “inject” a value into the IO monad: it’s just a way of turning a regular a value into a value of type IO a, which is the return type we need for imageURL.

Retrieving images

We can now use out imageURL function as part of a function to retrieve the latest image file for a given comic. We’ll call this writeImageToFile:

writeImageToFile :: Comic -> IO ()
writeImageToFile c = do
  putStrLn $ "Retrieving: " ++ (coName c)
  url <- imageURL c
  img <- fetchURL url
  writeBinary (normaliseName $ coName c ++ takeExtension url) img

The overall type of our function here is Comic -> IO (), showing that the function takes a Comic and returns an I/O action. The IO () return type means that the result of the I/O action is discarded: we are only interested in the I/O action for its side effects. Here again, we see sequencing of I/O actions using do. The first action writes a message to the terminal using the putStrLn function in the Haskell prelude (with type String -> IO ()). Then we use our imageURL function to download the base URL for the comic and extract the URL of the latest image file. Next, we use the fetchURL function to download the contents of the image URL, binding the result to the name img. Finally, we write the contents of the image URL into a file with a name built from the name of the comic using the normaliseName function and the takeExtension function from the standard System.FilePath library. Here is the definition for normaliseName:

normaliseName :: String -> String
normaliseName = map fix_spaces . map toLower
  where fix_spaces ' ' = '_'
        fix_spaces ch = ch

We write the image data to the file using the writeBinary helper function, which uses some System.IO library functions to write the file as binary data (i.e. no text encoding, no newline conversion — Haskell strings are written using UTF–8 encoding by default, which will mess up the binary image data we’re dealing with here):

writeBinary :: String -> String -> IO ()
writeBinary f s = do
  h <- openFile f WriteMode
  hSetBinaryMode h True
  hPutStr h s
  hClose h

The sequencing provided by do allows us to write this sort of I/O code in a very natural imperative-looking way. The Haskell compiler translates the do notation into pure code that threads the I/O state through the calls, but we don’t need to think about that to use the IO monad. We just need to remember two things:

1. To distiquish between pure functional expressions (which use let) and capturing the execution of I/O actions (which uses the <- notation);

2. That all assignments are immutable — once we assign a value using let or <-, that name cannot be reassigned.

Putting it together

Next, we want to take our writeImageToFile function and apply it to the list of comics defined above. We expect to do this with a function with type [Comic] -> IO (). We might initially be tempted to write something like map writeImageToFile comics, but this won’t work. To see why, consider the types of map and writeImageToFile:

map :: (a -> b) -> [a] -> [b]
writeImageToFile :: Comic -> IO ()

The type of map writeImageToFile would then be something like [Comic] -> [IO ()], which is not quite what we want. Composition of I/O actions should not result in a list of I/O actions, but in a single action that performs the individual actions in sequence. A suitable mapping function that provides this sequencing functionality is provided in the mapM_ function in the Control.Monad library. This library contains monadic equivalents of many common list processing functions (map, folds, etc.). Specialising to the IO monad, since that’s the only one we care about here, mapM_ has type (a -> IO b) -> [a] -> IO () and applies a function with an IO result to a list of values, returning an I/O action that discards its result (which is what IO () means).

We can then write a function to process all of our comics as:

processAll :: [Comic] -> IO ()
processAll cs = mapM_ writeImageToFile cs

Typing processAll comics at the GHCi prompt results in the current story image for each of the comics in our list being downloaded and saved to a file.

What next?

In Part 2, we’ll extend what we’ve done to organise our downloaded images a little, placing all the images we download into directories with names based on the current date — this will require us to do some I/O to create the directories, but also to determine the date (how could a getCurrentDate function always return the same value? i.e. how could it be pure?). We’ll also write some code to read a list of comics to download from an XML configuration file.

We moved from Vancouver Island on the west coast of Canada to Montpellier in March 2011. Montpellier is not so bad, but there are a lot of things I miss about Canada. Our friends, for one thing. But also the wide open spaces. One of the last sea kayaking trips I took in Canada was a solo outing to a small (very small!) island in the Gulf Islands National Park.

Whaleboat Island is a marine provincial park, but it’s unmanaged, which means you are completely on your own: no campsites, no hookups, no wardens, no helpful information boards, and yes, no toilets. There’s also the minor matter that the beach only exists at low tide and you have to climb up some cliffs to find a flat spot to pitch a tent. Apart from that, it’s a perfect destination. Perfect for me anyway. I’d landed on Whaleboat once before, as a possible lunch spot, but that time no-one else seemed interested in clambering over rocks to eat their sandwiches and we ended up on an admittedly very pretty beach on Pylades Island.

This time though, I was on a mission. I’d bought myself a kayak a couple of days before, and was out to try it out, as well as checking out the camping possibilities on Whaleboat.

I put in at Cedar-by-the-Sea, a small settlement just south of the town of Nanaimo, where there’s a boat ramp, a few houses and not much else but beautiful views of the islands. And yes, it’s islands. There are a lot of them (I have no idea exactly how many). On this trip, I paddled by Round Island, Mudge Island, Link Island, De Courcy Island, Ruxton Island, Whaleboat Island and Pylades Island plus countless nameless little rocks that support a couple of seals or a few birds.

Speaking of birds, there were a lot to see this weekend: gulls, cormorants, crows, eagles, some cute little black and white ducks, geese, plus a bunch of others. I never quite got used to seeing big bald eagles in the trees above me. They’re really surprisingly common. The other thing I never quite got used to was seeing all the marine mammals around. They’re quite skittish, and you need to make sure you keep a good distance from them, but families of seals are a regular sight (and it’s hard to avoid disturbing them sometimes as you paddle round a rock and all the seals who were happily basking in the sun decide that you’re Public Enemy #1 and they need to get in the water and look at you with accusing puppy-dog eyes until you go away). I also saw a family of otters I’d seen before, with the adults fishing and the pups “helping”. And a first for me, on the way back into Cedar, there was a big family of sea lions fishing for herring. Quite intimidating when you’re in a fragile little boat, since they’re not small animals and they get quite excitable when they’re fishing, with lots of angry-sounding honking from the big males as they herd the herring towards the other sea lions.

There’s always a lot of more exotic weird sea stuff to see as well. This time out, I saw lots of starfish, including the biggest one I’ve ever seen, which must have been over 50 cm across, as big as some “world record” ones… There were also millions and millions of jellyfish (they must have been spawning or whatever it is they do, but there were more of them than I have ever seen before), and there were some really weird dangly blobby things that looked like condoms, but which I think might be brown tunicates that got exposed at low tide.

Camping on Whaleboat Island proved possible, if slightly more amusing than most people might put up with. I though it was nice. You do need to be careful about where you put your boat overnight, since the beach disappears at high tide and you may find yourself with a tricky clamber along the cliffs to get back onto the water. Also, the “path” up the cliffs is pretty rough, and is more of a scramble than a real path. Once up on top, there are plenty of flat places to pitch a (small) tent, but you need to watch out for potentially ankle-breaking holes in among the moss-covered rocks. I didn’t explore too much of the upper part of the island because of those. I was on my own and didn’t much fancy the prospect of waiting out the night for someone to come and rescue me if I broke something.

But still, there was solitude and quiet, with just the sound of the wind whispering in the trees and the waves lapping on the beach to send me to sleep. What a great place. I miss it!

Along with about 160,000 other people, I’ve signed up for the Stanford Introduction to Artificial Intelligence course. My interest in this is two-fold: first for the material, which should be pretty cool if Peter Norvig’s book is anything to go by, but second for just how they (the two instructors plus presumably an army of TAs) are going to do this.

Perhaps I’m not a very good teacher in a lecture setting, but I have real trouble connecting even to a class of 90 or so students. Most effective teaching occurs one-on-one or in small groups in office hours. And most learning occurs when students are sitting quietly thinking about things themselves. How you make contact with and retain the attention of 85,000 students is a problem that boggles the mind. I’m very interested to see how it goes!

The first couple of modules are up on the website already, and it looks like the format of most of the course will be pretty nice and accessible. How assignments and exams will work, we’ll have to see. I really hope Stanford are supplying some additional resources to help with this project. It’s a very cool thing to be doing, and very exciting, but it has the potential to be a pretty high stress adventure for the instructors!

If AI doesn’t float your boat, Stanford are also running Intro to Machine Learning and Intro to Databases courses online as well.

Glanmore Lake, Beara Peninsula. On a holiday to Ireland some years ago.

Last time, we looked at the basics for a webcomic scraper. In this post, we’ll look at extending our code a little to incorporate some date handling and some simple XML processing.

The second version of the code is here.

Date based output directories

The first thing we’ll do is set things up so that the webcomic images we download are stored in a directory whose name is taken from today’s date. Date and time handling functions are in the Data.Time module, so we import that:

import Data.Time

Following our “does it always return the same value” heuristic, it seems pretty clear that a function to get today’s date has to live in the IO monad, since it wouldn’t be very useful for such a function to return the same value every time it’s called. The Date.Time module has a number of functions for dealing with dates and times and timezones, mostly modelled on the Unix C library functions for doing the same jobs. We can get a representation of the current date (the Day type is just a wrapper for an integer giving the modified Julian day) in the local timezone using the following code:

getToday :: IO Day
getToday = do
  tz <- getCurrentTimeZone
  tm <- getCurrentTime
  return (localDay $ utcToLocalTime tz tm)

Here, both getting the current time zone and getting the current time require IO actions (the current time zone can change from time to time if the machine where our code is running is moved, for instance), and obviously the current time changes with time… All of the functions getCurrentTimeZone, getCurrentTime, localDay and utcToLocalTime are defined in the Data.Time module, whose documentation you can read here.

Once we have a date, we’d like to turn it into a good name to use for a pathname. This a simple manipulation of the result of getToday, but because getToday is an IO action, our function to make the pathname returns an IO action too:

makeDateName :: String -> IO String
makeDateName base = do
  date <- getToday
  return (base ++ "/" ++ showGregorian date)

Here, the showGregorian function is a handy utility from Data.Time that formats a date in ISO YYYY-MM-DD format, and we pass in a “base directory” where all the webcomic download directories should go.

We can now modify the processAll function from last time. We’ll need to pass in a base directory name (parameter bd in the function below), plus a list of comics to get. We can determine the path to save the downloads to using makeDateName, and we then use the createDirectoryIfMissing function from System.Directory to make the relevant directory (the first True argument acts the same as the -p flag to mkdir, making parent directories as required), then change working directory to the new directory, then download the files:

processAll :: String -> [Comic] -> IO ()
processAll bd cs = do
  dateDirectory <- makeDateName bd
  createDirectoryIfMissing True dateDirectory
  setCurrentDirectory dateDirectory
  mapM_ writeImageToFile cs

As before, a lot of this looks just like what we would write in a traditional imperative language. The monadic structure of the code that the Haskell compiler generates from a do statement deals with all the details of routing the results of one step of the computation to the next in a way that preserves referential transparency.

An XML configuration file

Until now, we’ve been specifying the list of comics to read directly in our code. This isn’t very convenient, so it would be good to read the comic details from a configuration file. We’ll use a simple XML file that we’ll call comics.xml: an example is here.

Haskell has a number of very sophisticated libraries for dealing with XML documents, but sometimes these are slight overkill. If all you want to do is pull some information out of an XML file without too much fuss, then the TagSoup package is what you want. This has a nice simple interface for reading XML (or HTML) that may or may not be well-formed, from which you can extract the data items you need. We import the package as:

import Text.HTML.TagSoup

and we use it as shown in the getConfig function, which reads the contents of a configuration file and parses it using the TagSoup parseTags function. The result of this is a list of HTML/XML tags with attached attributes that you can process using Haskell’s usual list processing functions. Here, we first use filter to pick out the entry in the tag list having an open tag of baseDirectory, from which we extract the name attribute to use as our base directory. If that works, we pick out all the entries with an open tag of comic and process them with the makeComic function, which simply pulls the relevant items out of attributes in the comic tags.

getConfig :: FilePath -> IO (String, [Comic])
getConfig cfg_path = do
  fc <- readFile cfg_path
  let tags = parseTags fc
  let bdtags = filter (isTagOpenName "baseDirectory") tags
  case bdtags of
    [] -> return (error "baseDirectory field missing from comics.xml")
    otherwise -> do
      let bd = fromAttrib "name" $ head bdtags
      let cs = map makeComic $ filter (isTagOpenName "comic") tags
      return (bd, cs)
  where makeComic t = if n == "" || u == "" || r == "" || p == "" then
                         error "attribute missing in comic tag"
                         else Comic n u r p 
          where n = fromAttrib "name" t 
                u = fromAttrib "url" t
                r = fromAttrib "regex" t 
                p = fromAttrib "prefix" t

While it’s not up to more complex XML processing tasks that require walking and transforming the tree of entries in an XML file, for this type of application, TagSoup is just about perfect. It’s lightweight, easy to use, and interfaces with standard ways of working with lists in Haskell in a seamless way.

To make use of this, we just modify our main program to read the configuration information, which we pass right along to processAll.

main :: IO ()
main = do
  (bd, cs) <- getConfig "comics.xml"
  processAll bd cs

Simple!

In my day job, I work on the ecology of Mediterranean ecosystems in southern France, and if you work on a particular type of ecosystem, the first thing you need to know is where you can find your ecosystem! Seems like a simple problem, but it’s not.

How might we start? Go out and take a look? That’s fine, as far as it goes, but it’s labour-intensive and it can be tricky to compare manual observations from different time periods for changes in land cover. We’ll come back to that in a minute. What about aerial photos? They’re available for many regions of the globe, and they’ve been around a lot longer than satellite data. Here’s an aerial photo of a small region (about 8.5 km by 7 km) near an experimental station we have at Puéchabon in the Languedoc-Roussillon region of southern France, about 30 km from Montpellier.

IGN aerial photo

We can see a river, roads, some fields (easy to spot both because of their colour and their geometry), some areas that look less vegetated (the grey parts north of the fields), some parts that look more vegetated (mostly around the greyer parts). But can we tell which areas are predominantly covered by Quercus ilex (holm oak) woodlands? That’s a bit tougher. Detailed examination of high resolution images can be enough to identify vegetation assemblages, but there’s often just not enough information in a simple visible light photograph (in the usual digital rendering, just three spectral channels for red, green and blue) to pick out more subtle differences. And before the 1970s, most aerial photographs were black and white, which makes things even harder.

For some regions of the world, there are in fact pretty good inventories of the “go out and take a look” kind. For France, the Inventaire Forestier National (IFN) collects vegetation inventory information for forestry applications. Here’s a plot of their data for the same region as shown in the aerial photo:

IFN inventory data

There’s a pretty good correspondence between the units from the IFN data and the units that are visible on the aerial photograph, which lends some confidence to the classification that we see. There are three problems though. First, this data has been developed for forestry use. There are large white “unclassified” areas where there is no data, because there are no economically interesting trees there. For some ecological applications, that’s not much of a hindrance, but it can be a problem for others. Second, to understand what you’re seeing in a data set like this, you need to know quite a bit about the survey method used for the inventory and the statistical methods used to interpolate to a map view. For the IFN data, this is well documented, which makes the data relatively easy to use, but that level of documentation is more the exception than the rule. And finally, this data is difficult to collect: a ground-based survey of the entire country requires a lot of resources and so it can’t be done all that often. (For some reason, Spain does very well in this regard. They have really good forestry data collected on a regular basis across the whole of the country. Good for them!)

What about some combination of aerial photography and ground-based observations to serve as some sort of reference? This is a good approach in many cases: it allows for a more reasonable use of resources for surveying, and it can produce new data sets as new aerial photography comes in. I use two different data sets of this type, both produced in broadly similar ways, called CORINE (which covers the whole of Europe) and OCSOL (which covers only the Languedoc-Roussillon and PACA regions of southern France). Both of these data sets use ground-based observations with contemporaneous aerial photography to construct sets of rules for classifying land cover types from photographs. The rules tend to be quite complicated, relying on both basic spectral data (trees are green, water is blue, rock is grey, that sort of thing), geometric classification (fields tend to have square boundaries, rivers are roads are mostly linear features, and so on) and a combination of the two, possibly with other factors (greenhouses are squarish shiney things that often appear in regular groups). Along with some ancillary rules to control the geometry of the resulting data set (no near singular triangles, for example), this provides a system for going from aerial photographs of a region to a data set of polygons classified by land cover type. So how well does it work? Both CORINE and OSCOL use more or less the same classification scheme, with the following classes:

and here is what the data looks like for the same region as shown in the aerial photo and the IFN data above:

CORINE land cover

OCSOL land cover

Hmmm… Not so awesome. Land cover units in the data sets don’t bear a strong resemblance to the land cover units you can see in the aerial photograph, and they don’t correspond all that closely to the vegetation units in the IFN data (which is probably closer to the “truth”, since it’s survey-based everywhere, while the CORINE and OCSOL data uses rules calibrated in some parts of the region of application to determine land cover type in other parts of the region).

For a further comparison, we can think about what sort of remote sensing products we might try to use to determine the land cover type. With remote sensing data, we have a good chance of getting a regular time series of data (annual land cover maps from MODIS, for instance), which is harder to get via any other method. The downside? Spatial resolution is pretty terrible, and classification accuracy is patchy. Here’s the MODIS data for 2006 (same year as the CORINE data and the aerial photo) for the region we’re looking at:

MODIS land cover

The spatial resolution here is about 500 m, compared to about 25 m for the IFN data and whatever you want to believe for the CORINE and OCSOL vector data sets (their resolution is defined in terms of the minimum area of land cover feature that they include, rather than a regular grid-based resolution limit). It’s actually very unfair to compare MODIS data to data produced by land-based surveys, but we don’t really care too much about fair in this game: we just want good data.

Let’s look at a comparison between some of these data sets in a slightly more quantitative way. Eyeball is fine, but just how bad is the correspondence between these different data sets? Let’s compare CORINE with MODIS first. Gaze at this technicolour horror for a few seconds:

What this is showing is, in the vertical bar on the left, the overall classification across the whole of the Languedoc-Roussillon region, according to CORINE, and in the main part of the plot, how the portion of the land surface in each of those CORINE classes was classified according to the MODIS data. If we take a look at the two most common CORINE classes, broad-leaved and coniferous forest, we see that these are mostly, but not exclusively, classified as forest in the MODIS data, but there’s quite a lot of mixing of classifications between the different forest types. The same kind of phenomena is seen in the other classes: areas classified as urban by CORINE are about 50% classified as urban in the MODIS data and about 50% something else. One possible complaint about this comparison is of a resolution mismatch between CORINE and MODIS: the CORINE data is a vector data set with pretty high spatial resolution, while the MODIS data is on a 500 m grid (more or less). We can do a similar comparison between CORINE and OCSOL though, both of which are vector data sets, though OCSOL has a smaller minimum area for land cover units it will consider. Here’s the same sort of comparison plot as for the CORINE/MODIS comparison:

The comparison is a little easier here, since CORINE and OCSOL use more or less the same classification so we can use the same colour scales. That means that blocks in the main part of the plot that are the same colour as the bars in the left hand part of the plot are classified consistently between the two data sets. Even though these methods ostensibly use similar methods to classify land cover, the correspondence between the two data sets is not great. For instance, of the area that is classified as coniferous forest by OCSOL, only about 40% is also classified as coniferous forest by CORINE.

So, what do we do? It’s difficult to come up with any concrete recommendations that aren’t very tightly tied to specific applications, since everyone’s detailed needs are different. For my work, I’m most interested in assemblages including Quercus ilex, Quercus pubescens, Pinus halepensis and shrub species that can be bundled up into a generic “garrigue” classification. For the individual tree species, the IFN data is very good. For identifying garrigue sites, it’s a bit less good, so I use OCSOL for that. The same goes for grasslands, when I need them.

It’s a bit daunting that what would appear to be the simplest possible remote sensing/geostatistical data question, that is “What’s there on the ground?” turns out to be so difficult to answer. The geography and history of this part of France does make the disposition of land cover particularly complex, but something like this problem is going to be encountered more or less anywhere in the world.

day-job remote-sensing science

The first week of the online Stanford AI class has gone by. The format of the presentation is pretty good, with lots of short lecture videos chained together with little quizzes embedded. It’s really pretty neat. There are some minor errors in some of the lectures, which is normal, though in this kind of online setting, it’s more difficult for the keener students to correct the lecturer as the lecture goes along, so some people are likely to spend a bit of time confused because of that. Some people have complained on Reddit about the relatively superficial level of the coverage of the lectures, but as far as I’m concerned, that’s what the book (and papers) are for. The course is called “Introduction to Artificial Intelligence”, after all.

Motivation: Take me home, Romanian roads

The two things we covered this week were a very quick introduction to the field, then some stuff about search. Although there is a big pile of code available for download associated with Russell & Norvig’s AI book which would have made playing with some of this stuff easier, I decided to write some code of my own in Haskell, mostly for my own amusement. I wanted to implement A* search, with a clean interface for setting up problems.

The example used in the lectures, of finding driving routes on a simple road map of Romania, is a good simple problem to look at. I wanted the definition of this problem to look something like this:

import AStar

import Data.Maybe
import Data.List

data Rom = A | B | C | D | E | F | G | H | I | L | M | 
           N | O | P | R | S | T | U | V | Z
         deriving (Eq, Ord, Show)

initial :: Rom
initial = A

goal :: Rom -> Bool
goal = (== B)

distances :: [((Rom, Rom), Int)]
distances = [((A, S), 140), ((A, T), 118), ((A, Z), 75), ((B, F), 211),
             ((B, G), 90), ((B, P), 101), ((B, U), 85), ((C, D), 120), 
             ((C, P), 138), ((C, R), 146), ((D, M), 75), ((E, H), 86), 
             ((F, S), 99), ((H, U), 98), ((I, N), 87), ((I, V), 92), 
             ((L, M), 70), ((L, T), 111), ((O, S), 151), ((O, Z), 71),
             ((P, R), 97), ((R, S), 80), ((U, V), 142)]

euclidean :: [(Rom, Int)]
euclidean = [(A, 366), (B, 0), (C, 160), (D, 242), (E, 161), (F, 176), 
             (G, 77), (H, 151), (I, 226), (L, 244), (M, 241), (N, 234), 
             (O, 380), (P, 100), (R, 193), (S, 253), (T, 329), (U, 80), 
             (V, 199), (Z, 374)]

instance SearchState Rom where
  actions x = map const $ nub $ filter (/= x) $ (map fst ps) ++ (map snd ps)
    where ps = map fst $ filter (\((a, b), _) -> a == x || b == x) distances

  stepCost x y 
    | x < y = fromJust $ lookup (x, y) distances
    | otherwise = fromJust $ lookup (y, x) distances

heuristic :: Rom -> Int
heuristic x = fromJust $ lookup x euclidean

Here, we define a data type Rom to represent Romanian cities, define an initial state and a goal function, set up a couple of data structures to record city-to-city road distances and Euclidean distances of cities from Bucharest (our goal), then define Rom as an instance of the type class SearchState which includes code to find the possible actions we can take from any state (i.e. the cities we can travel to directly from a given city) and the costs (i.e. distance) associated with going from one state to another. Finally, we define a heuristic, which just returns the Euclidean distance from a given city to the goal city. Given that definition, I wanted to be able to type something like

astar initial goal heuristic

to run an A* search for the goal state, starting at the initial state and using the given heuristic. We might also want to look at astar initial goal (const 0) to do a uniform cost search. I’d like to both get a result and a count of the number of search tree nodes expanded (which will allow me to see how much of a difference different heuristics make to the efficiency of the search).

A* implementation

To make this work, the AStar module is defined as follows. It’s basically one type class, some supporting scaffolding, and the astar search function. We need a few data structures to implement A* search efficiently, so we import, in particular, Data.Set for sets and Data.PSQueue for priority search queues (priority queues that also support fast membership testing):

module AStar where

import Data.Maybe
import Data.List
import Data.Ord
import qualified Data.Set as S
import qualified Data.PSQueue as PS
import Debug.Trace

The SearchState type class is defined as:

class SearchState a where
  actions :: a -> [a -> a]
  pathCost :: [a] -> Int
  stepCost :: a -> a -> Int
  stepCost _ _ = 1
  pathCost sts = sum $ zipWith stepCost (tail sts) (init sts)

First, for a given search state, we want to find the actions that we can perform from that state. The actions method of the type class does this, returning a list of functions that will transform one state to another — we can then just apply these functions to a state to get the result of any action. We also want to be able to calculate the costs of paths composed of states, either in one go, from a list of states, using pathCost, or step by step, which is what stepCost does. We define defaults for stepCost (each state change has a cost of 1) and pathCost (the cost of a path through a list of states is just the sum of the individual steps).

To maintain a record of the search tree, we need a node type, holding a state, possibly a parent (Nothing if this is the initial node) and the cost at this node. Note that states can nominally appear in multiple nodes if a state is reached via multiple paths in the search graph (in fact, in the search function below, we only keep the best path to any particular state, so at any time, at most one node is associated with any state). We make our nodes an instance of Ord so that we can put them into data structures that require comparison. We set things up so that the comparison is done on the cost of the nodes.

data NodeInfo a = NodeInfo { state :: a,
                             parent :: Maybe (NodeInfo a), 
                             cost :: Int } deriving (Eq, Show)

instance Eq a => Ord (NodeInfo a) where
  compare = comparing cost

The main search function is a little complicated, but it has 7 main steps:

1. Initialise the search frontier with a node for the initial state (with no parent and zero cost).
2. If the goal has been reached, extract the path from the initial to the goal node from the search tree and return it, along with a count of the number of search tree nodes expanded to find the solution.
3. As long as there are nodes in the frontier, pick the lowest cost node for expansion (the frontier is represented using a priority search queue, so this minimum cost extraction is $O(logn)$ in time).
4. Add the node to be expanded to the “explored” set.
5. Determine the actions from the state of the node to be expanded, the result states of those actions and the costs of the possible result states (here, “cost” means the known path cost to the state of the node being expanded, plus the heuristic cost estimate for the result state).
6. Combine the existing frontier (minus the node being expanded) with the new states: any states not already in the frontier are added, and states that already exist in the frontier have their paths and costs updated if we have found a better path to the state than the one already recorded in the search tree.
7. Repeat from step 3.

Here’s a Haskell implementation:

astar :: (Show a, Eq a, SearchState a, Ord a) => 
         a -> (a -> Bool) -> Maybe [a]
astar init goalTest = helper initFrontier S.empty
  where -- Initial frontier contains just the single initial node (no
        -- parent, zero cost).
        initFrontier = PS.singleton init (NodeInfo init Nothing 0)

        -- Tail recursive helper.
        helper fr ex
          | PS.null fr = Nothing   -- Fail if the frontier is empty.
          | otherwise = if (goalTest s) 
                        then Just (reverse $ res s p) 
                        else helper fr'' ex'
            where -- Extract next state to expand and associated
                  -- information.  We produce trace output to show the
                  -- states on the frontier.
                  Just (s PS.:-> ni@(NodeInfo _ p c), fr') =
                    PS.minView (traceShow (PS.keys fr) fr)

                  -- Add to explored set.
                  ex' = S.insert s ex

                  -- Determine possible actions from this state and
                  -- resulting state, filtering out any in the
                  -- explored set.
                  rs = filter (\r -> not (r `S.member` ex')) $ 
                       map (result s) (actions s)

                  -- Calculate full costs of possible result states.
                  cs = map (\r -> c + stepCost s r + heuristic r) rs

                  -- Find which of the new states are already in the
                  -- queue.
                  exis = map (\n -> PS.lookup n fr) rs

                  -- Combine the existing states in the queue and the
                  -- new ones we've found.  Only ever keep one entry
                  -- in the queue for each state, the one with the
                  -- smallest cost.
                  toadd = catMaybes $ zipWith3 combine rs cs exis
                  combine r c Nothing = Just (r, NodeInfo r (Just ni) c)
                  combine r c (Just (NodeInfo _ _ c'))
                    | c < c' = Just (r, NodeInfo r (Just ni) c)
                    | otherwise = Nothing

                  -- Insert new states into new frontier.
                  fr'' = foldl (\q (k, p) -> PS.insert k p q) fr' toadd

                  -- Build the result by following the parent links
                  -- through the search graph.
                  res n Nothing = [init]
                  res n (Just p) = n : res (state p) (parent p)

This is a bit more verbose than the implementations in the Data.Graph module, but it works fine and is relatively efficient. Here’s what we get for the Romanian roads problem:

*Main> astar initial goal (const 0)
[A]
[S,T,Z]
[O,S,T]
[L,O,S]
[F,L,O,R]
[F,L,R]
[C,F,L,P]
[C,F,M,P]
[B,C,M,P]
[B,C,D,P]
[B,C,D]
[B,D]
[B]
Just ([A,S,R,P,B],13)
*Main> astar initial goal heuristic
[A]
[S,T,Z]
[F,O,R,T,Z]
[C,F,O,P,T,Z]
[B,C,O,P,T,Z]
[B,C,O,T,Z]
Just ([A,S,R,P,B],6)
*Main> 

First, we try the search using uniform cost search (i.e. a heuristic function of (const 0)). This gives us the right answer (as reported in the lectures), and expands 13 nodes in the search tree to find the solution. Then, using the Euclidean distance as a heuristic for the actual distance by road, we get the same answer (guaranteed since our heuristic is admissible, i.e. optimistic) by only expanding 6 search tree nodes. You can look at the order in which the nodes in the search tree are expanded and you’ll see that they correspond to the discussion in the lecture.

Homework problem

Let’s look at how to set up a couple of other problems. First, one of the homework problems dealt with a very simple setting: a 4 × 6 grid where we need to find a route from the top left to the bottom right moving only horizontally or vertically at each step, with a given heuristic function at each grid cell to estimate the remaining distance to the goal cell (numbers in the grid cells here show the heuristic value and the circles mark the initial and goal states):

In our framework, this problem looks like this:

import AStar

data HWState = HWState Int Int
             deriving (Eq, Ord)

instance Show HWState where
  show (HWState r c) = ("ABCD" !! (r-1)) : (show c)

instance SearchState HWState where
  actions (HWState r c) =
    map (const . uncurry HWState) $
    filter valid [ (r-1, c), (r+1, c), (r, c-1), (r, c+1) ]
      where valid (r, c) = r >= 1 && r <= 4 && c >= 1 && c <= 6

initial :: HWState
initial = HWState 1 1

goal :: HWState -> Bool
goal (HWState 4 6) = True
goal (HWState _ _) = False

heuristic :: HWState -> Int
heuristic (HWState 4 _) = 1
heuristic (HWState _ 6) = 1
heuristic (HWState 3 _) = 2
heuristic (HWState _ 5) = 2
heuristic (HWState 2 _) = 3
heuristic (HWState _ 4) = 3
heuristic (HWState _ _) = 4

The problem state, defined by the HWState type, just gives the row and column where we are at this step, we define a Show instance for the state so we can see what’s going on, we define an initial state (top left) and a function to test whether a state is the goal state (easy here: are we at the bottom right?). Then we get to the meat of the code, which is an instance definition for the SearchState class (defined in the AStar module). Here, this just has a method to find out what possible actions we can take from a given state (moving to any of the adjacent cells, module edge effects). We use the default path cost definition from the SearchState type class. Finally, we have a function to calculate the suggested heuristic for a given state. This all works as expected (I’ve taken out the traceShow call, so we don’t see the states that are expanded at each step), although here the heuristic doesn’t seem to help at all:

*Main> astar initial goal (const 0)
Just ([A1,A2,A3,A4,A5,A6,B6,C6,D6],24)
*Main> astar initial goal heuristic
Just ([A1,A2,A3,A4,A5,A6,B6,C6,D6],24)
*Main> 

Sliding blocks puzzles

Let’s try a more substantial example. The sliding blocks puzzles of the 15-Puzzle kind are a classic search problem. Things are quite a bit more complicated here than in the two previous examples. The boards for these puzzles look like this (for the 3 × 3 “8-puzzle” version on the left and the 4 × 4 “15-puzzle” version on the right):

First we need some imports (we’ll use a map to help represent the board state, and we’ll need some random numbers to generate scrambled initial board states).

import Prelude hiding (Left, Right)
import qualified Data.Map as M
import System.Random

import AStar

Next, we define our board state. We need to know how big the board is (typically, we’ll consider the 3 × 3 “8-puzzle” and the 4 × 4 “15-puzzle”), and what tiles are in each position, which we represent as a map from a (row, column) tuple to an integer tile value. For convenience, we also record the row and column where the blank is.

type Board = M.Map (Int, Int) Int
data BoardState = BoardState { size :: Int,
                               board :: Board,
                               bpos :: (Int, Int) } deriving (Eq, Ord)

We often need to generate the list of all coordinate pairs for the board, so let’s have a helper function that does that for a given board size.

coords :: Int -> [(Int, Int)]
coords n = [(r, c) | r <- [1..n], c <- [1..n]]

Moves in these puzzles are most easily thought of in terms of moving the blank, rather than moving a tile. Any single move just moves the blank by one square, either up, down, left or right. Here, if we try to apply an invalid move to a board state (e.g. moving the blank off the left hand side of the board), we define the result to be an unchanged board. This is convenient when we come to generate scrambled boards, but it’s not a problem in the A* search, since we will never try to apply invalid moves.

data MoveDir = Up | Down | Left | Right

move :: MoveDir -> BoardState -> BoardState
move dir (BoardState n m bpos@(brow, bcol)) = BoardState n m'' bpos'
  where m'' = M.insert bpos' 0 m'
        m' = M.insert bpos (m M.! bpos') m
        bpos' = moveBlank dir
        moveBlank Up = if (brow > 1) then (brow-1, bcol) else (brow, bcol)
        moveBlank Down = if (brow < n) then (brow+1, bcol) else (brow, bcol)
        moveBlank Left = if (bcol > 1) then (brow, bcol-1) else (brow, bcol)
        moveBlank Right = if (bcol < n) then (brow, bcol+1) else (brow, bcol)

We add a Show instance for the board state so that we can see what’s going on.

instance Show BoardState where
  show (BoardState n m _) = 
    unlines $ map concat $ chunks n $ map (format . (m M.!)) (coords n)
      where format x
              | x == 0 = " XX"
              | 1 <= x && x <= 9 = "  " ++ show x
              | otherwise = " " ++ show x
            chunks _ [] = []
            chunks n xs = (take n xs) : chunks n (drop n xs)

Then, the SearchState instance is a bit of an anticlimax, since all we need to do is to generate the possible valid moves given a board state. We use the default implementation of path costs from SearchState, with each moving having the same cost.

instance SearchState BoardState where
  actions (BoardState n m (brow, bcol)) = map move (lr ++ ud)
    where lr
            | bcol == 1 = [Right]
            | bcol == n = [Left]
            | otherwise = [Left, Right]
          ud
            | brow == 1 = [Down]
            | brow == n = [Up]
            | otherwise = [Up, Down]

The “initial” state (we’ll actually start our searches from a scrambled state derived from this):

initial :: Int -> BoardState
initial n = BoardState n 
            (M.fromList $ zip (coords n) ([1..n^2-1] ++ [0])) (n, n)

and the goal state is the same:

goal :: BoardState -> Bool
goal (BoardState n m _) = map (m M.!) (coords n) == ([1..n^2-1] ++ [0])

There are two possible heuristics that were mentioned in the lectures. The first just counts the number of tiles that are not in the correct position:

heuristic1 :: BoardState -> Int
heuristic1 (BoardState n m _) = sum $ zipWith 
                                (\x y -> if x /= y then 1 else 0) 
                                (map (m M.!) (init $ coords n)) [1..n^2-1]

The second measures the Manhattan distance of each tile from its correct position and uses the sum of these distances for all the tiles as the heuristic value:

heuristic2 :: BoardState -> Int
heuristic2 (BoardState n m _) = M.foldrWithKey folder 0 m
  where folder (r, c) v s = s + if (v > 0) then dist (r, c) (goodPos v) else 0
        dist (r1, c1) (r2, c2) = abs (r1 - r2) + abs (c1 - c2)
        goodPos v = ((v - 1) `div` n + 1, (v - 1) `rem` n + 1)

Finally, we need to be able to generate scrambled starting states. This is most easily done by generating a random list of moves and applying them to the initial board state. Since we ignore invalid moves in the move function above, this all works fine.

scramble :: Int -> BoardState -> IO BoardState
scramble n b = do
  g <- getStdGen
  return $ foldl (flip move) b (makeMoves g)
    where makeMoves gen = map ([Up, Down, Left, Right] !!) 
                          (take n $ randomRs (0,3) gen :: [Int])

And then everything works as you might expect:

*Main> b0 <- scramble 20 (initial 3)
*Main> b0
  5  1 XX
  4  2  3
  7  8  6

*Main> heuristic2 b0
6
*Main> astar b0 goal heuristic2
Just ([  5  1 XX
  4  2  3
  7  8  6
,  5  1  3
<< lines skipped >>
  7 XX  8
,  1  2  3
  4  5  6
  7  8 XX
],54)
*Main> 

Here, we generate a scrambled state for the 8-puzzle from the initial “perfect” state by applying 20 randomly selected moves. We display the scrambled initial state, the value of the Manhattan distance heuristic for this state (which gives us some idea of how much effort we’ll have to expend to find a solution), then run the A* search using the Manhattan distance heuristic. The result is a list of states (some of the output snipped) and a count of the number of search tree nodes expanded (54, in this case).

In a post in a day or two, I’ll look a little at how much of a difference using heuristics really makes in this problem…

by China Miéville

Can you unsee? Can you unhear? If I ask you not to think of a tiny green rhinoceros wearing a straw hat, can you do it?

No? Then you might not be welcome in Besźel, or Ul Qoma. Two cities, physically entwined, psychically distinct, located at the frontier between Europe and Asia. No-one knows how these twin cities, estranged at birth, came to be. No-one knows if there are only two. The shadowy interstitial force of Breach certainly exists in the gaps between the cities, the spaces unclaimed by the citizens of either Besźel or Ul Qoma. So why not a hidden third city? Or a fourth, a fifth? An infinity of fractally entangled citizenries, passing each other in their shared unshared streets, eyes sliding across the things that should not be seen, ears filtering out unbidden sounds from other countries?

And if you have the power to conjure such a place? What do you do with it? If you’re China Miéville, you use it as the setting for a police procedural, with a laundry list of well-worn tropes: we start with a murder scene, we have the mismatched cops from different jurisdictions forced to work together, we have mystery and slow unfolding. As a stylistic framework for a slow reveal of Besźel/Ul Qoma, it works well. The plot carries the process of discovery along, and Miéville has a light touch with the details of life in the conjoined cities.

There’s a pleasing thread of restraint all the way through the book. “Restraint” is not usually a word one might associate with China Miéville: he’s not one to shy away from the grotesque, the bizarre and the weird. Here though, it sometimes feels like he’s teasing himself and his readers, presenting opportunities to fly off into the wildest flights of fantasy but, at every step, adroitly turning away to a mundane explanation.

The only truly fantastic element in The City & The City is the stability of Besźel and Ul Qoma, even in the face of encounters with the wider world. Given that, everything else follows logically. No angels, no demons, no hidden dimensions, no magic. Just a social contract construed under different parameters than our own.

Try unseeing: walk the streets of your town and tell yourself that you will not see anyone wearing red. It’s hard. But we know that unseeing is possible. We do it every day: the disadvantaged among us are easy to ignore, our eyes slide past theirs, we easily unhear their requests for help. Perhaps with training, we could live in Besźel. Or Ul Qoma, for that matter.

by Roger S. Bivand, Edzer J. Pebesma & Virgilio Gómez-Rubio

I recently had to do a bunch of geostatistical analysis on some climate data (to be specific, using universal kriging to interpolate a time series of solar radiation data covering the region I’m working on to a different grid). I started off trying to use the geostatistical analysis toolbox in ArcGIS, which works fine as far as it goes, but seems to be very difficult indeed to access via ArcGIS’s Python scripting interface. Since I had 36 years of daily data to process, doing it by hand was not an option.

I did the job using R, eventually, more by trial and error than through any great process of inspiration. I’d never done much geostatistical processing with R before, although I use it quite a lot for “normal” statistics and data analysis. My little voyage through the land of R-GeoStatistica was quite an eye opener. You can really get a lot done this way. If you know what you’re doing. Which is where this excellent little book comes in.

Part of the Use R! series published by Springer, Applied Spatial Data Analysis With R is written by the authors of some of the main R geostatistical packages, in particular the sp package which is used for the representation of geostatistical data. The book is divided into two main parts, the first dealing with general issues related to the representation of spatial data, and the second part divided into three sections dealing with point data, interpolation and geostatistics, and areal data. The point data and areal data sections seem to be well done, but they’re about subjects I’m not too familiar with, and that I don’t really need to know about for my current work. I mostly just skimmed them, but they seem comprehensive and well-explained. The interpolation and geostatistics section was what I was really here to see, and it’s good.

If I’d had this book before I started, I could easily have saved myself a day or two of scrabbling around not quite knowing which packages to use or how to set things up. The examples in the book are realistic, comprehensive and clear, and they serve as a very good basis to build from. Applied Spatial Data Analysis With R is never going to be ranked among the greats of scientific literature, but that’s not its aim — its aim is to explain how to do spatial data analysis in R, and it does that perfectly. Highly recommended.

Winnie is a dog, probably half beagle, half something else. She lives with us now, but her life wasn’t always wall-to-wall belly rubs, salmon for breakfast and as much love as she can stand. We adopted her from the animal shelter where we work as volunteers on September 4, and for 3–4 months before that, she’d been cowering at the back of her cage, frightened even to show her little nose to the people outside.

We think Winnie was raised as a hunting dog, locked up in a kennel with other dogs and no humans to bond with, since she has a bad case of kennel syndrome: she’s scared of more or less everything except other dogs, and now, us. Loud or just sudden noises, cars, bikes, trams, other people, big boxes (no idea why), and so on. All send her into a panic. We think that she probably wasn’t aggressive and confident enough to satisfy the assholes who had her before, so they abandoned her.

To add injury to insult, when she was abandoned, her previous “family” decided that, in order to avoid any chance of being identified as the owners of a stray dog, they would remove Winnie’s ID tattoo. The tattoo was on her ear. So they cut it off. She was found wandering in the village of one of the other volunteers, covered in ticks, with an oozing fresh wound where her ear used to be. It’s all healed up now, and she just looks kind of funny, like she’s continually cocking her head to one side as if to say “Who? Me?”.

Anyway, Winnie is a little cutie, and adopting her has made our lives much richer, if rather dirtier and less well rested. She loves nothing better than running through the dewy meadow in the local park in the mornings, and it’s even better if there’s another crazy dog to chase. She sleeps on our bed, howls at the old lady who feeds the pigeons next to the dog park, loves kisses and crazy play, and is just starting to realise that not everything in the world of humans leads to pain and unpleasantness. She’s even learning that it can be fun to do what the funny two-leggers she lives with ask her to do.

If you ever feel the need for a dog in your life, head to your local shelter. You won’t regret it.

In an earlier post, I presented some code for solving tile puzzles using A* search. In the lectures, Peter Norvig talked about two different heuristics for these puzzles, one simply counting the number of misplaced tiles and the other giving the total Manhattan distance of all misplaced tiles to their correct positions. It’s interesting to find out how much of a difference using these heuristics really makes.

State space size

First question: how many states do these problems actually have? For an $n\times n$ puzzle, we generate an arbitrary state by placing $n^2-1$ tiles into $n^2$ positions, which means that there are, on the face of it, $(n^2)!$ possible states. However, states have a kind of parity, which means that the state space is partitioned into two mutually exclusive halves. Using only legal moves of the blank position in the puzzle, there are only $(n^2)!/2$ positions accessible from the “perfect” initial state.

For a $3\times 3$ puzzle, $(n^2)!/2=181,440$ and for a $4\times 4$ puzzle, $(n^2)!/2=10,461,394,944,000\approx {10}^{13}$, which is a big number. In the $3\times 3$ case, we can just about imagine exploring a reasonable fraction of the full state space to find the solution. For the $4\times 4$ case, we have no hope of exploring more than a tiny fraction of the state space. The number of states that any particular algorithm needs to expand to find a solution will obviously depend on the starting state: some states are only a move or two away from the goal state, so solutions are easy to find, while others are “a long way away” in state space (whatever the metric is that we might define).

What we do

Here’s what we do: we generate a whole bunch of random start states, each a given number of “scrambling” moves from the perfect state. We try to solve each of those using uniform cost search (Heuristic 0, i.e. a heuristic of zero for all states), the misplaced tile count heuristic (which we’ll call Heuristic 1) and the total Manhattan distance heuristic (Heuristic 2). We record the number of search tree nodes that need to be explored to find a solution for each case (I set a ceiling of 10000 nodes just so I can finish a reasonable number of computations on my aging laptop). What we end up with is a set of tuples $(n,s,h,c)$, where $n$ is the problem size (3 or 4), $s$ is the number of “scrambling” moves (between 10 and 100), $h$ is the heuristic (0, 1 or 2) and $c$ is the cost, i.e. the number of nodes expanded. We can then look at plots of $c$ for different combinations of $n$, $s$ and $h$. Below, I’ll show results from all the heuristics and “scrambling distance” together, with one plot for each problem size.

Results

Let’s look at the $3\times 3$ case first. This plot shows the number of nodes expanded for a range of scrambling moves for all three heuristics. For each heuristic and each scrambling move count, the box-and-whisker plots show the results of 1000 runs from random initial states:

Note the log scale for the number of nodes expanded! Using heuristics really does seem to make a difference. And the heuristic that we choose also makes quite a difference — Heuristic 2 is appreciably better than Heuristic 1. The story is the same for the $4\times 4$ case. Here, I wasn’t patient enough to wait for all the uniform cost search results, since they take a long time. I’ve also limited the number of nodes expanded to 10,000, so things finish up in a sensible timeframe.

One thing that’s clear on these plots is that there’s a very big spread in the cost of starting from different states the same “scrambling distance” from the goal state. That’s expected: consider two states at a “scrambling distance” of 3 from the initial state, one of which consists of moving the blank left, right, left and the other left, up, left. The first state has two moves that are inverses of one another, so the resulting state is only one move from the goal state, while the second one requires three moves to get back to the goal state. So perhaps “scrambling distance” isn’t the best metric to use on the state space. What other metrics could we use? Well, we have two ready-made, in the form of the heuristics. As well as perhaps giving us a better view of the performance of the heuristics in terms of actually helping to solve the puzzles, plotting the node expansion costs as a function of the heuristic values will also show us how well the heuristics do at measuring how hard it is to solve from a given initial state. Here are results for the 8-puzzle:

and the for 15-puzzle:

Although the spread at the top range of the costs is artificially compressed because I terminate any solves that expand more than 10,000 nodes, it seems pretty clear from these plots that both heuristics do a reasonable job of approximating the likely solution cost for a given initial configuration of the board. Heuristic 2 divides states up more finely than Heuristic 1, and is definitely a more accurate heuristic, in terms of being less optimistic while still admissible.

Conclusions

The conclusions are pretty clear: A* search with a good heuristic works much better on problems with realistically sized search spaces than a simple uniform cost search. We end up needing to explore a far smaller portion of search space to find solutions. The key thing here is that A* search is guaranteed to find the optimal solution, provided that the heuristic we use is optimistic. There is a partial order on admissible heuristics for a given problem, where we can produce heuristics that are more and more accurate, i.e. less and less optimistic, and thus produce a better and better assessment of how far a particular state is from the goal.

A couple of points about this example. First, we’re solving these tile puzzles in kind of a dumb way. We could introduce a much more structured way of thinking about permissible moves (we can think of “circulating” sets of tiles as a single move, for instance, which immediately allows us to jump around in state space in more interesting ways). Second, A* search runs out of memory sometimes. And in this problem, it runs out of memory unpredictably: given two initial states, both generated from the goal state by 50 scrambling moves, one may take only 500 node expansions to solve, the other may take more than 10,000. Given unlimited memory, A* search will find the solution eventually, but we keep hold of a lot of “junk” states in the search space that are really of no use at all. There are more sophisticated algorithms for this sort of search that can run in limited memory, by discarding some of those redundant states.

Anyway, the take home lesson is that heuristics are important for search!

Sundown on Wallace Island, Gulf Islands National Park. Thanksgiving weekend, 2009.

As blogging software, I use the really rather nice Hakyll, by Jasper Van der Jeugt. This is a static website generator written in Haskell, and it’s a great solution for smaller blogs or personal websites.

Hakyll works just fine out of the box, but the blog setup I wanted was a bit different from what I’ve seen other people do with it, so a bit of hacking was required. I wanted to share some of the things I’ve done, since they might be of interest to other Hakyllers (Hakyllites?). All the code is available on Github.

I was going to write some sort of extensive description of what I’d done, but it makes more sense just to point at some places in the code. There are three source files, blog.hs which has all the main Hakyll-required definitions, Overrides.hs which contains definitions of some functions from the Hakyll libraries that I override, and TikZ.hs which contains my slightly rough around the edges code for converting images represented as TikZ code into SVG for inclusion in web pages.

Here are the “highlights”:

• blog.hs lines 109–115, and functions makeIndexPages, makeIndexPage and indexNavLink: most of the material for splitting articles across index pages, generating the index pages and the links between them. The only tricky bit here was figuring out just how to get the Hakyll metacompilation feature working right.
• blog.hs, function addTeaser: this is used to generate the article extract that appears on the index pages, and is a slight elaboration of a suggestion of Jasper’s; the only nasty is fixing up the URLs for any images that appear in the teaser so that they refer to the right directory under the posts hierarchy.
• various places in blog.hs and Overrides.hs: code to allow me to use a more “Wordpressy” article naming convention, i.e. yyyy/mm/dd/title instead of Hakyll’s default yyyy-mm-dd-title.ext. I like to keep image files and other resources next to the text of articles, and this helps me to do that.
• everything in TikZ.hs: a while ago, when I was trying to write my own blog framework in Haskell, I had the idea of using TikZ to represent inline images in articles, and I cobbled up something to render TikZ code into SVG for embedding. It seemed like a good idea at the time, but I’m now undecided about it, mostly because I’ve not yet been able to get SVG font styling working properly. Getting the size of SVGs right in Chrome turns out to be a bit of a headache too. Still, the code is there: it uses htlatex to do the conversion, and it seems to work.

I’m not exactly the world’s best Haskell programmer, and some of this stuff could do with some polishing. The processTikZs function in TikZ.hs in particular is a bit grim, including a use of the pattern (id &&& f) >>> (arr (\(x, fx) -> blah x fx)) that I seemed to find myself using often enough that it looked like there ought to be a standard combinator for it, but for which I couldn’t find one in Control.Arrow. It’s the first time I’ve used arrows though, so I was just happy to end up with something that worked!

For what it’s worth, all the code is free for reuse.

“Pour avoir une voiture, sans avoir une voiture”

To have a car, without having a car

Car sharing/car club schemes are pretty common in European cities, but the one here in the south of France is the best I’ve heard of. In Montpellier, it goes by the name of Modulauto. We pay a (small) monthly fee, can book cars from a number of parking locations online, pay an hourly charge and charge per kilometre travelled. Fuel is included in the price. One location is about 5 minutes walk from our house, more or less all of them are right at tram stops, and the people who thought this out have done a really good job of locating the cars in spots where it’s easy to navigate to and from (for North American readers, this is a non-trivial accomplishment in European cities that grew up during eras when bullock-drawn carts were high technology and have road plans of a similar vintage).

We have an electronic card to unlock the car (only enabled when you’ve booked the car), the keys are inside, there’s a Total charge card to refuel the thing, and a very helpful call centre when things go wrong. You can book for as short or long a time as you like, and you can arrange special deals at the office in the centre of town for longer rentals over quiet periods. It’s generally more economical than renting a car for periods of less than a few days.

It’s actually kind of shocking that it works so well. Going to be a few minutes late bringing the car back? Give the office a call, and they’ll make sure you don’t pay any late charges (they’re helpful! and efficient! in France!). Had to refuel somewhere other than Total? Just drop the receipt into the office and you’ll get a refund on your account in a few days.

More or less every time we use the system, we comment about how good it is. I am regularly left reeling by the futuristic efficiency of it all. We may not have teleporters and robot servants yet, but we can have cars without having cars!

The second week of the online Stanford AI class was about probability, reasoning when there is uncertainty and, more specifically Bayes networks. I don’t much want to talk about the theory behind these things here since that’s covered in a lot of detail on the linked Wikipedia page, in the lectures and in AIMA.

Instead, I want to present a mildly amusing, although not very efficient, Haskell implementation of Bayes networks. It’s often said that if you know the full joint probability distribution function (PDF) for a system, you can calculate any marginal or conditional probabilities you want. Well, Haskell is a functional programming language, a PDF is a function, so can we represent the PDF explicitly and use it for calculating? The answer, of course, is yes.

The idea here is to implement something that will allow us to write the two-test cancer problem from the lectures as:

import BayesNet

data Event = C | T1 | T2 deriving (Eq, Ord, Show)

bn = fromList [(C, [], [0.01]),
               (T1, [C], [0.9,0.2]),
               (T2, [C], [0.9,0.2])]

where the entries in the list passed to the BayesNet.fromList function are triples giving the label for a node in the network, a list of other nodes that it is conditional on (i.e. nodes at the other end of incoming edges) and the probability table for the node. If a node $A$ is conditional on nodes $B$ and $C$, the probability table entries are stored in the order $P(A\mid +B,+C)$, $P(A\mid +B,-C)$, $P(A\mid -B,+C)$, $P(A\mid -B,-C)$. Given this definition, we can then calculate $P(C\mid T1,T2)$ as

p_3_20 = prob bn C [(T1,True),(T2,True)]

and $P(C\mid T1,\neg T2)$ as

p_3_21 = prob bn C [(T1,True),(T2,False)]

Data structures

A Bayesian network is basically just a directed acyclic graph with probability tables of the appropriate size associated with each node. In my Haskell code, I treat Bayesian networks as an abstract data type, exporting only a way to construct a network from a simple list-based description, and functions to calculate probability values given a network. (You can imagine lots of other possible things you might want to do, including updating networks, calculating conditional independence relationships, and so on, but here, we’re going to concentrate on the basics.)

module BayesNet(fromList, prob, jointProb, jointProbTF) where

import qualified Data.Map as M
import Data.List

The internal representation of a node in the network is as a list of nodes on which this node is conditional, and a probability table. Nodes are parameterised by type e (“event”) and the type of probability values is parameterised too, by type p, so that we can use, for instance, rational values if we want. (I’ll show an example of this later on, but basically, we just want to impose the type class constraint Fractional p everywhere to make sure that the probability values that get passed in are numbers with a fractional part.)

data BayesNode e p = BayesNode { cond :: [e], ps :: [p] } 
                   deriving (Eq, Ord, Show)

A Bayesian network is then just a map from node labels to nodes.

newtype BayesNet e p = BayesNet (M.Map e (BayesNode e p)) 
                     deriving (Eq, Ord, Show)

We can generate Bayesian network values from a simple list representation, which is a list of nodes of the form:

  (event, conditions, probabilities)

where:

• event is a label for the node (of type e in all the function type signatures here);
• conditions is a list of the events on which this event is conditional (i.e. a list of type [e]); and
• probabilities is a table of probability values, stored as a flat list. The type of these values (represented as type variable p everywhere) can be any fractional numeric type. The values are stored in lexical order of the truth value assignment to the conditions variables, where True sorts before False. For instance, if conditions is [R,S], then the probability values are in the order +R,+S; +R,-S; -R,+S; -R,-S.

Creating a network from its list representation is simple. The only small thing we need to do is to check that the probability table has the right number of entries, dependent on the number of conditional variables. It would be interesting to try to apply this constraint at the type level. No idea how to do that. It’s probably easy in something like Agda, with dependent types, or Qi, where you can write programs at the type level. In Haskell, I guess it might require Template Haskell. I don’t know though. In any case, let’s keep things simple here:

fromList :: (Ord e, Fractional p) => [(e, [e], [p])] -> BayesNet e p
fromList = BayesNet . (foldl doOne M.empty)
  where doOne m (ev, cond, ps) = 
          if (length ps /= 2^(length cond)) 
          then error "Invalid length for probability table"
          else M.insert ev (BayesNode cond ps) m

Probability calculations

Given a Bayesian network, we want to use it to calculate probabilities. A couple of special cases, first the probability that a single event is true, given conditions (true or false) on a set of other events.

prob :: (Ord e, Fractional p) => BayesNet e p -> e -> [(e,Bool)] -> p
prob bn ev cond = jointProb bn [ev] cond

Then, the joint probability that a set of events is true, given conditions (true or false) on a set of other events.

jointProb :: (Ord e, Fractional p) => BayesNet e p -> [e] -> [(e,Bool)] -> p
jointProb bn evs cond = jointProbTF bn (zip evs $ repeat True) cond

Then the general case: the probability that a set of events is true or false, as specified, given conditions (true or false) on a set of other events.

We do this by calculating the joint PDF of the whole network, then enumerating the possible assignments of truth values to the free variables in the numerator and denominator of the representation of $P(X | Y) = P(X ^ Y) / P(Y)$, applying the joint PDF to each of these variable assignments, summing the results and forming the fraction representing the conditional probability.

jointProbTF :: (Ord e, Fractional p) => 
               BayesNet e p -> [(e,Bool)] -> [(e,Bool)] -> p
jointProbTF bn@(BayesNet m) evs cond = eval (cond ++ evs) / eval cond
  where eval = sum . (map (jointPDF bn)) . (enumerate bn)

All the work here is being done in the computation of the joint PDF, which represented just as a regular Haskell function, and the enumeration of the possibilities (remember that we’re only dealing with binary random variables throughout, just as in the lectures).

The joint PDF

The joint PDF for the network

is:

$P(B,E,A,J,M)=P(B)P(E)P(A\mid B,E)P(J\mid A)P(M\mid A)$

where we have one factor per node in the network, each of which is conditioned on the variables in the nodes connected to the incoming edges. In the code below, the joint PDF is represented as a function that takes a map from node labels to Boolean values and returns a probability value. To calculate the joint PDF of the whole network, we form the product of the individual PDF factors for each node in the network, applying each of these factor functions to the input variable assignment, and forming the product.

jointPDF :: (Ord e, Fractional p) => BayesNet e p -> (M.Map e Bool -> p)
jointPDF bn@(BayesNet m) = 
  (\vals -> product $ zipWith (applyFactor vals) (M.keys m) factors)
  where factors = map (jointPDFFactor bn) (M.keys m)
        applyFactor vs k f = f (vs M.! k) (map (vs M.!) (cond $ m M.! k))

We then need to produce the constituent functions for the individual factors in the PDF. Thes functions, for a factor of the form $P(X\mid Y)$ are represented by a function that takes a single Boolean value representing the value of $X$, and a list of Boolean values representing the values of $Y$, and returns $P(X\mid Y)$. To do this, we need to know the predecessors of the node in question in the graph of the network, we need to extract the correct value from the node’s probability table, and we need to deal with complementary probability values.

jointPDFFactor :: (Ord e, Fractional p) => 
                  BayesNet e p -> e -> (Bool -> [Bool] -> p)
jointPDFFactor bn@(BayesNet m) ev =
  (\b bs -> let pval = pvals !! (length pvals - idx - 1)
                idx = foldl (\i b -> (if b then 1 else 0) + 2 * i) 0 bs
            in if b then pval else (1 - pval))
  where pvals = ps (m M.! ev)

Enumerating cases

The other element we need in order to calculate probabilities is to be able to enumerate the possible assignments of Boolean truth values to variables in the network, given that some variables may have their values fixed by the requirements of the calculation. Here, we assume that there are no extra logical constraints on the values that variables may take. In many more realistic problems, there may be consistency requirements that eliminate some possible assignments (see The Wumpus World Revisited, Section 13.6 of AIMA for a simple example where this is the case). Given that caveat, state enumeration is quite simple:

enumerate :: (Ord e, Fractional p) => 
             BayesNet e p -> [(e,Bool)] -> [M.Map e Bool]
enumerate bn@(BayesNet m) fixed = map compose msks
  where base = M.fromList fixed
        vars = (M.keys m) \\ (map fst fixed)
        msks = masks (length vars)
        compose = (M.union base) . M.fromList . (zip vars)

We have a little helper to generate combinations of Boolean values:

masks :: Int -> [[Bool]]
masks 0 = [[]]
masks 1 = [[False], [True]]
masks n = map (False :) xs' ++ map (True :) xs'
  where xs' = masks (n - 1)

Applications

Given the definitions above, the two test cancer example works as expected. We can also deal with Sebastian’s happiness example. The necessary definition of the Bayes network is:

import BayesNet
import Data.Ratio

data Event = S | R | H deriving (Eq, Ord, Show)

bn = fromList [(S, [], [7 % 10]),
               (R, [], [1 % 100]),
               (H, [S, R], [1, 7 % 10, 9 % 10, 1 % 10])]

Here, I’ve made the probability values in the Bayes network be rational numbers (remember that we can use any instance of typeclass Fractional for this). We can then answer some of the questions Sebastian posed in the lecture. First we load the definition into GHCi:

*Main> :load "happiness.hs"
[1 of 2] Compiling BayesNet         ( BayesNet.hs, interpreted )
[2 of 2] Compiling Main             ( happiness.hs, interpreted )
Ok, modules loaded: BayesNet, Main.

Then we ask the value for $P(R\mid S)$:

*Main> prob bn R [(S,True)]
Loading package array-0.3.0.2 ... linking ... done.
Loading package containers-0.4.0.0 ... linking ... done.
1 % 100

The answer is just $P(R)$ because $R$ and $S$ are independent. Notice that we get the result as a rational here. What about $P(R\mid H,S)$? Just as easy:

*Main> prob bn R [(H,True),(S,True)]
10 % 703
*Main> fromRational $ prob bn R [(H,True),(S,True)]
1.422475106685633e-2

Here, I’ve converted the result from a rational to a floating point for comparison with the next result, which is $P(R\mid H)$:

*Main> prob bn R [(H,True)]
97 % 5245
*Main> fromRational $ prob bn R [(H,True)]
1.8493803622497616e-2

Here, $P(R\mid H)>P(R\mid H,S)$, demonstrating the “explaining away” effect.

For sure, you can do the homework questions using the same setup.

Conclusions

This is not a good way to do probability calculations on Bayes networks in general. There are much better approaches available than enumeration. My goal here was more to investigate how easy it would be to construct an explicit programmatic representation of the full joint PDF of a Bayes network. Whether writing all this code to answer three homework problems is an efficient use of time, I’ll leave for the reader to decide…

The code is in: BayesNet.hs, cancer.hs and happiness.hs. There’s also code implementing the earthquake alarm example from the lectures in alarm.hs.

Morning paddling preparations on Wallace Island, Gulf Islands National Park. Thanksgiving weekend, 2009.

Week 3 of the Stanford AI class was about machine learning. This was kind of handy for me, since my PhD thesis was mostly about applying some unsupervised learning techniques to dimensionality reduction for climate model output. That mean that I’ve read hundreds of papers about this stuff so the basic ideas and even quite a few of the details are already pretty familiar. However, most of what I’ve read has been aimed at applications in climate science and dynamical systems theory. Not much from the huge literature on clustering methods, for instance.

Sebastian very briefly mentioned some of the nonlinear dimensionality reduction methods that have been developed for unsupervised learning applications, but it was nothing more than a mention (no time for anything else). I can’t resist the temptation to dust off a little of this stuff from the archives.

Sebastian mentioned two methods of nonlinear dimensionality reduction (the two most well known), locally linear embedding¹ and Isomap² (Isomap is one of the methods I used in my thesis³). There are dozens of other methods that have been developed, and I’d like to describe the characteristics of one of the more recent ones, called Hessian locally linear embedding (HLLE). As the name suggests, it has a lot in common with locally linear embedding, but it turns out to be easier to deal with from a theoretical point of view, and has some rather nice properties.

The basic idea of HLLE

HLLE⁴ is a manifold learning method. That means that, given a set of data points lying in $m$-dimensional Euclidean space, it tries to find (an approximation to) a $p$-dimensional manifold $M\subset {ℝ}^m$ close to which the data points lie. Such methods can be thought of as generalisations of principal components analysis (PCA), where PCA finds linear subspaces, instead of curved manifolds. The Isomap method does this by trying to build a distance measure between the data points that approximates distances along paths in the data manifold. This distance measure can then be used to “unroll” the manifold using multidimensional scaling. Locally linear embedding works by using a numerical approximation of the Laplace-Beltrami operator on the data manifold (a generalisation of the usual Laplacian operator to manifolds). An eigenvalue decomposition of this numerical Laplace-Beltrami operator is then used to generate “coordinates” for the manifold⁵. HLLE works in a similar way, but uses a numerical approximation to the Hessian matrix on the manifold (actually a sort of coordinate-invariant operator built from the Hessian), which is a generalisation of the matrix of second partial derivatives in Euclidean spaces.

The key observation here is that the null space of this Hessian operator is closely related to the existence of nice “flat” coordinates for the data manifold, in a much better way than the null space of the Laplacian. (Lots of functions have zero Laplacian without being “simple”. Functions with zero Hessian have all second derivatives zero, so are just linear, i.e. very simple.)

The details are complicated, both from a theoretical point of view and for the numerical implementation. All of these manifold learning algorithms are relatively similar in form: calculate some sort of matrix (in Isomap, it’s a distance matrix, in LLE, the Laplace-Beltrami operator and in HLLE, the Hessian), then do some sort of eigenvalue decomposition or similar computation on it. The matrices tend to be large (typically $N\times N$ for $N$ data points, hopefully reasonably sparse) which means that iterative methods are needed to solve the resulting eigenvalue problems. At the time I did this work, I was writing most of my working code in C++, so I made use of the wonderful Trilinos project, developed at Sandia. Lots of good stuff for dealing with complex matrix calculations, PDE discretisation problems, plus 101 other things. Check it out.

Application to some test data sets

So, how does it work? The traditional way to test manifold learning methods is with simple geometrical data sets, like these (the gray surfaces are the data manifolds, and the coloured points data points randomly sampled from these manifolds, perhaps with some added noise):

Given the three-dimensional coordinates of the coloured points, can the HLLE algorithm discover the intrinsic two-dimensional manifolds? Yes, it can:

The important thing to note here, particularly in the third example (the “Swiss roll with hole”), is the lack of distortion in the “unrolling” of the manifolds to a two-dimensional representation. Compare the equivalent results from Isomap and from a neural-network nonlinear PCA method:

Parameter sensitivity

I’ve not talked about the details of the HLLE algorithm at all, but in common with a lot of similar “computational differential geometry” algorithms, at one point, a neighbourhood calculation is required (specifically, for finding locally linear subspaces by doing a local singular value decomposition). The number of points to include in these neighbourhoods is a tunable parameter of the algorithm, and it’s interesting to observe the variations in the performance of the discovery of the intrinsic data manifold as the neighbourhood size is varied. The other important factor is the total number of data points, which we can easily vary by randomly subsampling our data set. The following image shows embeddings derived for the “Swiss roll with hole” with a small amount of Gaussian noise, for a range of total data point counts and neighbourhood size (click for a really big version):

There are some striking patterns in the images here. Good embeddings are those in the central part of the image (they look like a rainbow-hued square with a hole in the middle) and exist for some values of the neighbourhood size $k$ as long as we have enough data points (less than 1000 points and we don’t find good embeddings at all). When we don’t have enough data, embeddings for smaller $k$ are highly distorted while those for larger $k$ are degenerate, i.e. all data points are mapped to a single point. For intermediate values of the data point sampling density, $1000\le N\le 2750$, there is a maximum $k$ value below which good embeddings are seen. This maximum $k$ value increases more or less linearly from $k=14$ when $N=1000$ to $k=46$ when $N=2750$. As $k$ is increased above the threshold value, embeddings show increasing distortion and eventually become degenerate.

The upper limit on the value of $k$ to get a good embedding is imposed by the requirement that the neighbourhoods should represent “locally linear” subsets of the data manifold. When $k$ becomes larger than the threshold seen here, the neighbourhoods start to cover regions of the data manifold that show significant curvature at the scale of the mean distance between data points. This means that the “locally linear” part of the HLLE algorithm breaks down. The maximum $k$ value depends on the data sampling density because the neighbourhood size grows linearly with $k$ but inversely with $N$ — the more data points there are, the closer together they lie both in the original data space and in the lower-dimensional data manifold, and a smaller fraction of the total data manifold is encompassed by any given neighbourhood size, based on a simple count of neighbours. At the lowest data point sampling densities examined here, the Hessian LLE method does not produce good embeddings for any choice of $k$.

There is also a lower limit for $k$ below which no good embeddings are seen. This limit also depends on the data sampling density. For $N=1500$, the lowest neighbourhood size for which a good embedding is produced is $k=14$, while for $N=4000$, no good embeddings are seen for $k<22$. For values of $k$ below the lower limit, the Hessian LLE procedure identifies a one-dimensional manifold, rather than the true two-dimensional data manifold. The reason for this lower neighbourhood size limit is that, for smaller values of $k$, the neighbourhoods essentially sample only the noise variability in the input data, and do not capture any of the structure of the data manifold. Interestingly, the hue assignments shown in the one-dimensional manifolds appearing in these results are consistent, i.e. hues vary smoothly from one end of the embedded manifold to the other. The one-dimensional manifolds recovered from the Hessian LLE procedure appear to capture at least some aspect of the intrinsic geometry of the input data, albeit very crudely.

Application to real data

So much for carefully controlled toy data sets. The parameter sensitivity plots above show how difficult it can be to interpret the results from this kind of algorithm even for simple data (are we really seeing something intrinsic in the data or is it just an artefact of the dimensionality reduction algorithm that will go away if we change the algorithm parameters?). This problem only gets worse when confronted with realistic data. The basic problem is that, for a lot of phenomena, linear methods like PCA do surprisingly well. In particular for the climate system, there are some quite deep underlying reasons why this should be so — Gerald North showed some time ago⁶ that PCA actually calculates the normal modes of certain stochastically driven linear dynamical systems, which makes it a natural tool for investigating the dynamics of such systems. For certain conditions, the dynamics of the atmosphere (and to some extent, the coupled ocean-atmosphere system) can be approximated by the type of system considered by North, meaning that PCA can be a good fit. In such a situation, it’s a bit optimistic to expect your smart nonlinear method to do all that much better.

That said, these methods don’t necessarily do any worse than PCA for some climate applications. The plot to the right shows the results of applying HLLE to the analysis of interannual variability in tropical Pacific sea surface temperatures (SSTs). The two main modes of variability in this region are the annual cycle and the El Niño/Southern Oscillation (ENSO). PCA does a good job of picking out both these modes of variability, both in observational data and in the results of atmosphere-ocean general circulation models (GCMs). It turns out that HLLE can do the same (which is not something that’s immediately obvious, principally because the computation of the Hessian used in the HLLE algorithm is susceptible to problems because of numerical noise).

What the plots on the right actually show is the value of one of the components that comes out of the HLLE analysis, rotated into a coordinate frame that “unmixes” variations due solely to the annual cycle. The hope is that this procedure will pick out variability due to ENSO. In the plots, you can see an index of ENSO variability (in black, derived from mean SSTs over a specified region in the equatorial Pacific) and the rotated HLLE component time series in red, both for observational data and for four GCMs from the CMIP3 ensemble. What’s quite interesting here is that the HLLE algorithm manages to pick out the ENSO (or ENSO-like) variability in both the observations and most of the model data sets, without any prompting. This is not a particularly shocking result, since PCA does the same, but it’s quite encouraging to see that these more complex methods are at least no worse that linear methods when such methods are appropriate.

There are situations where there definitely is a lower-dimensional nonlinear manifold to be discovered, and in these cases, algorithms like Isomap or HLLE can do a lot better than a linear method like PCA. You need to experiment to find out whether that really is the case though: it’s very easy to pick a method because it’s “fashionable” and demonstrate the existence of a “nonlinear” manifold that’s just as easily discovered using a linear method. The literature on nonlinear dimensionality reduction is littered with studies of this sort.

The take-home lesson? Data analysis is hard, and clever methods add a extra burden of understanding on top that’s not always justified.

What does it mean when your dog yawns at you? She’s tired? She’s bored? She’s embarrassed of your choice of socks? If you don’t know, you probably want to read Turid Rugaas’s book about the body language signals that dogs use to display unease and to calm each other down.

Yawning is a very clear sign that a dog wants people (and other dogs) to back off a little and give them some space, but other signs are more subtle. Next time you see a photograph of a dog, look carefully: is the dog licking its nose? That nose lick means that it was kind of uncomfortable with all the attention and the camera pointing at it. He’s not just buffing his nose up so it looks nice and shiny on the photo…

There are lots of other signs, many to do with body posture and placement, that can help you to interpret how your dog is feeling. Our dog, Winnie, is quite fearful because of maltreatment when she was little, so it’s very useful to be able to watch her and get some idea when she’s beginning to get outside her comfort zone, before she transitions into tail-between-the-legs-must-run-away-now mode.

And this body language is more or less universal between dogs of different breeds and sometimes radically different shapes and sizes. Floppy ears and docked tails both restrict the messages that dogs can get across, but they still seem to understand each other pretty well most of the time.

by Mark Kac & Stanislaw Ulam

This interesting little book, published in 1968, is the result of a collaboration between two of the great figures of 20th century mathematics. Although known mostly for his work on the Manhattan Project and later associated applied mathematics efforts (I first learnt of his work after being told to read about the Fermi-Pasta-Ulam problem by my PhD supervisor), Ulam began his career as a pure mathematician, working on problems in general topology. (His autobiography, Adventures of a Mathematician, is well worth a read.) Kac, best known for asking “Can one hear the shape of a drum?”, made major contributions to probability theory. Kac and Ulam got to know each other in Poland, then both made the move to the United States during the Second World War.

Mathematics and Logic was originally commissioned by the Encyclopedia Britannica as a sort of long-form article to appear as an appendix to the encyclopedia. It was only published separately a little later. It’s particularly interesting because here we have two truly great mathematicians trying to answer the question “What is mathematics?”, aiming at the broader scientific community, trying to give an impression of what it is that mathematicians do. Mostly, to be honest, they don’t do a great job of answering their question. The entertaining aspect of the book is that they seem to have realised right from the start that they’re not going to manage to answer the question they pose as the first sentence of the introduction, so they instead embark on a “show-and-tell” voyage through the world of mathematics. This ends up working well, and to be honest, it makes sense. If you want to learn about a new country, where do you turn first? The CIA World Factbook, or something by Paul Theroux? If you want the raw statistics, the former is handy, but if you want a feel for the place, to take in the sights, you want something a bit more picaresque.

What you end up with here is 105 pages of examples of mathematics, then around 60 pages that tries to synthesise these examples into a coherent picture of what mathematics is, to try to tease out the essence of the subject. It’s a hard task! The examples that Ulam and Kac use range from the very traditional (the infinity of the primes, the irrationality of $\sqrt{2}$) through the more “modern” (transcendental numbers & Cantor’s argument, for instance) to topics that would be identified as resolutely modern: groups and transformations; homology groups; transformations, flows and ergodicity. In all, they give a good idea of the vast range of material encompassed by the word “mathematics” at the time. This part of the book is satisfying, although one often feels the limitations of a short form like this — tell us the details, guys! We want the details!

The second part of the book, I find a little less satisfying. We step from the picaresque to the analytic, to try to pin down just what this subject is. There are some general trends that are easy to pick out, but there’s no clear overarching principle (probably because there just isn’t one). The trends:

First, a growth of rigour, rendering the subject more formal and inward-looking.

Then, axiomatisation: a focus on structures, rules, consequences. For example, and as a possible indicator of one path by which new mathematics can arise, consider the field axioms, originally for real numbers, and think about how one might apply these axioms to continuous functions with convolution as the composition rule. Same axioms, different structures, different consequences, and hence new things to think about.

Next, algebraisation, in particular of topology, an approach with enormous ramifications and many applications.

Then, controversially, the liberation from intuition. The very striking example that Kac and Ulam use is the fact that continuous functions differentiable at at least one point form set of first category in set of continuous functions, i.e. they are infinitesimally rare! In some sense, “most” continuous functions are like the Weierstrass function, differentiable nowhere.

And then we get to the end results of axiomatisation, Gödel’s incompleteness theorem, Turing’s work on computability and an apparent crisis in the fundamentals of the subject. (Of course, the “crisis” wasn’t too critical: Gödel’s work didn’t much difference for most working mathematicians.)

There’s some material about the relationships between computers and mathematics (still only nascent in the 1960s) and relations with other disciplines.

And then we end. Does the book answer the original question? Not as such. It shows the answer, but this might be the only thing that it’s possible to do. Successful in its aim or not, it’s an interesting book, and Kac and Ulam are charming guides to the territory.

I kept trying to tell Rita that a hippo would make a perfect pet. She didn’t believe me. But then she sent me this. Point proven, I think you will agree…

Week four of the Stanford Introduction to AI class was about, well, what was it about? It was a bit of a mish-mash. Some stuff about logic, some stuff about planning. Nothing I can really pin some experiments on, so I’ve been thinking a bit about the format of the course instead.

I’m becoming less and less sure that the format that Peter and Sebastian have chosen for the course works well. First, the lecture format (hand written notes, done sort of live) is very susceptible to errors. This might be OK in a real live lecture, where the audience (if they’re paying attention) can help to correct mistakes, or can ask for clarification, but when the audience is disconnected and on the other end of an internet connection, it doesn’t work so well. I’m not quite sure why they elected to follow this route, since they have lecture slides prepared for the “real” Stanford course, which they could reuse with spoken narration.

Second, production values aren’t that great. Preparing e-learning materials is hard and always requires a lot more resources than expected. The impression that I get from looking at the course materials here is that some consultation with someone who’s worked on e-learning platforms before might have been a good idea.

Third, the “in video” quiz format works fine for quizzes, but it’s not that great for homeworks — too many errors and ambiguities. It seems as though the plan is to use the same format for the midterm and final, which seems like a recipe for disaster. I hope they manage to come up with an alternative.

I know that the course is very much an experiment, and I am 100% behind Sebastian’s desire to expand access to high-quality university teaching to as many people as possible. I just feel that the technical choices that they’ve made here don’t help them a lot.

I’ve heard from other people that the other Stanford engineering courses (machine learning and databases) are much more professionally produced and have had far fewer server issues. Perhaps that’s a result of the narrower focus and smaller student numbers, I don’t know.

All that said, I don’t really buy into the strand of complaining about things that appears in some threads on Reddit. My own experiences of teaching have given me a nice (post-traumatic…) appreciation of how hard it is to prepare good course materials and to really get through to students. And that’s when the students are right there in front of you. What Peter and Sebastian are trying to do is more than an order of magnitude more difficult. It’s rather tricky to assess in advance just how much resource you need for a project like this with open subscription, famous names, and very wide advertising. Expected enrollment of around 10,000 students; actual enrollment 160,000. See the problem?

I really believe in Sebastian’s mission, and if they can carry the experience they’re getting with this course forwards, I tihnk they could build something really worthwhile. I’m very happy indeed to be part of this experiment!

via the MIT Technology Review Physics arXiv Blog

Everything’s better in slow-motion, right? And everything’s even more betterer in super-slow-slow-slow slow-mo, am I right? (You know I’m right.) It turns out that if you have a high-speed (10,000 fps!) camera and a pile of C4 explosive, you can make some pretty cool movies. You can read about it here.

On a less explodey note, two other vids from this year’s Gallery are also pretty outstanding:

Bouncing balls on water!

Freezey trees!

science

Science Dog is tired

by John Bradshaw

Those of us who live with small furry friends have a vested interest in understanding what our beasties are thinking. They don’t talk, they won’t use SurveyMonkey, and they sometimes act like they’re not human at all, no matter how we might like to think of them as little people in dog costumes. But never fear. There is a whole industry out there that promises to help you interpret your dog’s slightest ear flick and nose twitch, that will allow you to correct all “undesirable” canine behaviours back to the acceptable human norms, that will turn you into a veritable Dr. Dolittle.

Just one little problem. Most of what you read is wrong, and quite a lot of it is harmful. Dominate your dog! Tame the wolf in your home! Show your pooch who’s boss! All complete crap, based on romantic story-telling and what makes good TV, rather than on decent science.

In In Defence Of Dogs, John Bradshaw, a researcher in canine behaviour at the University of Bristol (although he has appeared on TV, including The Colbert Report!) presents some of the recent thinking on dog behaviour, based on, of all things, scientific research.

The first thing to know is that, despite being genetically near identical to wolves, dogs are not wolves from a behavioural point of view. A good parallel is the comparison between chimpanzees and bonobos: very similar genetically, completely different social organisation and behaviour. Dogs have been radically altered by the process of domestication, having been selected, albeit haphazardly, for sociability and friendliness to humans. From that perspective, basing dog training methods on the idea that dogs are wolves is just silly. What we really need is to understand what motivates dogs to act the way that they do, bearing in mind the long process of selection.

Second, the position of dogs in society has changed. Most dog breeds started out as working dogs, selected for features useful for herding or guarding or hunting. Dogs as companions, as pets, are a relatively recent development. That has two consequences: first, many dog breeds formerly spent most of their time with a human handler, working, rather than being cooped up in an apartment while their owner goes off to play their part in the knowledge economy; and second, behaviours useful for hunting, guarding or herding may not be so useful for life in the city. Dachshunds want to dig. Who knows? There might be badgers hiding under your tulips! Collies want to herd. Something, anything. No sheep available? No problem! Children are about the right size! Beagles love to run. Four hours, five hours, no trouble. Restrict your dog from following those inbuilt behaviours without providing some alternative outlet and they won’t be happy. And when your dog isn’t happy, you’re going to know about it pretty quickly. Gnawed furniture, holes in the garden, anxious barking, peeing and pooping misadventures, all are possible.

The third important thing to think about is how we might breed dogs that are better suited to this life. Current breeding strategies based on conformation to breed standards have led to terrible inbreeding and the production of dogs who, in many cases, just aren’t suited to life as domestic pets. Why breed for ear shape or coat colour when you could be breeding for even temperament, friendliness and calmness (or any other combination of desirable behavioural traits you might list)? Partially because it’s harder — you can tell quite quickly whether a puppy is going to grow up to be a good show dog, but it requires longer-term monitoring to track a dog’s personality. But there’s also a strong “nurture” aspect to the behaviour a dog exhibits as an adult. Managing that requires a lot more effort from breeders, and is likely to make commercial breeding economically unviable in most cases.

Bradshaw’s book is something of a (quiet, polite) call to arms, to try to establish better practices for dog breeding, to breed dogs that are more likely to be happy sharing their lives with us busy modern humans. His views are not universally shared in the dog training community (to put it mildly; some of these dog people can be a little catty), but what he says makes a lot of sense.

Great Mosque, Kairouan, Tunisia. April, 2005.

Haskell

Stanford CS240h: Functional Systems in Haskell
Stanford is really pushing e-learning at the moment, what with the AI, machine learning and database classes that the engineering department is running. This is another good course with content available online.

Reddit: What does your company use Haskell for?
What it says on the tin: quite a few people chime in. Seems like Haskell isn’t so much of an “academic-only” language any more!

Storage and Identification of Cabalized Packages
Very helpful guide to GHC package management from Albert Lai.

Unused constraints in GHC
This week, we spotted an interesting thing in some of the diagrams code. There are a bunch of places where an earlier implementation of a feature required certain type class constraints. With a more recent implementation, that requirement has now gone away, but the constraints remain in the code. That makes using these particular functions trickier than it needs to be. We’d like for the compiler to warn about these extra un-needed constraints, since otherwise they just hang around like a bad smell. Brent describes it as “a nice project for someone wanting to dig into hacking on GHC”…

The other culture

Money (via Crooked Timber)
Where does it come from? There’s a nice little Just-So story that money arose naturally out of barter economies as a natural consequence of Immutable Economic Laws. It’s thus slightly embarrassing that anthropologists haven’t found any evidence at all for barter economies of the required type. Cue immense academic pissing contest, of course. Read it all. It’s good.

Cool stuff

Visual 6502
Remember the 6502? Come on, no need to be shy. The other 8-bit processor from the 1980s. I was a Z80 boy myself, but I did a bit of 6502 assembler programming back in the day (for my A-Level computer studies project, I wrote a little data capture package for an infra-red spectrometer in our school’s chemistry lab using the A/D ports on a BBC Micro). Any other fans of retro 8-bit should take a look at this site. They’re building sub-gate-level simulations and visualisations of old microprocessors. The visualisations are very neat, but the way they’re doing it is the kicker — working directly from dies, they’re photographing them, doing some image analysis to get the patterns of the chip layers out and using these to build transistor-level models of the chips. Lots of fun!

Circos
You may have seen those super sweet circular charts floating around in some biology papers in Nature or Science. This is where they’re made, and you can make them too.

Quasicrystal animations
This is “Haskell” too, but it lives in “Cool stuff”.

I’ve recently started making some tiny contributions to Brent Yorgey’s Haskell diagrams library, mostly bug fixes. The bug fixes are primarily just a way of familiarising myself with the diagrams codebase though. I have my eye on a rather chunkier task in the open issues list. That’s writing a constraints solver for diagram layout. I’ve been interested in this sort of problem for some time, and have never had a good reason to get down to it, so I’m going to see what I can come up with.

And I’m going to try something I’ve not done before, which is to document experiments, design and development as blog articles. It may all turn out to be hideously embarrassing, it may end up that none of what I do actually makes it into the mainstream of the diagrams library, but having to write about what I’m doing will keep me honest, force me to be clear about what I’m doing, and will provide an incentive to work on this stuff.

Constraints 101

OK, so what do we mean here when we talk about “constraints”? The word “constraint” is used in lots of ways: we have type class constraints in Haskell, people talk about constraint logic programming, in mechanics constraints are extra side-conditions on Newton’s equations of motions imposed by the structure of the problem we’re solving¹ (think bead-on-wire problems), and so on. Here, we’re talking about something closer to the last case. Given a bunch of geometrical objects (points, lines, polygons, circles, arcs, Bézier curves, whatever), we want to be able to specify geometrical relationships between our objects, for instance that the end-point of one line segment is coincident with the corner of a rectangle, or that a certain circle is tangent to a certain line segment.

Why do this? My first exposure to constraint-based drawing was through AutoDesk’s Mechanical Desktop, which was then part of the AutoCAD product suite². This had a sketching tool that was just magical. Draw a very rough sketch of approximately what a component looked like, then say, “OK, this line should be horizontal, this one should be perpendicular to this one. This curve is a fillet and should be tangential to this line and this line. These two circles are concentric, these two lines are parallel. This point lies on this line, not quite sure where yet.” What you end up with is a drawing that has a smaller number of free parameters (usually the major dimensions of whatever part you’re designing). Specify those dimensions and the rest of your drawing jumps around to make everything right with the constraints you defined³. I was using Mechanical Desktop for work, but I got a real kick out of just playing with the constraint-based drawing tool. It really seemed like magic. Set up your drawing with some constraints, drag some points around and watch as AutoCAD updates the rest of the drawing to maintain the constraints you specified. Cute and powerful.

The same power can be applied to layout in the diagrams library. You might not want to go the whole hog with the complex geometric constraints available in CAD packages (although why not?), but you can certainly say things like: “These three items lie on a horizontal line; the gaps between them should be equal; I don’t know how wide the items are yet.” This is a very common situation in GUI layout and lots of GUI toolkits provide the capability to specify layout in this fashion. Since the diagrams library is intended to produce images more complex than typical GUI layout, it seems to me that something closer to the AutoCAD experience is what we want. (It will also be more fun!)

Goals

Given all that spiel, how do I get from nothing to a usable constraints module for the diagrams library?

1. The first thing to do is probably to implement some existing constraint solver algorithms in Haskell and do some experiments. Cassowary seems like a good starting point. Other interesting possibilities are the system used in Juno–2 and Mike Gleicher’s snap-together mathematics. This shouldn’t be too hard, and would result in a constraint solver working at the level of individual equations and inequalities for sets of real variables.

2. The diagrams library already has primitives for drawing various geometric objects, but to start with, I’d like to abstract away from that. Two reasons for this. The first is to reduce the dependence on the details of the representations used in the base diagrams library in order to get an idea of exactly what is required in the constraints system. The second reason is that there are multiple ways of looking at geometrical objects. A point can be identified by its Cartesian coordinates or by its polar coordinates. A line segment can be identified by two points or by a single point, a direction and a length. A rectangle can be identified by giving two corner points and a rotation angle, or by a single corner point, a width and height and a rotation angle. Each of these “views” of these objects is equivalent, and it should be possible to express constraints in any convenient view. For instance, I might want to constrain the centre of a circle $C$ to lie on the circle $r=1$. I should be able to give a constraint $r(\mathrm{\text{centre}}(C))=1$ to achieve this. I have an idea of some sort of “constrainable lens” floating around in my head but it’s going to require some experimentation to pin down just how this should work, and keeping this independent from diagrams to start with should help to reduce my confusion to manageable levels. (There are some relevant thoughts about this idea of different views of geometrical objects in Mike Gleicher’s PhD thesis.)

3. The most important part of a constraint system from the user’s point of view is the interface for defining and managing constraints. The constraint that two lines are perpendicular, or that a circle is tangent to a particular line segment, can be expressed as sets of equations and/or inequalities that can be handled by a constraint solver, and we’d like a way to work at this higher level: in the most trivial case, I want to be able to say “Point $P_1$ is coincident with point $P_2$”, not “$x(P_1)=x(P_2)\wedge y(P_1)=y(P_2)$”. More complex geometrical constraints can be broken down and expressed as equations in similar ways. Providing an easy-to-use interface between this geometrical view of the world and the equational view required by the constraints solver will be key to making the constraint system useable. Part of that will also involve finding a good way to deal with underconstrained and overconstrained systems.

4. The last job to tackle will be taking whatever I come up with and trying to massage it into a state where it’s usable in the diagrams library. I’m going to keep this final goal in mind, but I’m not going to let it constrain what I do too much⁴. Chances are, someone else will develop something usable for diagrams before I’m too far out of the starting gate on this, and it’s mostly for my own amusement anyway…

First things first

OK, so I’ll start with reading the Cassowary papers and will implement the relevant algorithms in Haskell to act as a baseline for experimentation. Off we go!

I miss Canada.

I miss sea kayaking.

I miss mountain biking.

I miss CrossFit Taranis and all the awesome people there.

We moved to France. Dumbasses. We exchanged a fantastic life in Canada for what? Basically: cheese. Feel free to mock us. We deserve it.

Looking for a prod to jolt me out of an (involuntary) 45 day blogging hiatus, I’m going to follow a scheme that Rita is using quite successfully: the Thirty Day Challenge. The idea is simple. Just do that thing, whatever it is, every day for thirty days. The timescale is short enough not to be daunting, but long enough to be habit-forming. Or that’s the theory anyway.

I have a few things lined up, including a slew of book reviews of my holiday reading and the creakingly slow commencement of my constraints project, but we’ll see how it goes.

OK, one down, 29 to go. That wasn’t so hard after all!

One thing that came to an end over the Christmas holidays was the Stanford Artificial Intelligence course. The final exam was on the weekend of 17/18 December and final grades were distributed a couple of days later.

Overall, it was an interesting experience, and I certainly learnt some things. The instructors (Peter Norvig and Sebastian Thrun) were enthusiastic and clearly extremely knowledgeable, and they took on an extremely challenging task in trying to run an online class for tens of thousands of students. For that, I’m very grateful. There are a few things that weren’t 100% ideal about the presentation of the class material, but that’s to be expected the first time out with something like this.

The highlight of the course was definitely Sebastian’s presentation of his work on self-driving cars. I’d heard a bit about the DARPA Grand Challenge before, but I hadn’t known about the progress that the Stanford team had made with self-driving cars in traffic. That’s really impressive work, and Sebastian did a nice job of relating the algorithms used for guiding cars to the simplified versions considered in the class.

This word, algorithms, is kind of key to what I want to talk about here. The content of the course was, to my mind, very focused on algorithms for practical applications. For instance, machine learning, planning, and perception under uncertainty were all covered in a lot of detail (particle filters are a good example). What wasn’t really covered at all was what I tend to think of when I think of “Artificial Intelligence”, which is general reasoning and problem solving.

All of the examples considered in the course were quite constrained, and very far from the early dreams of “intelligent machines”. This is clearly just a reflection of where the field of AI is today: there is much less emphasis on the (very hard) problems of general intelligence, and much more focus on solving practical problems. “Putting AI to work” is the name of the game. I understand why things are this way. It’s much easier to get a research grant for something with an obvious industrial application, rather than a vague programme of research that might have applications in 50 years or 100 years or just… sometime. It’s a little disappointing all the same.

The disappointment stems not so much from the fact that we’re thinking about practical problems (that’s a good thing!), but a feeling that most of the methods being used (probability theory, most of the machine learning algorithms) aren’t really about intelligence as such, but are just mathematical or statistical methods that are useful for some applications. That word “just” is doing a lot of work there: some of these methods are highly nontrivial and very interesting in themselves.

But where are our intelligent machines? You could definitely make a case that an internet search engine capable of finding relevant pages for a vaguely expressed search term is intelligent in some sense, and the same goes for a self-driving car. But you can’t engage them in conversation about what they do. They have no power of introspection. They don’t know that they’re doing what they’re doing. This is all a long way from the idea of “human level AI”.

Some people have expressed this sense of disappointment more strongly than I would. Here’s John McCarthy, a man who knew a thing or two about artificial intelligence, writing in 2007¹:

The computer science world is still suffering from a 1990s fit of pseudo-practicality that is inimical to the solution of difficult scientific problems. Lip service is given to basic research, and a lot of basic research is done, but the initiation of ambitious research by young people is hampered by the now prevalent doctrine that “basic research” should be done in connection with applied problems that have been identified by the competent committees. I think that Newell and Minsky and I would have had a much harder time initiating AI research if the atmosphere of the 1950s had been like that of the 1990s.

So, we should still hold out for our intelligent machines! McCarthy’s paper lays out some of the requirements to get there from here. It’s a daunting list, and introspection is one of the features he highlights.

I did a spot of reading over the Christmas and New Year holidays. In fact, reading was more or less all I did. Apart from walking the dog, spending time with Rita and eating myself silly. I read enough to write a lot of book review posts, but I’ll restrain myself to three or so… First off, spy stories.

I’d never read anything by John Le Carré, having once picked up Smiley’s People, read the first page, not liked the tone of the narrative and so put it right down. For some reason though, Amazon insisted on recommending Le Carré’s books to me, and I was tempted to give them a go. I read Tinker Tailor Soldier Spy then The Honourable Schoolboy, which follows right on from Tinker Tailor. I still find the narrative voice uncomfortable. The off-handedness of some of the comments, the descriptions of characters, all make me think of some establishment patrician type pronouncing from his favourite chair in his Mayfair club. On reflection though, the voice makes perfect sense. Much of the action is set in Whitehall, many of the characters really are those old school tie types, and from that point of view, the voice works.

And anyway, maybe it’s supposed to make us feel uncomfortable. The atmosphere of Le Carré’s books is one of claustrophobia and bleakness, where no-one can truly be trusted, where betrayal and treason are the bread-and-butter of the business. Not a recipe for comfort.

I can’t really say that I like Le Carré. I understand why many do, and I see a lot of value in his depiction of the Cold War and the unlikely warriors who fought a large part of it. But still. The way that the death of Jerry Westerby at the end of The Honourable Schoolboy was reported gave the impression that the narrator really didn’t give a toss. Par for the course in the intelligence services? Maybe? Le Carré ought to know, having worked for both MI5 and MI6.

Apparently, Le Carré himself listed Tinker Tailor as being one of his four best novels. Maybe I should read the other three on his list (The Spy Who Came In From The Cold, The Tailor Of Panama and The Constant Gardener).

In a similar, although different, vein is Anthony Price’s Other Paths To Glory, which I read on the recommendation of Charlie Stross, who quoted it as an inspiration for some of the style of The Fuller Memorandum. I didn’t really know what to expect. I was wondering if the title was a reference to Kubrick’s Paths Of Glory, because of the First World War connection, so wasn’t quite in the mindset for a 1970s spy thriller. However, it’s a good read, and grabs hold very quickly, with an entirely unexpected bit of violence and the protagonist’s reaction to it, perhaps borne more of shock than anything else (having just been the victim of a quite professional murder attempt, he’s more worried what his mother will think of him coming in dripping wet from being in the river…). The historical detail of the battlefields of the Somme feels nicely authentic, not that I know anywhere near enough to comment on that, and the connection to the present afforded by the visit of a tour group of veterans works well. The “mystery” at the heart of the plot isn’t all that mysterious, and it’s a little surprising that none of the experts were quite aware of it before the denouement of the action.

My last spy novel of the holiday was a more modern one, The Tourist by Olen Steinhauer. Of the four books described here, this one does the best job of getting across the bleakness and moral ambiguity of the whole spy game. Milo Weaver is a “tourist” for the CIA, travelling the world on the orders of a voice on the phone and performing at best morally ambiguous tasks at the behest of his masters in New York. Burnt out and worn out, he’s back at an office job for the middle of the book, but is never quite able to leave his Tourism behind him. By the end of the novel, it’s no longer very clear who the good guys are. Or if there ever were any. The bad bad guys are picked out fairly clearly (psychopathic paedophile Russian oligarchs, anyone?), but apart from that, no-one has entirely clean hands. Weaver’s (to be honest, rather unlikely) family history eventually gets him out of a little pickle with home office, but he ends up annoying enough people that he’s demoted back to Tourist status.

Not a cheerful book. None of these four are. They’re all bleak and amoral (The Tourist perhaps even more than the others), and it’s never quite clear which side we should cheering for. In The Honourable Schoolboy, one character explicitly, if briefly, mentions this, commenting more or less that “Well, we are better than them, aren’t we? Our system is better, isn’t it?”. “Them” refers to global communism, in the shape of the USSR and China. The question of who are we and who are they is even more obliquely treated in Other Paths To Glory where it’s never clear who the “heroes” end up protecting from the “baddies”.

I’d like to know just how accurate, in particular, is Le Carré’s depiction of the spying life. I guess I’ll need to wait some years before records are declassified (if they ever are) so that someone can write a definitive history of the Cold War…

Pathetically slow in starting work on my little constraints project as I am, here’s the first of what should be a long series of posts…

One of the constraint solvers I’m looking at starting from, Cassowary, uses what is essentially an extended version of one of the most venerable of optimisation algorithms, Dantzig’s simplex algorithm. I’m going to start off thinking a little bit about the general setup of the kind of linear programming problems that the simplex algorithm is designed to solve, just to get a geometrical feeling for how these algorithms work and to understand the issues that might arise from relaxing some of the assumptions used in them.

Setup

The basic idea of linear programming problems is to find a vector $\mathbf{\text{x}}\in {ℝ}^N$ that maximises a linear objective function

subject to $M$ linear constraints of the form ${\mathbf{\text{a}}}_i\cdot \mathbf{\text{x}}\ge b_i$, ${\mathbf{\text{a}}}_i\cdot \mathbf{\text{x}}=b_i$ or ${\mathbf{\text{a}}}_i\cdot \mathbf{\text{x}}\le b_i$. There is lots of specialised terminology surrounding the field of linear programming, but this is the basic idea.

What’s the connection between this sort of problem and the problem of laying out diagrams? In a diagram, the variables that we are interested in determining, i.e. the components of the vector $\mathbf{\text{x}}$, are the coordinates of certain points in our diagram: perhaps the end points of line segments, the centres of circles, whatever points we’ve included in our constraints. The dimension of the solution space, $N$, is then just the total number of coordinates we need to find. If we can express our geometrical constraints as linear equations or inequalities between these coordinates, and if we can express a “good” layout for underconstrained situations as a linear function of our coordinate values, then we have a linear programming problem to solve.

Most of the time, we hope that the constraints we impose are sufficient to precisely locate all the elements of the diagram, in which case there is only one possible solution to the constraint equations and our linear programming problem degenerates to the solution of a system of linear equations to determine the unique point satisfying the constraints. On the other hand, if we are building a system of constraints interactively, we will often find ourselves with an underconstrained system — the user has placed a number of shapes, has applied some constraints between them, but those constraints are not sufficient to fix the positions of all the elements of the diagram. We might then choose some optimisation criterion to decide what is a good layout for the underconstrained drawing¹.

What sort of constraints can we treat? Suppose we have three points $p_1=(x_1,y_1)$, $p_2=(x_2,y_2)$ and $p_3=(x_3,y_3)$. We can make two points be coincident, for example with the constraint $x_1=x_2\wedge y_1=y_2$. We can make the point $p_3$ lie on the line segment connecting points $p_1$ and $p_2$ using the constraints

where we introduce an auxiliary variable $t$. We can express the condition that the line segment between points $p_1$ and $p_2$ be horizontal by saying $y_1=y_2$, that it be vertical by saying $x_1=x_2$, or in general that it lie at any given angle $\theta$ by saying

where again we introduce an auxiliary variable $t$.

However, we can’t demand that the distance between points $p_1$ and $p_2$ is at most a particular value, $d$ say, since to express this constraint, we would need to say something like

which is not a linear constraint. That seems like a bit of a shame, since distance-based constraints are very natural from a geometrical point of view. I’ll talk below about what it might mean to lift the restriction to linear constraints.

Finding solutions

First of all, let’s think about how the constraints that we impose restrict the space of solutions to our optimisation problem. If $\mathbf{\text{x}}\in {ℝ}^n$, then the linear equality ${\mathbf{\text{a}}}_i\cdot \mathbf{\text{x}}=b_i$ defines an $N-1$-dimensional hyperplane in ${ℝ}^N$. For concreteness, for the moment let’s set $N=3$. In 3-dimensional Euclidean space, the equation $\mathbf{\text{r}}\cdot {\mathbf{\text{n}}}_1=d_1$, where $\mathbf{\text{r}}$ is an unknown vector, ${\mathbf{\text{n}}}_1$ is a given unit vector, and $d_1\ge 0$ is a real number, defines a plane with normal vector ${\mathbf{\text{n}}}_1$ and perpendicular distance from the origin $d_1$, i.e. solutions $\mathbf{\text{r}}$ to this equation lie in the given plane. So the solutions to our optimisation problem have to lie in this plane, not just in ${ℝ}^3$. If we have a second equality constraint, defined by the equation $\mathbf{\text{r}}\cdot {\mathbf{\text{n}}}_2=d_2$, there are three possibilities: either ${\mathbf{\text{n}}}_2={\mathbf{\text{n}}}_1$ and $d_2=d_1$ so both equalities refer to the same plane; or ${\mathbf{\text{n}}}_2={\mathbf{\text{n}}}_1$ and $d_2\ne d_1$ or ${\mathbf{\text{n}}}_2=-{\mathbf{\text{n}}}_1$ and $d_1{d}_2\ne 0$, a situation where we have two distinct parallel planes and there are thus no consistent solutions for $\mathbf{\text{r}}$; or ${\mathbf{\text{n}}}_1$ and ${\mathbf{\text{n}}}_2$ are not collinear, the planes are distinct and meet in a line, to which our solutions $\mathbf{\text{r}}$ are constrained. In higher dimensional spaces, adding further equality constraints works in the same way: either we have a degenerate case, so there are no admissible solutions, or the additional constraints restrict the admissible solutions to smaller and smaller linear subspaces of the original ${ℝ}^N$ solution space.

In the end, after considering all of the linear equality constraints, we thus end up with a situation where we need to seek solutions in some linear subspace of ${ℝ}^N$, subject to our inequality constraints. Further, there always exists a linear transformation $(x_1,x_2,\dots ,x_N)\to (u_1,u_2,\dots ,u_N)$ that can simplify our view of this linear subspace so that we end up considering an optimisation problem in a lower-dimensional Euclidean space, ${ℝ}^M$ say, with $M\le N$, with only inequality constraints of the form ${\widetilde{\mathbf{\text{a}}}}_i\cdot \mathbf{\text{u}}\le {\tilde{b}}_i$ or ${\widetilde{\mathbf{\text{a}}}}_i\cdot \mathbf{\text{u}}\ge {\tilde{b}}_i$.

Next, let’s think about the role of the inequality constraints in restricting the solution space. Each inequality constraint of the form ${\widetilde{\mathbf{\text{a}}}}_i\cdot \mathbf{\text{u}}\ge {\tilde{b}}_i$ or ${\widetilde{\mathbf{\text{a}}}}_i\cdot \mathbf{\text{u}}\le {\tilde{b}}_i$ divides the space ${ℝ}^M$ into two half-spaces, one in the permissible solution region and one not. The final permissible region is the intersection of the permissible half-spaces from each of these constraints. A little thought shows that this is a convex polytope² in ${ℝ}^M$.

Here’s an example in ${ℝ}^2$. Let’s call our coordinates $x$ and $y$. Suppose we have the constraints $x\ge 0$, $y\ge 0$ (these two are normally included by default in the “traditional” statement of the setup for the simplex algorithm), $x+y\le 4$ and $x+2y\le 6$.

The figure to the left shows the constraints as blue boundaries, with the interior of the polygon bounded by the constraint lines being the set of permissible solutions. I’ve also show the contours of a particular gradient function (the red lines) and the resulting best solution (point marked in green). It’s pretty clear that adding more linear inequality constraints can’t make the permissible region anything other than a convex polygon.

It’s also pretty clear from this image that the optimal point, i.e. the maximum of the objective function $z$, is going to be found on the boundary of the polygon defined by the constraints. To see this, suppose that we select some point in the interior of the polygon as our putative “best” point. Then we can always go downhill along the contours of the objective function until we reach the boundary of the permissible polygon, thus finding a better solution than our originally proposed one.

In fact, the optimal solution, except in degenerate cases, is found at one of the vertices of this polygon. This is the key point that makes this type of optimisation problem more tractable than it might initially seem: even with large numbers of dimensions, we basically only need to solve a combinatorics problem over the vertices of our permissible polygon. In particular, we don’t need to think about what happens to our objective function in the interior of the polygon. This is a big deal, and we’ll see how lifting the linearity requirement on either the constraints or the objective function renders the problem much more difficult.

In essence, the simplex algorithm is a smart way of doing this combinatorial search along the edges of our permissible polygon, in a way that works for larger problems and with some special cases to deal with degenerate problems and to detect insoluble problems.

Lifting assumptions

What happens if we lift the requirement that constraints be linear? A general nonlinear equation $f(\mathbf{\text{x}})=0$ in ${ℝ}^N$ defines a manifold³ embedded in ${ℝ}^N$, so nonlinear equality constraints, instead of reducing our solution space to a nonlinear subspace of ${ℝ}^N$, will reduce it to a manifold $U\subset {ℝ}^N$ on which possible solutions live. Theoretically, this isn’t too much of a problem, but from a practical point of view, it might be tricky to deal with. In principle, we can find some sort of coordinate transform to make nice charts on our manifold $U$ we can use to think about our inequality constraints, but even in simple cases, care is required to make sure that our charts make a good atlas for the manifold. Suppose $N=3$, we call our coordinates $x$, $y$ and $z$ and we have a constraint that $x^2+y^2+z^2=1$, i.e. solutions lie on the surface of a sphere of radius one centred on the origin. Geometrically, this is no problem, but if we seek a single coordinate transformation to represent the two-dimensional surface of the sphere, $(x,y,z)\to (u,v)$ say, we get into trouble: we need at least two distinct charts to form an atlas for the 2-sphere. It may thus be necessary to continue to work in the original $(x,y,z)$ coordinate system, even though these are not independent degrees of freedom in the problem.

Nonlinear inequality constraints pose another sort of problem. If we have an inequality like $x^2+y^2\le 4$, i.e. $(x,y)$ lies inside a circle of radius 2 centred on the origin, although, given a linear objective function, the optimal point still lies on the boundary of the region defined by the inequality, finding that point is rather more difficult and is no longer a simple combinatorial optimisation over the vertices of a polygon, as in the simplex algorithm.

There is another problem with nonlinear inequalities. Recall that for the linear constraint case, the permissible region was always a convex polytope. In general, nonlinear constraints do not guarantee convex regions (think of the constraints $0\le x\le 1$, $0\le y\le 1$, $xy\le 1$, for instance) and this can lead to kinds of degenerate problem that do not exist in the linear case (for this set of constraints, optimising the objective $z=x+y$ leads to two solutions, one on each of the “corners” of the concave arc defined by the hyperbola equation $xy=1$). It seems like it might be possible to restrain ourselves to inequalities defining convex regions, which would mean that the permissible region formed from the intersection of these regions would also be convex⁴. Something to think about a bit more, particularly since there are optimisation methods designed for these convex situations.

Lifting the assumption that our objective function is linear means that we can no longer be sure that the optimal value lies on the boundary of our permissible region. An arbitrary nonlinear function can have all sorts of bumps and maxima within the permissible region. There are classes of nonlinear functions (in particular harmonic functions) where we can make definitive statements about extremal values within the permissible region compared to on the boundary, but these don’t seem like practical classes to use to restrict the choice of objective. All other things being equal, a linear objective is probably the best approach for now.

Conclusions and what next?

• Linear constraints and linear objective function are nice and easy to understand geometrically.
• General nonlinear constraints are potentially nasty.
• Convex nonlinear constraints might be easier to handle (there’s a whole field called convex optimisation about dealing with this situation) and it seems intuitively likely that most of the nonlinear constraints we’re interested in might be convex — distance constraints essentially define balls in the coordinate space, for example.
• Nonlinear objective functions break the whole “optimal solution on the boundary” idea. Best avoided if possible…

Next, I’m going to set things up to play with an implementation of the simplex algorithm in Haskell, along the way experimenting with an interface for specifying constraint systems.

After the rather dreary spy novels, time for something a bit more fluffy.

First, a couple by John Scalzi, who is an all-round good guy with a great blog, and a fine line in fairly traditional sci-fi. Zoe’s Tale is an alternative view of some of the events in the universe of Old Man’s War, told from the perspective of the teenage daughter of the hero of the original book. It’s an interesting experiment, to rewrite events already told from one perspective from a different viewpoint, one with quite distinct emotional responses and interests. Scalzi says that it was among the hardest things he’s ever written! It’s also interesting in that the protagonist and narrator is a teenage girl. I don’t think I’d have the balls to try to write anything substantial from that point of view, but Scalzi does, as far as I can tell, an excellent job. He says that he ran the book by a lot of female friends to check that he’d got the tone right.

The Android’s Dream, also by Scalzi, is a standalone novel introduced by a diplomatic incident precipitated by farting. The book gets sillier from then on. It’s amusing, if you’re prepared to accept the main premise of the book, which is to do with a sheep-human hybrid developed through non-consensual zoophilia (is zoophilia ever consensual) that plays a critical part in the ceremonial involved in the handover of power of the government of an alien power, sort of though not quite allied with humanity. They’re big lizards. Who communicate partially by scent. Hence the farting. Seriously, it’s quite a diverting little book, but very very silly.

Snuff: Pratchett, Discworld, Vimes. Does more need to be said? This is another Discworld book that deals with otherness and racism, people who are even more marginalised than dwarves and trolls. For me, the hero of the show? Willikins, Vimes’s manservant, a man with a history so shady you could use it as sunscreen for an albino rhino.

I’ve now read all of the Twenty Palaces novels by Harry Connolly, the latest being Twenty Palaces, A Prequel, which covers some of the backstory of Ray Lilly and the events that occurred before the “first” novel, Child Of Fire. I haven’t yet decided one way or another whether I think they’re any good. But I’m still reading them, so that says something, eh? They have some of the same feeling as spy novels, this sense of another world lying parallel to our own everyday world, where unpleasant people do unpleasant things to each other and everyday folks just suffer the consequences. I’m interested to see where he’s going with it all though, especially if we get to learn some more about where the spell books come from (and what they hell they actually are, since they certainly aren’t books!).

Similarly indecisive critical opinion surrounds Richard Kadrey’s Sandman Slim books. The latest is Aloha From Hell. It has all the usual stuff, with demons and angels (and other weirder things) doing battle, this time in a weirdly skewed Los Angeles in Hell which is an echo of the real L.A. (and you might argue, not all that much different). Kadrey’s books are more “mind candy” than Connolly’s: I get the feeling that Connolly is going somewhere with his stuff, but Kadrey is much more in-your-face with everything at once, with less of the mysterious slow reveal of the Twenty Palaces books. But what do I know? I’m still reading these too, although this may be the last one.

Door, Kairouan, Tunisia. April, 2005.

I was born in the UK in 1972. The “world leaders” who defined my youth were Margaret Thatcher, Ronald Reagan and Leonid Brezhnev. This may have something to do with why I am so bitter, cynical and generally misanthropic nowadays, although the star turns we’ve seen in the US, UK and former USSR since then may have helped too.

Before Reagan, there was Jimmy Carter. I remember him from the TV news at the time only very vaguely. Since missing re-election in 1981, Carter has worked on a vast range of issues, mostly to do with disease eradication in developing countries and election supervision in fragile democracies. That his name can send the US right wing into a frothing ball of fury seems to indicate that he’s been doing good things.

A recent interview in The Guardian brought home just what a great man he is. The description of his house and his manner make me think more of a Greek senator elected by lot than a modern American politician with all the schmaltz and money that goes with it. He’s a humble man who has lived his beliefs. That deserves a lot of respect.

The other thing that comes across in the interview is his intellectual engagement. I loved the story from his chief of staff that he only once was able to tell Carter something that he didn’t already know. If you have any sense, you don’t choose a leader you’d be happy to have a beer with (don’t know if Carter even drinks); you choose a leader who is intelligent, motivated and honest. Carter is all of those things.

A couple of more substantial (in length) books I read over Christmas were Neal Stephenson’s latest and the first of Tim Powers’ Fisher King series. I liked them both.

Reamde by Neal Stephenson

I can’t help feeling that, after Anathem, Neal Stephenson just wanted to write something with lots of guns and stuff blowing up. He does it very entertainingly, although the gun fetishism is a bit weird for someone unused to it¹. The plot is up to the usual Stephensonian standards of twistedness, and it’s pretty long, although it feels much less long than any of the Baroque Cycle books, probably as a result of rollicking along at a rather frantic pace from page one onwards. The settings, both real and virtual, are nicely done. My only real complaint is that some of the characterisation is a bit two-dimensional. The hero of the piece is a slightly enigmatic man-with-a-past-but-a-heart-of-gold; we have a taciturn former Spetsnaz man working in “security” (there really should be a place in the world for novels featuring former Spetsnaz operatives who now make balloon animals at children’s birthday parties); we have homicidal Russian gangsters; we have an intellectual super-villian (who happens to be a Muslim terrorist, a little disappointed with the hired help his pals in the U.S. are able to round up). James Bond meets World Of Warcraft via the Washington State Tourist Board, perhaps.

Anyway, it’s a lot of fun and it whiled away a few happy hours, but don’t expect another Cryptonomicon.

Last Call by Tim Powers

The first Tim Powers novel I read was Declare, which is a hard act to follow. Last Call is good, although I enjoyed it less than Declare, mostly I think because the connection to real history (the story of Kim Philby, in Declare) and the slight twisting thereof is where Powers really excels. Last Call isn’t set around any major historical events (that I know of, though I may very well be wrong), but it is connected to some other books by Powers, in particular Expiration Date and Earthquake Weather, which form a sort of loose trilogy with Last Call, and The Drawing Of The Dark, which is also related to the Fisher King myth (and, more importantly, beer and the Turkish siege of Vienna).

One aspect of all of Powers’ novels that I’ve read so far (the other one I’ve finished recently was The Anubis Gates) is “secret history”. Well-known and well-reported historical events are given a new gloss, where the most salient aspects are never known to the general populace, and the characters in the novels live in a parallel world (much like that of spy novels) where minor everyday happenings take on enormous significance. This otherworldliness or extraworldliness is at the forefront of Last Call: in this world, the arcana of the Tarot are real entities, Archetypes that underly all human striving. Since a regular deck of playing cards is basically just a cut-down tarot deck, Las Vegas is a centre of power for the operation of these entities, and for the establishment of the reign of a Fisher King for the West. Powers does a great job of depicting the power of chance and ritual in this world, with some genuinely creepy moments. The climax of the novel is entertaining, although perhaps a little predictable in some ways.

The kind of secret history that Powers writes is very engaging because it offers lots of hooks for a reader to anchor themselves: fantasy written in invented worlds can often leave one adrift, without any points of reference, but once it becomes clear what the rules are, the real history that underlies the secret provides signposts to help keep the reader oriented. The disadvantage, of course, is that it can be difficult not to stretch credibility. Not all of history can be secret history! Some things did happen more or less the way they are reported in the history books, without the intervention of supernatural forces or other strangeness².

It’s a very tempting approach to fantasy writing though, and I look forward to reading more from Tim Powers. (I’m slightly stalled on Expiration Date at the moment, as it’s not quite grabbing me, but I read The Anubis Gates in about three sittings.)

In 1998, the mathematical physicist David Ruelle wrote an odd little article, entitled Conversations on Mathematics with a Visitor from Outer Space, which appeared as a chapter of the very interesting looking book Mathematics: Frontiers and Perspectives, published by the IMU in 2000. I’ve not read the whole volume, although I think I’ve read at least one of the other chapters as a preprint. Most of the 30 authors seem to have written more or less “straight” articles, but Ruelle’s is different. He wanted to explore the constraints imposed on human mathematics by the structure and capabilities of the human brain. Perhaps constraints is too strong a word — “predispositions” might be better: human mathematics is necessarily tied to human brains, and human brains have evolved to their current state to solve problems that are quite different from the problems encountered in mathematics. One might thus expect human mathematics to have followed the “fault lines” in the Platonic edifice of Mathematics that are most easily appreciated using the mental tools that evolution has given us.

To explore this idea, Ruelle introduces the conceit of a visitor from outer space, a “galatic mathematician” pursuing doctoral studies investigating the nature of human mathematics. She describes some of the characteristics of the human mental apparatus that are relevant: limited short-term memory, and hence inability directly to execute complex algorithms; predisposition to see symmetry and pattern; human brains operate very slowly, even compared to human computers; the importance of geometrical and visual reasoning; and so on. Ruelle comments that despite our limitations, humans have managed to do some fairly complex mathematics, but the question is always: how much further could we go if we could transcend the limitations of our history?

Ruelle’s visitor tells him that it would be impossible for her to explain Galactic Mathematics, since galactic mathematicians have already undergone this transcendence: “an ancient civilization would have fixed such intellectual inadequacies by assisted coevolution, biological engineering, and so on”. Humanity must necessarily do the same if they are to go beyond Human Mathematics.

This reminded me very strongly of an after-dinner talk by Freeman Dyson at a conference at Caltech I attended some years ago. The conference was about the exploration of Mars¹, and Dyson’s chosen topic was, naturally enough, the human exploration of space. The main point of his talk, as I remember it, was that he didn’t think that humans would get all that far into space. At least, not humans as we are now. Too much effort expended on life support, too much danger from radiation or extended periods of weightlessness. Would it not make much more sense, he asked, for us to modify ourselves for life in outer space? Not overnight, of course, since such a process would involve a long period of experimentation and gradual adaptation, but eventually the idea would be to have people able to survive life in a weightless environment, with better resistance to ionising radiation, and perhaps even limited ability to survive in vacuum.

And if we might do that to travel and live in outer space, why not follow a similar process to allow us to explore the further reaches of mathematics? If arithmetic was no longer a question of rote learning but just came naturally, if complex symbolic manipulations were “obvious” instead of a lottery of sign errors, transpositions and copying slip-ups, if complex logical arguments flowed effortlessly at first sight, what might mathematics feel like to us, and how might it feel to be freed from the bondage of our (relatively) plodding squishy brains? What sort of mathematics would be produced by an entity for whom, for instance, the classification of finite groups could be held in the mind’s eye like a gem to be rotated and admired from every angle?

Of course, it’s pretty useless to speculate what it would be like to be such an “extended” mathematician: not so different from the archetypal example of imagining what it might be like to be a bat, although perhaps more like the bat imagining life as a human. Nevertheless, thinking about “extended” or modified human intellects is tied to another topic I’ve been idly thinking about recently, mostly as a follow-on from the Stanford AI course. That’s the question of “common sense”. I’ll write about that another time.

I’ve been very lucky so far in my life, and have had many opportunities to work and study in interesting places. I’m not sure I’ve made the best use of those opportunities — there’s much that feels unfinished or that didn’t quite work out the way I’d hoped it would — but these “past lives” have given me a lot of experience working in different fields, living in different places, hanging out with different kinds of people.

Blogging is inherently self-indulgent: you have to assume that someone out there is interested in reading your maunderings. (Not really. Honestly, I do this for my own amusement.) The ultimate in self-indulgence has to be autobiographical blogging and reminiscing, where some old fart bends everyone’s ear about the “good old days”. Anyway, since this is for my own amusement and for a bit of writing practice, I don’t really care. I’m going to be brazenly self-indulgent and start writing “Past Lives” articles to see if I can dredge up any interesting memories.

In October 1997, I went to Cambridge to spend 9 months at DAMTP¹ to do the course that’s generally called Part III Maths, although it has the official title of the Certificate of Advanced Study in Mathematics. It’s a taught masters’ in mathematics, where you can choose from around 100 different courses on all aspects of maths (pure and applied). It’s examined at the end, and has a reputation as being one of the harder mathematics courses in the world.

It’s also used as a gatekeeper to determine who is offered maths PhD places in Cambridge. I went to Cambridge fresh from a stint in the delightful world of international finance², all bright-eyed and bushy-tailed with the idea of doing a PhD in theoretical particle physics. Which meant that I needed to work hard and do well to have any chance of getting a studentship.

Part III is a funny sort of course. Cambridge undergraduates can do it as an optional fourth year of the standard undergraduate maths programme, but people also come in from outside to do it (like me). It’s very much a sink or swim affair. The courses are hard, the exams are harder, and there’s little or no pastoral guidance or help. Go to lectures, sit at home thinking hard and doing maths, go to tutorial classes, rinse and repeat (possibly with a pub visit or two in there somewhere). The lecturing ranges from the abysmally poor (no names…) to the very very good (Peter Goddard lecturing on conformal field theory sticks out as being one of those³), no prisoners are taken, and you quickly get the impression that you’re surrounded by some very smart people indeed.

I mostly took courses in theoretical physics (quantum field theory, general relativity, elementary particle physics and group theory, more quantum field theory, conformal field theory, plus some other stuff that I don’t remember any more). Mostly it was fun, and I learnt a lot. Some of it was less fun, and I got quite seriously turned off the idea of a particle theory PhD. I think the straw that broke the camel’s back was talking to one of a PhD student who proudly showed me a six-inch stack of paper that was his work for the last couple of months. A single calculation. He described to me what it was about, and I understood perhaps 10% of what he said. “Could it be real?” I asked. “Oh no — no way. Definitely not real. Just a thought experiment. Just something that hadn’t been worked out.” And there was the rub. At that time, and maybe still, the life of a student wasn’t likely to be much more than filling in corners left by the big names. Now, I can look back and recognise that that wasn’t necessarily a bad approach to take for PhD students: they get to learn about the subject on a well-constrained problem, a problem with a definite end-point that’s attainable in three years. At the time though, it seemed pretty uninspiring, so I kicked back and didn’t work terribly hard for the rest of the year, then lit out for Oxford to work on remote sensing of the atmosphere of Mars…

In retrospect, Part III Maths for me served as a good interregnum, a short period between working and starting as a PhD student. It was also a period where I went through big changes in my personal life, splitting up with a long-term girlfriend (fiancée at that time), with all the trauma that that entailed. I wasn’t really in the right frame of mind to truly benefit from the teaching offered in Cambridge at that point, but it was still useful experience. I still don’t know what the $L$s are though.

Speedy Dog goes where she likes!
Well, maybe not any more, little girl…

Yesterday evening, we had a meeting with Martine, the dog behaviourist who’s helping us with Winnie. Winnie has come a long way from the terribly frightened little dog that we picked up from the animal shelter four months ago. She’s no longer terrified of everything in sight, she runs around with her tail up, likes to play with other dogs, likes to sniff in the bushes, likes to run crazily through the fallen leaves in the forest. She’s still jumpy and easily gets scared of people in the street or sudden noises, but she’s come a long way.

Unfortunately, we might have taken the gently-gently treatment we were pursuing to help her stop being so fearful a little bit too far. She doesn’t come when called, unless it suits her, and we had been escalating the treats we offered her to a slightly ridiculous degree. Time for a slightly stricter regime, to let her understand her position in the household. In the long run, it will be better for all of us: no more frustrating walks in the park where she won’t come back, no more worries as she wanders off for 15 or 20 minutes at a time, well out of sight, and safer walks near the road. She has no idea at all that cars and roads and human things in general can be dangerous, and will try to bolt into the road if she sees something really scarey, a bin bag, for instance…

So, we went to a nearby park with a dog area and let her run around with Jedi, Martine’s “regulator” dog-in-training, who is a beautiful Staffie cross. Martine’s diagnosis? Laughing, “She doesn’t give a toss, does she? She just doesn’t care! She’s a confident little dog. She thinks she’s the boss!” We tried walking around the park and hiding behind trees to see if Winnie would notice the absence of her human protectors and come looking for us. Eventually, eventually… Once she had exhausted the sniffing possibilities and was wondering why it was so quiet, she came looking. But not enough.

The prescription is Puppy Purgatory for 10 days or so. No talking to her, no cuddles, no games. Commands, OK. Praise if she comes when called, OK. Day-to-day chat, incessant “Good girl!”, belly rubs, tug games, no. Not for a week and a half. It’s hard. Harder for us than for her, I think. She just doesn’t quite know what’s going on, but we have to ignore her soulful eyes and wagging tail.

But hard as it may be, it seems to be paying off a little already. We’ve only had one day of this regime, and Winnie is already showing more interest in us, at home at least. I did a little bit of clicker training with her in the flat after dinner, and she was doing sit-stay-come pretty nicely: quite a bit better than usual. We’re going to go for a walk in the garrigue near Martine’s house next weekend, and we’ll see how much better Winnie is in terms of recall and staying near us then.

In the meantime though, Puppy Purgatory for all three of us…

Mt. Cain, Vancouver Island. New Years, 2010/2011.

I’m a sucker for nature photos. One of our annual outings when we lived in Bristol was to the Nature Photographer Of The Year exhibition at the city museum. Going by the crowds there, we aren’t the only ones who like this stuff.

So, here’s some eye candy.

First, The Boston Globe has The 50 Best Photos From The Natural World. I don’t know about the 50 best, but they’re pretty good!

Like old stuff? Here’s some old stuff. Some of the oldest there is. Photos of some of the oldest living things in the world. And if you like thought-provoking images of old organisms, check out Mike Iverson’s rendition of “Bristlecone Pine”, an image in sound.

And lastly, let’s look at some eyes. And what eyes. Suren Manvelyan took a simple idea (photograph ideas) and produced something truly beautiful. Be sure to check out both the images of animal eyes and those of humans. The variation in patterns is fascinating.

I’ve been thinking for a long time, mostly privately in my idle moments, about the sort of house I’d one day like to build for myself. I have some ideas that I think are pretty cool: multiple small wooden buildings, connected by walkways, with the main building on a slope with a stepped living area; a separate Japanese-style bathhouse; an octagonal library/study with windows all around; a cellar dug into the side of the hill.

Of course, all of this is nothing more than idle daydreaming. Recently though, Rita sent me a link to some pictures of cabins (sheds, huts, that sort of thing) that capture a lot of the spirit of what I’ve been thinking of. You can see them here (yes, the link is safe!). Some of these places look heavenly.

Uh-oh. Getting a bit behind on this 30-day challenge. Time for some shorties…

Not really a “past life” as such, but something interesting I did a while ago that was brought back to mind by the Costa Concordia cock-up. I’ve not spent a lot of time on big ships, indeed until 2007, I think the only larger vessels I’d been on were ferries, cross-Channel or around the Greek islands.

In April 2007 though, I went to Canada for a big trip, mostly to visit some potential future places to live and work and to attend a workshop at the Banff International Research Station. I flew out to Vancouver, took a ferry to Vancouver Island where I visited UVic (ended up working there for a couple of years afterwards), then travelled east by train. I made it as far east as Halifax, Nova Scotia, all by train. That was a pretty cool experience in itself, but I’d had a wacky idea for how to get home from Canada to the UK. It’s possible to book passage on container ships under some circumstances, which I decided to do. It’s a tricky process, not super cheap, and the logistics of making the rendezvous with the ship turned out to be a bit more “interesting” than I expected.

At this time, the M/V Flottbek was sailing a triangular route between Montreal, Liverpool and Antwerp, so my plan was to meet the ship in Montreal and spend a week travelling to Liverpool. Four days before the ship was supposed to sail, I called the port agent from Halifax, letting him know who I was, that I was sailing with the Flottbek in a few days and asking what to do about getting to the ship. “Eh, if you’re sailing on the Flottbek, you won’t be sailing in a few days! You’ll be sailing tomorrow. She’s loading in Montreal right now!” Oh dear. Massive panic. Ran to a travel agent, luckily got a ticket for a flight back to Montreal that evening, taxi to airport, flew back to Montreal, six hours sleep then presented myself at the gates of the container dock in the morning (after a complicated explanation in French to a Québecois taxi driver that yes, I really did want to go to the container port).

I spent seven days on the Flottbek, under the capable care of the German master and chief engineer and they Philippino crew. It was a very interesting experience, though the life of a crewman on a container ship isn’t one that I would want to share any time soon: 9 months on, 3 months off for the men, or a generous 3 months on, 3 months off for the senior officers. Being away from home for that length of time, all the time, sounds more like something out of a Patrick O’Brien novel than real life. Still, for seven days, in spring, with calm seas and beautiful sunsets, it’s something I’d recommend to anyone who has the time.

Everyone on the Flottbek seemed extremely competent and the captain was a serious serious man who’d been a seafarer all his life, having grown up on Heligoland and having a little boat of his own as a boy. I can’t, not in a thousand years, imagine him or any of his crew doing what the master of this Italian cruise ship did. The Flottbek had run aground once, on a shoal in the St. Lawrence River, on the way from Montreal to the open sea, but that was the fault of the Lawrence River pilot (thankless job: shifting shoals and big ships don’t make for a stress-free life).

Along with a lot of other people, I’m a big fan of cephalopods. I’ve dived with squid and cuttlefish and have watched octopus while snorkelling. I particularly remember following one cuttlefish across a reef in the Philippines, watching its mesmerising pattern display, until it got bored of the clumsy and noisy thing with too few appendages plodding along behind it and made off for deeper water.

Octopus, in particular, are smart little critters. One of the saddest things I’ve ever read was a section of a book about octopus physiology¹ that talked about the effect of certain nervous system lesions on the behaviour of octopus — these were lesions induced by human experimenters, of course. The writer talked about how the cephalopod victims cowered at the back of their tanks and clearly were less than keen on being used as experimental subjects. It stuck in my mind as the only place in the book (otherwise a good and thorough treatment of octopus physiology) where the author seemed tempted in any way to anthopomorphise or to ascribe emotions or feelings to the octopus. That’s why Octopus vulgaris is the only invertebrate protected under the UK’s animal experimentation laws…

Anyway, octopus are very cool and a fascinating model for non-human (and non-vertebrate!) intelligence. A recent article in Orion magazine does a great job of getting across just how amazing these creatures are. Go and read it. If they lived a bit longer and could be trusted not to molest the dog, I would love to have an octopus to live with us. Alas, I don’t like to imagine what would be the result of Winnie versus a Pacific giant octopus. Messy, for sure.

Carnedd Llywelyn, North Wales. January 2006.

Some interesting things that passed through my RSS reader recently:

Food pairing and flavour networks
This is kind of interesting. Why do cheese and bacon (if you eat that kind of thing) go together so well? Or asparagus and butter? Or caviar and chocolate? (Apparently. If you’re Heston Blumenthal.) The hypothesis was that these “paired” foods had many flavour compounds in common. The data presented in this preprint seems to confirm that idea for Western cookery, but contradict it for Asian cuisine. There’s a lot more in there about the evolution of recipes, clusters of ingredients, and other network analysis goodness.

Economists doing something right?
The people at Crooked Timber always have a lot of interesting things to say. This is a summary and some discussion of a meeting at the New Economics Foundation talking about the need to cut consumption and spread wealth around by redistributing working hours. From a personal point of view, the idea of earning a reasonable salary from a 21-hour working week and having time to work on personal projects, do some volunteering, spend time with Rita and Winnie, all sounds great. From a social point of view, a gradual redistribution of working hours to reduce unemployment and spread income around more fairly also sounds fine. It seems unlikely to happen, if only because NEF seems to be the only group of economists who can bear to think about the end of economic growth and a transition to a steady-state economy. When I listen to mainstream economists speak, I have this image in my mind of a train racing along a bridge which is being frantically cobbled together bit by bit as the train approaches. Sometimes the train gets closer to the edge, sometimes it backs off a little way. But in the long run, the bridge builders can’t win. They’re just going to run out of stuff to build more bridge.

Some good news
To leaven the economics misery, this is really good. Polio is on the way out in India: not one new case in the last year!

For a brief period some years ago, I worked for an engineering firm that did sonar work for the (UK) navy. Many of the people in the company were ex-submariners, who had served both on the Swiftsure/Trafalgar attack boats and on the Vanguard missile boats. These were all pretty solid people, and I remember one in particular who delighted in telling stories about his time working his way up from a “baby sailor” to naval attaché to the British embassy in Washington, D.C. His most amusing tales included a reenactment (with actions) of the joys of carrying soup tureens around crowded submarines, and the entertaining ability of US spy satellites to dip into the atmosphere to take a closer look at interesting sights, like Soviet mini-subs stuck on their motherships in their pens.

A slightly less amusing story revolved around the hubris of submarine commanders. He sent some of us a photograph showing £20 million of towed sonar array snarled up around a mooring buoy, all because the sub commander couldn’t be bothered to wait for the divers to help reeling the thing in. A sad navy man with a big beard stood gazing at the pile of expensive spaghetti, looking like he might burst into tears at any moment. Not the navy’s finest moment.

One of my main jobs was to work on sonar system performance measures, which were used by the Ministry of Defence to try to ensure that they weren’t being diddled by the main contractor for the sonar system in question. Terrifyingly large amounts of money were involved, and some terrifyingly bad engineering decisions. The contractor decided to “buy British” for microprocessors for the beamforming hardware and designed a system using the Inmos Transputer. Oops. Inmos were assimilated into SGS Thomson, the Transputer shortly thereafter became extinct, and a sharp about-turn was needed to redesign the whole processing chain to use off-the-shelf PC components instead of the custom hardware that they had planned for.

It was an interesting little period for me, though I hadn’t wanted to work on the naval side of things. The company also did a lot of aerospace work, and I had nominally been hired to work on that. The structure of the company meant that I got poached for sonar stuff, got grumpy and left.

This company did have one extremely good thing they did though, one that I’ve not yet seen anywhere else (although it may be a standard approach in some kinds of engineering companies). All of the details of all non-classified projects that the company had going, in all of the fields where it worked, were available for perusal via the company’s accounting and time tracking software. Each engineering employee kept track of the time they spent on whatever projects they were assigned to (no surprises so far), but it was also possible to look through any other projects, see what work remained to be done, to see who was assigned to do the work, to see any open slots where people with particular skills were needed, and if you had spare time, to put yourself down to do that work. It was a very efficient system, and it actively encouraged people to talk and cooperate across project boundaries within the company. I think that was probably one of the biggest lessons I took from my time there. That, and, if you have 1500 m of towed array to reel in, wait for the divers…

Today, we had an early morning outing to the Plage de l’Espiguette, a big beach about 30 km east of Montpellier. We got up at 6:15, picked up the Modulauto car at 7:00, persuaded Winnie into the back (she’s still a bit scared of cars…) and off we went. We arrived at the car park before sunrise and were the only people there! Miles and miles of dunes were ours alone! Much frolicking there was. And climbing up and down the dunes. And digging. And chasing of sticks. And rolling around pretending to be a dog (just Ian, since Winnie is a dog, and Rita isn’t quite as silly as me). It was a lovely morning, and the dunes were very pretty indeed. We’ll be going back there again, some time before the summertime beach dog ban comes in. And hopefully, with more doggie company next time, since I think Winnie could have done with a four-legged buddy to run around with. I do my best, but I don’t quite have what it takes for serious boisterous play, since I can’t run at 40 km/h for hours and hours at a time…

After referring to a paper in Agricultural and Forest Meteorology in the last post, I remembered something I read on Crooked Timber the other day. Many people have a pretty good idea of how toxic the world of academic publishing is — if you don’t, take a look at this recent Guardian piece by Mike Taylor, who lays out the issues pretty clearly.

But what can individual researchers do to fight the power and toxic influence of the big publishers? Personally, I only submit papers to journals with an open-access policy (the EGU journals like Biogeosciences are good here, especially with their “open peer review” process), and try to encourage co-authors to do the same. But I am among the tiniest of tiny fish in a very big pond, so what I do has an influence barely measurable in pico-Seldons¹. Now some of the big fish are taking an interest, and have started a movement! You should sign up if you’re involved in publishing scientific results and care at all about the free dissemination of knowledge.

One of the things I’m responsible for in my day job is a phenology webcam at our experimental site at Puéchabon. The idea of this is to observe colour changes in the canopy of the forest up there, with a view perhaps eventually to replacing manual phenological observations with information drawn from digital photos. In this sense, phenology means things like when flowers come out, when fruit forms, and so on. The Quercus ilex (holm oak) forest at Puéchabon is evergreen, so you don’t get the same spectacular seasonal changes in leaf colour that you see in deciduous forests, so we’re not sure whether this is going to work — the changes we’ll be looking at will be a bit more subtle.

We installed the webcam on the 20 metre tower at Puéchabon in July and have been collecting images every half hour since, apart from a few outages. The images aren’t all that exciting (below left), but it’s quite nice to watch the changing angle of the sun on the canopy throughout the day. Very preliminary data analysis indicates that we will at least be able to see something changing through the seasons, even if it’s not abundantly clear to the eye from the images. A recent paper by Sonnentag et al. (unfortunately behind the evil Elsevier paywall…) gives some ideas about how to analyse these images. In the plot below right is a time series of the mean chromatic green component of the part of the webcam field of view that covers the forest canopy. This is defined as $g_{cc}=G/(R+G+B)$, where $R$, $G$ and $B$ are the red, green and blue components of the digital image. There is definitely some seasonal variation, although with a lot of noise. I want a full year of data, so that we can see the new growth in the spring, before doing some more detailed analysis, since there are definitely still some scene illumination artefacts in what we’re seeing. It’s encouraging that we can at least see something though!

Webcam image

Chromatic green time series

The potential difficulties in data analysis aren’t helped much by the local criminal low-lives. We’ve had four major thefts at Puéchabon in the last six months, the latest of which involved break-ins at both of the sheds on site and theft of a load of gas analysis and data processing equipment (including the PC collecting the webcam images), as well as power conditioning equipment and batteries used with the generator we use to power the site. Until the third theft of the year, we did have a nice big set of photovoltaic panels for power, but some enterprising lads took a truck up there and carted them all away. Probably 20 square metres of panels, gone in one night. It’s pretty weird. Much of the stuff these idiots have taken is basically worthless to them. It’s hard to imagine them successfully fencing a gas analyser or a fluorometer at one of the local flea markets! They seem to be taking stuff just because it’s there and it’s shiny.

It’s not quite clear what’s going to happen at Puéchabon now going forwards: we can’t afford to have guards at the site 24 hours a day to protect stuff, and it’s getting kind of silly when Alain and Jean-Marc, the technicians who go up there weekly, come back more often than not with long faces and another tale of broken locks and missing stuff. The whole point of a site like this is to have continuous observations, and if those keep getting interrupted by equipment walking away, it reduces the value of the remaining data quite a lot. We’ll have to see what happens.

day-job

Trident is the UK’s independent nuclear deterrent. Except that it isn’t independent and it doesn’t deter anyone. It is nuclear though, so we do have that. Apparently, Trident is our ticket to sit down with the big boys, to strut around as part of the Nuclear Club, to be on the right side of the Non-Proliferation Treaty (the side that does all the shouting about non-nuclear states and quietly does nothing about its own disarmament obligations). Narrow-minded political commentators on the right in the UK regularly pull out this argument as a reason for keeping or renewing Trident, despite its obscene cost and highly dubious (at best) morals. And that tells you more or less all you need to know about international politics at the highest levels. It’s just Playground Bullies Redux. He who has the biggest stick wins.

Trident isn’t independent (the missiles are leased from the US, the warheads, although assembled at Aldermaston, are based very firmly on US technology, the re-entry vehicle navigation software is American, and so on), but Scotland might soon be. Which might pose a tiny little problem for the Royal Navy, since the only places in the UK where they can harbour and replenish their missile boats are in Scotland. Oops.

From the Guardian this morning:

Asked during the referendum debate in the Scottish parliament last week whether the government of an independent Scotland would do a deal to keep Trident, the first minister Alex Salmond replied: “It is inconceivable that an independent nation of 5.25m people would tolerate the continued presence of weapons of mass destruction on its soil.”

I don’t know if it will really happen. People tend to get all serious about nuclear weapons, and there might be some sort of “deal” done, backed by threats, but I honestly love the idea of Alex Salmond wagging his finger at David Cameron yelling “Oot! Oot, ya wee eejit! An’ tek yon dam’ missals wi’ ye!”.

My Freude is quite quite schaddly tonight…

I recently read an interesting article by John McCarthy that set me thinking. The paper is called The Well Designed Child¹ and talks a bit about about the old nature versus nurture debate: to what extent is a newborn child a blank slate, and to what extent are human intellectual capabilities intrinsic and instinctual?

To me, the whole question of nature versus nurture has always seemed odd. It’s clear that both aspects are important. We are all limited to one extent or another in what we can do purely by the physical parameters of our existence: as much as I would like to teleport myself to Europa to frolic in the sunless sea beneath the ice, I can’t, and never will be able to. Equally though, the potential that exists in all of us at birth can be squandered, and without education and opportunity, none of us can make of ourselves all that we might. So, we need a bit of both.

Perhaps the source of the debate lies in observation of other creatures: beavers don’t go to Beaver Engineering School to learn how to build dams, and spiders don’t attend art college to learn how to make beautiful webs. These abilities are intrinsic, evolved, part of what Richard Dawkins called the “extended phenotype” of these creatures, behavioural manifestations of developmental and neural structures that are passed down from generation to generation. Humans don’t appear much like that. We do have to go to school to learn how to build dams or weave webs or to do any of the thousands of things that we do. The question then is: what intrinsic capabilities are we born with? What is on our slate, since it seems as though it ought not to be completely blank?

Human babies don’t seem all that capable, compared to the young of other species. That’s something of an illusion though, since humans give birth to developmentally very premature infants, mostly as a result of evolutionary compromises to allow us to have large brains combined with an upright posture. I don’t know exactly what human developmental stage would be a good one to use for comparison with other species, to try to decide what intrinsic capabilities we might have. However, there are three fairly obvious capabilities that do spring to mind. These are the capability to learn, the apparently intrinsic facility for language acquisition, and a basic grasp of what, for want of a better term, I’ll call “folk physics”.

The first two are aspects of human development that are quite well known and explored, although none of the questions are yet really settled, as far as I can tell. I’d like to concentrate on the third, this idea of “folk physics”. What do I mean by that? I mean the kind of common sense knowledge of the world of physical objects that we all have — two objects cannot occupy the same space; if one object passes behind another, it does not cease to exist; objects further away appear smaller; gravity gives a definite up-down orientation to the world. You can come up with lots more examples. These things seem almost too obvious to state, but they are still knowledge of a kind that either has to be intrinsic or acquired, and they constitute a large base of shared facts that we all need and use for living in the physical world.

What McCarthy’s paper really set me to thinking about though, was what might minds be like that had different sets of common sense to our own? Uncommon sense, if you like. A couple of examples will give an idea of what I’m talking about. First, think about an entity evolved to live in vacuum, in a micro-gravity environment. Human-default common sense physics seems to be Aristotelian more than Newtonian, but that’s hard to imagine for a creature that never feels air resistance and doesn’t have a large local gravity field to give it a sense of up and down. Just for an example, how would such a creature refer to directions? No up and down may lead to no left and right either, although near and far seem as though they should still be usable. How about referring to directions towards items, rather than abstracting into categories like up and down? That might work, but it would have a fundamental impact on the way that such entities might use language to refer to each other and to objects and places. It’s interesting to speculate what sort of idiomatic usage might arise from this different perspective on the world.

Second, another example, more remote from human experience in many ways. What sort of common sense, and what sort of mind, might an entity inhabiting a computer network (the internet, say) develop? Here, there is no “physics”, since there is no physical. There are no physical directions at all, although the environment has a rich topological and metric structure. The more I think about it, the more I find myself at a loss to imagine what the existence of such an entity might be like. It led me to think that true artificial intelligences, if we ever succeed in creating them, may be even stranger than they are usually portrayed in fiction².

Over the course of the month of January, I:

• published 24 blog articles. Less than thirty, but not a crushing failure.

• had 13 days when I didn’t write anything, compensated by two days when I wrote 3 articles and 3 days when I wrote two.

• wrote about 17 articles that I would consider “proper” articles, i.e. involving some actual writing. The others were just links or photos.

• was ill for a few days, which could be a viable excuse for around 5.5 missing articles, which means that I (morally) only missed my target by half an article. Compared to people who liveblog their gall bladder surgery, I feel kind of small. Maybe it’s not so much of a moral victory after all…

So, was it a useful thing to do? Did I learn anything? I certainly did:

1. It’s quite hard to come up with substantial and substantive articles every day. I could wiffle on about random subjects without too much difficulty, but things that involve thinking take a bit more time.

2. The writing, in terms of putting words down one after another, isn’t something I find too hard. (I think I knew that before, but it’s nice to have it confirmed.)

3. Writing reviews of other peoples’ work (books, papers, etc.) is an order of magnitude easier than generating original ideas yourself. There’s an in-between stage that’s quite good for blogging, I think, where you get some interesting ideas from reading or thinking about something, not substantial enough to turn into a longer piece of writing, but perhaps enough to blog about. That provides quite a nice incentive for working through my enormous literature backlog…

4. Ultimately, the most satisfying things for me to write are more meaty. I didn’t manage anything I would consider in this category this month, mostly because of the perceived pressure of at least thinking about producing something every day, but I have a few ideas I’m going to work on.

Given the other things that I want to do with my (limited) time, I think a more reasonable goal is 2–3 articles per week, and that’s what I’m going to try to stick with for the next couple of months. We’ll see how that goes.

What about other 30-day challenges? Well, I have one lined up already. Starting tomorrow, I’m going to try to do 20–30 minutes of German study per day. We’re going to be moving to Austria in a few months, and I’d like to be able to do a bit more than order beer and pretzels once we’re there!

I spent the period from October 1998 until November 2001 as a graduate student in the atmospheric physics department in Oxford. This all started out swimmingly.

The project I was working on was a space mission to remotely sense water vapour in the atmosphere of Mars, using a novel infrared radiometer on the Mars Climate Orbiter (MCO) spacecraft¹. I was going to write software to perform retrievals of temperature and water vapour profiles. MCO was slated to arrive at Mars near the end of my first year, so there would be real data to work on. Exciting stuff.

The first year was great. I learnt a lot of new and interesting things about remote sensing and Mars, and started developing algorithms and software. I visited the Jet Propulsion Laboratory in Pasadena, went to a conference at CalTech, and even won a departmental prize for my first year report. All very sweet.

A week or so before Mars Orbit Insertion, I went on holiday to the Canary Islands with my girlfriend, where we explored the cloud forests of Gomera, drank lots of cheap wine and stayed with some friends of her family. It was lovely and relaxing. We got back to my girlfriend’s place in Brighton afterwards and settled down to read the Sunday newspaper. “Here, isn’t this something to do with your work?” she asked. “‘Mars mission lost’, it says.” Uh-oh. A quick phone call to my supervisor revealed the gory details. This was the famous “metric-imperial mix-up” (it was more complicated than it sounded, but that was no excuse) that resulted in MCO being steered into the planet instead of into an orbit around the planet. Everyone in Oxford was furious, as this was the second time a version of this instrument had been lost.

Back in Oxford, I pondered my future. Friday lunchtime “planetary” beers were somewhat legendary in the department, “lunchtime” often stretching until about seven in the evening. At least one respectable visiting scientist had been discovered sleeping under a bench in the University Parks on Saturday morning, having got “lost” and being unable to find his way home or back to the department (the physics department in Oxford is next to the University Parks). This same respectable scientist afterwards referred to the planetary group as “animals” and refused to visit again. Needless to say, MCO-related sorrows were drowned and drowned again, with an additional toast or two to send them on their way.

Everyone in the department was depressed about the turn of events, and this led me to make a decision, that in the light of hindsight², was rather foolish. I could have stuck with the work I was doing, built an instrument simulator to generate synthetic data based on likely Martian atmospheric profiles, finished developing retrieval algorithms which I could have tested on the synthetic data. The result would have been a perfectly respectable PhD thesis. That seemed like a dull and unadventurous approach to take. Oh, the folly of (relative) youth.

Instead, I chose to switch projects (and supervisors). I was going to develop optical delay line hardware and associated control software for a space interferometry mission. When I think back on this decision now, my reactions are something along the lines of “What? What what? What the fuck?”. It’s easy to enumerate the ways in which this was stupid: I had only a physics undergraduate’s grasp of optics and mechanical engineering, knew little about technical drafting and next to nothing about actually making things, I hated electronics as an undergraduate and had avoided it like the plague ever since, there was no money to fund this “project”, I had already spent a year on learning all about Mars and all things Martian, as well as having a lot of software already written for doing atmospheric retrievals from the PMIRR instrument. This was an idea that should have been smothered in the cradle.

As it was, I didn’t get very far. First thing I needed was a platform to test whatever delay line hardware I came up with. That meant an interferometer. Did we have money to buy a nice off-the-shelf metrology instrument? No, we didn’t. My supervisor said, “Well, I suppose you’d better make one then.” Right-oh. High precision machine shop work, optical system design, radio-frequency electronics, data processing electronics. What could possibly go wrong?

I accumulated piles of bits for an interferometer, most of which I made. I designed, built and tried to make work complicated electronics. (They never worked.) At the end of three years, I quit without a PhD and feeling pretty bitter about it all.

Despite that, looking back now, I don’t feel as though those three years were wasted. The outcome was less than ideal, admittedly, but I picked up an amazingly varied set of skills. I spent a lot of time in the workshop alongside people who really knew what they were doing, I learnt a lot about CAD, I learnt more about electronics than I ever really thought I wanted to know³. None of these were things I would have been exposed to otherwise.

Ten years later, part of me still wishes that I had had the good sense to stick with the remote sensing project so I could get a nice quick PhD and go and work for NASA. But the greater part of me doesn’t regret anything. Apart, perhaps, from some of those Friday “lunch” outings.

by Francis Spufford

How do you make a book about central economic planning in the Soviet Union into an entertaining page-turner? You do what Francis Spufford did with Red Plenty, a book that is almost impossible to classify, a mix of fact and fiction and dramatised might-have-beens, plausible if not quite verisimilitudinous¹ narratives populated by a melange of real historical personages and imagined characters.

It’s not quite fiction, not quite history. Some reviews that I’ve read weren’t comfortable with the changeling mood of the book, but I thought it was great. It’s hard to bring to life the enormity of the project, and subsequent events and the all too real lies and corruption that surrounded much of what went on in the Soviet era have obscured the visionary nature of the enterprise, but Spufford gets it across through uncovering the motivations of some of the main actors. Kantorovich and his colleagues really did want to make the world a better place, to alleviate the suffering that they saw as arising from the unequal distribution of wealth in market economies, and to use science and mathematics to make the lives of ordinary working people more pleasant.

Eventually, the project failed, for many reasons, but there was a period in the 1950s where the expansion of the Soviet planned economy genuinely had western economists and politicians worried. The stagnation of the Brezhnev era that followed had many causes, some related to the failure to diversify the economy of the USSR beyond heavy industry and basic agriculture, but the very real achievements of the Soviet planners shouldn’t be understated. Red Plenty gives a realistic and sometimes cynical portrayal of life in a system supposedly based on equality and scientific planning where a word from Khruschev over the telephone condemned a whole village of striking workers to death.

The cynicism is brought into stark focus by the frequent use of genuine Soviet era jokes throughout the book, all of them taken from the PhD thesis of Seth Benedict Graham². Here’s a typical example:

Yuri Gagarin’s daughter answers the phone. “No, mummy and daddy are out,” she says. “Daddy’s orbiting the earth, and he’ll be back tonight at 7 o’clock. But mummy’s gone shopping for groceries, so who knows when she’ll be home.”

A lot of research went into the book: the representations of daily Soviet life feel very authentic, and the characters of some of the historical personalities who appear are finely detailed. Highly recommended.

To compensate for the psychopathic driving habits of the French, the Mediterranean climate is often offered up as a benefit of life here. For those of us who are not all that keen on temperatures in the 30s for weeks on end in the summer, that’s not much of a compensation. However, today, I would offer another piece of evidence that this “Mediterranean climate” isn’t all it’s cracked up to be. Here’s the swimming pool in our residence, with a nice layer of ice all across it. And yes, Rita is wearing a down jacket, an article of clothing more common on the ski slopes and high mountains than the Mediterranean coast…

Here’s a rag-bag of entertaining things…

Whatever: John Scalzi’s blog
John Scalzi writes entertaining science fiction books and is also an all-round good egg. His blog, Whatever, has been around since forever (in internet terms). Scalzi has a magic touch when it comes to moderating comment threads: Whatever must be the politest place on the internet, all without threats or banning or rudeness from John. Scalzi’s books tend towards the light and funny, but his blog posts are sometimes more serious. Here are a couple that really stray dangerously close to thought-provoking: Being Poor, Things I Don’t Have To Think About Today.

Mathematical Fiction
Yes, everything is there to be found on the internet. Even a page detailing more or less every work of fiction involving mathematics even tangentially, scored for “Mathematical Content” and “Literary Quality” (although they do need some graphs to show whether there’s any sort of correlation between these scores!). There’s some good stuff on there, and it’s quite interesting just to browse along the “similar story” links to see where you end up.

Structured Procrastination
I’ve never been a fan of the Getting Things Done type of self-help/self-organisation book, but John Perry, a philosophy professor at Stanford, has what sounds like a perfect recipe for making progress on all those tasks that get pushed aside in favour of more interesting things. The key idea seems to be to make a task list with tasks at the top that you’re never really going to do (“Write novel.” “Learn Icelandic.” “Found religion.”) then use your other tasks as excuses to avoid these. That way you can trick yourself into actually doing things. It sounds like a great way to go, and it fits very nicely with the way that a lot of natural procrastinators already work. We don’t do nothing, we just do something more interesting and perhaps less important that what we’re supposed to be doing…

How Your Cat Is Making You Crazy
Finally, here is one of the freakiest things I have heard about for a long time. You think you have free will? You think you’re in control of the decisions you take? Maybe you should think again. Rats infected with the parasite Toxoplasma gondii, which they pick up from cat faeces, exhibit a strange mix of behavioural changes, the most striking of which is that they no longer fear cats. Male rats infected with T. gondii find the smell of cat urine sexually exciting. Result: infected rats get eaten by cats and the parasite goes on its way to explore the next part of its charming life cycle. Big deal, you say. Who cares about freaky rats? The big deal is that T. gondii is a zoonose — it infects humans too. There’s not a perfect match between the parasite and human brains, since T. gondii evolved to jump between cats and rats, but the parasite has enough affinity for human grey matter to cause significant changes in behaviour in some carriers. It gets a little scary when you hear just what those changes are. One is increased recklessness (particularly in males), leading to detectable differences in car accident statistics for carriers compared to non-carriers. Another is schizophrenia, which shows strong correlations with T. gondii status. How weird is that? A parasite that lives in cats and rats may be at least partially responsible for one of the most mysterious and terrifying of mental illnesses. There is still a lot of work to be done on this, but there are some serious people involved in the research, and it sounds pretty solid.

For me, I guess there are two separate “whoah!” moments that come out of this. The first is about the power of evolution. The coevolution of parasites and their hosts is already strange, even before you get to mind-altering parasitic cysts that make one of the parasite’s carriers more likely to be eaten by another. I find it quite hard to get my head around this: it’s like the parasites are farming the rats and cats (albeit unintentionally). The second thing is the idea that schizophrenia in humans may just be collateral damage in the T. gondii/Felis catus/Rattus norvegicus arms race. It would be one thing for humans to suffer the by-blows of some cataclysmic war of the gods, but these are cats, rats and protozoa!

If you want to get even more scary, think about the fact that T. gondii is just one environmental parasite, one of the relatively well-studied ones. There might be dozens of other little suckers shaping you to their ineffable monocellular will. Still think you’re the one driving up there?

My second attempt at a PhD went off rather better than my first. It was all done, dusted and examined in the allotted three years, without too much psychic suffering. Even more remarkable was that, in a period of four years, both my partner Rita and myself finished, wrote up and graduated. Without any domestic bickering at all (well, more or less).

I put most of this down to the excellent atmosphere in the research group I was in in Bristol, founded by Paul Valdes, one of my two supervisors. We had frequent group meetings, both in the department and in the pub, there was a lot of good cross-fertilisation of ideas, and, best of all, the group was full of people I was happy to spend time with (not at all a common situation for me, as anyone who knows me will testify). It was great. Paul had a super attitude for a group leader, letting people mostly get on with what they wanted to do, never coming the big boss, and it worked really well. The BRIDGE group is one of the things I miss most about Bristol.

My other supervisor was Steve Wiggins, in the maths department. He was responsible for enormously broadening my horizons in all kinds of ways. There are mental states I would never have entered without Steve’s prodding, mostly in the form of late night emails with bundles of papers attached, saying “You really ought to know something about this stuff…”. Steve has a work ethic that would kill an ox, and I don’t think he ever missed one of our Friday morning meetings throughout the three years, despite being head of the maths department and running an active research group. I guess someone who has written five or six fat books about dynamical systems probably knows a thing or two about time management. I’m very grateful that he agreed to take me on as a student.

As for the work I did during my PhD time, well, I started off with all sorts of grandiose plans, and ended up producing a workmanlike though not terribly inspiring thesis about nonlinear dimensionality reduction methods. I did learn a lot along the way, and the thesis probably represents only about 20% of the work I did. Since Paul and Steve mostly left me to get on with things, I ended up doing real research, i.e. cocking up repeatedly and having to start over with slightly modified goals and preconceptions. I think that with more guidance, I could probably have written a more impressive end product, but I don’t think I would have learnt as much as I did during my sometimes rather pathetic flailing.

All in all, it was a good time. I met a lot of cool people, did a lot of caving, some kayaking, climbed some mountains, and had a lot of thinking time, something that seems to be sorely lacking as a post-doc!

We saw something cool on the way to work this morning. It turns out that they’re considered pests in this part of the world, but a five metre string of nose-to-tail caterpillars counts as cool in my books even if they are pests. Apparently, these little critters make nests high in pine trees (we think we saw the nest — looked like a big ball of spiderweb), then troop down the tree and off into the world to make caterpillary mischief of one sort or another.

We first spotted the caterpillar chain coming out onto the pavement as we were walking up to Rita’s office. Keeping Winnie well out of the way (Rita takes her to work a couple of days a week), since the caterpillars are hairy and would be bad news for a little dog’s nose, we had a good look at what was going on. The caterpillars really were stuck nose-to-tail, wriggling their way along to who knows where.

Most of them were meeting a sad fate on the pavement, as they got stepped on and squashed, so leaving a confusing pile of caterpillar guts that led the caterpillars behind to make strange loops and knots of themselves as they could no longer tell where to go.

We followed the trail of caterpillars inside the fence to the tree they were coming from. The trail stretched all the way through the pine needle litter from the tree to the fence, a distance of about three metres. With the metre of caterpillars still coming down the tree and the metre already on the pavement, that gives us five metres of caterpillars, 500 centimetres. Each caterpillar was about 2.5 cm long, so we had about 200 individuals in this troop.

There’s a (not very good) video here showing the extent of the procession.

Later on, walking home, we saw caterpillar remains in a couple of other places along the road, suggesting that there were quite a few processions that morning. The caterpillars normally come out of their nests together at night, to feed on pine needles, but they leave the nest in the same way when it’s time to pupate, so I guess that was what we were seeing today.