A New Start!

It’s hard to believe that I’ve not written anything here since 2017. That’s a long break. But now I’m back, with lots of new ideas and projects and stuff!

Anyway, what’s happened in the last couple of years? Quite a lot. Here are the highlights:

• Houses sold: 1

• Jobs quit: 2

• Passports surrendered: 1

Six Months of Women Writers

Last spring, I was taking a look at what I’d been reading recently. I read a lot of novels. Reading a novel seems to be my default state. I read a lot of science fiction, some crime novels, some historical fiction, some “straight” fiction (whatever that really is). But what I noticed was that the authors I tend to read skew very male. I decided I needed to do something about that, if only because I was probably missing some great stuff. So for six months I decided to read only novels written by women.

And I was right! I was missing some great stuff. I read about 60 novels in those six months, and discovered some amazing things that I really wish I’d known about before. Here are a few of the stand-outs.

Setting up dual boot Arch Linux/Windows 10

I had to do a slightly weird bit of system admin recently, and there was one step that was kind of sneaky and not something I’d seen mentioned anywhere else. So I thought I’d better write it up...

What I wanted to do was make my desktop machine dual boot Arch Linux and Windows 10 (I needed Windows to use some CAD software). That’s not so unusual, but the recommended procedure seems to be to start from a clean machine, install Windows, then install Linux. I was starting from a pre-existing Arch Linux installation and didn’t want to lose anything. I also had things set up using syslinux in BIOS mode, which wasn’t going to work with Windows. So I needed to switch to UEFI booting first.

New Job with Memcachier

I’ve just finished my first week in a new job, which is looking like it’s going to be a lot of fun. I’m working for MemCachier, a small company (I’m employee #4) that offers a multi-tenant cloud-based memcache service. The code is mostly Haskell and Go (and maybe in the future some Rust), the people are all very smart and friendly and committed to trying to do things the right way. I’m still (obviously, after one week) in that initial state of confusion when dropping into a new codebase, but it looks very promising indeed.

The work should be a good mix of web programming, DevOps automation things and distributed systems problems. There’s going to be lots to learn, which is ideal for me! I’m also very happy to be back working in Haskell after a year-long (or a bit more) diversion into Python.

So, as far as I’m concerned, 2017 has started off very well.

Back again...

Well, that’s pretty poor. It’s a year and a half since I last wrote an article here. A lot has happened in that time, and I have some cool new things coming up that I want to write about, so I’d better dust this off.

Quick summary of the last 18 months:

I started a new job in September last year, and had my last day there on December 9. It was OK, but obviously not perfect, otherwise I wouldn’t have left. There were some great people there, but the management of the organisation left a lot to be desired. I’m starting something new tomorrow (!) which I’m quite excited about and that I’ll write about in the next couple of weeks.

We bought a house! It has some doer-upper aspects to it, but it’s pretty great. Very big garden, plenty of space (we have more rooms here than in all the places we’ve lived before now put together!), good dog walking nearby, quiet but handy for transport (we’re about two minutes drive from a motorway, but you don’t hear it much), work for Rita nearby, etc. We moved in in July and had an initial spasm of renovation work where we transformed the rather manky kitchen into something nice. We’re just finishing up renovating the living room, and have lots of ideas of more things to do. One thing is to build a separate office building for me to use, which is going to my big project for the spring. Lots of things to learn! (As well as having more rooms now than we’ve ever had before, we also have more power tools than I’ve ever owned in my life...)

A Haskeller in Pythonland

I recently did some work for Andy Ridgwell, an old colleague from Bristol, writing a build and configuration system and GUI for a medium-sized climate model called GENIE. GENIE is an EMIC, an Earth system Model of Intermediate Complexity. It’s about 55,000 lines of Fortran and includes models of the atmosphere and ocean plus models of atmospheric chemistry and biogeochemistry in the ocean and ocean sediments.

This model had been in use for some years by different groups, and the infrastructure around it had become quite baroque. Andy wanted this tidied up and made nice (i.e. rewritten...) to make the model easier to set up and use. He also wanted a cross-platform GUI for configuring and running the model, allowing you to keep track of the model state in real-time, to pause and restart model runs, changing the model configuration in between, and so on.

A major consideration for this work was that as well as being easy to use the new system had to be easy to install (on both Linux and Windows) and easy for scientists to hack on. That ruled out Haskell, my usual tool of choice. I decided to use Python instead, for a couple of reasons.

C2HS Tutorial Ideas

One of the things that C2HS is lacking is a good tutorial. So I’m going to write one (or try to, anyway).

Q1 2015 Review

I’ve started doing a new thing this year to try to help with “getting things done”. I normally have a daily to-do list and a list of weekly goals from which I derive my daily tasks, but I’ve also now started having a list of quarterly goals to add another layer of structure. Three months is a good timespan for medium-term planning, and it’s very handy to have that list of quarterly goals in front of you (I printed it out and stuck it to the front of my computer so it’s there whenever I’m working). Whatever you’re doing, you can think “Is this contributing to fulfilling one of my goals?” and if the answer is “No, watching funny cat videos is not among my goals for this quarter”, it can be a bit of a boost to get you back to work.

So, how did I do? Not all that badly, although there were a couple of things that fell by the wayside.

Non-diffusive atmospheric flow #15: Wrap-up

OK, so we’re done with this epic of climate data analysis. I’ve prepared an index of the articles in this series, on the off chance that it might be useful for someone.

The goal of this exercise was mostly to try doing some “basic” climate data analysis tasks in Haskell, things that I might normally do using R or NCL or some cobbled-together C++ programs. Once you can read NetCDF files, a lot of the data manipulation is pretty easy, mostly making use of standard things from the hmatrix package. It’s really not any harder than doing these things using “conventional” tools. The only downside is that most of the code that you need to write to do this stuff in Haskell already exists in those “conventional” tools. A bigger disadvantage is that data visualisation tools for Haskell are pretty thin on the ground–diagrams and Chart are good for simpler two-dimensional plots, but maps and geophysical data plotting aren’t really supported at all. I did all of the map and contour plots here using UCAR’s NCL language which although it’s not a very nice language from a theoretical point of view, has built-in capabilities for generating more or less all the plot types you’d ever need for climate data.

I think that this has been a reasonably useful exercise. It helped me to fix a couple of problems with my hnetcdf package and it turned up a bug in hmatrix. But it went on a little long–my notes are up to 90 pages. (Again: the same thing happened on the FFT stuff.) That’s too long to maintain interest in a problem you’re just using as a finger exercise. The next thing I have lined up should be quite a bit shorter. It’s a problem using satellite remote sensing data, which is always fun.

Non-diffusive atmospheric flow #14: Markov matrix calculations

This is going to be the last substantive post of this series (which is probably as much of a relief to you as it is to me...). In this article, we’re going to look at phase space partitioning for our dimension-reduced $Z_{500}$ PCA data and we’re going to calculate Markov transition matrices for our partitions to try to pick out consistent non-diffusive transitions in atmospheric flow regimes.

C2HS 0.25.1 “Snowmelt”

I took over the day-to-day support for C2HS about 18 months ago and have now finally cleaned up all the issues on the GitHub issue tracker. It took a lot longer than I was expecting, mostly due to pesky “real work” getting in the way. Now seems like a good time to announce the 0.25.1 “Snowmelt” release of C2HS and to summarise some of the more interesting new C2HS features.

Constraint kinds and associated types

This is going to be the oldest of old hat for the cool Haskell kids who invent existential higher-kinded polymorphic whatsits before breakfast, but it amused me, and it’s the first time I’ve used some of these more interesting language extensions for something “real”.

Non-diffusive atmospheric flow #13: Markov matrix examples

(There’s no code in this post, just some examples to explain what we’re going to do next.)

Suppose we define the state of the system whose evolution we want to study by a probability vector $\mathbf{p}(t)$–at any moment in time, we have a probability distribution over a finite partition of the state space of the system (so that if we partition the state space into $N$ components, then $\mathbf{p}(t) \in \mathbb{R}^N$). Evolution of the system as a Markov chain is then defined by the evolution rule

$$ \mathbf{p}(t + \Delta{}t) = \mathbf{M} \mathbf{p}(t), \qquad (1) $$ where $\mathbf{M} \in \mathbb{R}^{N \times N}$ is a Markov matrix. This approach to modelling the evolution of probability densities has the benefit both of being simple to understand and to implement (in terms of estimating the matrix $\mathbf{M}$ from data) and, as we’ll see, of allowing us to distinguish between random “diffusive” evolution and conservative “non-diffusive” dynamics.

We’ll see how this works by examining a very simple example.

Non-diffusive atmospheric flow #12: dynamics warm-up

The analysis of preferred flow regimes in the previous article is all very well, and in its way quite illuminating, but it was an entirely static analysis–we didn’t make any use of the fact that the original $Z_{500}$ data we used was a time series, so we couldn’t gain any information about transitions between different states of atmospheric flow. We’ll attempt to remedy that situation now.

What sort of approach can we use to look at the dynamics of changes in patterns of $Z_{500}$? Our $(\theta, \phi)$ parameterisation of flow patterns seems like a good start, but we need some way to model transitions between different flow states, i.e. between different points on the $(\theta, \phi)$ sphere. Each of our original $Z_{500}$ maps corresponds to a point on this sphere, so we might hope that we can some up with a way of looking at trajectories of points in $(\theta, \phi)$ space that will give us some insight into the dynamics of atmospheric flow.

Non-diffusive atmospheric flow #11: flow pattern visualisations

A quick post today to round off the “static” part of our atmospheric flow analysis.

Now that we’ve satisfied ourselves that the bumps in the spherical PDF in article 8 of this series are significant (in the narrowly defined sense of the word “significant” that we’ve discussed), we might ask what to sort of atmospheric flow regimes these bumps correspond. Since each point on our unit sphere is really a point in the three-dimensional space spanned by the first three $Z_{500}$ PCA eigenpatterns that we calculated earlier, we can construct composite maps to look at the spatial patterns of flow for each bump just by combining the first three PCA eigenpatterns in proportions given by the “$(x, y, z)$” coordinates of points on the unit sphere.