A(nother) New Start: Pollen!

16 May 2021colophoniapollen

The alternative title for this article was “The FrankenBlog: It Lives!”. You’ll see why “FrankenBlog” in a minute.

The idea here was to convert my website and blog over to use Pollen, Matthew Butterick’s marvellous Racket-based authoring system. The main reason for wanting to do that is that I have a couple of longer-form writing projects in mind, and Pollen gives much more flexibility than writing in Markdown or something similar.

Starting a new blog with Pollen is easy. Unfortunately, I had some history I wanted to preserve, some new ideas I wanted to try, and a whole row of yaks lined up waiting for a shave.

I’ve heard it said that people are more likely to change their spouse than to change their bank, and something similar seems to be the case for blogging platforms. At least for reasonable people. I have commitment issues there. Pollen is my third blogging platform.

First there was Hakyll, with whom I had a long and happy relationship. It was a time of excitement and experimentation, some of it perhaps slightly misguided, but it lasted a long time and we were very happy together. Looking back now, I’m not sure why I left. Isn’t that always the way with our youthful dalliances?

Then there was Zola. Younger, flashier, even, dare I say, a little louche? Zola and I never really got along. Oh, there were some happy days, but it always felt as though we wanted different things from the relationship and were destined to part some day.

Now though, it feels like I’m ready to settle down. At this time of life, some people would be looking for something staid, middle-of-the-road, a blogging platform that Clive from the bowls club would nod at approvingly. Not me. I still retain at least a smidgeon of adventurousness. So, on to the mysterious and exotic yet surprisingly practical Pollen.

Why Pollen?

It’s Scheme, it’s infinitely extensible and programmable, and it’s explicitly designed for web-based long-form writing. Pollen’s author, Matthew Butterick, has used it to write three books so far.

Pollen has the nicest approach to markup that I’ve ever seen. Basically, you can write anything as markup and it gets passed through unchanged–make a div tag and you get an HTML div in your output. But you can intercept any of those tags and replace their behaviour with custom behaviour that you program in Racket. This is really ergonomic. I often find myself wanting to write what’s essentially HTML in the middle of a page, and it’s honestly a great big hassle with Markdown-based systems. You end up living in two worlds, one of “real” Markdown, and one of your custom HTML markup. And never the twain shall meet. In Pollen, all markup, whether it’s just a bunch of HTML or whatever custom markup you define, all goes through the same intermediate representation, which is easy to manipulate in Racket (it’s just a specialised s-expression format, so Lispy manipulation of it is completely natural).

The way that Pollen handles arguments to custom tags is also really neat. You have normal arguments, keyword arguments, and “rest” arguments, as you normally do in Scheme and similar languages. But there’s an alternative syntax for calling functions/referencing tags in Pollen documents that maps the “rest” argument to the “contents” of the tag. It’s quite hard to get across how useful this is until you’ve played with it a bit. Take a look at some of the source files for my blog on GithHub:

ian-ross/website

I’m only using a small fraction of the capabilities of Pollen. You can do multi-platform publishing with it, i.e. turning your Pollen sources into HTML, PDFs, via LaTeX, or whatever else you want. In all cases, your Pollen source is parsed, the tagsAll represented using a cute little lozenge character: ◊. and text around them are converted into “tagged x-expressions”, which you can mess around as you want to, then those are converted into your output format. (Conversion to HTML is direct, since the intermediate format is more or less just a different syntax for HTML.)

Why “FrankenBlog”?

That all sounds sweet, right? Of course, the honeymoon lasted about five minutes, because I was intent on doing things that weren’t quite what Pollen was intended for. Online books are one thing, but I wanted to use it both for project and tutorial pages (fairly bookish), and for my blog. That means lots of auto-generated content, for blog index pages, category pages, RSS feed, maybe full-text search index, and so on.

You can do all of these things in Racket, of course, but I’m a real beginner with itI did write quite a lot of Scheme at one point, but that was a while ago, and Racket is quite a bit more sophisticated and full-featured than any Scheme system I ever used., so I ended up taking a brutally direct and simple approach, which was to write a Python program that generates all the dynamic content from the Pollen source files. I’ve managed to set things up so that all of that can be done without really parsing the Pollen source files: I’ve sneakily got it so that all the information needed for the dynamic content extraction can be done using regular expressions.

The build process thus goes:

  1. Run a Python script called make-dynamic.py to generate all the dynamic content from the current set of blog articles.

  2. Render all the Pollen pages by running raco pollen render -p.

  3. Generate a full-text search index from the rendered HTML pages.

That search index generation is the last part of Frankenstein’s BlogMonster. I’m using minisearch to do client-side search in the browser, and of course that’s in Javascript, so it doesn’t make any sense to use anything other than Javascript to build the index while building the site.

So, we have Python, then Racket, then Javascript. I honestly didn’t intend it to come out that way. These things were just the most pragmatic choices along the way. But that’s everyone’s excuse.

Why so very very complicated?

Before I could even get onto the dynamic content generation though, there was the small matter of about 200 old blog articles from two different blogging systems to be converted to Pollen. It’s at moments like this that I’m forced to look at myself and ask “Why can you never do things the simple way?”. These 200 blog articles have the following exciting and amusing features that needed to be converted:

Converting all of those was done with a mixture of automation (shell scripts, sed, and Python using the marko Markdown parser), and hand editing. I estimate that automation covered about the first 80% of the work, and the remaining 80% was done by hand. That’s what it felt like anyway!

The TikZ stuff? I just had to do it over in Scheme. It was relatively painless as these things go. I’m still very fond of it as an idea, although I’ve not used it much recently.

With all the individual articles more or less converted and with a more or less working Pollen setup, I could do this dynamic content generation. That wasn’t all that hard, and I quite quickly got to the point where I had an almost working site. I had to do some of my least favourite job in the world along the way, which is writing CSS. Seriously, I’d rather be digging ditches in the rain.

Once I was at that point, I had a couple of other things I wanted to do, which I’ll write about separately. Those are server-side analytics (no cookies! no tracking!) and full-text search. I think they’re worth looking at in more detail.

But seriously now, why?

That all sounds like a lot of work, eh? Why not just bin the old blog, get a Wordpress account and get on with writing?

Partially it’s a matter of wanting to have an integrated site that I can use as a contact point for the contracting work I’ve started doing this year, as well as a place to put writing about various projects. I have a few things in the pipeline right now, and I wanted a good platform to host them. I’d been putting project stuff on my old website, and I never thought it looked all that great.

I was also pretty taken with Pollen when I started reading about it, and I have no regrets about switching over.