All software should have an off-switch

It's a bit of cliché, I know, but I spent much of my childhood buried in books.  

I was a mediocre athlete — not last to be picked, but definitely the bottom quartile. And while small-town life back when I was a young 'un had much to recommend it, this was a time not only prior to ubiquitous internet but also where three channels of black-and-white television were what one could expect on a good day, when the antenna was pointed in just the right direction. Newspapers still had morning and evening editions and the world was awash in paper and ink. So, if you weren't running wild outside in your free time, piles of books were (at least for me) the equivalent of the World Wide Web.

While I read widely — whatever was in my academic parents' shelves, over which they provided no censorship — it was the escape to other times and planets that was my relief, and I spent many of my free hours and much of my spare change in the science fiction shelves of our local used book shop. I tended towards spaceships, time travel, robots, and my shelves were full of the likes of Robert Heinlen, Isaac Asimov, Phillip Dick, Harry Harrison and Stanislaw Lem, among others, and piles of comic books.

A common theme and trope that I grew up on — also a cliché perhaps at this point — is the nightmare future where machines battle and best their human masters. These themes are also mirrored in film, television and popular culture — the Terminator, the Matrix, Blade Runner, Battlestar Galactica are just a few. It's a long list.

Sometimes, the bleak dystopia takes a comic turn. For a comic take on machines too smart for our own good, Stanislaw Lem's short story "The Electronic Bard" is hard to beat. And maybe there's a lesson to learn while you chortle. This is one of my personal all-time favorites.

The story is about Trurl, an inventor who decides to build a machine that writes poetry. He realizes quickly that it's no small task, as "[t]he program found in the head of an average poet, after all, was written by the poet's civilization, and that civilization was programmed by the civilization that preceded it, to very Dawn of Time . . ." Undaunted, Trurl proceeds, creating a poetry machine that's fed all human knowledge (and primordial history that precedes it) and given human foibles. There are plenty of fits and starts, and narcissism is added as a final tweak:

"....just as he was ready to give up and take a hammer to it, Trurl was struck by an inspiration; tossing out all the logic circuits, he replaced them with self-regulating egocentripetal narcissistors. The machine simpered a little, whimpered a little, laughed bitterly, complained of an awful pain on its third floor, said that in general it was fed up, through, life was beautiful but men were such beasts and how sorry they'd be when it as dead and gone. Then it asked for pen and paper."

So formed, the Electronic Bard writes such moving verse that the disruption it wreaks (and its massive power consumption) forces its hapless inventor to disassemble it and let it be towed to a distant galaxy. My synopsis doesn't do the story justice — it's hilarious and wise.

Other than the fact that it's a great story, and a masterpiece of translation from the original Polish (with poems about algebra that survive translation!), why should anyone who thinks about law, policy, poetry or systems design care about this story?

For one thing, as is almost predictable as my own origin story at the introduction of this piece, Lem's story had more than a nubbin of predictive truth in it. There are now and have been for several years computer systems that, when given a prompt, can actually write almost credible poetry. 

Here's an example from the talk to transformer website, where I provided the first line as a writing prompt:

The morning sun brooks no dispute

A few bushels to the ripe
And no grow-up tom catz In court.
The grandson's boy
Takes his hickory bow
And sets up a rich hue
And's hit by some falling Wood.
My grandsons repine
'Till the catz are picked off,
And the youth falls a-drowned.

O Maine O Maine! O Maine O Maine!
But it is too late: 't is too late
The maiden is sung and abeam.
The manner she's ben blasted
By a cockatrice or a bifocole

Look, I am not saying that this is Robert Frost or an intelligent machine, but it's not terrible and I am saying that if science fiction imagines it, often the vision proves at least partially correct. And this is, at least, a little bit similar to the Electronic Bard, even without sentience (and still with an off switch).

*           *           *

So, what does the wisdom of Stanislaw Lem offer for system designers?  

Obviously, machine augmentation of intelligence is not a new or bad thing. Machines have made people smarter long before computers were invented. Abacuses augment human intelligence. Electronic calculators did, too. So do digitized databases (and they save you the time of going to a brick, mortar and paper library). Substances like coffee and prescription Adderall are also, arguably, a form of augmented intelligence. But of course, humans who augment their intelligence with a machine or substances still make decisions and can be held legally liable for bad ones.  

The law and attendant risks become less clear when machines can act autonomously, or where they are smarter, faster or (in a Battlestar Galactica-like meme) have a soul that they believe is more human than their human creators. Lem's comic proposition is that to write like a human, you have to leaven the mix with a little narcissism and with that comes some autonomy. 

It's unlikely that AI developers are writing narcissism functions into AI frameworks or deep learning systems (should they be?). But the amount of autonomy that ought to be built into intelligent systems is a topic top of mind for many and for years, including the likes of Elon Musk, Steven Hawking and others, as this an article from a few years back ("The Real Problem with Artificial Intelligence") notes:  

"Some so-called AI systems incorporate many different components like computer vision, speech recognition, tactile feedback and touch systems. All of these sensory modalities give computers the ability to sense as well as, or even better than humans. The collected data can then be used to plan or take action. For example, the autopilot systems used on commercial aircrafts are AI systems that help safely fly the plane. But when people like Musk or Hawking warn about AI, they are cautioning against giving AI systems complete autonomy — which isn't something that happens naturally, Dietterich said."

The issue they're concerned with isn't AI or automation per se — it's autonomy and control. And maybe here's where the Lem Rule (which I describe below) could come into play.

As a lawyer, one learns to work within a system of laws that has taken centuries (and sometimes more) to develop, and deal with equally thorny questions — like parental responsibility for children, employer-employee liability, liability for products once they are in the stream of commerce.  

In short, concepts like vicarious liability, respondeat superior, strict product liability (and many others) provide rubrics for responsibility and rules of the road that may be subject to improvement but don't need to be reinvented.

My working definition of the Lem Rule has three parts:

1. Always build an off switch
2. At least one human actor must always be responsible for a machine's actions or inactions
3. Consider existing principles of legal liability — if you don't know who will be legally responsible for a machine after it leaves the lab, don't let it out, and assume that you will be responsible, regardless. 

We haven't seen a great deal of deep or meaningful thought here when it comes to so-called autonomous blockchain systems. It's been mostly Klieg lights and governance theater. Still, someone has to identify and write the rules of the road. The failure to do so means that the rules will write themselves.

My good friend and law partner Preston Byrne wrote a recent thoughtful piece for CoinDesk entitled "Bitcoin Doesn't Take Sides: Why Apolitical Solutions Are the Internet's Future." While Preston and I agree on much, we disagree here. There are no apolitical or non-ideological software systems. The notion of ideologically neutral or apolitical system design and deployment is a myth and a mirage. 

Even bitcoin, friends: there's nothing neutral about it. The decision to make something putatively unstoppable is, by nature, a political and ideological choice. If you create a system that can act or think on its own without human intervention, this is a consequential and non-neutral position. So, while bitcoin might create a future where financial transactions aren't easily censorable, that's a political and ideological choice, a non-neutral one.

Regardless — and like Captain Adama in Battlestar Gallactica — I am going to keep a rotary phone around, just in case.