One Level Up, One Level Down

I have spent years building rule-based systems.

I did not start by thinking of them that way. I was trying to run a medical practice, understand financial data, find real estate investments, and eventually build software around problems that had become too complicated to manage reliably by hand.

A few weeks ago, I started watching MIT's old 6.034 artificial intelligence lectures in the sauna.

There is something useful about watching an old AI course now. The machinery can look almost primitive beside what we call artificial intelligence today. But that simplicity makes the ideas easier to see.

In Lecture 2 (opens in a new tab), Patrick Winston works through a symbolic integration program written by James Slagle in 1960. The program takes a difficult integration problem and repeatedly transforms it into simpler problems. Some transformations are safe. Others are heuristic. The resulting structure is what Winston calls a problem-reduction tree, an and/or tree, or a goal tree.

The program was remarkably capable. Tested against 56 difficult problems drawn from MIT calculus finals, it solved 54 of them. The average depth of problem reduction was only about three levels.

Near the end of the lecture, Winston asks what all of this says about intelligence.

He tells a story about showing the integration program to a student who believed computers could not be intelligent. After watching the program solve difficult problems, the student reconsidered.

Then Winston explained how the program worked.

The student changed his mind again. If the machine solved integration essentially the same way he did, apparently it wasn't so intelligent after all. Winston's larger observation was that once we understand how something works, its apparent intelligence tends to disappear.

I thought that was funny.

But another line from the lecture stayed with me:

Knowledge about knowledge is where the real power is.

Yesterday, I watched Lecture 3 (opens in a new tab).

That one hit me like a lightning rod.

A world made of blocks

Winston begins with a small world made of blocks.

The task might be to put one block on another. To accomplish that, the program may need to find space, grasp a block, move it, and release it. If another block is in the way, that creates another goal. Satisfying that goal may create another one.

The program itself is surprisingly simple. Recursion produces behavior that looks much more complicated than the underlying machinery.

As it works, the program leaves behind a trace of what it did.

Winston asks how that trace could allow the program to answer a question about its own behavior.

Why did you get rid of this block?

Go up one level.

Because I was trying to clear the top of another block.

Why did you clear that block?

Go up again.

Because I was trying to grasp it.

Why did you grasp it?

Because I was trying to put it somewhere else.

A why question moves upward toward the goal that gave an action its purpose.

Then Winston asks how the program answers a how question.

Go down one level.

The children beneath a goal show the actions used to accomplish it.

That was it.

Move down one level and ask how.

Move up one level and ask why.

I had spent years building systems full of rules, states, conditions, and actions.

I had never seen the relationship reduced that cleanly.

Once I did, I started seeing it everywhere.

Rules aren't reasons

At our medical practice, a patient had to provide a credit card to book a consultation.

The consultation itself did not carry a fee.

The card existed so that we could charge a no-show fee.

If someone inherited only the rule—

Credit card required to schedule consultation

—they might reasonably conclude that the card was required because the consultation cost money.

That wasn't why.

Move one level up.

The goal was to protect the doctor's schedule.

An empty consultation block was physician time that could not be recovered. The no-show fee was simply one mechanism for reducing the probability that the patient would leave that block empty.

We had another rule:

Surgery must be paid in full two weeks before the procedure.

There was nothing intrinsically important about fourteen days.

We had learned that once a patient had paid for surgery, the probability of a last-minute cancellation fell dramatically.

Why collect payment two weeks before surgery?

Not because the practice needed the cash on that particular day.

Move up.

The goal was to protect a much more valuable block of operating time from disappearing at the last minute.

If the system remembers only:

Collect payment fourteen days before surgery

then it has preserved a rule.

If it remembers:

Reduce late cancellation of committed operating time

then it has preserved the reason for the rule.

That difference matters.

Circumstances change. Maybe advance payment remains the best mechanism. Maybe a different deposit structure works better. Maybe reminders, scheduling practices, or another mechanism produces the same result with less friction.

Once the goal survives, the rule becomes replaceable.

Winston makes essentially this criticism of rule-based systems later in Lecture 3. A grocery-bagging program might know the rule that potato chips belong on top, while knowing nothing about why: putting them on the bottom crushes them, and customers don't want crushed potato chips. The rule captures behavior without necessarily capturing the knowledge beneath it.

I had seen versions of that problem for years.

I just hadn't had a name for it.

Sometimes you misunderstand what you built

Years ago I wrote a monthly sales report for the medical practice in PowerShell.

At runtime, it asked:

Do you want the current month? Y/N

If the answer was yes, it ran the current month. If the answer was no, it asked for a start date and an end date.

There was no grand architectural principle behind this design.

I simply did not feel like writing an algorithm to figure out the first and last day of whatever month somebody wanted.

The report was almost always used monthly. Usually we wanted this month's sales or last month's sales. Occasionally an employee would think there had been an error on a paycheck several checks earlier, and we would reconstruct another period.

So I thought I had built a monthly sales report.

Then somebody asked a different question.

Could we calculate operating-room and anesthesia fees for a quarter?

I realized the same program could.

It could calculate them for a year.

Or several years.

Nothing about the software had changed.

My understanding of what I had built had.

I thought the parent goal was:

Produce a monthly sales report.

Move one level up.

The more general problem was closer to:

Understand financial activity over a defined period of time.

Monthly reporting was only one branch beneath it.

Almost accidentally, I had chosen a more general representation—start date and end date—than the immediate problem required.

Once the representation changed, an entire family of apparently different questions became versions of the same problem.

The wrong parent

My son's favorite toys are Magna-Tiles.

He builds houses, buildings, zoos, elevators, hospitals—you name it. He builds something, smashes it, and starts again on something else.

Suppose I decide his goal is:

Have a hospital.

Now smashing the hospital looks irrational.

He has just destroyed the thing he successfully created.

But maybe I chose the wrong parent goal.

Maybe the goal is simply:

Build.

Or perhaps:

Discover what I can build.

I don't know which description is actually in his head. That's part of the point.

The observable action doesn't contain its own explanation.

If his goal is exploration rather than possession of the finished object, destroying one structure may make perfect sense. The Magna-Tiles are finite. While they are committed to the hospital, they cannot become a zoo or an elevator or whatever comes next.

Smashing one state returns the pieces to a state in which other possibilities become available.

The same behavior can look irrational under one assumed goal and perfectly coherent under another.

You cannot necessarily explain an action merely by recording that it happened.

You have to understand what goal sits above it.

Enough of the world

One of the things that struck me about Winston's lectures was how aggressively the problems were simplified.

Symbolic integration became transformations and a relatively small table.

A world of physical action became blocks, goals, and a handful of procedures.

Yet complicated behavior emerged.

In Lecture 3, Winston invokes Herbert Simon's ant: an ant's path across a beach may look enormously complicated, not because the ant contains an enormously complicated program, but because it is navigating a complicated environment. Winston summarizes the point as complex behavior emerging from a simple program interacting with a complex problem.

That felt familiar too.

I had spent years dealing with environments that were impossibly complicated if I tried to represent all of them.

Real estate is like that.

A property has effectively unlimited facts associated with it.

Years ago I built tools that collected foreclosure opportunities into structured data. I could query those records for combinations of characteristics that seemed worth investigating.

One of those processes brought a property on Blinka Road to my attention.

The query did not understand the investment.

It did something much smaller.

It told me:

Look here.

Then the representation changed.

The property had significant road frontage.

The acreage mattered.

The price mattered.

The auction deadline mattered.

Ownership mattered.

Title mattered.

Financing mattered.

A subdivision became possible.

At each stage, enormous amounts of reality were irrelevant to the immediate question. A smaller set of facts determined what I needed to understand or do next.

The trick wasn't to model the world.

It was to model enough of the world to reason about the question.

What the system is allowed to believe

Shortly before watching these lectures, I had been working on an underwriting system at Upperline.

One of the problems I was thinking about was how to bring sourced assumptions into an underwriting model without pretending that finding a number made the number true.

A system might find an assumption and preserve where it came from.

Useful.

But not enough.

I wanted a human to be able to inspect the assumption, accept it, reject it, or replace it—and for the system to preserve that distinction.

Sourced was not the same as verified.

I had encountered a simpler version of the same problem years earlier. A property-data workflow could navigate to a nearby parcel, but the software itself warned that proximity did not establish that it had found the correct parcel.

A human still had to confirm it.

The machine had done useful work.

It had narrowed the problem.

But it had reached the boundary of what its information justified.

That makes some actions different from the obvious actions in the blocks world.

Sometimes an action isn't intended to change the outside world.

It changes what we are justified in believing about it.

Research.

Verify.

Reconcile.

Confirm.

Reject.

If an assumption has been sourced but not verified, some downstream action may not yet be justified.

Verification doesn't change the property.

It changes what the system is allowed to do with what it knows.

That brought me back to Winston's line from the previous lecture:

Knowledge about knowledge is where the real power is.

Leaving a trace

I recently wrote an essay called The System Has to Remember.

I was thinking about research that disappears, decisions whose reasoning gets lost, contacts separated from the opportunities that gave them meaning, and work that has to be reconstructed because a system retained a record without retaining enough of the context surrounding it.

Lecture 3 gave me another way to think about that problem.

Winston's blocks program can answer questions about its own behavior because it leaves behind a trace organized as a goal tree. The trace preserves relationships between goals and the actions undertaken to achieve them.

Memory may therefore be only the first requirement.

A system can remember what happened and still forget why it happened.

And once the why disappears, rules can harden into rituals.

Collect the credit card.

Collect payment fourteen days before surgery.

Run the monthly report.

Postpone the auction.

Each action can survive long after the purpose that made it sensible has disappeared from the system.

But if the relationship survives, we can interrogate it.

Why did we do this?

Go up.

Because we were trying to accomplish that.

How did we accomplish that?

Go down.

These were the actions.

Now memory and explanation begin to touch.

And explanation can become planning.

If the goal remains but circumstances change, perhaps the system can consider different children beneath the same parent.

The rule can change without losing the reason the rule existed.

Where the system cracks

There was one more part of Lecture 3 that felt familiar.

Winston talks about extracting knowledge from experts. One of his recommendations is to study specific cases because experts tend to give vague generalities when asked to describe what they know. Concrete cases expose distinctions they would not otherwise think to articulate.

Then he gives another heuristic:

Build the system and see where it cracks.

When the system fails or comes to a halt, you may have discovered a missing rule—a piece of knowledge that your representation failed to capture.

I smiled when I heard that.

That is approximately how I have learned to build software.

Not by sitting down and trying to model an entire domain.

By encountering a specific case.

Why did this parcel fail to match?

Why did this report fail to answer the question?

Why does this assumption require verification?

Why does this workflow keep breaking here?

The exception teaches you something about the domain.

You add a distinction.

You acquire a new piece of vocabulary.

You change the representation.

Then you run it again until you find the next place where your understanding is incomplete.

One level

I don't know whether literal goal trees are the right representation for the systems I will build next.

That isn't what I took from the lectures.

What stayed with me was the discipline of asking what a system needs to represent before asking how sophisticated the system needs to become.

If I ask a machine what to do next, I increasingly want its answer connected to something I can inspect.

What are we trying to accomplish?

What does the system believe to be true?

Where did that belief come from?

Has it been verified?

What actions are available?

Why does this action serve the goal?

What does the system expect to become true afterward?

Those questions don't require the machine to model the world.

They require it to model enough of the world to reason about the decision.

Maybe that is why those blocks stayed with me.

I had spent years building increasingly capable systems around complicated environments.

Winston showed me a tiny world and a simple program capable of producing behavior complicated enough that the program could be asked to explain itself.

The lesson I carried out of the sauna wasn't that complicated worlds are simple.

It was almost the opposite.

Complicated worlds may only become tractable after we decide what we're willing to leave out.

And if we preserve the right relationships, sometimes understanding requires moving only one level.

Down:

How?

Up:

Why?