Reading The Concept of Mind from a Software Engineering perspective

I recently read philosopher Gilbert Ryle’s book The Concept of Mind, as it was one of the books referenced by Peter Naur’s text Programming as Theory Building (which I wrote about here previously). I wanted to read it because I was interested in the background of Naur’s arguments, and was not disappointed, as The Concept of Mind is a quite fascinating book. It provides a lot of insight into Naur’s text also.

Although Ryle wrote the book in the 1940s when programming as we know it today wasn’t really a thing, I think we can still draw a couple of interesting parallels from what he writes, and some aspects of software develpment. In particular, we can learn more about Naur’s ideas in theory building, some things about learning programming, interviewing software developers, and a couple of AI-related things.

Background

The Concept of Mind discusses the topic of Mind-body dualism, which put simply, suggests that the mind exists in a separate non-physical place from the body and the rest of the physical world. In the book, Ryle argues that this is nonsense. The book is in general fairly approachable, although being written in the 1940s, it used some terminology I had to look up.

I should note that I’m only a casual philosophy enjoyer, and as such, my perspective or understanding may differ from those who have studied the subject more. Regardless, I find philosophical works quite interesting. In addition to Ryle, I’ve read f.ex. Plato and Aristotle, which had less practical implications, but they all have an interesting way of looking at things from different perspectives, and finding attributes and properties even in ordinary things you might never have thought of. At least for me, this “thinking about ordinary things from new perspectives” is interesting. Sometimes some of it feels like it’s obvious in retrospect, but you realize you never thought about it in that particular way before.

Theory Building

Since the Concept of Mind is one of the books referenced in Programming as Theory Building, we can unsurprisingly find clarification on what Naur might have meant. One particular point that is somewhat contested in Naur’s text is that the theory of a program cannot be recovered. Many smart people say that you can recover the theory just fine - so was Naur wrong?

From reading Ryle’s book, we can surmise that it’s a somewhat more philosophical point, relating to how Ryle describes different types of knowledge and how that knowledge is defined and learned.

The book describes two types of knowledge, “knowing that” and “knowing how”. There’s an entire chapter dedicated to what this means and the distinction, but put simply, “knowing that” is facts, “I know that X is Y”, and “knowing how” is practical knowledge, “I know how to ride a bike”. Programming falls into the category of knowing how. You can know all the facts about programming languages, but that doesn’t mean you can actually write a program: you have to learn how to apply the facts in practice. Similarly, when you work on a specific program, you need to know facts about it, but you must also know how to put those facts into practice.

An important thing about knowing how is that you cannot express it in terms of facts. Just like it’s impossible to learn to ride a bike without actually doing it, you can’t learn to program without programming. To be able to express this type of knowledge as a set of facts, we would have to create some kind of rules about it. When this happens, do that, when that happens, do this instead. But for a skill like programming (or riding a bike), this is impossible, because you would end up with rules about selecting rules about selecting rules, which just goes on infinitely.

This is probably what Naur meant - while you could recover “knowing that” about the theory of the program from the code, you can’t recover the “knowing how”. I’m not sure if Naur ever elaborated on what he meant when he wrote the theory can’t be recovered, so this is of course speculation, but given what Ryle wrote, this feels like a plausible explanation: Naur also describes this aspect of expressing things in terms of rules in Programming as Theory Building, although he does not explicitly connect this to the lack of recoverability.

Many ideas behind Naur’s “theory building” seem to stem from this book, as Ryle describes the concept of theory building itself as well, along with some related concepts. So for anyone interested in the background of Naur’s text, reading The Concept of Mind would be highly recommended. Reading this book provides a lot of clarity into what Naur writes.

Learning programming

A topic that sometimes comes up online is whether it’s better to go to school to learn programming, or just learn it yourself. I’m entirely self-taught, but nowadays I’ve been of the opinion that if you have the option, you might as well learn it in school - nothing says you can’t also learn it on your own at the same time if you’re so inclined.

Something related to this that Ryle writes about in the book is that it’s possible to learn something without knowing the rules of it. You can figure out how you’re supposed to do something by observing others and by trial and error. When you do it this way, you don’t know the “rules” of the activity, but you can still do it. An example Ryle gives is chess - in theory it’s possible to learn chess by observing how others play it, and whether your attempts at playing certain moves are allowed or not. At that point, you would be able to play a game of chess, but you might struggle to explain the exact rules of the game.

In some ways, we could say that learning programming outside of school, you’re learning it without learning the rules. For example, when I was learning, I could write programs but I would struggle with certain parts because I couldn’t explain them properly. I disliked strong typing because I didn’t know the rules, and they felt arbitrary. I only learned various concepts much later, despite writing code for a relatively long time - I had learned programming without understanding the surrounding system of rules and practices, which had made doing certain things harder for me.

I think learning programming in school, or at least supplementing your self-learning with a more formal education, would allow you to learn these types of rules and practices faster. I think the deeper you go in your skill level, the more important it is to have a balanced knowledge of both the rules and facts, and the practical knowledge on how to apply them. If we use chess as an example a second time, top level chess requires not only deep study of chess and the games of the masters, but also knowing how to apply those strategies in practice.

Job interviews

Ryle also writes about how the intelligence of some activity can be evaluated. How do you determine someone actually understood something, instead of mindlessly repeating something they heard/read word for word? It occurred to me that this is essentially what we do when interviewing candidates in a job interview - we are trying to determine how much the candidate actually knows about something, which is, in essence, trying to determine the “intelligence” of the interviewee in this context.

Evaluating the intelligence doesn’t involve just looking at a specific performance, as it could be a fluke. We have to take into account multiple performances, but also their explanations and excuses. In an interview, to evaluate multiple performance, we can look at the candidate’s experience and previous employment, but to understand their explanations and excuses, we have to ask questions that allow us to gain insight into this. We need to ask more probing questions that uncover the thinking process, of why the interviewee answered the way they did.

Especially if you interview for more senior roles, discovering the depth of the knowledge of the interviewee becomes increasingly important. For juniors it can be enough to verify that they have “knowing that” type of factual knowledge about the desired tools. But for senior roles, their value is more in the “knowing how” side of things, so you have to take this into account, and ask questions that require them to explain their thinking process.

This is why I’m not sure if score-based evaluations, or evaluating a candidate based on asking technical trivia, helps that much. Sure - you might be able to use those as a filter to weed out some candidates, but the same filter might end up weeding out talented self-taught developers. For a self-taught developer, answering these types of trivia based questions could be more difficult, even if they were perfectly able to do the work. For example, if you ask me about big O notation, you’ll get very little beyond “it’s a way to show the cost of an algorithm”, because I never really learned it. But I could still optimize your program just fine.

AI tools

There are also certain implications to AI / LLM tools from what Ryle writes. I don’t think we can really say that LLM’s have practical knowledge about things, or any real understanding either. If we take the fact that you can’t create a set of rules to teach someone practical knowledge, where does this leave us with LLM’s? It seems to me that LLM’s will always be an imperfect system compared to actual human programmers.

Another reason for this is that you cannot recover the original design intention of code from just reading the code. This again relates to the earlier point about recovering the theory - in order to truly understand the design intent, you would need to understand the original thought process, which cannot be put into a set of steps or a set of rules, because the set would become infinite. This means that LLM’s cannot recover this information either, and therefore cannot make decisions about the code that are in line with the original intention.

I’m increasingly seeing the usage of Architectural Decision Records or other types of Markdown-based guides or skills in an attempt to bolster the LLM’s understanding of these types of choices. But if the above is true, and you cannot create a set of rules about rules, then creating those kinds of documents seems somewhat futile. Of course, an LLM with at least some inkling of the design choices is better than one with nothing at all, but the resulting system is still not going to match an actual human.

In closing

The Concept of Mind was a quite fascinating book. I’ve only mentioned a few things here that I thought had a some relation to programming, but there’s a lot more in there that are interesting in other ways as well.

Naur’s Programming as Theory Building text also mentioned Thomas S. Kuhn’s book The Structure of Scientific Revolutions. This is next on my reading list around this topic, and hopefully it will also have something interesting to say.