Some musings on human brutality and human evil

[epistemic status: semi-poetic musing]

I’m listening to Dan Carlin’s Hardcore History: Supernova in the East this week. The biggest thing that’s struck me so far is the ubiquity of brutality and atrocity. In this series, Carlin describes the Rape of Nanjing in particular, but he points out that the “police reports” from that atrocity could just as well describe the Roman sack of Cremona, or the Turkish conquest of Byzantium, not to mention the constant brutality of the the Mongol hordes.

I’m left with an awareness that there’s an evil in human nature, an evolutionary darkness, inextricably bound up with us: in the right context, apparently decent, often god-fearing, young men will rape and plunder and murder en mass. There’s violence under the surface.

Luckily, I personally live in a democratic great power that maintains a monopoly on the use of force. At least for me (white and middle class), and at least for now (geopolitics shifts rapidly, and many of the Jews of 1940 Europe, felt that something like the Holocaust could never happen [in their country]), power, in the form of the largest, most technological advanced military ever, and in the form of nuclear weapons, is arrayed to protect me against that violence.

But that protection is bought with blood and brutality. Not just in the sense that America is founded on the destruction of the Native Americans that were here first, and civilization itself was built on the backs of forceful enslavement (though that is very much the case). In the sense that elsewhere in the world, today, that American military might is destroying someone else’s home. I recently learned about the Huế Massacre and other atrocities of the Vietnam war, and I’m sure similar things (perhaps not as bad), happen every year. Humans can’t be trusted not to abuse their power.

It’s almost like a law of nature: if someone has the power to hurt another, that provides opportunity for the darkness in the human soul to flower in violence. It’s like a conservation law of brutality.

No. That’s not right. Brutality is NOT conserved. It can be better or worse. (To say otherwise would be an unacceptable breach of epistemic and ethics). But brutality is inescapable.

So what to do? I the only way I can buy safety for myself and my friends is with violence towards others?

The only solution that I can think of is akin to Paretotopian ideas: could we make it so that there is a monopoly on the use of force, but no human has it?

I’m imagining something like an AGI whose source code was completely transparent: everyone could see and read the its decision theory. And all that it would do is prevent the use of violence, by anyone. Anytime someone attempts to commit violence the nano-machines literally stay their hand. (It might also have to produce immortality pills, and ensure that everyone could access them if they wanted too.) And other than that, it lets humans handle things for themselves. “A limited sovereign on the blockchain.”

I imagine that the great powers would be unwilling to give up their power, unless they felt so under threat (and loss averse), that this seemed like a good compromise. I imagine that “we” would have to bully the world into adopting something like this. The forces of good in human nature would have to have the underhand, for long enough to lock in the status quo, to banish violence forever.



Two models of anxiety

[This is a confused thought that feels like it is missing something.]

I have two competing models of anxiety.

The first one, is basically the one I outlined here. There’s a part of me that is experiencing a fear or pain, and that part seeks distraction and immediate gratification to compensate for that pain.

But after reading about [[Physiological Arousal]], I have a secondary hypothesis. Instead of postulating a “part” that is motivated to seek distractions, maybe it is just that the fear triggers a fight or flight response, which increases arousal, which causes decreased attentional stability.

These different models suggest different places for intervention: in the one case, I ought to dialogue with the part that is seeking distraction or relief (?), and in the second case, I need to lower my arousal.

Or maybe both of those are mistaken, and I should just intervene on my scattered attention directly, perhaps by holding my attention on some external object for a minute (a kind of micro [[meditation]]).



Some musings about exercise and time discount rates

[Epistemic status: a half-thought, which I started on earlier today, and which might or might not be a full thought by the time I finish writing this post.]

I’ve long counted exercise as an important component of my overall productivity and functionality. But over the past months my exercise habit has slipped some, without apparent detriment to my focus or productivity. But this week, after coming back from a workshop, my focus and productivity haven’t really booted up.

Here’s a possible story:

Exercise (and maybe mediation) expands the effective time-horizon of my motivation system. By default, I will fall towards attractors of immediate gratification and impulsive action, but after I exercise, I tend to be tracking, and to be motivated by, progress on my longer term goals. [1]

When I am already in the midst of work: my goals are loaded up and the goal threads are primed in short term memory, this sort of short term compulsiveness causes me to fall towards task completion: I feel slightly obsessed about finishing what I’m working on.

But if I’m not already in the stream of work, seeking immediate gratification instead drives me to youtube and web comics and whatever. (Although it is important to note that I did switch my non self tracking web usage to Firefox this week, and I don’t have my usual blockers for youtube and for SMBC set up yet. That might totally account for the effect that I’m describing here.)

In short, when I’m not exercising enough, I have less meta cognitive space for directing my attention and choosing what is best do do. But if I’m in the stream of work already, I need that meta cognitive space less: because I’ll default to doing more of what I’m working on. (Though, I think that I do end up getting obsessed with overall less important things, compared to when I am maintaining metacognitive space). Exercise is most important for booting up and setting myself up to direct my energies.

[1] This might be due to a number of mechanisms:

  • Maybe the physical endorphin effect of exercise has me feeling good, and so my desire for immediate pleasure is sated, freeing up resources for longer term goals.
  • Or maybe exercise involves engaging in intimidate discomfort for the sake of future payoff, and this shifts my “time horizon set point” or something. (Or maybe it’s that exercise is downstream of that change in set point.)
    • If meditation also has this time-horizon shifting effect, that would be evidence for this hypothesis.
    • Also if fasting has this effect.
  • Or maybe, it’s the combination of both of the above: engaging in delayed gratification, with a viscerally experienced payoff, temporarily retrains my motivation system for that kind of thing.)
  • Or something else.


Some notes on Von Neumann, as a human being

I recently read Prisoner’s Dilemma, which half an introduction to very elementary game theory, and half a biography of John Von Neumann, and watched this old PBS documentary about the man.

I’m glad I did. Von Neumann has legendary status in my circles, as the smartest person ever to live. [1] Many times I’ve written the words “Von Neumann Level Intelligence” in a AI strategy document, or speculated about how many coordinated Von Neumanns would it take to take over the world. (For reference, I now think that 10 is far too low, mostly because he didn’t seem to have the entrepreneurial or managerial dispositions.)

Learning a little bit more about him was humanizing. Yes, he was the smartest person ever to live, but he was also an actual human being, with actual human traits.

Watching this first clip, I noticed that I was surprised by a number of thing.

  1. That VN had an accent. I had known that he was Hungarian, but somehow it had never quite propagated that he would speak with a Hungarian accent.
  2. That he was middling height (somewhat shorter than the presenter he’s talking too).
  3. The thing he is saying is the sort of thing that I would expect to hear from any scientist in the public eye, “science education is important.” There is something revealing about Von Neumann, despite being the smartest person in the world, saying basically what I would expect Neil DeGrasse Tyson to say in an interview. A lot of the time he was wearing his “scientist / public intellectual” hat, not the “smartest person ever to live” hat.

Some other notes of interest:

He was not a skilled poker player, which punctured my assumption that Von Neumann was omnicompetent. (pg. 5) Nevertheless, poker was among the first inspirations for game theory. (When I told this to Steph, she quipped “Oh. He wasn’t any good at it, so he developed a theory from first principles, describing optimal play?” For all I know, that might be spot on.)

Perhaps relatedly, he claimed he had low sales resistance, and so would have his wife come clothes shopping with him. (pg. 21)

He was sexually crude, and perhaps a bit misogynistic. Eugene Wigner stated that “Johny believed in having sex, in pleasure, but not in emotional attachment. HE was interested in immediate pleasure and little comprehension of emotions in relationships and mostly saw women in terms of their bodies.” The journalist Steve Heimes wrote “upon entering an office where a pretty secretary was working, von Neumann habitually would bend way over, more or less trying to look up her dress.” (pg. 28) Not surprisingly, his relationship with his wife, Klara, was tumultuous, to say the least.

He did however, maintain a strong, life long, relationship with his mother (who died the same year that he did).

Overall, he gives the impression of being a genius, overgrown child.

Unlike many of his colleagues, he seemed not to share the pangs conscience that afflicted many of the bomb creators. Rather than going back to academia following the war, he continued doing work for the government, including the development of the Hydrogen bomb.

Von Neumann advocated preventative war: giving the Soviet union an ultimatum, of joining a world government, backed by the threat of (and probable enaction of) nuclear attack, while the US still had a nuclear monopoly. He famously said of the matter, “If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o’clock, I say why not 1 o’clock.”

This attitude was certainly influenced by his work on game theory, but it should also be noted that Von Neumann hated communism.

Richard Feynman reports that Von Neumann, in their walks through the Los Alamos desert, convinced him to adopt and attitude of “social irresponsibility”, that one “didn’t have to be responsible for the world he was in.”

Prisoner’s dilemma says that he and his collaborators “pursued patents less aggressively than the could have”. Edward Teller commented, “probably the IBM company owes half its money to John Von Neumann.” (pg. 76)

So he was not very entrepreneurial, which is a bit of a shame, because if he had the disposition he probably could have made sooooo much money / really taken substantial steps towards taking over the world. (He certainly had the energy to be an entrepreneur: he only slept for a few hours a night, and was working for basically all his working hours.

He famously always wore a grey oxford 3 piece suit, including when playing tennis with Stanislaw Ulam, or when riding a donkey down the grand canyon. But, I am not clear why. Was that more comfortable? Did he think it made him look good? Did he just not want to have to ever think about clothing, and so preferred to be over-hot in the middle of the Los Alamos desert, rather than need to think about if today was “shirt sleeves whether”?

Von Neumann himself once commented on the strange fact of so many Hungarian geniuses growing up in such a small area, in his generation:

Stanislaw Ulam recalled that when Von Neumann was asked about this “statistically unlikely” Hungarian phenomenon, Von Neumann “would say that it was a coincidence of some cultural factors which he could not make precise: an external pressure on the whole society of this part of Central Europe, a subconscious feeling of extreme insecurity in individual, and the necessity of producing the unusual or facing extinction.” (pg. 66)

One thing that surprised me most was that it seems that, despite being possibly the smartest person in modernity, he would have benefited from attending a CFAR workshop.

For one thing, at the end of his life, he was terrified of dying. But throughout the course of his life he made many reckless choices with his health.

He ate gluttonously and became fatter and fatter over the course of his life. (One friend remarked that he “could count anything but calories.”)

Furthermore, he seemed to regularly risk his life when driving.

Von Neuman was an aggressive and apparently reckless driver. He supposedly totaled his car every year or so. An intersection in Princeton was nicknamed “Von Neumann corner” for all the auto accidents he had there. records of accidents and speeding arrests are preserved in his papers. [The book goes on to list a number of such accidents.] (pg. 25)

(Amusingly, Von Neumann’s reckless driving seems due, not to drinking and driving, but to singing and driving. “He would sway back and forth, turning the steering wheel in time with the music.”)

I think I would call this a bug.

On another thread, one of his friends (the documentary didn’t identify which) expressed that he was over-impressed by powerful people, and didn’t make effective tradeoffs.

I wish he’d been more economical with his time in that respect. For example, if people called him to Washington or elsewhere, he would very readily go and so on, instead of having these people come to him. It was much more important, I think, he should have saved his time and effort.

He felt, when the government called, [that] one had to go, it was a patriotic duty, and as I said before he was a very devoted citizen of the country. And I think one of the things that particularly pleased him was any recognition that came sort-of from the government. In fact, in that sense I felt that he was sometimes somewhat peculiar that he would be impressed by government officials or generals and so on. If a big uniform appeared that made more of an impression than it should have. It was odd.

But it shows that he was a person of many different and sometimes self contradictory facets, I think.

Stanislaw Ulam speculated, “I think he had a hidden admiration for people and organizations that could be tough and ruthless.” (pg. 179)

From these statements, it seems like Von Neumann leapt at chances to seem useful or important to the government, somewhat unreflectively.

These anecdotes suggest that Von Neumann would have gotten value out of Goal Factoring, or Units of Exchange, or IDC (possibly there was something deeper going on, regarding a blindspots around death, or status, but I think the point still stands, and he would have benefited from IDC).

Despite being the discoverer/ inventor of VNM Utility theory, and founding the field of Game Theory (concerned with rational choice), it seems to me that Von Neumann did far less to import the insights of the math into his actual life than say, Critch.

(I wonder aloud if this is because Von Neumann was born and came of age before the development of cognitive science. I speculate that the importance of actually applying theories of rationality in practice, only becomes obvious after Tversky and Kahneman demonstrate that humans are not rational by default. (In evidence against this view: Eliezer seems to have been very concerned with thinking clearly, and being sane, before encountering Heuristics and Biases in his (I belive) mid 20s. He was exposed to Evo Psych though.))

Also, he converted to Catholicism at the end of his life, based on Pascal’s Wager. He commented “So long as there is the possibility of eternal damnation for nonbelievers it is more logical to be a believer at the end”, and “There probably has to be a God. Many things are easier to explain if there is than if there isn’t.”

(According to wikipedia, this deathbed conversion did not give him much comfort.)

This suggests that he would have gotten value out of reading the sequences, in addition to attending a CFAR workshop.


Basic Double Crux pattern example

Followup to: The Basic Double Crux Pattern


This is a continuation of my previous post, in which I outlined the what I currently see as the core pattern of double crux finding. This post will most be an example of that pattern in action.

While this example will be more realistic than the intentionally silly example in the last post, it will still be simplified, for the sake of brevity and clarity. There’s a tradeoff between realism and clarity of demonstration. The following conversation is mostly-fictional (though inspired by Double Cruxes that I have actually observed). It is intended to be realistic enough that this could have been a real conversation, but is also specifically demonstrating the basic Double Crux pattern, (as opposed to other moves or techniques that are often relevant.)


As a review, the key conversational moves of Double Crux are…

  1. Finding a single crux:
    1. Checking for understanding of P1’s point.
    2. Checking if that point is a crux for P1.
  2. Finding a double crux:
    1. Checking P2’s belief about P1’s crux.
    2. Checking if P2’s crux is also a crux for P1.

As an excise, I encourage you to go through the example below below and track what is currently happening in the context of this frame: Which parts of the conversation correspond to which conversational move?

Note that, in general, participants will be intent on saying all kinds of things that aren’t cruxes, and part of the facilitator’s role is to keep the conversation on track, while also engaging with their concerns.

Example: Carol and Daniel on gun control legislation

[Important note: This example is entirely fictional. It doesn’t represent the views, of anyone, including me. The arguments that these two character’s make are constructions of arguments that I have heard from CFAR participants, and things that I heard in podcasts years ago. I have no idea if these are in fact the most important considerations about gun control, or if the facts that these characters reference are actually true, only that they are plausible positions that a person could hold.]

Two people, Carol and Daniel, are having a discussion about politics. In particular, they’re discussing new gun laws that would heavily restrict citizens ability to buy and sell firearms. Carol is in favor of the laws, but Daniel is opposed.

Facilitator: Daniel, do you want to tell us why you think this law is a bad idea?

Daniel: Look every time there’s a mass shooting, everyone gets up in arms about it, and politicians scramble over each other to show how tough they are, and how bad an event the shooting was, and then they pass a bunch of laws that don’t actually help, and add a bunch of red tape for people who want to buy firearms for legitimate uses.

Facilitator: Ok. There were a lot of things in there, but one part of is you said that these laws don’t work?

Daniel: That’s right.

Facilitator: you mean that they don’t reduce incidence of mass shooting?

Daniel: Yeah, but mass shootings are almost a rounding error.

Facilitator: [suggesting an operationalization] So what do you mean when you say they don’t help? That they don’t reduce the murder rate?

Daniel: Yeah. That’s right.

Facilitator: Ok. If you found out that this law in particular did reduce the US murder rate, would you be in favor of it?

Daniel: Well, it would have to have a significant impact. If the effect is so small as to be basically unnoticeable, then it probably isn’t worth the constraint on millions of people across the US.

Facilitator: Ok. But if you knew that this law would “meaningfully” reduce the murder rate, would you change your mind about this law? (Where meaningfully means something like “more than 5%”?)

Daniel: Yeah, that sounds right.

Facilitator: Great. Ok. So it sounds like whether this law would impact the murder rate is a crux for you.

Carol, do you think that this law would reduce the murder rate?

Carol: Yeah.

Facilitator: By more than 5%?

Carol: I’m not sure by how much specifically, but something like that.

Facilitator: Ok. If you found it that this law would actually have no effect (or close to no effect) on the murder rate would you change your mind?

Carol: I think it would reduce the murder rate.

Facilitator: Cool. You currently think that this law would reduce the murder rate. I want to talk about why you think that in a moment. But first, I want to check if it’s a crux. Suppose you found really strong evidence that this this law actually wouldn’t reduce the murder rate at all (I realize that you currently think that it will), would you still support this law?

Carol: Yeah. I guess there wouldn’t be much of a point to it in that case.

Facilitator: Cool. So it sounds like this is a double Crux. You, Daniel, would change your mind about this law if you found out that it would improve the murder rate, substantially, and you, Carol, would change your mind, if you found out it wouldn’t improve the murder rate. High-five!

The participants high-five, and write down their cruxes.

[insert picture]

Facilitator: Alright. So it sounds like we should talk about this double crux, why do each of you think that this would improve the murder rate?

Carol: Well, I’m aware of other countries that implemented similar reforms, and it did in fact improve incidents of violent crime in those countries.

Facilitator: Ok. So the reason why you think that this law would reduce the murder rate is that it worked in other countries?

Carol: That’s right.

Facilitator: Ok. Suppose you found out that actually laws like this did not help in those other countries? Would that change your mind about how likely this reform is to work in the US?

Carol: I’m pretty sure that it did work. In Sweden, for instance.

Facilitator: Ok. Suppose you found out that the evidence in those countries, like Sweden, didn’t hold up, or it turns out you misread the studies or something.

Carol: If all those studies turned out to be bunk…Yeah. I guess that would cause me to change my mind.

Facilitator: Cool. So it sounds like that’s a crux for you.

Daniel, presumably, you think that laws like this don’t lower the murder rate in other countries. Is that right?

Daniel: Well, I don’t know about those studies in particular. But I’m generally skeptical here, because this topic is politicized. I think you can’t rely on most of the papers in this area.

Facilitator: Ok. Is that a crux for you?

I don’t think so. There are a lot of gun law interventions that are effective in other countries, but don’t work in the US, because there are just so many more guns in the US, than in other countries in the world. Like, because there are so many guns already in circulation, making it harder to buy a gun today, makes almost no difference for how easy it is to get your hands on a gun. In other countries where there are fewer weapons lying around, things like gun buybacks, or restrictions on gun purchases, have more of an effect, because the size of the pool of the existing guns is larger, and so those interventions have a much smaller proportional impact.

Like, you have to account for the black market. There’s a big black market in weapons in the US, much bigger than in most European countries, I think.

Facilitator: [Summarizing] Ok. It sounds like you’re saying that you don’t think making harder to buy guns has an impact on the murder rate because, if you want to kill someone in the US, you can get your hand on a gun one way or another.

Daniel: Yeah.

Facilitator: Ok. Is that a crux for you? Suppose that you found out that, actually reforms like this one do make it much harder for would-be criminals to get a gun. Would that change your mind about whether this reform would shift the murder rate?

Daniel: Let’s see…I might want to think about it more, but yeah, I think if I knew that it actually made it harder for those people to get guns, that would change my mind, but it would have to be a lot harder. Like, it would have to be hard enough that those people end up just not having guns, not like they would have to wait an extra 2 weeks.

Facilitator: Ok. Cool. If laws like this made it so that fewer criminals had guns, you would expect those laws to improve the murder rate?

Daniel: yeah.

Facilitator: Carol, do you expect that this new gun law would cause fewer criminals to own guns?

Carol: Yeah. But it’s not just the guns. The other thing is the ammunition. This law would not just make it harder to buy guns, but would make it harder to buy bullets. It doesn’t matter how criminals got their guns (they might have inherited them from their grandmother, for all I know), if they can’t buy bullets for those guns.

Facilitator: OH-k. That’s pretty interesting. You’re saying that the way that you limit gun violence is by restricting the flow of ammunition.

Carol: Exactly.

Daniel: Huh. Ok. I hadn’t thought about that. That does seem pretty relevant.

Facilitator (to Daniel): Is that a crux for you? If this law makes it harder for criminals to get bullets, would you expect it to impact the murder rate?

Daniel: Yeah. I hadn’t thought about that. Yeah, if would-be murderers have trouble getting ammunition, then I would expect that to have an impact on the murder rate. But I don’t know how they get bullets. It might be mostly via black market channels that aren’t much effected by this kind of regulation. Also, it maybe that it’s similar to guns, in that there are already so many bullets in circulation, that it

Facilitator (to Daniel): Cool. Let’s hold off on that question until we check if it is also a crux for Carol.

Facilitator (to Carol): Is that a crux? Suppose that you found out that criminals are able to get their hands on ammunition via “unofficial”, illegal methods. Would that change your mind about how likely this policy is to reduce the murder rate in the US?

Carol: Let me think.

. . .

I’m still kind of suspicious. I would want to see some hard data of states or countries that have the a similar gun culture to the US, and also have this sorts of restrictions-

Daniel [Interrupting]: Yeah, I would obviously want that too.

Carol: …but in the absence of evidence like that, then it makes sense that if criminals  get guns and ammo via illegal channels, then a reform that makes it harder to buy weapons wouldn’t have any impact on the murder rate.

Facilitator: Ok. Awesome. So it sounds like we have a Double Crux, “would this regulation limit the availability of ammunition for would-be killers.” Separately, you Carol, have a single crux about “Would this make it harder for criminals to buy guns?” We could talk about that, but Daniel doesn’t think it’s as relevant.

Carol: No. I’m excited to just talk about the ammo for now.

Facilitator: Cool. That also makes sense to me.

And maybe underneath “Would this regulation limit the availably of ammunition to would-be killers”, there’s another question of “How do criminals mostly get their ammunition?” If they get their ammo from sources that are relevantly affected by these sorts of restrictions, then we expect it to restrict the flow of ammo to criminals. But if they have a stockpile, or get it from the black market (which get’s it from other countries, or something), then we wouldn’t expect the availability of ammo for criminals to be impacted much.

Does that sound right?


Facilitator: Ok then. Cool. I think you guys should high five.

Carol and Daniel high five, and the write the new cruxes on their piece of paper.


Facilitator: Also, it sounds like there was maybe another Double Crux there? That is, if there was evidence from a country or state that tried a policy like this one, and also had a similar number of guns in circulation?

Does that sound right?

Daniel: Yeah.

Carol: Yep.

Facilitator: Ok. That just seems good to note, even  if it isn’t actionable right now. If either of you encounter evidence of that sort, it seems like it would be relevant to your beliefs.

But it seems like the most useful thing to do right now is to see if we can figure out how criminals typically get ammunition.

Daniel and Carol researched this question, found a bunch of conflicting evidence, but settled on a shared epistemic state about how most criminals get their ammo. One or both of them updated, and they lived with a more detailed model of this small part world ever after.


Hopefully this example gives a flavor of what a Double Crux conversation is (often) like.

Again, I want to point out that this example uses the person of the facilitator for convenience. While I do usually recommend getting a third person to be a facilitator, it is also totally possible for one or both of the conversational participants to execute these moves and steer the conversation.

This conversation is specifically intended to demonstrate the basic Double Crux pattern (although the normal order was disrupted a bit towards the end of the conversation), but nevertheless, we see some examples of other principles or techniques of productive disagreement resolution.

Distillation: As is common, Daniel said a lot of things in his opening statement, not all of which are relevant, or cruxes. One of the things that the facilitator needs to do is draw out the key point or points from all of those words, and then check if those are crucial. Much of the work of Double Crux is drawing out the key points each person is making, so you can check if they are cruxes.

Operationalizing: This particular example was pretty low on operationalization, but nevertheless, early on, the conversation settled on the operationalization of “reducing the murder rate”.

Persistent crux checking: Every time either Carol, or Daniel made a point, the facilitator would ask them if it was a crux.

Signposting: The facilitator frequently states what just happened (“It sounds like that’s a crux for you.”). This is often surprisingly helpful to do in a conversation. The facilitator

Distinguishing between whether a proposition is a crux, and what the truth value of a crux is: One thing that you’ll notice is that several times, the facilitator asks Carol if some proposition “A” is a crux for her, and she responds by saying, in effect, “I think A is true”. This is a fairly common thing that happens, even for quite smart people. It is important to make mental space between the questions “is X a crux for Y?”, and “is X true?”. It is often very tempting to jump in and refute a claim, before checking if it is a crux. But it is important to keep those questions separate.

False starts: There were several places where the facilitator checked for a Double Crux, and “missed”. This is natural and normal.

This conversation was weird on in that most of the considerations named were cruxes for someone. People are often inclined to share consideration, that upon reflection, they agree don’t matter at all.

Conversational avenues avoided: This is maybe the key value add of Double Crux. For every false start, there was a long, and ultimately irrelevant conversation that could have been had.

It might have been tempting for Daniel to argue about whether gun control regulation actually worked in other countries. (It is just sooo juicy to tell people exactly how they are wrong, when they’re wrong.) But that would have been a pointless exercise, (for Daniel’s belief improvement, at least), because Daniel thinks  that the data from other countries is irrelevant. At every point they stay on a track that both parties consider promising for updating.

(Of course, sometimes you do have information that is relevant to your partner’s crux, but not to your own, and it is good and proper to share that info with them.)

Inside view / outside view separation: Towards the end, we saw Carol distinguishing between outside view reasoning based on empirical data, and inside view reasoning based on thinking through probable consequences. She clearly separated these two, for herself. I think this is sometimes a useful move, so that one can feel psychologically permitted to do blue-sky reasoning, without forgetting that you would defer to the empirical data over and above your theorizing.


I also again want to clarify my claims:

  • I am not claiming that this pattern solves all disagreements.
  • I am not claiming that this is the only thing a person should do in conversation.
  • I am not claiming that one should always stick religiously to this pattern, even when Double Cruxing. (Though it is good to try sticking religiously to this pattern as a training exercise. This is one of the exercises that I have people do at Double Crux trainings.)

I am saying…

  1. This is my current best attempt to delineate how finding double cruxes usually works, and
  2. I have found double cruxes to be extremely useful for productively resolving disagreements.


Hope this helps. More to come, soon probably.


[As always, I’m generally willing to facilitate conversations and disagreements (Double Crux or not) for rationalists and EAs. Email me at eli [at] rationality [dot] org if that’s something you’re interested in.]

The Basic Double Crux Pattern

[This is a draft, to be posted on LessWrong soon.]

I’ve spent a lot of time developing tools and frameworks for bridging “intractable” disagreements. I’m also the person affiliated with CFAR who has taught Double Crux the most, and done the most work on it.

People often express to me something to the effect, “The important thing about Double Crux is all the low level habits of mind: being curious, being open to changing your mind, paraphrasing to check that you’ve understood, operationalizing, etc. The ‘Double Crux’ framework, itself is not very important.”

I half agree with that sentiment. I do think that those low level cognitive and conversational patterns are the most important thing, and at Double Crux trainings that I have run, most of the time is spent focusing on specific exercises to instill those low level TAPs.

However, I don’t think that the only value of the Double Crux schema is in training those low level habits. Double cruxes are extremely powerful machines that allow one to identify, if not the most efficient conversational path, a very high efficiency conversational path. Effectively navigating down a chain of Double Cruxes is like magic. So I’m sad when people write it off as useless.

In this post, I’m going to try and outline the basic Double Crux pattern, the series of 4 moves that makes Double Crux work, and give a (simple, silly) example of that pattern in action.

These four moves are not (always) sufficient for making a Double Crux conversation work, that does depend on a number of other mental habits and TAPs, but this pattern is, according to me, at the core of the Double Crux formalism.

The pattern:

The core Double Crux pattern is as follows. For simplicity, I have described this in the form of a 3-person Double Crux conversation, with two participants and a facilitator. Of course, one can execute these same moves in a 2 person conversation, as one of the participants. But that additional complexity is hard to manage for beginners.

The pattern has two parts (finding a crux, and finding a double crux), and each part is composed of 2 main facilitation moves.

Those four moves are…

  1. Clarifying that you understood the first person’s point.
  2. Checking if that point is a crux
  3. Checking the second person’s belief about the truth value of the first person’s crux.
  4. Checking the if the first person’s crux is also a crux for the second person.

In practice

The conversational flow of these moves looks something like this:

Finding a crux of participant 1:

P1: I think [x] because of [y]

Facilitator: (paraphrasing, and checking for understanding) It sounds like you think [x] because of [y]?

P1: Yep!

Facilitator: (checking for cruxyness) If you didn’t think [y], would you change your mind about [x]?

P1: Yes.

Facilitator: (signposting) It sounds like [y] is a crux for [x] for you.

Checking if it is also a crux for participant 2

Facilitator: Do you think [y]?

P2: No.

Facilitator: (checking for a Double Crux) if you did think [y] would that change your mind about [x]?

P2: Yes.

Facilitator: It sounds like [y] is a Double Crux

[Recurse, running the same pattern on [Y] ]

Obviously, in actual conversation, there is a lot more complexity, and a lot of other things that are going on.

For one thing, I’ve only outlined the best case pattern, where the participants give exactly the most convenient answer for moving the conversation forward (yes, yes, no, yes). In actual practice, it is quite likely that one of those answers will be reversed, and you’ll have to compensate.

For another thing, this formalism is rarely so simple. You might have to do a lot of conversational work to clarify the claims enough that you can ask if B is a crux for A (for instance when B is nonsensical to one of the participants). Getting through each of these steps might take fifteen minutes, in which case rather than four basic moves, this pattern describes four phases of conversation. (I claim that one of the core skills of a savvy facilitator is tracking which stage the conversation is at, which goals have you successfully hit, and which is the current proximal subgoal.)

There is also a judgment call about which person to treat as “participant 1” (the person who generates the point that is tested for cruxyness). As a first order heuristic, the person who is closer to making a positive claim over and above the default, should usually be the “p1”. But this is only one heuristic.


This is an intentionally silly, over-the-top-example, for demonstrating the the pattern without any unnecessary complexity. I’ll publish a somewhat more realistic example in the next few days.

Two people, Alex and Barbra, disagree about tea: Alex thinks that tea is great, and drinks it all the time, and thinks that more people should drink tea, and Barbra thinks that tea is bad, and no one should drink tea.

Facilitator: So, Barbra, why do you think tea is bad?

Barbra: Well it’s really quite simple. You see, tea causes cancer.

Facilitator: Let me check if I’ve got that: you think that tea causes cancer?

Barbra: That’s right.

Facilitator: Wow. Ok. Well if you found out that tea actually didn’t cause cancer, would you be fine with people drinking tea.

Barbra: Yeah. Really the main thing that I’m concerned with is the cancer-causing. If tea didn’t cause cancer, then it seems like tea would be fine.

Facilitator: Cool. Well it sounds like this is a crux for you Barb. Alex, do you currently think that tea causes cancer?

Alex: No. That sounds like crazy-talk to me.

Facilitator: Ok. But aside from how realistic it seems right now, if you found out that tea actually does cause cancer, would you change your mind about people drinking tea?

Alex: Well, to be honest, I’ve always been opposed to cancer, so yeah, if I found out that tea causes cancer, then I would think that people shouldn’t drink tea.

Facilitator: Well, it sounds like we have a double crux!

In a real conversation, it often doesn’t goes this smoothly. But this is the rhythm of Double Crux, at least as I apply it.

That’s the basic Double Crux pattern. As noted there are a number of other methods and sub-skills that are (often) necessary to make a Double Crux conversation work, but this is my current best attempt at a minimum compression of the basic engine of finding double cruxes.

I made up a more realistic example here, and I’m might make more or better examples.


Some things I think about Double Crux and related topics

I’ve spent a lot of my discretionary time working on the broad problem of developing tools for bridging deep disagreements. I’m also probably the person who has spent the most time explicitly thinking about and working with CFAR’s Double Crux framework. It seems good for at least some of my high level thoughts to be written up some place, even if I’m not going to go into detail, defend, or substantiate, most of them.

The following are my own beliefs and do not necessarily represent CFAR, or anyone else. I, of course, reserve the right to change my mind.

Here are some things I currently belive:


  1. Double Crux is one (highly important) tool/ framework among many. I want to distinguish between the the overall art of untangling and resolving deep disagreements and the Double Crux tool in particular. The Double Crux framework is maybe the most important tool (that I know of) for that goal, but it is only one tool/framework in an ensemble.
    1. Some other tools/ frameworks, that are not strictly part of Double Crux (but which are sometimes crucial to bridging disagreements) include NVC, methods for managing people’s intentions and goals, various forms of co-articulation (helping to draw out an inchoate model from one’s conversational partner), etc.
    2. In some contexts other tools are substitutes for Double Crux (ie another framework is more useful) and in some cases other tools are helpful or necessary compliments (ie they solve problems or smooth the process within the Double Crux frame).
    3. In particular, my personal conversational facilitation repertoire is about 60%  Double Crux-related techniques, and 40% other frameworks that are not strictly within the frame of Double Crux.
  2. Just to say it clearly: I don’t think Double Crux is the only way to resolve disagreements, or the best way in all contexts. (Though I think it may be the best way, that I know of, in a plurality of common contexts?)
  3. The ideal use case for Double Crux is when…
    1. There are two people…
    2. …who have a real, action-relevant, decision…
    3. …that they need to make together (they can’t just do their own different things)…
    4. …in which both people have strong, visceral intuitions.
  4. Double Cruxes are almost always conversations between two people’s system 1’s.
  5. You can Double Crux between two people’s unendorsed intuitions. (For instance, Alice and Bob are discussing a question about open borders. They both agree that neither of them are economists, and that neither of them trust their intuitions here, and that if they had to actually make this decision, it would would be crucial to spend a lot of time doing research and examining the evidence and consulting experts. But nevertheless Alices current intuition leans in favor of Open Borders , and Bob’s current intuition leans agains Open Borders. This is a great starting point for a Double Crux.)
  6. Double cruxes (as in a crux that is shared by both parties in a disagreement) are common, and useful. Most disagreements have implicit Double Cruxes, though identifying them can sometimes be tricky.
  7. Conjunctive cruxes (I would change my mind about X, if I changed my mind about Y and about Z, but not if I only changed my mind about Y or Z) are common.
  8. Folks sometimes object that Double Crux won’t work, because their belief depends on a large number of considerations, each one of which has only a small impact on their overall belief, and so no one consideration is a crux. In practice, I find that there are double cruxes to be found even in cases where people expect their beliefs have this structure.
    1. Theoretically, it makes sense that we would find double cruxes here: if a person has a strong disagreement (including a disagreement of intuition) with someone else, we should expect that there are a small number of considerations doing most of the work of causing one person to think one thing and the other to think something else. It is improbable that each person’s beliefs depend on 50 factors, and for Alice, most of those 50 factors point in one direction, and for Bob, most of those 50 factors point in the other direction, unless the details of those factors are not independent. If considerations are correlated, you can abstract out the fact or belief that that generates the differing predictions in all of those separate considerations. That “generating belief” is a crux.
    2. That said, there is a different conversational approach that I sometimes use, which involves delineating all of the key considerations (then doing Goal-factoring style relevance and completeness checks), and then dealing with each consideration one at time (often via a fractal tree structure: listing the key considerations of each of the higher level considerations).
      1. This approach absolutely requires paper, and skillful (firm, gentle) facilitation, because people will almost universally try and hop around between considerations, and they need to be viscerally assured that their other concerns are recorded and will be dealt with in due course, in order to engage deeply with any given consideration.
  9. About 60% of the power of Double Crux comes from being specific.
    1. I quite like Liron’s recent sequence on being specific. It re-reminded me of some basic things that have been helpful in several recent conversations. In particular, I like the move of having a conversational paint a specific, best case scenario, as a starting point for discussion.
      1. (However, I’m concerned about Less Wrong readers trying this with a spirit of trying to “catch out” one’s conversational partner in inconsistency, instead of trying to understand what their partner wants to say, and thereby shooting themselves in the foot. I think the attitude of looking to “catch out” is usually counterproductive to both understanding and to persuasion. People rarely change their mind when they feel like you have trapped them in some inconsistency, but they often do change their mind if they feel like you’ve actually heard and understood their belief / what they are trying to say / what they are trying to defend, and then provide relevant evidence and argument. In general (but not universally) it is more productive to adopt a collaborative attitude of trying to sincerely trying to help a person articulate, clarify, and substantiate the point your partner is trying to make, even if you suspect that their point is ultimately wrong and confused.)
  10. Many (~50%) disagreements evaporate upon operationalization, but this happens less frequently than people think: and if you seem to agree about all of the facts, and agree about all specific operationalizations, but nevertheless seem to have differing attitudes about a question, that should be a flag. [I have a post that I’ll publish soon about this problem.]
  11. You should be using paper when Double Cruxing. Keep track of the chain of Double Cruxes, and keep them in view.
  12. People talk past eachother all the time, and often don’t notice it. Frequently paraphrasing your current understand of what your conversational partner is saying, helps with this. [There is a lot more to say about this problem, and details about how to solve it effectively].
  13. I don’t endorse the Double Crux “algorithm” described in the canonical post. That is, I don’t think that the best way to steer a Double Crux conversation, is to hew to those 5 steps in that order. Actually finding double cruxes is, in practice, much more complicated, and there are a large number of heuristics and TAPs that make the process work. I regard that algorithm as an early (and self conscious) attempt to delineate moves that would help move a conversation towards double cruxes.
  14. This is my current best attempt at distilling the core moves that makes Double Crux work, though this leaves out a lot.
  15. In practice, I think that double cruxes most frequently emerge not from people independently generating their own list cruxes (though this is useful). Rather double cruxes usually emerge from the move of “checking if the point that your partner made is a crux for you.”
  16. I strongly endorse facilitation of basically all tricky conversations, Double Crux oriented or not. It is much easier to have a third party track the meta and help steer, instead of the participants, who’s working memory is (and should be) full of the object level.
  17. So called, “Triple Crux” is not a feasible operation. If you have more than two stakeholders, have two of them Double Crux, and then have one of those two Double Crux with the third person. Things get exponentially trickier as you add more people. I don’t think that Double Crux is a feasible method for coordinating more than ~ 6 people.
  18. Double Crux is much easier when both parties are interested in truth-seeking and in changing their mind, and are assuming good faith about the other. But, these are not strict prerequisites, and unilateral Double Crux is totally a thing.
  19. People being defensive, emotional, or ego-filled does not preclude a productive Double Crux. Some particular auxilary skills are required for navigating those situations, however.
    1. This is a good start for the relevant skills.
  20. If a person wants to get better at Double Crux skills, I recommend they cross-train with IDC. Any move that works in IDC you should try in Double Crux. Any move that works in Double Crux you should try in IDC. This will seem silly sometimes, but I am pretty serious about it, even in the silly-seeming cases. I’ve learned a lot this way.
  21. I don’t think Double Crux necessarily runs into a problem of “black box beliefs” wherein one can no longer make progress because one or both parties comes down to a fundamental disagreement about System 1 heuristics/ models that they learned from some training data, but into which they can’t introspect. Almost always, there are ways to draw out those models.
    1. The simplest way to do this (which is not the only or best way, depending on the circumstances, involves generating many examples and testing the “black box” against them. Vary the hypothetical situations to triangulate to the exact circumstances in which the “black box” outputs which suggestions.
    2. I am not making the universal claim that one never runs into black box beliefs, that can’t be dealt with.
  22. Disagreements rarely come down to “fundamental value disagreements”. If you think that you have gotten to a disagreement about fundamental values, I suspect there was another conversational tact that would have been more productive.
  23. Also, you can totally Double Crux about values. In practice, you can often treat values like beliefs: often there is some evidence that a person could observe, at least in principle, that would convince them to hold or not hold some “fundamental” value.
    1. I am not making the claim that there are no such thing as fundamental values, or that all values are Double Crux-able.
  24. A semi-esoteric point: cruxes are  (or can be) contiguous with operationalizations. For instance, if I’m having a disagreement about whether advertising produces value on net, I might operationalize to “beer commercials, in particular, produce value on net”, which (if I think that operationalization actually captures the original question) is isomorphic to “The value of beer commercials is a crux for the value of advertising.  I would change my mind about advertising in general, if I changed my mind about beer commercials.” (In this is an evidential crux, as opposed to the more common causal crux. (More on this distinction in future posts.))
  25. People’s beliefs are strongly informed by their incentives. This makes me somewhat less optimistic about tools in this space than I would otherwise be, but I still think there’s hope.
  26. There are a number of gaps in the repertoire of conversational tools that I’m currently aware of. One of the most important holes is the lack of method for dealing with psychological blindspots. These days, I often run out of ability to make a conversation go well when we bump into a blindspot in one person or the other (sometimes, there seem to be psychological blindspots on both sides). Tools wanted, in this domain.

(The Double Crux class)

  1. Knowing how to identify Double Cruxes can be kind of tricky, and I don’t think that most participants learn the knack from the 55 to 70 minute Double Crux class at a CFAR workshop.
  2. Currently, I think I can teach the basic knack (not including all the other heuristics and skills) to a person in about 3 hours, but I’m still playing around with the how to do this most efficiently. (The “Basic Double Crux pattern” post is the distillation of my current approach.)
    1. This is one development avenue that would particularly benefit from from parallel search: If you feel like you “get” Double Crux, and can identify Double Cruxes fairly reliably and quickly, it might be helpful if you explicated your process.
  3. That said, there are a lot of relevant compliments and sub-skills to Double Crux, and to bridging disagreements more generally.
  4. The most important function of the Double Crux class at CFAR workshops is teaching and propagating the concept of a “crux”, and to a lesser extent, the concept of a “double crux”. These are very useful shorthands for one’s personal thinking and for discourse, which are great to have in the collective lexicon.

(Some other things)

  1. Personally, I am mostly focused on developing deep methods (perhaps for training high-expertise specialists) that increase the range of problems (in this case, disagreements) that the x-risk ecosystem can solve at all. I care more about this goal this than developing shallow tools that are useful “out of the box” for smart non-specialists, or in trying to change the conversational norms of various relevant communities (though both of those are secondary goals.)
  2. I am highly skeptical of teaching many-to-most of the important skills for bridging deep disagreement, via anything other than ~one-on-one, in-person interaction.
  3. As such, I am publishing my drafts of Double Crux stuff here, over the next few weeks. But making writeups for the public internet, particularly of content that I don’t yet know how to teach via writing, is not a priority for me.

I have a standing offer to facilitate conversations and disagreements (Double Crux or not) for rationalists and EAs. Email me at eli [at] rationality [dot] org if that’s something you’re interested in.

On Double Crux tests and tournaments

Most of the tests I’ve heard people pitch for DC don’t seem very valuable to me, and I want to at least gesture at why.

Other folks seem to be thinking of Double Crux as a complete method, to be directly compared with other methods: “which one works better”. I think of Double Crux as one (very important) pattern in an ensemble for the overall goal of bridging disagreements. “Testing Double Crux”, as I often hear people talk about it, sounds to me a little like “testing bank shots” in basketball: it is clearly useful sometimes, it isn’t always the right thing to go for, and it depends heavily on personal skill.

I think that example overstates it somewhat: Double Crux is more of a broad framework for disagreement bridging than bankshots are for basketball. And that’s not to say that you can’t test bank shots: it’s plausible that there are superstitions about it, and it isn’t as effective as many practitioner’s belive. But the value of information seems lower to me (at least at this stage, where approximately no one has put in more than 20 hours in explicitly training disagreement bridging, compared to basketball, which has hundreds of highly skilled experts.)

I would be more excited in organizing a “disagreement resolution tournament”, where experts who have developed their art and trained to excellence, compete, rather than (for instance) a setup where we give 20 undergrads a 30 minute long double crux lecture with 30 minutes of practice, and compare them to a control group.

(That second things isn’t useless, but I care a lot less about developing shallow tools that are helpful for ~0-skilled folks, out of the box, than I do about deep experts who increase the range of problems (in this case, disagreements) that humanity  / the x-risk ecosystem can solve at all.)

The logistics of such a tournament seem hard to make work, because there’s not an obvious way to standardized disagreements to resolve, and in practice there are very few highly skilled experts of differing schools. So the value of information in 2019 still seems low. But it seems more promising than most of the tests I hear proposed.

Capability testing as a pseudo fire alarm

[epistemic status: a thought I had]

It seems like it would be useful to have very fine-grained measures of how smart / capable a general reasoner is, because this would allow an AGI project to car-fully avoid creating a system smart enough to pose an existential risk.

I’m imagining slowly feeding a system more training data (or, alternatively, iteratively training a system with slightly more compute), and regularly checking its capability. When the system reaches “chimpanzee level” (whatever that means), you stop training it (or giving it more compute resources).

This might even be a kind of fire-alarm. If you have a known predetermined battery of tests, then when some lab develops a system that scores “at the chimp level” at that battery, that might be a signal to everyone, that it’s time to pool our resources and figure out safety. (Of course, this event might alternatively precipitate a race, as everyone tries to get to human-level first.)

Probably the best way to do this would be for both training data, and compute / architecture. Start with a given architecture, then train it, slowly increasing the amount or quality of the training data, with regular tests (done on “spurs”, the agent should never have eposodic memory of the . When increasing training data plateaus, iteratively improve the architecture in some way, either by giving the systems it more compute resources, or maybe making small adjustment. Again train the new version of the system, with regular tests. If you ever start to get very steep improvement, slow down and run tests more frequently.

Naively, it seems like a setup like this would prevent an AI team from overshooting and making a system that is much more capable that they think (which gives rise to all kinds of problems, like treacherous turns), regardless of how close “chimp” is to “human” on some absolute intelligence scale.

Some considerations:

  • This of course, depends on having metrics that work. Of course, it seems very likely that early general AIs will be minds with a very unusual “shape”, such that it has sophisticated reasoning abilities along dimensions that are not natural categories to humans, while being weak on many of the dimensions we’re testing for.
    • Although, this seems a bit surprising, since presumably we would be training it to hit those metrics? Somehow it would get better, faster, at a kind of reasoning we were not intending to train, than at the kind of reasoning that we were optimizing for?
    • Also, can we just design metrics that get around this problem, by measuring exactly what we care about, i.e. danger? Something like a metric for consequentialist reasoning?
  • This idea depends on the system not deceiving you, or intentionally underperforming on your metrics.
    • Naively, this doesn’t seem like that much of an issue, if you are doing tests regularly enough. I’m presuming (perhaps wrongly), that a chimpanzee is not smart enough to infer that it is an agent in a simulation that is overseen by human agents, and that it should try to decide those humans. That is, we stop our progression before the agent is at the point where deception is a consideration.
      • Which means we really need to be confident about how smart something needs to be before it tries to deceive us?
        • Really? Its first attempts at tricking humans will be bad. Just as the attempts of animals to trick us will be pretty bad.
  • At least one “danger point”, is when the system is capable enough to realize the instrumental value of self improving by seizing more resources.
    • How smart is this?
      • My guess, is really smart. Animals come pre-loaded with all kinds of instincts that cause them to seek out food, water, etc. These AI systems would not have an instinct to seek more training data / computation. Most humans don’t reason their way into finding ways to improve their own reasoning. If there was a chimp, even loose in the internet (whatever that means), would it figure out to make itself smarter?
      • If the agent has experienced (and has memories of) rounds of getting smarter, as the humans give it more resources, and can identify that these improvements allow it to get more of what it wants, it might instrumentally reason that it should figure out how to get more compute / training data. But it seems easy to have a setup such that no system has episodic memories previous improvement rounds.
        • [Note: This makes a lot less sense for an agent of the active inference paradigm]
          • Could I salvage it somehow? Maybe by making some kind of principled distinction between learning in the sense of “getting better at reasoning” (procedural), and learning in the sense of “acquiring information about the environment” (episodic).