Paul Christiano
3 min readNov 18, 2016

--

I am not yet convinced that “lack of philosophical understanding” is a really significant driver of risk; we have disagreed about this a few times before.

I agree that there are some risks associated with a lack of philosophical understanding, just as there are risks associated with a failure to coordinate. But to me those risks look quite small, by comparisons either to the risks posed by AI or to other coordination failures.

In a previous discussion you gave one example of such a risk — in order to remain competitive coalitions must adopt some agreeable collective decision-making procedure, and perhaps without philosophical understanding they would inadvertently adopt a procedure that reflected none of their values (e.g. one suggested by an AI system). I appreciated this example, but the expected damage caused by this particular philosophical shortcoming looks to be very small — the scenario where this creates irreversible damage seems relatively unlikely, and to be far off in time, the required levels of philosophical sophistication don’t seem high, and so on. While I can see an intuition that there are more such failures I don’t see the intuition that their total magnitude is very large.

There is another possible failure I’d regard as conceptually separate: people will lock in values / beliefs / philosophical decisions, even in cases where it offers no competitive advantage, because they cannot appreciate the value of further deliberation or are actively opposed to updating their views. I am also more skeptical of this doing damage — I don’t see many people who exhibit the necessary antipathy towards reflection, especially in such a coordinated way that they would actively avoid it for a very long period of time.

Even if we endorse the view that philosophical understanding is a key driver of risk, I don’t yet see how global coordination exacerbates the problem (though I agree that it wouldn’t fix the problem either).

On the “we have to make philosophical commitments to remain competitive” it seems clear that global coordination is helpful, for the typical reasons. So I assume that you are concerned about this second class of risk.

Previously the world’s resources were divided in some way, and some of those resources would be used in an unreflective way. It seems that even in the worst case coordination would result in something like: the probability of an unreflective global outcome is comparable to the fraction of resources that were previously going to be used unreflectively. But from the utilitarian standpoint, that doesn’t sound like a big deal.

Possible counterpoints might include: you might think that the majority of resources will be used unreflectively, and that coordination might give that majority a superlinear chance of determining the outcome. Or you might think that improved coordination would shift the balance of influence over whether or not reflection occurs (e.g. away from economic influence and towards political influence, or towards individuals with an authoritarian bent), and think that the new recipients are on balance less reflective. But overall I do not yet find any of these stories very compelling — I suspect that it is a small minority rather than a majority that would intentionally eschew reflection, which will receive sublinear influence in a political equilibrium, and I expect improved global coordination to in expectation decrease rather than increase the tendency for individuals with unsavory characteristics to obtain power.

(I’m sorry if this is badly misunderstanding your point. I did not make an extended effort to understand where you are coming from, it seemed easier to just have a round of back and forth to clarify.)

--

--