Scott explores what constitutes a deontological bar (hard moral rule) by examining when consequentialist reasoning should be constrained, using debates within AI safety about working with AI companies versus pursuing regulation as his main examples.
Longer summary
Scott examines the concept of deontological bars - hard moral rules that shouldn't be broken even for good consequences - and tries to develop a framework for determining what counts as such a rule. He starts with the classic example of not assassinating leaders, then explores various formulations like 'act as if your maxim would become a general law' and 'don't defect from functioning norms,' testing them against cases like military disarmament and spreading misinformation. The post is motivated by debates in AI safety between those working with AI companies and those pursuing pause/ban regulations, with each side suspecting the other might be violating deontological bars. Scott proposes that the rule might be 'don't do something which would be bad if universalized, unless the norm is non-functioning in such a way that you'd be playing cooperate while your enemy plays defect,' though he acknowledges this requires interpretive work and common sense to apply.
Shorter summary