AI regulations are Good actually

Broadly Anthropic’s policy seems to be something like – AI in it’s current form is insufficiently aligned and we don’t have a good enough answer to the alignment problem to proceed safely. Unfortunately we are locked in an arms race with China and if we slow down and China doesn’t then the superhuman AI we’re worried about will come into existence anyways but will instead be representative of the interests of the Chinese government which is a marginally worse world to be in than if such a superhuman AI comes into existence in one of the US companies (which would still be pretty bad).

Usually people seem to end at the thought of ‘They are stuck in an arms race’ as proof that these AI companies are irredeemable and evil and probably wrong. The problem with that is that’s exactly how each nation should behave if they genuinely believe that they are about to build something truly dangerous. Absent some sort of overseeing entity a mad dash to the finish is exactly the ‘correct’ move for each individual player.

Now there are some possible solution to the arms race problem, namely transparency into the exact internals of what is going on in each of these AI labs including the specific details of their top of the line internal models along with constraining the ways compute flows into each of these countries. Thankfully we already have a historical playbook of how such oversight might be implemented by looking at how nuclear arms treaties work. But, crucially, all of these options require an understanding that the technology that is being built is indeed dangerous, on par with if not more dangerous than nukes.

A guy named Jacob Coxon resigned from Anthropic a couple of days ago, the primary reason he cites is that he doesn’t believe that a mad dash to the finish line will lead to the best possible world. Now the reason a resignation like this is effective is that he’s explicitly abdicating financial stake in the company by doing this. That removes any incentive he personally might have to con investors for a big Anthropic IPO or whatever. The simplest explanation of why he might do something like this is that he actually does believe it. That this is in fact the honest to good internal truth that people at these big AI labs subscribe to.

I guess the broader point I want to drive home here is that this is not a PR stunt. These people who are supposed to be at the top of their field actually believe that this technology might end the world. They are explicitly working against their financial best interest to tell people about it. Believing that this is some sort of grand conspiracy to trick the public into throwing more money at them is exactly the wrong stance because if the technology isn’t dangerous then there is no reason to lobby towards regulation.