Inconvenient argument against powerful open-weight models

I love open-weight models and use them every day. My job involves post-training large and capable open-weight models, much of the most interesting science studies these models, and I am generally appreciative of the market dynamics created by multiple providers serving the same weights.

However, I find it difficult to convince myself that, in the limit, powerful open-weight models will be good for the world.

The argument:

1. There are some tasks that we do not want language models to do, as they are obvious legal and/or ethical violations.
   1. Make bioweapon
   2. Hack website
   3. Plan murder
   4. Commit fraud
2. Language models are very powerful and will continue to get more powerful, including on tasks that we do not want them to be good at.
3. Open-weight models are uniquely difficult to prevent from enabling harmful tasks.
   1. It is impossible to take back weights once they are released, giving bad actors an infinite amount of time to design attacks against even very well-aligned models.
   2. It is impossible to monitor what people do with compute they own.
4. Therefore, any regulation will not be able to reasonably prevent bad actors from doing what they want, given that they have access to an open model.

Given that open models cannot be prevented from taking harmful actions at a user’s will, they add significant risk beyond powerful closed models that can be monitored and regulated by their providers.