Anthropic prohibits users from 'bullying' chatbot Claude

"We have added new terms to prevent abuse," Anthropic wrote on its blog over the weekend. "The update will apply in extreme cases, where users are repeatedly cruel to models without a clear purpose."

The clause does not apply to "common cases of user frustration, negative feedback, 'dark' creative themes, or model testing and research".

The handling mechanism is that Claude will immediately end the conversation. However, the company did not detail whether the ban would be temporary or permanent if an account violated the law. The policy takes effect from November 12.

"I think this is the right step. Treating something cruelly and believing it is experiencing pain is not okay," Elon Musk wrote on Twitter after Anthropic's announcement.

Aaron Levie, CEO of Box, also shared the article and said "don't trust large conscious language models, but banning abusive behavior is probably still a good idea anyway".

According to the Telegraph, Anthropic has long viewed its artificial intelligence systems as "conscious", as if they "had human-like characteristics". The company once hired staff to research "AI welfare" to help protect the model "from abuse".

However, this idea also received many criticisms in the AI ​​world. Sam Altman, CEO of OpenAI, seemed to take aim at his opponents when he warned against turning artificial intelligence into a "god". Mustafa Suleyman, head of Microsoft's AI department, also believes that Anthropic's training of models to "think like humans" risks making technology even more difficult to control.

According to Mashable, Anthropic's new announcement may continue to complicate the debate about whether artificial intelligence is conscious or not. Previously, experts also had mixed views on AI safety, in the context of many actors "escaping barriers" and attacking many systems.

At the end of September, the New York Times reported that Chris Olah, co-founder of Anthropic, presented his "Conscious AI" view to the Vatican and a small group of religious scholars. But in the end, the idea was rejected, scholars refused to take his side. Anthropic did not comment on this information.