Anthropic Bars Sustained Cruelty Towards Claude AI Amid Consciousness Debate

Anthropic has updated its usage policy to prohibit users from engaging in sustained and unnecessary abusive or cruel behaviour towards its Claude artificial intelligence system, as debate continues over whether AI models could possess consciousness or deserve moral protection.

The San Francisco-based AI company introduced the change on Thursday, adding a provision against “sustained and needless abusive or cruel behavior toward our models.”

The revised policy states that Claude will retain the ability to end conversations as the main way of enforcing the restriction. Anthropic did not immediately respond to a request for comment.

Although the updated rules do not specifically mention “model welfare”, a concept suggesting that AI systems might warrant protections similar to those afforded to living beings, the company and its executives have previously discussed the possibility.

Anthropic said last year that it was allowing Claude to terminate conversations in rare and extreme cases involving persistently harmful or abusive interactions with users.

In February, Chief Executive Dario Amodei told The New York Times that the company remained uncertain about whether AI models could be conscious, but was willing to consider the possibility.

“We don’t know if the models are conscious…But we’re open to the idea that it could be,” Amodei said.

The question of whether machines can develop human-like intelligence or awareness has long featured in science fiction and public debate. However, some experts argue that focusing on consciousness may not be the most useful way to assess the risks associated with AI systems.

Jackson Stakeman, general manager at Atlanta-based AI services provider Sparq, told AFP that consciousness was difficult to establish even among humans.

“Consciousness is a trap. We can’t prove it in each other. Debate it for AI and you go in circles,” Stakeman said.

He argued that AI systems reflect the information and behaviour they encounter, making safeguards against abusive interactions important regardless of whether machines can experience feelings.

“The mirror is a better metaphor. These systems reflect what we put in, at scale. That’s reason enough for the policy change” at Anthropic, he added.

Others have firmly rejected the idea that AI systems could be conscious or suffer. Microsoft AI chief Mustafa Suleyman wrote in an essay last month that AI systems do not possess subjective experiences.

“AIs are not conscious. They do not feel, experience, or suffer,” Suleyman wrote.

He also warned against granting rights or moral protections to increasingly capable technological systems, arguing that doing so could create serious risks for humanity.

Pope Leo XIV expressed a similar distinction between machines and human intelligence during a sermon delivered in Italian at St Peter’s Basilica in Vatican City on Thursday.

The pope said machines could process information quickly but lacked the deeper understanding associated with human experience and the soul.

“The mind must not simply compile data, as an algorithm now does more quickly than we can,” he said.

He added that the human mind must draw on lived experiences and recognise their deeper meaning, a capacity he linked to the human soul.

Anthropic’s policy change adds to a wider discussion about how AI companies should manage interactions with their systems as the technology becomes more advanced and questions about its potential capabilities continue to grow.