OctoML says it has ended its business relationship with Civitai days after an investigation by 404 Media revealed the text-to-image platform was being used to generate images that “could be categorized as child pornography.” Civitai previously relied on OctoML’s OctoAI for image generation. While OctoML initially indicated it would continue working with Civitai and introduced new measures to curb the creation of harmful images, 404 Media reported on Saturday that it has now decided to cut ties with the platform altogether.
According to 404 Media’s December 5 report, internal communications showed that OctoML was aware some Civitai users were creating sexually explicit material that included nonconsensual images of real people and pornographic depictions of children. In a followup report this weekend, the publication noted that OctoML rolled out a filter to block the generation of all NSFW content on Civitai before announcing its decision to pull out. Civitai also added new moderation methods in response to the investigation earlier this week, including a mandatory embedding called Civitai Safe Helper (Minor) that bars the model from generating images of children if “a mature theme or keyword is detected,” according to 404.
Andreessen Horowitz-backed Civitai has previously come under scrutiny for its “bounties” feature that challenges users to generate realistic images of real people for rewards. In November, 404 Media found it was being used to make nonconsensual deepfakes of celebrities, influencers and even a private citizen that were, in many cases, sexual in nature. The subjects were primarily women. According to 404 Media, Civitai has since added a filter to prevent the creation of NSFW content featuring “certain” celebrities.
Nevertheless, OctoML — which uses Amazon Web Services’ servers — no longer wants to be involved. In a statement to 404 Media on Saturday, OctoML said, “We have decided to terminate our business relationship with Civitai. This decision aligns with our commitment to ensuring the safe and responsible use of AI.”
This article originally appeared on Engadget at https://www.engadget.com/controversial-ai-image-platform-civitai-has-been-dropped-by-its-cloud-computing-provider-195530538.html?src=rss
Trending Products
![Cooler Master MasterBox Q300L Micro-ATX Tower with Magnetic Design Dust Filter, Transparent Acrylic Side Panel…](https://m.media-amazon.com/images/I/51WfytAtGCL._SS300_.jpg)
Cooler Master MasterBox Q300L Micro-ATX Tower with Magnetic Design Dust Filter, Transparent Acrylic Side Panel…
![ASUS TUF Gaming GT301 ZAKU II Edition ATX mid-Tower Compact case with Tempered Glass Side Panel, Honeycomb Front Panel…](https://m.media-amazon.com/images/I/41JUuW8Yc5S._SS300_.jpg)
ASUS TUF Gaming GT301 ZAKU II Edition ATX mid-Tower Compact case with Tempered Glass Side Panel, Honeycomb Front Panel…
![ASUS TUF Gaming GT501 Mid-Tower Computer Case for up to EATX Motherboards with USB 3.0 Front Panel Cases GT501/GRY/WITH…](https://m.media-amazon.com/images/I/41j9qzlOi2L._SS300_.jpg)
ASUS TUF Gaming GT501 Mid-Tower Computer Case for up to EATX Motherboards with USB 3.0 Front Panel Cases GT501/GRY/WITH…
![be quiet! Pure Base 500DX Black, Mid Tower ATX case, ARGB, 3 pre-installed Pure Wings 2, BGW37, tempered glass window](https://m.media-amazon.com/images/I/41xW6xrbicL._SS300_.jpg)
be quiet! Pure Base 500DX Black, Mid Tower ATX case, ARGB, 3 pre-installed Pure Wings 2, BGW37, tempered glass window
![ASUS ROG Strix Helios GX601 White Edition RGB Mid-Tower Computer Case for ATX/EATX Motherboards with tempered glass…](https://m.media-amazon.com/images/I/41T-2v3IuML._SS300_.jpg)
ASUS ROG Strix Helios GX601 White Edition RGB Mid-Tower Computer Case for ATX/EATX Motherboards with tempered glass…
![Bgears b-Voguish Gaming PC with Tempered Glass ATX Mid Tower, USB3.0, Support E-ATX, ATX, mATX, ITX. (Note: Fan NOT…](https://m.media-amazon.com/images/I/41p2u3NJN6L._SS300_.jpg)