Keeping Inventions Safe with AIRS

Rec Room Official Announcements & News
1

We have millions of players making millions of rooms, inventions, and avatar items every week. Your creations make Rec Room a hub of creativity and fun. Our job is to make sure all of you get to enjoy all of those creations without the risk of coming across content or behavior that could negatively impact your experience. Online, that can sometimes be a tough job. So, we’re always looking for ways to make your time in Rec Room more positive.

We have made good progress on our player safety systems - our voice moderation system is 99.999999999% accurate, and we have increasingly sophisticated child safety systems in place. Over the last few months, we’ve been turning more of our attention to all of the content made by our players. We’ve provided better tools for creators to moderate their rooms, improved our avatar item moderation and appeal system, and are taking action on more rooms and repeat offenders than ever before.

Now we’ve turned our attention to inventions. Each day, tens of thousands of inventions are created. Over 96% of those inventions are good inventions, but even a small fraction of bad inventions means potentially thousands of opportunities to find content that shouldn’t be in Rec Room. Finding these manually is like searching for a needle in a haystack.

So, we built something to make that easier. Today, we’re sharing our new moderation system: Automated Invention Review System (AIRS). AIRS is specifically designed to ensure inventions meet our Creator Code of Conduct and help us keep pace with all of your creative output.

How AIRS Works

AIRS is a complex system that involves scanning inventions and taking automated action when needed. We’ve been testing AIRS for the last few weeks, and we’ve found that it’s very good at catching sexual and discriminatory content.

When an invention is published or reported, we immediately start the scanning process. We evaluate not only the name and description of the invention, but also the images the invention contains, such as printed photos and canvas art. This is all tuned to content that breaks our Creator Code of Conduct.

What Happens When a Violation is Found

When AIRS identifies that an invention contains inappropriate content, it triggers a series of automatic enforcement actions designed to prevent its spread:

  1. Invention Ban
    The violating invention is immediately banned. This also places a two-week restriction on the creator’s ability to create new inventions.

  2. Lineage Scan
    We scan all parent and child inventions connected to the banned invention. We also scan other inventions created by the same player. This ensures that related content doesn’t slip through the cracks and means we prioritize reviewing content from creators who are known to have broken the Creator Code of Conduct.

  3. Room Content Cleanup
    Any room saves that include the banned invention are automatically removed. If the invention is part of the room’s latest or current save, the room is closed to prevent exposure.

AIRSFlowChart
AIRSFlowChart1115×339 19 KB

What This Means for You

AIRS will start taking automatic action against violating inventions and rooms that contain them at the end of next week. Why are we telling you this now? Well, if you have an invention in your room that violates our Creator Code of Conduct, please remove it before then to avoid losing your content.

We will be reviewing how the system works and listening to your feedback. We’ll come back with any updates or changes we need to make.

So, let us know what you think. And keep reporting when you find inappropriate content - our systems are getting better and better every day. Together, we can make Rec Room an even safer place to create. [|=)]

April 28th Update

We further tested AIRS today by letting it run for multiple hours. What were the results of the test? The system ran flawlessly and took the right action against harmful inventions. Our moderators checked through all the content the system flagged during this multi-hour test, and there were no false positives. Again, we designed this system to detect the very worst content, and it’s doing just that.

We will re-enable the system tomorrow, April 29th, and plan to keep it on. We appreciate all the feedback and questions you’ve shared–we will keep monitoring. This system, alongside our hard-working moderators, helps keep Rec Room a safe place to create.

8 Likes
2

Rec Room Is Insane They’re Really Cooked

3 Likes
3

This is worrying.

I don’t have any content that violates the code of conduct, but I also don’t believe the other forms of moderation are foolproof.

I myself have been victim of the voice moderation system without breaking COC. Once when it was first being trialled in Rec Con, and another this year. First time I was reading a message I was sent out loud, second time I was reading the names of items from the shop, and poof, to an empty dorm, having to acknowledge a warning.

Yet kids in public lobbies can shout all kinds of profanity without a problem. Definately not 99% accurate…

…So with this new system, what happens if one of my inventions, which doesn’t break the COC, is caught up on it? How quickly can I appeal, by sending a ticket probably, and how soon will staff manually rectify any mistakes? How negatively will it affect my account and standing, if I am indeed falsely caught up within this?

8 Likes
4

This is the best news! Now I won’t see the 18+ content that was released 10 seconds later in the invention catalogs. :grin:

I hope some of my inventions with music will pass the test of this moderation.

5

Sophisticated child safety systems??? I mean half the people in the rec center are under the age of 10!

3 Likes
6

If those music inventions are being sold and are not music made by you, that is theft. A shocking amount of people get away with that, if you did that on any other platform and it gained great popularity and you made money off of it, it could even go so far as to earn you a lawsuit.

1 Like
7

It is not immediate. Those players tend to suffer their bans sometime after. That’s also where alot of the “i did nothing wrong and got banned” videos come from where a player is “randomly” sent to their dorm while doing nothing. They may have been doing nothing then, at the time of recording, but earlier they easily may have been screaming slurs and other profanities.

2 Likes
8

Like I said thx you Rec Room for keeping everyone safe

3 Likes
9

I hate paid inventions with music, I follow the rules and create such inventions for free.

10

How is this bad?! They are fighting against inappropriate inventions.

3 Likes
11

I have high hopes for the lineage and creator’s other inventions checks for this!! Lots of players get away with spreading a LOT of inappropriate or otherwise harmful stuff and by the time enough people manage to report it, the damage is already done and the invention has spread far and wide. Hopefully this thing is accurate! I see alot of people already unhappy with this on Reddit, but the only real alternatives are you guys spend crazy amounts on hiring people moderators to constantly scan through all this stuff which would be no easy or efficient task, or they give some kind of invention rating system that we won’t actually engage with much and that—if it had the power to get inventions taken down—would absolutely be abused.

3 Likes
12

Great questions and perfectly valid concerns. I want to reassure most creators that this will not affect them. Most of our creators do not create content that violates our Creator Code of Conduct.

When this first goes live, we will keep a close eye on what’s being actioned and make adjustments as needed. The system is only taking automatic action on content that exceeds a high severity score that we’ve set. It’s targeting the worst of the worst content.

We also wanted to design the system to be as least disruptive as possible while still removing the harmful content. So, rather than closing all rooms that used a flagged invention in past saves, we’re only removing access to those specific saves.

We’ll send creators a notice including the name of the invention that was closed if they need to appeal.

2 Likes
13

That seems like a backwards system. If the voice moderation system takes that long to act, someone who is getting verbally abused will feel like no moderation action was taken against their abuser. And also as you say, creates a lot of confusion when someone witnesses the moderation action take effect, with seemingly no recent inciting incident. You know it’s because the moderation takes time, and I know it’s because the moderation takes time, but what about all of the players who don’t interact with the forums?

2 Likes
14

There’s not alot that can be done about that, i’m sure if Rec Room could safely and reasonably make it immediate, they would! But i assume for some reason they just can’t. I dunno if they get a real person to look at the flagged language, or if it’s just a really high volume of flagged moments slowing it down, no idea, but i’m sure it isn’t just for no reason. Also if those kids aren’t smart enough to realise that there is some correlation between their being a horrible slurring swearing person, and them getting banned sometime later/often, they are probably too young to be playing Rec Room.

15

To clarify, we are over 99% confident when we issue a ban or mute for voice toxicity violations. We only ban or mute players if they’ve had multiple violations. If one thing gets flagged, we simply issue a warning that does not affect your account as a light-touch intervention (which seems to be what happened in your case).

16

Thank you for clarifying.

With the automated moderation, I wish it was clear what exactly was flagged up, for example the exact word or phrase, as to avoid triggering it again.

17

This is cool. Rec Room does have an issue with inventions breaking the CoC. Placing this system into the game is cool, but with all things automated, it can make mistakes. This is a step in the right direction I feel however. It’s great to hear that they are taking action against this conduct, but what if the system gets something wrong? What then?

18

The system will only take action against inventions that have high severity scores. Our moderators have manually reviewed inventions that score high, and we’ve found a lot of bad stuff. We have high confidence in it. Of course, we’ll keep a close eye on what it’s catching and make adjustments as needed–we’ve already done a lot of tuning during our testing, though.

19

Does this mean that the save will be deleted even if the room is unpublished?

20

As @aliceinthapalace said, I am also a victim of the voice moderation system. More than 70% of the time, the moderating system takes what I said without any context from my previous voice dialogue, but I see kids being able to yell the r slur, showing that rec room doesn’t do well with AI moderating systems. However, since this AIRS system immediately bans, I don’t see how it’s 100% foolproof. The AI that processes the AIRS system could be closely related to the voice moderation system, so this seems pretty bad as a creator on Rec Room (even though I publish inventions occasionally)

1 Like