The RBot Helps People!
Experiments come with testing and validation. The Rbot is no different. Our first user-tests showed that the RBot helps people to resolve their legal problems on Reddit: a palpable success!
We started our tests by checking how a response generated by ChatGPT would be handled by Reddit moderators. The moderators immediately identified that the response was generated by ChatGPT, flagged the response and removed it. When we asked for more information on why the post was removed, the moderators rightly explained that while ChatGPT sometimes provides accurate information in response, it also often provides incorrect information, and sometimes it even makes up information (something known as “hallucination”).
We wanted to test if RBot provided better information than ChatGPT, and if the RBot’s responses would be accepted by the moderators. We thought the RBot would be successful on both fronts for two reasons:
- RBot is carefully designed to use only accurate, well-curated content from community legal support websites in British Columbia (we describe this as the RBot using “AI with guardrails”); and
- RBot uses AI to provide answers which sound sympathetic and natural.
We manually inputted six (6) redditor questions to Rbot and posted the answers back to Reddit. All six questions were about housing issues, since (so far) that is what the RBot is trained to answer. We breathlessly waited to see what would happen.
All responses were successfully posted to the subreddit. No responses were flagged as an AI answer or removed. Of the six tests responses, the responses to Test Question 3 and Test Question 4 received an upvote, and the response to Test Question 3 received a positive response from the author themselves. Take a look:
Test Question 3: Question on letting family members stay in a house while it’s going through probate.
RBot Response:
Test Question 4: Landlord evicted me so ex husband can move into home they own together. Help!
RBot Response:
Why This Matters
Our goal with the RBot is to help people resolve their legal problems by giving them accurate information that answers the questions they are asking, and by directing them to community legal support in their province. While the sample size is small and the Reddit moderator’s actions may be incidental, we are seeing that the Rbot helps people. We hypothesize that by providing empathetic, accurate answers that contain phone numbers and links to websites, there is a possibility that we can integrate AI into the subreddit, and give people the help they are asking for.