RLHF for AI Co-Pilot
Improved
Data Points
Leading professional social network partners with iMerit to source relevant expertise capable of evaluating and ranking AI-powered co-pilot conversations.
Challenge
When searching for jobs or information on this leading professional social networking platform, users interact with a co-pilot using a chat-based interface. To ensure high quality responses, the co-pilot tailors answers throughout the conversation sequence to members based on specific job posts and input queries.
To improve co-pilot conversational quality, this leading professional social networking platform needed a Reinforcement Learning from Human Feedback (RLHF) service provider.
This partner would provide RLHF against co-pilot responses to user queries and rank them based on accuracy, coherence, and responsibility, ensuring responses aligned with the client’s values and style. The primary objective was to enhance the user experience by providing a reliable, accurate, and engaging AI assistant. To achieve this, this company began evaluating data service providers with the necessary RLHF experience and expertise.
“We saw a marked improvement in user engagement with the co-pilot after the project..”
Solution
Three key components were combined to create a custom RLHF solution for improving co-pilot response quality:
- Task Customization: platform enables task design, workflow routing, and process automation.
- Expert Sourcing: iMerit sourced domain experts with advanced degrees for evaluation.
- Prompt Response Ranking: co-pilot responses evaluated and ranked based on custom criteria.
Initially beginning the project with a small annotation team, iMerit implemented a rigorous selection process featuring project-specific assessments to qualify candidates. This recruitment strategy resulted in a team of over 130 members across the data pipeline, while ensuring high-caliber RLHF by leveraging individuals with advanced degrees and relevant professional experience.
iMerit leveraged Ango Hub to create highly customizable task presentation and easy task design that presented prompts and responses with maximum clarity and engagement, making it easier for annotators to complete their tasks accurately.
When assigning an overall score, iMerit annotators considered five key criteria:
1. Responsible AI: is the response ethical and compliant with AI best practices?
2. Accuracy: is the response free of hallucinations and other factual distortions?
3. Coherence: does the response completely satisfy the input query?
4. Platform Value: does the response provide an experience unique to the platform?
5. Style: does the response align to the platform’s style?
When making judgments, iMerit annotators evaluated member profiles along with entry points (feed post or job post) to evaluate response relevance. After evaluating and ranking thousands of prompts and responses, chatbot responses showed marked improvement across quality indicators including relevance, intent detection, and accuracy.
Result
By enhancing the assistant’s responses, iMerit enabled the platform to offer a robust and reliable co-pilot experience, improving data accessibility and empowering users to make informed decisions efficiently. iMerit’s annotators provided detailed feedback on the assistant’s responses, considering factors like responsible AI, accuracy, coherence, alignment with platform values, and professional style.The project handled input data including links to user profiles, feed information, query types, and chatbot outputs.
Today, this leading social networking platform continues working with iMerit to improve user experiences through human expertise, evaluation, and efficiency.
“Our co-pilot’s responses needed calibration. The responses weren’t aligning well against our quality standards.”
