OpenAI has introduced a groundbreaking new model, Critic GPT, which is causing a stir in the AI community. Critic GPT is designed to critique other AI models, particularly focusing on errors in code generated by ChatGPT. The need for such a tool arises from the growing sophistication and complexity of AI systems like ChatGPT, which makes it increasingly difficult for human reviewers to spot mistakes.
(Credit: Open AI’s official site)
According to Open AI researchers: “We’ve trained a model, based on GPT-4, called Critic GPT to catch errors in ChatGPT’s code output. We found that when people get help from Critic GPT to review ChatGPT code they outperform those without help 60% of the time”.
How it improves AI Code Generation?
ChatGPT, powered by the GPT-4 series of models, continuously improves through reinforcement learning from human feedback (RLHF). This process involves human trainers reviewing ChatGPT’s responses and providing feedback for refinement. However, as these models become more advanced, identifying errors becomes more challenging. Critic GPT addresses this by acting as a second layer of review, catching errors that human reviewers might miss.
Based on the GPT-4 architecture, Critic GPT was trained similarly to ChatGPT but with a unique approach. OpenAI researchers manually inserted errors into code generated by ChatGPT and provided feedback on these mistakes. This training helped Critic GPT learn to identify and critique errors more accurately.
In tests, Critic GPT’s critiques were preferred over ChatGPT’s in 63% of cases when dealing with naturally occurring bugs.
Removing Bugs
The dual approach of evaluating human-inserted and naturally occurring bugs provides a comprehensive understanding of Critic GPT’s performance. Moreover, agreement among annotators was significantly higher when they had a reference bug description, emphasizing the importance of clear evaluation contexts.
Critic GPT not only reduces unhelpful complaints, often referred to as nitpicks, but also minimizes the occurrence of hallucinated problems—errors that don’t exist.
The reason why ChatGpt and other Models will be soon Unbeatable
In practice, Critic GPT enhances the quality of critiques, helping human reviewers write more comprehensive and accurate feedback. This synergy between human expertise and AI assistance is crucial, as Critic GPT helps humans write better critiques while producing fewer hallucinated bugs. The ultimate goal is to integrate Critic GPT into the RLHF labelling pipeline, providing AI trainers with explicit AI assistance, and thereby improving the accuracy and reliability of the data used to train AI models.
Sampling Beam Search (FSBS) Method
OpenAI also implemented a method called Forced Sampling Beam Search (FSBS) to balance the trade-off between finding real problems and avoiding hallucinations. FSBS allows Critic GPT to generate longer and more comprehensive critiques by using an additional test-time search against the critique reward model, ensuring thorough yet precise feedback.
In addition to coding tasks, Critic GPT has shown promise in critiquing general assistant tasks. It successfully identified issues in tasks initially rated as flawless by a human reviewer, which were later found to have substantial problems.
Still, it needs Human
While Critic GPT significantly enhances human capabilities, it cannot completely replace human expertise. Some tasks and responses are so complex that even experts with AI assistance may struggle to evaluate them correctly. However, by combining human and AI efforts, greater accuracy and performance can be achieved.
