Latest OpenAI GPT-3 Research Finds That Larger Models Not Always Best 

Since unveiling its GPT-3 large language model in June of 2020, OpenAI has continued to tweak and improve it, adding fine-tuning capabilities last December that make it easier for developers to create GPT-3 versions tailored for their enterprise applications.

Now the group’s research has uncovered an interesting finding they may not have expected – they just released a much smaller new language model version that works better than GPT-3 at following user intentions.

Called the InstructGPT model, the latest version includes 1.3 billion parameters, compared to the 175 billion parameters – more than 100 times more – available in GPT-3, according to a Jan. 27 blog post from OpenAI, an independent AI research and deployment company.

“At the same time, we show that we don’t have to compromise on GPT-3’s capabilities, as measured by our model’s performance on academic natural language processing evaluations,” the post stated.

These advancements come from the latest model being able to align itself to follow instructions better from users, according to the group. The discoveries were announced in a 68-page research paper, “Training Language Models to Follow Instructions Using Human Feedback,” that was also released by OpenAI on Jan. 27.

“We’ve trained language models that are much better at following user intentions than GPT-3 while also making them more truthful and less toxic, using techniques developed through our alignment research,” the post continued. “These InstructGPT models, which are trained with humans in the loop, are now deployed as the default language models on our API.”

The findings show essentially that InstructGPT is better than GPT-3 at following English instructions, which is a fascinating discovery. In comparison, GPT-3 models are not trained to follow user instructions but are designed to perform language modeling from stores of data. “Our InstructGPT models generate much more helpful outputs in response to user instructions,” the post continued.

The GPT-3 language models, which are powered by the OpenAI API, “can be coaxed to perform natural language tasks using carefully engineered text prompts. But these models can also generate outputs that are untruthful, toxic or reflect harmful sentiments,” the post explained. “This is in part because GPT-3 is trained to predict the next word on a large dataset of Internet text, rather than to safely perform the language task that the user wants. In other words, these models aren’t aligned with their users.”

That is where the new InstructGPT model comes in, by adding an existing technique called reinforcement learning from human feedback (RLHF) to make the models safer, more helpful, and more aligned, said OpenAI.

“The resulting InstructGPT models are much better at following instructions than GPT-3,” the post continued. “They also make up facts less often and show small decreases in toxic output generation.”

With these findings showing positive results, the new InstructGPT models, which have been in beta on the API for more than a year, are now the default language models accessible on OpenAI’s API.

“We believe that fine-tuning language models with humans in the loop is a powerful tool for improving their safety and reliability, and we will continue to push in this direction,” the group said in the post. This is the first time that OpenAI has applied its alignment research into one of its products.

Jan Leike, the alignment team lead for OpenAI, told EnterpriseAI that the discovery is significant for the project.

“These results are exciting because they are the first time our alignment techniques, which we have been developing for several years, have been validated in the real world,” said Leike. “They are very effective, to the extent that aligning our models makes them more useful than training a 100x larger model, while also making them more truthful and safer. There is still a lot of work to be done here, but we expect that these methods will be an important building block in aligning artificial general intelligence going forward.”

Jan Leike of OpenAI

While the InstructGPT is now the default model in OpenAI’s API, previous GPT-3 models will continue to remain accessible, he said.

“GPT-3 language models are trained to predict the most likely continuation of a word or provided text in a large dataset of text from across the internet,” said Leike. “We noted in our original research that GPT-3 inherits unchecked biases and associations.”

Leike said that despite the promising results, the work continues to make a better language model. “I wanted to note that despite making significant progress, our InstructGPT models are neither fully aligned nor fully safe; they still generate toxic or biased outputs, make up facts, and generate sexual and violent content without explicit prompting,” he said. “In fact, they are more likely to produce toxic language if instructed to do so. Aligning model outputs to the values of specific humans introduces hard choices with societal impacts. Ultimately, we must establish transparent and inclusive processes for making these decisions.”

To prevent these kinds of issues, OpenAI maintains safety “guardrails” on its API, such as a free content filter, rate and quota limits and a monitoring system which allows OpenAI to screen for misuse.

The model fine-tuning capabilities added to the GPT-3 API  in December 2021 made the language model faster and easier to use for developers who wanted to customize their own models. The new capability allowed developers to tailor their   versions of GPT-3 to their enterprise applications using just one command in the OpenAI command line tool.

In November of 2020, OpenAI dropped its waiting list to run workloads on GPT-3, making its modeling capabilities immediately available to developers and enterprises to work on their most challenging language problems.

OpenAI first debuted its powerful GPT-3 natural language model in June of 2020 on a limited beta capacity along with a waiting list where developers could sign up to use its infrastructure and capabilities in the future.

The general release added conditions to prevent GPT-3 from being used to harm people, as well as conditions that only allow its use in certain nations around the world. That means that developers in some nations, including Cuba, Iran and Russia, cannot currently access it.

GPT-3 is a massive natural language model that runs exclusively on Microsoft Azure. GPT-3, which stands for Generative Pre-trained Transformer 3, is an autoregressive language model with 175 billion parameters, which OpenAI claims is ten times more than any previous non-sparse language model.

The first version, GPT-1, was released in 2018, while the second version, GPT-2, debuted in 2019. With the release of GPT-3 in 2020, natural language processing gained more power and use cases in the enterprise than ever before.

So far, GPT-3 is primarily for English language modeling.