OpenAI’s GPT-2: Results, Hype, and Controversies

The nonprofit AI research company, OpenAI, recently released a new language model, called GPT-2, which is capable of generating realistic texts in a wide range of styles. In fact, the company stated that the model is so good at automatic text generation that it can be used for nefarious purposes; therefore, it did not publicize the trained model.

The dangerous-to-release model was simply trained to predict the next word based on a large-scale of 40 gigabytes of text data from the Internet. In fact, the success of the GPT-2 comes from two key factors: (1) lots and lots of data; (2) huge compute power. However, not every learning technique scales well when more compute is added. Thus, making sure that this learning algorithm scales well, is a great contribution by its own.

Are you interested in receiving more in-depth technical education about language models and NLP applications? Subscribe below to receive relevant updates.  

What’s more, the model was capable of performing the natural language processing (NLP) tasks without explicit supervision; it ploughed the Internet and learned the intricacies of the human language by itself.

For example, here is a screenshot of the results generated when the model was fed with a few words of text and asked to complete the rest of the article on its own.

Sample generated by GPT-2 (Source)

As you can see on the screenshot above, GPT-2 can generate coherent continuations about a given topic, and the text feels close to human quality.

Although the GPT-2 model is a text generator at its core, it also showed incredible capabilities at completing several language modeling tasks, such as reading comprehension, summarizing texts, answering questions, and translation.

Here is a screenshot of the model’s results at performing various tasks:

The graph above reveals that the model is not yet perfect and still needs more improvements to achieve human-level standards. Actually, the results are cherry-picked in the sense that you need to take a few tries before a good result is realized. Anyway, we can say that language modeling has reached the moment when an algorithm is capable of generating content that’s (almost) indistinguishable from human-generated texts. But what’s more, AI text generators are much more effective than human writers – the algorithms can write articles on a chosen topic all day long and much quicker than human-beings.

This monumental achievement could be used for both beneficial and malicious ways. Thus, the OpenAI decided to keep the full source code and trained model undisclosed. There was a huge discussion in the AI community regarding this decision but it’s anyway a great benefit of this case that potential misuse of AI text generators is acknowledged and widely discussed.

Of course, AI-based text generators can have many useful applications. For example, in our premium research summaries, we show how text generators can be leveraged in marketing and advertising.

If you want to learn more details about the GPT-2 model as well as the controversies that surround it, check out the video prepared by Two Minute Papers.

See also how Jesse Vig explains GPT-2 through visualizations.

We are also featuring the most recent AI approaches to text generation with a particular focus on marketing and advertising use cases in our premium research summaries for intermediate and advanced practitioners.

Enjoy this article? Sign up for more AI and NLP updates.

We’ll let you know when we release more in-depth technical education.