TOPBOTS Guide To NeurIPS 2019

This year’s Annual Conference on Neural Information Processing (NeurIPS 2019) will be held in Vancouver, Canada from December 8th to 14th, 2019. The popularity of this conference is growing rapidly, with the number of submitted papers getting to the record-breaking number of 6743, almost 40% more than last year.

The TOPBOTS team is also going to NeurIPS again this year. To make your own experience easier, we’ve put together this guide sharing with you major trends across AI research areas as well as the research papers, talks, tutorials, and workshops we are most excited about.

If you’d like to skip around, here is the table of contents:

1. Major AI & Machine Learning Trends
2. Inspiring Invited Talks
3. Highlighted Research Papers From NeurIPS 2019
4. Tutorials Worth Your Attention
5. Interesting Workshops
6. Further Reading

Major AI & Machine Learning Trends

We want to start by discussing the major trends we observed in different AI research areas over the last year. In addition, we want to suggest the key research papers you should read to capture these trends. These are the papers we recommend in addition to the NeurIPS 2019 papers we are excited about.

Natural Language Processing (NLP)

Trending topics:

• The new NLP paradigm is “pretraining + finetuning”. Transfer learning has dominated NLP research over the last two years. ULMFiT, CoVe, ELMo, OpenAI GPT, BERT, OpenAI GPT-2, XLNet, RoBERTa, ALBERT – this is a non-exhaustive list of important pretrained language models introduced recently. Even though transfer learning has definitely pushed NLP to the next level, it is often criticized for requiring huge computational costs and big annotated datasets.
• Linguistics and knowledge are likely to advance the performance of NLP models.The experts believe that linguistics can boost deep learning by improving the interpretability of the data-driven approach. Leveraging the context and human knowledge can further improve the performance of NLP systems.
• Neural machine translation is demonstrating visible progress. Simultaneous machine translation is already performing at the level where it can be applied in the real world. The recent research breakthroughs seek to further improve the quality of translation by optimizing neural network architectures, leveraging visual context, and introducing novel approaches to unsupervised and semi-supervised machine translation.

Breakthrough research papers to read:

• Language Models Are Unsupervised Multitask Learners (OpenAI GPT-2)
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Conversational AI

Trending topics:

• Dialog systems are improving at tracking long-term aspects of a conversation. The goal of many research papers presented over the last year was to improve the system’s ability to understand complex relationships introduced during the conversation by better leveraging the conversation history and context.
• Many research teams are addressing the diversity of machine-generated responses.Currently, real-world chatbots mostly generate boring and repetitive responses. Last year, several good research papers were introduced aiming at generating diverse and yet relevant responses.
• Emotion recognition is seen as an important feature for open-domain chatbots. Therefore, researchers are investigating the best ways to incorporate empathy into dialog systems. The achievements in this research area are still modest but considerable progress in emotion recognition can significantly boost the performance and popularity of social bots and also increase the use of chatbots in psychotherapy.

Breakthrough research papers to read:

• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study
• Jointly Optimizing Diversity and Relevance in Neural Response Generation

Computer Vision (CV)

Trending topics:

• 3D is currently one of the leading research areas in CV. This year, we saw several interesting research papers aiming at reconstructing our 3D world from its 2D projections. The Google Research team introduced a novel approach to generating depth maps of entire natural scenes. The Facebook AI team suggested an interesting solution for 3D object detection in point clouds.
• The popularity of unsupervised learning methods is growing. For example, a research team from Stanford University introduced a promising Local Aggregation approach to object detection and recognition with unsupervised learning. In another great paper, nominated for the ICCV 2019 Best Paper Award, unsupervised learning was used to compute correspondences across 3D shapes.
• Computer vision research is being successfully combined with NLP. The latest research advances enable robust change captioning between two images in natural language, vision-language navigation in 3D environments, and learning hierarchical vision-language representation for better image caption retrieval and visual grounding.

Breakthrough research papers to read:

• A Theory of Fermat Paths for Non-Line-of-Sight Shape Reconstruction
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Reinforcement Learning (RL)

Trending topics:

• Multi-agent reinforcement learning (MARL) is rapidly advancing. The OpenAI team has recently demonstrated how the agents in a simulated hide-and-seek environment were able to build strategies that researchers did not know their environment supported. Another great paper received an Honorable Mention at ICML 2019 for investigating how multiple agents influence each other if provided with the corresponding motivation.
• Off-policy evaluation and off-policy learning are recognized as very important for future RL applications. The recent breakthroughs in this research area include new solutions for batch policy learning under multiple constraints, combining parametric and non-parametric models, and introducing a novel class of off-policy algorithms to force an agent towards acting close to on-policy.
• Exploration is an area where serious progress can be achieved. The papers presented at ICML 2019 introduced new efficient exploration methods with distributional RL, maximum entropy exploration, and a security condition to deal with the bridge effect in reinforcement learning.

Breakthrough research papers to read:

• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Emergent Tool Use From Multi-Agent Autocurricula

If you want to be alerted when we release new research overviews, subscribe to our AI Research mailing list at the bottom of this article.

Inspiring Invited Talks

All the keynote presentations selected for this year’s NeurIPS conference are definitely worth your attention:

• From System 1 Deep Learning to System 2 Deep Learning by Yoshua Bengio. In this lecture, the famous AI researcher will talk about how new machine learning tools can help us address System 2 tasks, which, in contrast to System 1 tasks, are done consciously. These include reasoning, planning, and capturing causality, among others.
• Veridical Data Science by Bin Yu. In this talk, the researcher will introduce the PCS framework that is based on key principles of data science: Predictability, Computability, and Stability.
• How to Know, where Celeste Kidd will talk about the core cognitive systems people use to learn new things about the world without being overloaded with the tons of information available.
• Machine Learning Meets Single-Cell Biology: Insights and Challenges, where Dana Pe’er from Columbia University will show how machine learning can assist in uncovering new biology through constructing a human cell atlas.
• Mapping Emotions: Discovering Structure in Mesoscale Electrical Brain Recordings by Kafui Dzirasa, where a neurobiology researcher will explain how machine learning can help us get a better understanding of the brain signals indicating emotional pathology.
• Agency + Automation: Designing Artificial Intelligence into Interactive Systems by Jeff Heer. In this talk, the Professor of Computer Science & Engineering from the University of Washington will introduce the benefits of interactive systems where computational agents augment and enrich people’s intellectual work rather than replace it.
• Social Intelligence by Blaise Aguera y Arcas. Here the speaker from Google AI will explain why the optimization framework for ML might not be the best choice and will introduce some alternative and less explored approaches to general intelligence.

Highlighted Research Papers

Out of 6743 submissions received, 1428 papers were accepted, giving a 21% acceptance rate. Out of the accepted papers, 36 were selected for oral presentation, and the authors of these papers will have 15 minutes to present their research. Additionally, 164 papers were selected for short 5-minute presentations.

We at TOPBOTS mainly focus on research that has practical business applications or is of paramount importance for its research area. So, here are the papers that we want to highlight:

• XLNet: Generalized Autoregressive Pretraining for Language Understanding

The researchers from Carnegie Mellon University and Google have developed a new model, XLNet, for natural language processing (NLP) tasks such as reading comprehension, text classification, sentiment analysis, and others. XLNet is a generalized autoregressive pretraining method that leverages the best of both autoregressive language modeling (e.g., Transformer-XL) and autoencoding (e.g., BERT) while avoiding their limitations. The experiments demonstrate that the new model outperforms both BERT and Transformer-XL and achieves state-of-the-art performance on 18 NLP tasks.

• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems

The MIT research team investigates the problem of evaluating open-domain dialog systems. Humans are the ultimate authority in evaluating the quality of dialog systems, but getting human ratings is usually quite an expensive and difficult process. To overcome these challenges and still get reliable evaluations, the MIT team introduces a novel framework to estimate a dialog quality score, which has a high and statistically significant correlation with human ratings. Specifically, they propose a series of psychology-motivated metrics and then fit a function to predict human evaluation of conversation quality given these metrics. Bot quality is evaluated through self-play over a fixed number of turns, in which the bot generates utterances that are fed back as input in the next turn. The experiments confirm that the introduced self-play framework, together with psychology-motivated automated metrics, provides a good proxy for conversation assessment.

• Average Individual Fairness: Algorithms, Generalization and Experiments

The researchers from the University of Pennsylvania suggest combining statistical and individual notions of fairness to generate a new family of fairness definitions for classification problems. First of all, they assume that each individual is subject to decisions made by many classification systems. Then, they require that the error rates, or false-positive rates, or false-negative rates, are equal across all individuals. Finally, to satisfy this guarantee, they derive a new oracle-efficient algorithm for learning Average Individual Fairness, called AIF-Learn. The algorithm solves the fair empirical risk minimization task with the solution being generalizable to both new individuals and new classification tasks. The empirical evaluation verifies the effectiveness of the introduced algorithm.

• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models

In this paper, the Stanford University research team addresses the evaluation of image generative models. They introduce a gold standard human benchmark Human eYe Perceptual Evaluation (HYPE) to evaluate the realism of machine-generated images. The first evaluation method, called HYPEtime, evaluates the realism of image by measuring the minimum time, in milliseconds, required to distinguish the real image from the fake one. The second method, called HYPE∞, measures the rate at which humans mistake fake images and real images, given unlimited time. The experiments with six state-of-the-art GAN architectures and four different datasets demonstrate that HYPE provides reliable scores that can be easily and cheaply reproduced.

• Blind Super-Resolution Kernel Estimation using an Internal-GAN

The researchers argue that synthetically generated super-resolution (SR) datasets are not good enough for training SR systems that deal with real-world low-resolution (LR) images. To address super-resolution in a real-world setting, they introduce an image-specific Internal-GAN, called KernelGAN, that trains only on low-resolution test images at test time. The Generator of this GAN is trained to generate a downscaled version of the LR test image so that the Discriminator cannot distinguish between the patch distribution of the generated image and the patch distribution of the original LR image. The experiments demonstrate that KernelGAN leads to state-of-the-art performance in Blind-SR when incorporated into existing SR algorithms.

• Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs

In this paper, the authors draw our attention to the fact that even though deep neural networks were inspired by brain anatomy, typical deep networks these days are usually hard to map onto the brain’s anatomy because of the vast number of layers and missing important connections, such as recurrence. Therefore, they developed CORnet-S, a shallow artificial neural network with four anatomically mapped regions and recurrent connectivity. The evaluation results demonstrate that CORnet-S achieves top results on Brain-Score and outperforms similarly compact models on ImageNet.

• Guided Similarity Separation for Image Retrieval

In this paper, the Layer6 AI team introduces a novel approach to image retrieval. Specifically, they suggest leveraging graph convolutional networks to encode neighbor information into image descriptors, and thus promote local consistency. Furthermore, inspired by clustering and manifold learning approaches, the researchers propose an unsupervised loss that is based on a pairwise clustering of similarity scores. The experiments on five different datasets demonstrate the effectiveness of the introduced approach with up to 24% relative improvement in mAP over leading baselines.

• Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning

A discount factor plays the role of a hyperparameter in reinforcement learning by helping to avoid some optimization challenges that arise when optimizing an undiscounted objective directly. It was believed that low discount factors performed poorly because of the too-small action-gaps (i.e., the difference between the values of the best and second-best actions). In this paper, the authors show that this perception needs revision, and in fact, the primary factor defining the performance of the discount factor is the size difference of the action gap. The researchers introduce a new method that ensures more homogeneous action-gap sizes for sparse-reward problems. The experiments demonstrate that this method achieves much better performance for low discount factors than previously possible, thus supporting the theoretical analysis.

Tutorials Worth Your Attention

The tutorial program is an important part of the NeurIPS conference. The Tutorial Chairs were looking for “fresh perspectives from speakers outside of our community.” Additionally, diversity and inclusion were key factors when choosing topics and speakers.

In total, 9 tutorials were selected for NeurIPS 2019. Of those, we are most excited about these three:

• Imitation Learning and its Application to Natural Language Generation, by Kyunghyun Cho and Hal Daume III
• Representation Learning and Fairness, by Moustapha Cisse and Sanmi Koyejo
• Human Behavior Modeling with Machine Learning: Opportunities and Challenges, by Nuria M Oliver and Albert Ali Salah

Interesting Workshops

This year, the Workshop Chairs admitted the high number of innovative proposals. The selection process was guided by new rules with a specific focus on diversity. Finally, 53 exciting workshops were accepted for NeurIPS.

Here are the ones that we are looking forward to:

• Visually Grounded Interaction and Language
• Emergent Communication: Towards Natural Language
• The Third Conversational AI workshop: Today’s Practice and Tomorrow’s Potential
• Machine Learning for Creativity and Design 3.0
• Human-Centric Machine Learning
• Safety and Robustness in Decision-making
• Minding the Gap: Between Fairness and Ethics
• Joint Workshop on AI for Social Good
• Machine Learning for the Developing World (ML4D): Challenges and Risks
• AI for Humanitarian Assistance and Disaster Response
• Real Neurons & Hidden Units: Future Directions at the Intersection of Neuroscience and AI
• Context and Compositionality in Biological and Artificial Neural Systems

Further Reading

To get more in-depth understanding of the latest trends in AI, check out our curated lists of top AI and ML research papers:

• Top AI & Machine Learning Research Papers From 2019
• What Are Major NLP Achievements & Papers From 2019?
• 10 Important Research Papers in Conversational AI From 2019
• 10 Cutting-Edge Research Papers In Computer Vision From 2019

Enjoy this article? Sign up for more AI research updates.

We’ll let you know when we release more summary articles like this one.