Baidu announced a new language representation model ERNIE (Enhanced Representation through kNowledge IntEgration) and open-sourced its code and pre-trained model based on PaddlePaddle, Baidu’s deep learning open platform (https://github.com/PaddlePaddle/LARK/tree/develop/ERNIE). The model made significant performance on various natural language processing (NLP) datasets for tasks such as natural language inference, semantic similarity, named entity recognition, sentiment analysis and question-answer matching. The results show that ERNIE outperforms Google’s BERT in all Chinese language understanding tasks mentioned above.

In recent years, unsupervised pre-trained language models have made great progress on various NLP tasks. Early work in this field focused on context-independent word embedding. Later, models such as Cove, ELMo, and GPT have constructed sentence-level semantic representation. BERT achieved better results by predicting shielded words and utilizing Transformer's multi-level self-attention bidirectional modeling capability.

All of these models mainly focused on the original language signals not on semantic units in the text. This problem is more obvious in Chinese language. Take BERT for example, it is difficult for BERT to learn the complete semantic representation of larger semantic units since it processes Chinese language by inferring Chinese characters. When it comes to terms such as 乒[mask]球(a three-character term meaning table tennis with the middle character masked) , [mask]颜六色(an idiomatic phrase meaning colorful and we mask the first character),BERT can easily infer the masked characters based on the character collocations but not on the concepts and knowledge. 

We considered that if the model can learn the implicit knowledge from texts, its performances on various tasks will be further improved. Therefore, we train ERNIE with knowledge integration enhancement. ERNIE learns the semantic representation of complete concepts by masking semantic units such as words and entities. ERNIE directly models priori semantic knowledge units and, as a result, enhances the model's ability to learn semantic representation.

Learnt by BERT ：哈[mask] 滨是[mask] 龙江的省会，[mask] 际冰[mask] 文化名城。

Learnt by ERNIE：[mask] [mask] [mask] 是黑龙江的省会，国际[mask] [mask] 文化名城。

For example, BERT identified the character 尔(er) by the local co-occurring characters 哈（ha）and 滨（bin）in哈尔滨(Harbin) not by any knowledge related to Harbin. However, ERNIE learned that Harbin is the capital of Heilongjiang Province in China and that Harbin is a city with ice and snow by learning the expression of words and entities. 

ERNIE is character-based and it is more versatile and scalable. Compared with word-level modeling, character-level modeling can comprehend the compositional meaning of characters. For example, when modeling words like 红色,蓝色, 绿色(meaning red, blue and green respectively, with the same final character) , it can learn the semantic relationship between words through the compositional meaning of the characters.      

In addition, multi-source data containing different knowledge is introduced to ERNIE’s training data. Besides encyclopedia articles, ERNIE has been fed with data from news and dialogues in forums. In forum dialogue modeling, dialogue data is important for semantic representation, since the corresponding query semantics of the same replies are often similar.

Based on this hypothesis, ERNIE uses DLM (Dialogue Language Model) to model the Query-Response dialogue structure. It takes adjacency pairs in dialogues as input, and introduces dialogue embedding to identify the roles in the dialogue. It uses Dialogue Response Loss to learn the implicit relationship in dialogues, which also enhances the model'sability to learn semantic representation.

To verify ERNIE's knowledge learning ability, we tested the model on the machine comprehension task by filling some questions with answers. The experiment setting is that the entity knowledge is removed from the paragraphs and the model need to infer what they are. 

From the results above, it is clear that ERNIE performs better in context-based knowledge reasoning than BERT. 

To verify ERNIE’s ability of knowledge reasoning, we conducted further experiments on natural language inference tasks. XNLI was built by researchers from Facebook and New York University to assess a model's multilingual sentence comprehension. Its goal is to judge the relationship between two sentences (contradiction, neutrality and entailment). We have compared the performance of BERT and ERNIE on its Chinese data：

The result shows that ERNIE outperforms BERT in natural language inference.

In order to verify the model's capability, we validated its effectiveness on a number of open Chinese language datasets. Compared to BERT, ERNIE has achieved better results.

1.Semantic Similarity Task LCQMC

LCQMC is a question similarity dataset constructed by Harbin Institute of Technology in COLING 2018, one of the top international conferences in Natural Language Processing. Its goal is to determine whether the two questions share the same meaning

2.    Sentimental Analysis Task ChnSentiCorp

ChnSentiCorp is a Chinese sentimental analysis dataset, aiming to judge the emotional attitude of a paragraph.

3. Name Entity Recognition MSRA-NER

MSRA-NER dataset is designed for the research competition of named entity recognition, published by Microsoft Research Asia. 

4.Retrieval Question Answer Matching Task NLPCC-DBQA

NLPCC-DBQA is a task released by NLPCC (an International Conference on natural language processing and Chinese computing) in 2016. Its goal is to select correct sentence which can answer the questions.

ERNIE team said that this breakthrough technology will be applied to a variety of products and scenarios to enhance user experience. Next, Baidu will further conduct research on integrating knowledge into pre-training semantic representation models, such as using syntactic parsing or weak supervised signals from other tasks or validating this idea in other languages.

Baidu NLP group aims to "Understand languages, acquire intelligence and change the world". The team will continue to develop cutting-edge NLP technologies, create leading technology platforms and build innovative products to serve users.