Baidu introduced ERNIE Bot, its latest generative AI product and knowledge-enhanced large language model (LLM), on March 16, 2023. This advanced technology can comprehend human intentions and deliver accurate, logical, and fluent responses approaching human level. During a press conference at Baidu’s headquarters in Beijing, Baidu Co-founder, Chairman, and CEO, Robin Li showcased the comprehensive abilities of ERNIE Bot, in five scenarios: literary creation, business writing, mathematical calculation, Chinese language understanding, and multi-modal generation.

ERNIE Bot is a culmination of years of research and industry practices by Baidu, said Haifeng Wang, CTO of Baidu. Below is a deep dive into the technology behind ERNIE Bot.

ERNIE Bot: New-generation knowledge-enhanced LLM

ERNIE Bot is a new generation of knowledge-enhanced LLM and the latest generative AI product following Wenxin Yige, Baidu’s text-to-image platform. It can interact in dialogue, create content, reason with knowledge, and generate multiple modes of output. Said Wang, ERNIE Bot was developed based on the ERNIE (Enhanced Representation through Knowledge Integration) and PLATO (Pre-trained Dialogue Generation Model) series models. Its key technologies include supervised fine-tuning, reinforcement learning with human feedback, prompt learning, knowledge enhancement, retrieval enhancement, and dialogue enhancement. The first three are commonly used technologies in this type of LLM, which have been applied and accumulated in ERNIE and PLATO and have been further strengthened and refined in ERNIE Bot. The latter three are innovations built on Baidu's existing technological advantages and provide a solid foundation for ERNIE Bot's future growth.

In knowledge enhancement, ERNIE Bot mainly adopts two approaches: knowledge internalization and external utilization. Knowledge internalization involves learning from large-scale knowledge and unlabeled data based on semantic units, constructing training data using the acquired knowledge, and incorporating the knowledge into the model parameters. External utilization involves integrating multiple sources of heterogeneous knowledge for knowledge reasoning and prompt construction.

In search enhancement, ERNIE Bot benefits from a new-generation search architecture with semantic understanding and matching as its core technology. Introducing search results can provide the LLM with timely and accurate reference information, better meeting user needs.

In dialogue enhancement, based on accumulated experience in dialogue technology and applications, ERNIE Bot has memory mechanisms, contextual understanding, and dialogue planning abilities, achieving better continuity, coherence, and logic in dialogues.

Baidu possesses a diverse range of training data focused on the Chinese language, service applications, and knowledge. With supervised fine-tuning, ERNIE Bot has gained a more precise and nuanced understanding of the Chinese language and its practical applications. Additionally, a feedback and reward mechanism with strategic optimization further enhances the model's capabilities as it receives more feedback from real users. ERNIE Bot integrates various types of data and knowledge to automatically generate prompts, including examples, outlines, standards, key concepts, and thought chains. It provides rich reference information that feeds into the model's relevant knowledge and generates high-quality results.

Wang emphasized that the PaddlePaddle deep learning platform supports ERNIE Bot with better performance, higher efficiency, and stronger capabilities. For development and training, PaddlePaddle's unified development paradigm for both dynamic and static modes and its adaptive distributed architecture enable flexible development and efficient training of big models. In inference deployment, PaddlePaddle supports the efficient inference of big models and provides service-oriented deployment capabilities, including computation fusion, sparse quantization with software-hardware collaboration, model compression, and more. The PaddlePaddle platform has now attracted 5.35 million developers, serving 200,000 enterprises and institutions and creating 670,000 models based on PaddlePaddle.

The combination of the ERNIE Big Model and the PaddlePaddle deep learning platform has laid a solid foundation for industrial intelligence. With further integration and development of ERNIE Bot and PaddlePaddle, the R&D of AI technology and applications will become more standardized, automated, and modular, accelerating the industrial-scale production of AI, and driving the iterative evolution of ERNIE Bot to be applied in more scenarios and industries, bringing inexhaustible driving force to the intelligent upgrading of numerous industries.

Full AI stack built on years of technology development

In the era of AI, the IT technology stack can be divided into four layers: the chip layer, framework layer, model layer, and application layer. Since 2010, Baidu has been comprehensively deploying artificial intelligence and is one of the few AI companies in the world to offer a full-stack layout. From Kunlun AI chips and the PaddlePaddle deep learning platform to the Big Model ERNIE and numerous applications, Baidu has key self-developed technologies that lead the industry in each layer of the technology stack, achieving feedback between layers, end-to-end optimization, and significantly improved efficiency.

At the framework layer, PaddlePaddle is the first open-source, industry-level deep learning platform developed independently by Baidu in China. It includes a core framework, industry-level model library, development kit, tool components, and learning and training communities to support model production and their application by standardization and automation. At the model layer, the Big Model ERNIE includes basic big models such as NLP, CV, cross-modality, big task models such as dialogue, cross-language, search, information extraction, biology computing field big models, big industry models, and tool platforms that support big model applications. It forms a three-level big model technology system of basic-task-industry with two characteristics: knowledge enhancement and industrial level.

Haifeng Wang said that ERNIE Bot is a natural result of Baidu's many years of technological development and industrial practice, especially the joint optimization of the PaddlePaddle deep learning platform and the Big Model ERNIE, which provides solid technical support for ERNIE Bot. PaddlePaddle effectively supports flexible development, efficient training, and inference deployment of large models. Since its launch in 2019, the Big Model ERNIE has developed from an initial natural language understanding capability to become a big model platform with complete cross-language, cross-modality, cross-task, and cross-industry capabilities.