Baidu released a wave of new AI technologies this week, with 11 new AI Big Models, the latest version PaddlePaddle 2.4 and two AI applications. Together, these technologies are designed to drive AI adoption across industries and further lower the AI threshold. The releases came as part of Wave Summit+, Baidu’s flagship deep learning developer conference jointly hosted with China’s National Engineering Laboratory for Deep Learning Technology and Applications on November 30, 2022.

At the event, Baidu CTO Dr. Haifeng Wang announced that as of November 2022, Baidu’s deep learning platform PaddlePaddle is now home to 5.35 million developers, serving over 200,000 enterprise clients, with more than 670,000 industry models built based on the platform. PaddlePaddle has now become a robust and inclusive ecosystem where developers, academic institutes, enterprises, government administrations, and hardware OEMs can partner up and achieve success collaboratively.

“Working together, deep learning platforms and Big Models are able to run through the entire AI industry chain, from hardware adaptation to model training, inference deployment and application, consolidating the foundation of industrial intelligence and accelerating the pace of intelligent upgrading,” said Dr. Wang.

New releases of Big Model ERNIE

Tian Wu, Corporate Vice President of Baidu, revealed 11 new models under the Big Model ERNIE family, bringing the total number to 36. Big Model ERNIE is characterized by industry-level performance and specialization in knowledge enhancement.

Among the new releases are five general Big Models, one task Big Model, and five industry Big Models. The five general Big Models include:

Lightweight Big Model ERNIE 3.0 Tiny: ERNIE 3.0 Tiny is a lightweight model that transfers knowledge from ERNIE 3.0 Zeus, a 100 billion-parameter model serving as the "teacher" model, through multi-task knowledge distillation. ERNIE 3.0 Tiny has been trained with different sizes, from 1 billion parameters to 100 million parameters and 10 million parameters. ERNIE 3.0 Tiny generalizes well and speeds up inference by tens to hundreds of times compared to ultra-large-scale models, significantly reducing the cost of implementation.

Cross-modal Understanding Big Model ERNIE ViL 2.0: ERNIE ViL 2.0 has achieved state-of-the-art understanding ability in both Chinese and English by utilizing a multi-view contrastive learning framework, while building intra-modal and inter-modal representation and alignment. ERNIE-ViL 2.0 is now available on EasyDL, a PaddlePaddle low-code development platform, and supports one-stop fine-tuning training and inference, which can be used in a wide range of visual-language applications. For more: https://arxiv.org/abs/2209.15270.

Cross-modal Generation Big Model ERNIE ViLG 2.0: ERNIE ViLG 2.0 is a large-scale Chinese text-to-image diffusion model, which progressively upgrades the quality of generated images by incorporating fine-grained textual and visual knowledge of key elements in the scene, utilizing different denoising experts at different denoising stages. For more: https://arxiv.org/abs/2210.15257.

Intelligent Document Big Model ERNIE-Layout: ERNIE-Layout is a novel document pre-training solution with layout knowledge enhancement in the whole workflow, able to learn better representations that combine the features from text, layout, and image. ERNIE-Layout supports up to 96 languages. For more: https://arxiv.org/abs/2210.06155.

MSA-Free Protein Prediction Big Model HelixFold-Single: HelixFold-Single is the industry’s first open-source protein structure prediction model based on single sequence language modeling. It extracts information from nearly 300 million unlabeled protein data and models the relationship between proteins, thus implicitly learning MSA homology information in the pre-trained Big Model, which in turn effectively replaces the MSA information retrieval module and improves the model inference speed by hundreds of times. For more: https://arxiv.org/abs/2207.13921.

The one task Big Model is ERNIE-Code. Based on massive code and text data for pre-training, ERNIE-Code uses multilingual and multi-code joint learning, and a translation language model based on intermediate languages, with semantic understanding and generation capabilities across multiple natural and programming languages. The results have set leading levels on multiple public evaluation benchmarks. Currently, ERNIE-Code has been applied to applications such as intelligent code completion in PaddlePaddle AI Studio to improve software development efficiency.

The five industry Big Models are the results of Baidu’s collaborations with multiple companies and organizations, including Shenzhen Gas-Baidu·Wenxin, Geely-Baidu·Wenxin, Taikang-Baidu·Wenxin, TCL-Baidu·Wenxin, and Cihai-Baidu·Wenxin. Up to now, Big Model ERNIE includes 11 industry Big Models, covering industries and fields such as electricity, gas, finance, aerospace, media, city, film and television, manufacturing and social science.

At the event, Wu also introduced an upgraded Big Model ERNIE development kit, upgraded ERNIE API services, and new features from PaddlePaddle’s enterprise version EasyDL (a low-code AI development platform) and BML (a full-featured AI development platform).

Two AI applications built on Big Models

In August, Baidu released its AI-generated art platform Wenxin Yige, which allows users to complete a painting by simply entering a prompt. At Wave Summit+ 2022, Wu announced three new features on Wenxin Yige: image generation based on image inputs, image editing, and one-click video generation. With these new features added, Wenxin Yige is equipped with stronger creativity and further able to reduce content production costs.

The conference also saw the debut of an industry-level search system, Wenxin Baizhong, a joint development product of Baidu Search and Big Model ERNIE. This new search system can construct an online search engine within just three steps. It has a powerful semantic understanding ability, representing a major leap forward compared to traditional models. It can efficiently build search system from scratch, reducing labor costs by more than 90% compared with traditional search systems. Additionally, leveraging Big Model ERNIE’s few-shot learning, Wenxin Baizhong only requires a small amount of data to optimize search results in different industries.

New release of PaddlePaddle 2.4

The newly released PaddlePaddle 2.4 enables a more flexible and convenient framework development as well as the industry’s leading Big Model distributed training performance and helps to realize high-caliber deployment covering all scenarios. For more details, please check out: https://github.com/PaddlePaddle/Paddle/releases/tag/v2.4.0

Development: PaddlePaddle 2.4 is enhanced with new sparse matrix computation and graph learning APIs. It comes with upgraded higher-order automatic differentiation capabilities to support scientific computing applications. It also features upgraded dynamic graph to static graph technology to support complex model export and deployment.

Training: PaddlePaddle 2.4 releases PGLBox, a distributed hierarchical architecture for efficiently training super-large scale Graph Neural Networks (GNN) containing tens of billions of nodes and edges using a single machine with AI accelerators. With the ultimate optimization of distributed training performance, PaddlePaddle tops the world's authoritative AI training performance benchmark test MLPerf.

Deployment: Baidu released AI deployment tool FastDeploy, which comes with highly scalable, highly automated, high-performance inference capabilities to help support Big Model applications.

User Experience Enhancement: Baidu introduced the industry's first one-stop open-source Big Model development kit PaddleFleetX, which supports the whole process throughout the production to application of Big Models. The number of open-source low-code models in PaddlePaddle’s industry-level model library has increased to 600+. The number of industry-specific PP-series models has increased to 42. PaddlePaddle also includes a one-stop portal integrating all industrial models and tool for quick deployment. The number of industry flagship examples (which showcase the whole process throughout the industry implementation) has increased to 68, covering key industries such as finance, industry, transportation, and Internet.