Kubernetes community announced today that PaddlePaddle, the open source deep learning framework originally developed by Baidu, is now compatible with Kubernetes, the cluster management system, making PaddlePaddle the only deep learning framework that officially supports Kubernetes to date.
The compatibility will allow developers to conveniently train large models on all major global cloud service providers and on-premise clusters. The project was jointly developed by Baidu and CoreOS, one of the primary contributors to Kubernetes.
Developers usually deploy AI programs together with web services, log collectors, and data processors on the same general-purpose cluster in order to implement highly efficient data pipelines. To manage this process, developers use tools like Kubernetes, which is one of the most sophisticated general purpose cluster management systems. By making PaddlePaddle compatible with Kubernetes, developers can now build highly efficient deep learning-powered applications.
“Using frameworks like Kubernetes, developers don’t need to worry about writing unnecessary code to configure and deploy a deep learning training system on a standard cloud platform,” said Yi Wang, tech lead of the PaddlePaddle project. “This ultimately helps them get their projects off the ground faster.”
Kubernetes makes full use of cluster hardware by packaging PaddlePaddle jobs that require GPUs with other jobs requiring different resources, like large memory or disk I/O throughput, on the same set of physical computers. It also automatically scales out online services in the daytime when there are many active users, and frees resources for PaddlePaddle jobs in the evening.
Originally developed by Baidu, PaddlePaddle is an easy-to-use deep learning framework applied in a range of Baidu’s AI products and technologies, including search ranking and machine translation. The framework is well suited for training recurrent neural networks, making it especially effective for applications like natural language understanding, speech, and multimedia. After being open sourced in September, PaddlePaddle is now one of the fastest growing deep learning platforms.
For more details on how to run deep learning with PaddlePaddle on Kubernetes, please go to Kubernetes tech blog.