Supermicro Contributes to the MN-3 Supercomputer Earning #1 on Green500 list

Sophisticated Systems Supporting Deep Learning Projects at Preferred Networks Earn International Recognition 

 

Supermicro and Preferred Networks (PFN) collaborated to develop the most efficient supercomputer anywhere on earth, earning the #1 position on the Green500 list. This supercomputer, the MN-3, comprises Intel® Xeon® Scalable processors and MNCore™ boards developed by Preferred Networks. The Green500 list is compiled semi-annually and recognizes the most efficient supercomputer in the world based on real-world scientific application benchmarks. The MN-3 delivers over 21 GFlops of computing power per watt of electricity.

When looking at the performance per watt of electricity, the MN-3 is ranked #1 in the world in the Green500 ranking. The MN-3 delivers a whopping 21.11 Gigaflops of performance per watt on a benchmark run that showed a total of 1.62 Petaflops performance. In recognition of this engineering effort between Preferred Networks and Supermicro, the MN-3 was awarded the "Number 1 in the Green500" certificate at the annual (virtual in 2020) ISC conference in June 2020.

 

Challenges

 

• PFN existing technology could not solve particular machine/deep learning (DL) applications with the agreed-upon Service Level Agreements (SLAs) for their customers.
• They were concerned with the energy usage that such a large system would require.
• PFN needed to develop technology that would drastically speed up their DL training applications used in various domains. Some of the projects that PFN specializes in have to do with computer vision applications. Applications designed to recognize a face, scene, or object require a training phase, where thousands to millions of images are used to train the application.
• The massive amounts of images are typically stored or archived on an external storage system. For performance reasons, the images are first brought into solid-state drive (SSD) drives from an image database before the processing and training occur. 
• PFN was trying to understand and plan for the overall energy footprint that a new supercomputer would require. With increasing power costs, a large system of the size that PFN was going to need, the operating costs of both the power and associated cooling would exceed the allocated budget. Therefore, the energy efficiency of the new solution would have to be designed into the system and not become an afterthought.

PFN decided that partnering with a known supplier whose expertise included power-efficient designs would be critical to this project. Both Supermicro and PFN were excited that the MN-3 supercomputer is now recognized as the most efficient supercomputer globally in the June 2020 Green500 rankings.

 

 

Solution

 

The mechanical solution required out-of-the-box thinking by Supermicro -- the ability to fit not only two CPUs, four MNCore boards, but also up to 6TB of DDR4 memory. Each server would also have to house the Intel® Optane™ persistent memory modules, each containing 3TB of memory. Image AA below shows the overall architecture and workflow of an MN-Core server, including Intel Xeon scalable processors, Intel® SSDs, Intel® Optane™ persistent memory, and the MN-Core board, all integrated on to a Supermicro motherboard.

The solution that PFN designed is based on the Supermicro GPU server that contains the X11DPG-OT-CPU motherboard. This advanced system is ideal for housing multiple GPUs or accelerators, as well as the interconnects that enable ultra-fast communication between GPUs. The PFN customized server is shown below, containing all this technology, and is housed in a 7U high rack-mounted unit.

The server's four MN-Core boards are connected to PCIe x16 slots on a Supermicro motherboard and to the MN-Core Direct Connect board that enables high-speed communication between the MN-Core boards.

 

                                            

 

In Summary

 

With the server designed, PFN built a supercomputer called MN-3 that would serve many constituents at once. The initial cluster would consist of 40 servers; four interconnect nodes, and five 100GbE switches. A total of 2080 cores delivers 1,621.1 TFlops of performance while using just 77 kW of power. The supercomputer ranked #393in the world Top500 list, based on the LINPAC benchmark as of June 2020.

 

 

 

Available from BIOS IT  - contact us to discuss your requirements.