CSU shortlisted for Nobel Prize in Supercomputing

On the afternoon of November 15, local time, the supercomputing application result "2.5 Million-Atom Ab Initio Electronic-Structure Simulation of Complex Metallic Heterostructures with DGDFT", shortlisted for the international Gordon Bell Award nomination by China University of Science and Technology (CSUST), participated in the online defense of the Global Supercomputing Conference (SC22) held in Dallas, Texas, USA, reporting on CSUST's use of low-scalar This is the second time that CSU has been nominated for the Gordon Bell Award as the first completer.

The Gordon Bell Award is the highest international academic award in the field of high performance computing, known as the "Nobel Prize in supercomputing", and is selected and awarded annually by the ACM (American Computer Society) for outstanding achievements in the field of high performance computing. The award is given annually by the ACM (American Computer Society) for outstanding work in the field of high performance computing and has great international influence. This achievement was jointly developed by Hu Wei's group of academician Yang Jinlong at the National Research Centre for Microscale Matter Science in Hefei and Professor An Hong's group at the School of Computer Science and Technology, in close collaboration with the Laoshan Laboratory (formerly the Qingdao Pilot National Laboratory for Marine Science and Technology), the Institute of Computing Technology of the Chinese Academy of Sciences, Peking University, the Institute of Software Research of the Chinese Academy of Sciences, Qilu University of Technology, and the National Parallel Computer Engineering Technology Research Centre. The research was carried out in close collaboration with the researchers of the National Parallel Computer Engineering Technology Research Centre.

e3dff3edfb137eca540b3a229e2624b7

Figure 1 ACMGordonBell Award nomination

Advanced materials are the cornerstone of the national economy and an important foundation for achieving the transformation and upgrading of manufacturing industries. Due to the difficulty of developing new materials, the physical experiments used to verify the properties of materials are complex and expensive to design. The advent of quantum mechanics and high performance computing technology has fundamentally changed this situation. By inputting structural information about a material, the first-principles calculations of quantum mechanics can be used to predict the ground state structure and fundamental physicochemical properties of a known material with relatively high accuracy and control at the atomic level. This is the most competitive method and technological approach for solving experimental theoretical problems and predicting the structural properties of new materials in the 21st century. This approach, which does not require real experiments to be carried out, not only offers significant savings in experimental costs and shortens the development cycle of new materials, but also provides effective theoretical guidance for the preparation and modification of materials, the development of new materials, and the study of material properties in extreme environments.

In 1933, Schrödinger and Dirac were awarded the Nobel Prize in Physics for their development of the Schrödinger-Dirac equation for quantum mechanics. Although Dirac had predicted at the time of the establishment of quantum mechanics that the search for the fundamental laws of physics and chemistry was largely complete. However, because the equations describing these fundamental laws were so complex to solve, it was still very difficult to use the basic principles to solve practical problems. Until the advent of high performance computing (HPC) in the 1960s, we were only able to solve Schrödinger's equation with a few dozen atoms, which was far from a truly complex system.

In 1998, Kohn and Sham won the Nobel Prize in Chemistry for developing density generalized function theory (DFT) based on the Kohn-Sham equation, which reduced the 3N-dimensional wave function problem to a three-dimensional particle density problem, thereby reducing the computational complexity of ab initio electronic structure simulations to O(N^3). in 2013, three chemists, Martin Karplus, Michael Levitt and Arieh Warshel were awarded the Nobel Prize in Chemistry for developing methods for multi-scale modelling of complex systems on modern supercomputers. the Kohn-Sham equations introduced quantum mechanics for the first time to truly complex systems containing thousands of atoms. However, as the computational complexity of first-principles material simulations increases dramatically with the scale of the material being simulated, researchers are placing increasing demands on software performance and computational resources.

Rapid advances in supercomputers and high performance computing technologies have provided opportunities for the development of first-principles computing to play an increasingly important role in research areas such as condensed matter physics, chemistry, materials and biology. In the post-Moore era, ab initio electronic structure simulations have become the only way to understand our modern information technology. Full quantum mechanical simulations are essential for the design of next-generation field-effect transistors below 10 nm, but require simulations at scales of at least hundreds of thousands of atoms. For nearly three decades, modern high-performance computing techniques have brought ab initio modelling to the real physical world. This capability is important for most areas of science and engineering, such as energy, materials, biomedicine, catalytic reactions and strong correlation physics.

So how large a scale system can we actually simulate using the Kohn-Sham equation on the most advanced supercomputers of our time? There are three main computational approaches to achieving large-scale ab initio electronic structure simulations, namely the linear scalar algorithm, the artificial intelligence algorithm and the universal cubic scalar algorithm. Research work using all three algorithms has been awarded the Gordon Bell Prize or nominated for the Gordon Bell Prize. The linear and artificial intelligence algorithms rely excessively on approximation principles for the simulation of molecules, semiconductors and insulators, and therefore cannot be applied to complex metallic systems. In contrast, the current state-of-the-art DFT-FE software, which uses a generalised cubic scalar algorithm, can only simulate solid materials with 11K atoms as of 2019, which is much smaller than the linear scalar algorithm and the AI algorithm.

The use of basis group discrete Kohn-Sham equations is fundamental to the implementation of first-principles electronic structure calculations. Traditional first nature principle material simulations based on plane wave basis groups have a third order computational complexity and are difficult to apply to large complex systems by massively parallel acceleration. Linear scaling algorithms based on local atomic orbital basis groups allow for large scale simulations, but are often not accurate enough and difficult to apply to metallic systems, and also face the problem of non-regular sparse matrices that are difficult to compute in a highly parallel way. The new AdaptiveLocalBasis (ALB), which is strictly truncated and orthogonal in real space, combines the advantages of plane waves (orthogonal completeness) with those of numerical atomic orbitals (locality), providing both high accuracy comparable to plane waves and the use of linear scaling algorithms suitable for large scale computations. Moreover, the Hamiltonian matrix constructed from the ALB basis group has a fixed tri-diagonal block sparsity property, which is suitable for achieving a high degree of parallelism.

In collaboration with the Institute of Computing, Chinese Academy of Sciences and the Institute of Software, Chinese Academy of Sciences, CSU has developed first-principles computing software DGDFT (Discontinuous Galerkin density functional theory) based on ALB basis groups and interrupted Galerkin finite element methods, combined with multi-level parallel optimisation design and the state-of-the-art sparse matrix solver PEXSI's low-scalar diagonalisation algorithm in numerical algorithms, sparse The DGDFT software, designed in-house on a new generation of Shenwei supercomputers, has enabled the first high-precision first-principles electronic structure simulation of 2.5 million atoms for complex metal systems, reaching the mesoscopic scale (>100nm), which can be used to design a new generation of electronic transistors based on two-dimensional materials.

Fig. 2 Base-group discrete Kohn-Sham equations, field-effect tube heterojunction system, solution time and performance

The advanced features of DGDFT software are as follows: (1) Compared with similar first principles calculation software in the world, DGDFT software has the advantages of low scalability, high accuracy and high parallel scalability, overcoming the difficulties of applying conventional DFT methods to complex metallic systems and poor parallel scalability of sparse matrices, which have been plaguing the field of materials simulation and high performance computing. (2) Although the interrupted Galliogin method is widely used to solve partial differential equations, only DGDFT software has adopted this method to solve the Kohn-Sham equation. (3) Matrices discretized by ALB basis groups have a fixed format of triangular block sparsity, and related research is still very much at the forefront, with no comparable first principles software available internationally. (4) The low-scalar PEXSI algorithm is suitable for metallic systems and has a computational complexity of 1.5 times for quasi-two-dimensional systems. The team's optimised PEXSI algorithm can achieve 64 PFLOPS and 5% peak on the new generation of Shineway supercomputers, far exceeding the current record of the first HPCG sparse matrix performance benchmarks (16 PFLOPS, 2.9 PFLOPS and 5.9 PFLOPS for Fugaku, Summit and New Shineway respectively, corresponding to 3.6%, 1.5% and 0.5% of the peak).

Thanks to these innovations, DGDFT's simulations are 175 times larger than the current state-of-the-art high scale DFT-FE software (which uses 23 K GPUs to simulate 11K atoms), simulates atomic scales far beyond those of similar software (for example, the most popular VASP can only simulate 1K atoms), and can simulate 2.5 million metal systems with guaranteed chemical accuracy; at the same time, the DGDFT has extremely fast computational speed, with a solution time 2054 times faster than DFT-FE. Compared with the research results published by the project team in 2021 (Science Bulletin, 2021, Vol. 66, Issue 2), which can simulate the electronic structure properties of a two-dimensional metallic graphene system with about 10,000 carbon atoms on the Shenwei-TaizhouLight, the simulation scale on the new generation of Shenwei is 250 times higher than the previous simulation results.

In the future, with the supercomputer computing power reaching 10E level, the high scalability of DGDFT software can further enhance the simulated system and can further extend the first-principles material simulation to macroscopic scale (>1000nm), thus realizing the simulation of real materials and devices and paving the way for the industrial application of integrated software and hardware for first-principles material simulation.

Professor Hong An, Associate Researcher Weile Jia and Professor Jinlong Yang are co-corresponding authors of this paper; Researcher Wei Hu, PhD student Zhuoqiang Guo and Associate Researchers Qingcai Jiang and Xinming Qin are co-first authors. The research was co-funded by the National Natural Science Foundation of China, the National Key Research and Development Program, the University of Science and Technology of China (USTC) "Double First Class Project" Research Fund, and the Strategic Key Research Program of the Chinese Academy of Sciences. The project was supported by supercomputing resources from the Laoshan Laboratory, the Supercomputing Centre of the University of Science and Technology of China, and the National Supercomputing Centre of Jinan.