CASTEP on HECToR
The UK’s previous national supercomputing facility, HECToR, has been heavily used by scientists needing capability supercomputing resources since its official launch in October 2007. A substantial part of the Research Councils’ six-year funding for this facility is devoted to the Computational Science and Engineering (CSE) support provided by NAG. An important part of the CSE Service is the distributed CSE (dCSE) programme which, through a lightweight peer review process, delivers dedicated multi-month performance and scalability development projects in response to proposals from the user community. The first dCSE project to complete has proven to be an excellent example of what can be achieved through dedicated CSE effort, with dramatic improvements in code performance and scalability which could potentially save millions of pounds and allow significant new science to be undertaken for the UK Car-Parrinello Consortium (UKCP).
The objective of the dCSE project was to develop an improved, more scalable version of CASTEP – a software package which uses density functional theory with a plane wave basis set to calculate electronic properties of solids from first principles. The key task of the project was to implement band-parallelism in order to allow the code to scale to more than 1000 cores on HECToR. CASTEP is used on HECToR to model a range of materials or molecules at the atomic level. In particular scientists run CASTEP to obtain information about total energies, forces and stresses on an atomic system, as well as calculating optimum geometries, band structures, optical spectra, phonon spectra as well as molecular dynamics simulations. Dr Keith Refson from the Computational Materials Science Group at the Rutherford Appleton Laboratory was the Principal Investigator on the project, and Dr Martin Plummer from Daresbury Laboratory and Dr Matt Probert of The University of York were the Co-Investigators. NAG contracted Dr Phil Hasnip of the Department of Physics at the University of York to carry out the code development work in collaboration with both the wider CASTEP team and the NAG CSE team.
The results of this work were excellent. The improved code has a speed-up factor of between 2 and 4 times the original and now scales to over 1000 cores against 256 previously.
The UKCP Chairman, Dr Matt Probert of the Department of Physics at the University York estimated that the CASTEP consortium was using around 10m Allocation Units (AU)s per annum on HECToR – a nominal cost of around £640k. Making the code 2-4 times more efficient could result in a saving of £320k-£480k per annum (a saving of around say £1.6m-£2.4m over the remaining life of HECToR); all for around 8 person months of effort!
Commenting on the massive return on investment, Dr Probert said
“I guess it goes to show the value of centrally supporting key software packages, and that there is a considerable saving to be made due to scale of usage. Also the HECToR dCSE scheme is well worth supporting and continuing – and that the dCSE postdoc (Phil Hasnip in this case) was very good value for money!”
CASTEP user and PI on the project, Dr Keith Refson added
“The performance and scaling gains achieved by the band-parallel CASTEP represent a very substantial advance in the efficiency of utilisation of CPU cycles on HECToR. This will not only result in a lower time and cost and more rapid turnaround for jobs already planned, but as intended will permit larger and more complex simulations, using more processors, which were not previously feasible.”
Dr Probert predicts that due to the speed and scaling gains resulting from CASTEP’s improvements the software can now be utilized for larger scale scientific work, i.e. bigger atomic systems for less wall-clock time and/or more simulations– within their existing budgets. In fact, several research projects have been waiting in anticipation of the planned release of the improved CASTEP in 2009.
Since the writing of this case study, ARCHER has replaced HECToR as the UK’s national supercomputer.
The UKCP is an association of academic research groups collaborating on the first-principles computer simulation of condensed matter. Their aim is to use the creativity and strength of collaboration and shared expertise to produce outstanding science and simulation software.
CASTEP is a software package which uses density functional theory with a plane wave basis set to calculate electronic properties of solids from first principles. CASTEP is a fully featured first principles code and as such its capabilities are numerous. Aiming to calculate any physical property of the system from first principles, the basic quantity is the total energy from which many other quantities are derived. For example the derivative of total energy with respect to atomic positions results in the forces and the derivative with respect to cell parameters gives stresses. These are then used to perform full geometry optimizations and possibly finite temperature molecular dynamics. Furthermore, symmetry and constraints (both internal and external to the cell) can be imposed in the calculations, either as defined by the user, or automatically using in-built symmetry detection.