Lejaeghere et. al studied reproducibility of DFT codes (softwares) by comparing 15 different codes employing 40 different potentials. The study concluded that most codes agree very well, with pairwise differences that are comparable to those between different high-precision experiments.
Why do we have so many different codes? Are they redundant?
There are many reasons why so many different scientific packages have been developed
These packages were developed by individual researchers who were in competition with each other and also work mostly independently. It was natural that different packages sprung out of different regions (e.g. NWChem stands for NorthWest Chem, because it’s based at Pacific Northwest National Lab (PNNL)) and for different purposes or specialties (e.g. Molpro for excited-states). Also remember that this was before version control or the idea of open-access was a thing, so sharing code was not so easy.
The opportunity to make money with the software was also a huge draw and further enticed people to start there own company, or in contrast distance themselves from the company because of their own personal beliefs. For example, the company Gaussian has a well known history of controversy with some scientists over how the company was ran (perhaps even with the founder of Gaussian itself, John Pople).
Despite the code similarity most scientific software is very poorly managed because the developers work independently and are not computer programmers. So in many cases, people wanted to start something fresh in an attempt to make it better, or easier for them to write their own custom code. Again, open-access and object oriented programming is fairly recent. A great example of a software package that was developed to make reading and improving the code is OpenMM. And because of this, OpenMM now has superior GPU kernels and object oriented design making it one of the best and fastest codes for molecular dynamics today.
There is also a difference in preferred languages. Historically scientific software was written in Fortran. Then people started using C/C++. Nowadays people are even using python!
It is my opinion that all types of codes should be continued to be developed independently but hopefully they follow good coding practices like object-oriented programming. If that is the case then it's easy to mix and match code from different developers.
There are many reasons why we have so many different density functional theory programs, and it would be nigh on impossible to give a full answer encompassing all of them. A brief, select few:
Scientific reasons, because you need something that can handle some very specific physics or chemistry accurately and efficiently. At a simple level it might be that a particular basis set is most appropriate for the task you want, but there are many other considerations. You might want to treat solvation as an integral part of the problem (e.g. JDFTx), or not use the Born-Oppenheimer approximation etc.
Technical reasons, for example you really need methods which are efficient on some very particular hardware (e.g. with extreme parallelism, or accelerators, or very long vector machines), or are written in a particular language in order to integrate with some other software or workflow.
Philosophical design reasons. Perhaps you want a single program which can do all the kinds of calculations you want, or perhaps you prefer a suite of smaller, more specialised programs. Maybe you want something you can rapidly prototype new methods in and don't care about performance or features.
Personal reasons. Perhaps you want some software to your name to enhance your career, or perhaps you fell out with the authors of the "usual" choice (or vice versa). Perhaps your intended use of the software isn't compatible with the licence of the usual programs, e.g. because you're doing research which is commercial or classified.
Inertia. Now that we have all of these different programs, what is the incentive for developer communities to merge and settle on one particular software and approach?
Pedagogical reasons, for example someone wants to really understand how the theory and algorithms work. Many of my own PhD students write their own density functional theory program for this reason, though these are not used for real applications.
It isn't very hard! Writing a very basic, full-potential, all-electron density functional theory program is relatively straightforward. It will not be fast or scalable, nor will it compute very much -- probably just the ground state energy, density and Kohn-Sham states -- but it will work.
Finally, I'd like to note that having several implementations, even when they make the same major design choices (e.g. basis set; wavefunction or Green's function approaches), gives some competition, and this can be healthy. The Science paper referenced in the question demonstrates the benefit of this: the good agreement between the programs has not always been there, indeed it is partly due to the work behind this paper that the programs do agree so well; when we found an outlier in our tests, we worked hard to understand why and fix any issues. Reproducibility is a serious issue in research, and the ability to apply two (or more) completely independent implementations of the theory to a scientific problem is extremely valuable.
Another benefit of competition is that developers don't generally like it when a different program can do something theirs can't, or is faster, or scales better or... so competition can lead to improvements for everyone.
So in summary: there are lots of reasons why people write their own programs. Having a variety of design choices is good, but even having several implementations with similar design choices is healthy. Are there "too many" implementations? Possibly, but the "ideal" number of implementations is more than one or two.
Another thing not mentioned yet is that there are, broadly speaking, two camps of codes: those that are primarily meant for periodic (often but not exclusively plane-wave) DFT (VASP, Quantum Espresso, etc.), and those that are primarily meant for finite systems like molecules (e.g. Gaussian, ORCA). Also, there are different algorithms in each package, some of which may better suit a given user. There's also a number of packages recently that are meant to be open-source (or at least free) alternatives of paid packages that have dominated the field.
I would like to stress the importance of the choice of the numerical algorithm, which was briefly touched on above. The choice of the representation (plane waves, augmented plane-waves, finite elements, finite differences, multiresolution grids, or atomic orbitals: Slater-type orbitals, Gaussian-type orbitals, or numerical atomic orbitals) determines the algorithms one can use and the properties one can study; in addition with a given numerical representation one may have to choose between different styles of pseudopotentials. No code will cover all possibilities, since traditionally scientific software is not very modular and cleanly structured, and the different approaches may not have a lot of common ground infrastructure wise.
Fortunately, modularization has happened, and e.g. common ground has been established for the evaluation of density functionals which is now typically handled by a common purpose-built library.
Another reason can be the choice of Hamiltonian, for example relativistic vs. non-relativistic (especially spin-orbit including). In relativistic theories, the orbitals are complex as opposed to real orbitals in non-relativistic codes, so it has to be programmed accordingly. Moreover, relativistic orbitals possess a multi-component spinor structure, meaning that some objects suddenly become vectors or matrices and do not commute any more. This means that many core routines, especially when aiming for an optimized efficient code, will end up different in relativistic and non-relativistic codes.
