Why are Norm-conserving Pseudopotentials considered so important?

Question

So I am aware that a norm-conserving pseudopotential enforces the conditions that:

Inside the cut-off radius, the norm of each pseudo-wavefunction is identical to its corresponding all-electron wavefunction and all-electron and pseudo wavefunctions are identical outside the cut-off radius.

My question is why is the norm-conserving pseudopotential such a generally used and important pseudopotential that it is often referred to in the literature.

I initially assumed that it was because it was some sort of compromise between a pseudopotential that has a good level of softness and is also quite transferable. I must stress that from what I have read it seems like Norm-conserving pseudopotentials have a high level of transferability (why this is the case I am still unsure so any explanation would be greatly appreciated) but I am unsure if Norm-conserving also implies it has an acceptable level of softness (the papers I have read have not been too clear on this point, or perhaps I have not fully understood what I have read.)

Apologies if what I have written is unclear or does not make much sense as I am very new to this topic.

Answer 1

I think the best place to start is the original paper¹ proposing norm conserving pseudopotentials (NCPPs). It's very short and gives a nice explanation of why they were developed. I'll just give a brief summary here.

Norm conservation (specifically of the charge density $\rho$) ensures that the electrostatic potential for $r>r_c$ is accurate as a result of Gauss' Flux Theorem $\nabla^2V_E=\frac{\rho}{\epsilon_0}$. Matching the electrostatic potential makes these pseudopotentials more physically consistent and thus transferable.

Norm conservation also affects how well the scattering behavior of an atom is reproduced: $$2\pi[(r\phi)^2\frac{d}{d\epsilon}\frac{d}{dr}\ln(\phi)]_R=4\pi\int_0^R r^2\phi^2dr$$ The logarithmic derivative of the wavefunction is related to the scattering phase shift and energy derivative then gives the linear change in this scattering as the energy varies from the atomic energy. The right hand side is the norm-conservation condition in a shell of radius R. This means that norm conservation reduces the error in scattering behavior when using these pseudopotentials in different contexts than the bare atom they were defined from.

1. D. R. Hamann, M. Schlüter, and C. Chiang Phys. Rev. Lett. 43, 1494 (1979)