This is actually an important question regardless of the code you're using (I use SIESTA and NWChem).
To obtain a good phonon spectrum, the first thing you need is a truly relaxed/optimized structure. This requirement entails several parameter optimizations (mesh cutoff, force tolerance, basis set ..). Afterwards, other requirements are also important. I will address them while answering your specific queries.
- For the DFPT or finite displacements question, it has actually been answered before here :
Are there differences in accuracy and reliability between the frozen phonon method and Density Functional Perturbation Theory?
Here is a quote from that answer that might be all you need :
In summary, these two methods lead to comparable accuracies. If DFPT is available, then the calculations will be cheaper and DFPT should be the method of choice"
For the supercell size, the general rule is of course the larger the better, of course time and computational costs will play a role here (just as they would in choosing your optimization/relaxation parameters in the first step). Here is a short post on ResearchGate that tackles this same question.
The Pseudopotential question has also already been answered. Here is the link for that answer: What is the difference between Ultrasoft, ONCV and PAW Pseudopotentials? Which is better for a spin-orbit coupled calculation?
Let me add also a quote from that answer :
There is no systematic manner for deciding which pseudopotential is "better" as a pseudopotential that works well for obtaining one property may not work well for another property
- For the VASP specific enquiries, it would probably be better for you to look into those in the VASP documentation and actually edit this answer to add them so that we could have a complete account of all the parameters.
Here is some advice when doing phonon calculations where you obtain several negative (or imaginary, depending on the code you are using) modes:
- Take the lowest of these modes and displace the atoms along their respective eigenvectors. As this should reduce the total energy of your system and since you don’t know in which sense to displace, try both. From whichever displacement gives you lower energy, you can restart a new structure relaxation run followed by phonon spectrum.
- Check the accuracy of your atomic masses.
Finally here is an article, which covers your issue specifically but in a 3D structure instead of a monolayer.
- Evarestov, R. A.; Losev, M. V. All-electron LCAO calculations of the LiF crystal phonon spectrum: Influence of the basis set, the exchange-correlation functional, and the supercell size. J. Comput. Chem. 2009, 30 (16), 2645–2655. DOI: 10.1002/jcc.21259.