I want to perform
HSEsol (PBEsol + HSE06) calculation on a material (monolayer) that doesn't have experimental data available. Could you please tell me how to implement this in the
INCAR file for relaxation purposes ?
You want the following flags in your INCAR to specify HSEsol:
GGA = PS LHFCALC = .TRUE. HFSCREEN = 0.2
I also recommend setting
Algo = All for smoother SCF convergence. The VASP manual suggests setting
LASPH=.TRUE. as well for hybrid calculations, but that's a decision for you to make.
In the future, if you are ever curious about how to define a given density functional, you can check out all the pre-defined recipes in the Atomic Simulation Environment. The example for HSEsol is here.