I recommend reading "Electronic Structure Modeling of Metal–Organic Frameworks" by Mancuso and coworkers for an overview of some subtle considerations when carrying out electronic structure calculations of MOFs.
If I want to relax the geometry of the MOF materials, which algorithm
do I take (ISIF=?) in VASP?
There is no single best answer for this. In prior work, I had tested a variety of optimization algorithms for a diverse set of MOF structures. Empirically, I found that using the conjugate gradient (CG) algorithm (ibrion=2) is generally both robust and fairly efficient in most cases. When the forces get small (i.e. you get close to the local minimum in the potential energy surface), the CG algorithm can often fail though with MOFs. In these cases, I have empirically found that switching to FIRE (iopt=7, ibrion=3 with VTST) is generally the best choice. I have used CG $\rightarrow$ FIRE to optimize tens of thousands of MOF structures with great success. The only exception is if the forces are extremely high in the initial geometry optimization steps. In these cases, I have empirically found that using ASE's BFGSLineSearch algorithm for a few steps does a nice job at smoothly resolving the high forces without causing the structure to "explode" or anything like that.
As a side-note, you mentioned the isif flag in your question. That does not choose the type of geometry optimization algorithm but rather which degrees of freedom to relax. In general, you should ensure that your material is the minimum energy structure with respect to the atomic positions and cell shape/volume. In other words, use isif=3. Typically, when modeling adsorbates with MOFs, the guest-free MOF structure will be optimized with isif=3 and then the lattice constants fixed thereafter (i.e. isif=2 is used). This inherently assumes that the lattice constants of the MOF do not change when adsorbates are introduced, which is typically (although not always) true.
If I want to perform a self-consistent calculation with VASP, what do
I need to take care of?
In general, nothing special has to be done for MOFs. If you're using VASP as you mention, ensure that your property of interest is converged with respect to the plane-wave kinetic energy cutoff and $k$-point grid. Make sure you enable spin-polarization if there is any possibility of unpaired electrons in your system. Beyond this, you may find my prior answer to the question "What are good ways to reduce computing time when working with large systems in VASP?" to be helpful.
If I want to study the absorption of MOF on atoms, what do I need to
take care of?
You will need to ensure that you consider various adsorption modes for a given adsorbate. If you don't do this, you may end up modeling a configuration that is not the minimum energy configuration. If the adsorbate induces a change in oxidation state of the MOF, keep in mind that your typical GGA functionals like PBE will probably be fairly insufficient. You can check out this article for some suggestions about modeling the adsorption of species that oxidize metal centers of MOFs. Also, in DFT, people often only include one adsorbate per unit cell, but in reality you should remember that it is very possible multiple adsorbates would be present and there could be adsorbate–adsorbate interactions or coverage-dependent effects.
To systematically add adsorbates to MOFs, you may be interested in the MOF Adsorbate Initializer for small molecule adsorbates or the MOF Big Adsorbate Initializer for larger ones.
