I built a prediction model with fingerprints from 300 molecules and got an R2 of 0.9 However when I go to perform a cross-validation I get a very low R2. How can I improve this result? I'm using sklearn and PaDEL descriptors, maybe I'm doing something wrong though. Where can I learn how to do it correctly?
You don't give a lot of details (e.g., type of model, target property, etc.), but with only 300 molecules, it's very likely that your model is over-fit. Thus when you cross-validate, you get low $R^2$.
Consider that each descriptor you add is at least one parameter. You'll want more data than you have parameters.
- You want to look for more data if possible
- You'll want to use LASSO or other method to remove descriptors that don't add much information
- You'll want to use cross-validation (e.g., 5-fold or 10-fold CV) from the start to minimize the chance of an over fit model
But you also should think carefully about descriptors. Consider whether the descriptors you use would be meaningful for the model you want.
Let's say I wanted a model for solubility prediction. I can think of a few properties / descriptors that might be important:
- Molecular size (molecular weight or volume)
- Molecular shape (like dissolves like)
- Hydrogen bond donors and acceptors
- Dipole moment / electrostatics
Yet for other properties (e.g., redox potential) most of these are probably less relevant.
If you want a good model, you need to be careful with what descriptors you use.