General Techniques for Smart Sampling in Matter Machine Learning?

Question

This question is somewhat broad, but hopefully I can convey my point and elicit some worthwhile discussion.

One of the fundamental difficulties of machine learning is trying to develop a model that will work beyond your training set. The issue is that it has proven very challenging to develop a machine learning model that can extrapolate (e.g. derive insight about unseen, "outer" regions of the input space, with "outer" meaning input parameters larger/smaller than any in the test set). So instead, one often tries to develop the training set such that they can interpolate (e.g. derive insight about unseen, "inner" regions where the input parameters are between ones from the training set). However, the topography of a given input space and how it relates to some output property are precisely the problems one is hoping to solve with machine learning!

So my question is, what are some general techniques that Matter Modelers use to "smartly" sample input? What sort of intuition can we apply to ensure our training set has a wide boundary and that we are interpolating, rather than extrapolating, when applying our model to a test set? For some properties, this is fairly intuitive. For example [1], in developing a training set to model a potential energy surface, chose their training set by iteratively adding points that maximized the minimum distance from any existing point in the set. Here, the sample space is over physical distance, so there is an intuitive way to determine a boundary. But what sort of guidelines are available to make this selection more generalizable to other types of input without as clear a notion of distance?

References:

1. Dral, Pavlo O.; Owens, Alec; Yurchenko, Sergei N.; Thiel, Walter J. Chem. Phys. 146, 244108 (2017); DOI:10.1063/1.4989536

Answer 1

This is not an exhaustive answer. This is an evolving research area in terms of applying ML to dataset generation. I am most familiar with the use case for constructing atomistic potential energy surfaces.

The most common techniques I have seen all fall under the category of active learning. The goal of active learning in this context is essentially to determine whether or not a newly proposed structure for a dataset is redundant. Of course, no structure will be truly redundant, so the algorithm is essentially building a continuum that discriminates the similarity between structures in terms of how much new information they bring to the dataset. If a structure brings a lot of new information, then the uncertainty in the current models prediction should be large.

There are many ways one can train the agent in an active learning model. I am not an expert on any of them, so I'll just give some references and a description of my understanding.

In Ref. [1], a Gaussian process regression model is used to determine which configurations should be sampled for a many-body potential energy surface (PES). That is, one would have a separate fit for the 1-body, 2-body, etc. pieces of the PES, and these are combined via the many-body expansion. The way this model works is by having the active-learning model try to estimate the uncertainty in the prediction of the model which has been fitted to some initial dataset. So, a good candidate structure to add to the dataset is one which has a large uncertainty in the prediction (likely because it falls in the extrapolation category, rather than the interpolation category).

In Ref. [2], the criteria for including a new data point in the training set is determined using a model called query-by-committee. What I have gathered is that one keeps around many different models which are generated from different subsets of the total available dataset. That detail could be wrong, but in any case, there is some way in which you keep around multiple similar models. One then proposes a new candidate structure, and each model labels that structure (predicts the energy/forces in the context of a PES). Then, the new structure is added to the dataset based on a parameter which related to the standard deviation of the predicted labels. Again, higher standard deviation means you want to add that structure more.

Ref. [3] is sort of an on-the-fly application of any active learning method, really, in which the new structures are determined via a molecular dynamics simulation (based on an ab initio method in this case). So, one uses the active learning method to determine if the new structure is similar to old structures, and if so, then you use a trained model to propagate the dynamics, and otherwise you do the full ab initio evaluation, and add the new labeled data to the training set. I guess part of the simulation is pausing to re-train the model on-the-fly, but I imagine it would be more efficient to do this in batches or something. So, this is basically a way that one can start out doing AIMD and end up doing classical MD with an ab initio trained PES. That will be quite cool if this works out in the long-term. I can imagine some interesting things you can do if you have a split CPU/GPU architecture for doing the training and force evaluation in parallel without having to fully stop either at any point (then like rewinding the dynamics at certain points).

Ref. [4] is another example of a similar procedure where the researchers refine a neural-network potential on-the-fly using a committee-based approach. In this case, they definitely do random sub-sampling of an initial ab initio simulation of water, and then train many neural network potentials, all of the same form, and then determine the uncertainty in a prediction based on the standard of all the predictions from the members of the committee, which are the various NN potentials.

References:

1. Zhai, Y.; Caruso, A.; Gao, S.; Paesani, F. Active learning of many-body configuration space: Application to the Cs+–water MB-nrg potential energy function as a case study. J. Chem. Phys. 2020, 152 (14), 14410
2. Smith, J. S.; Nebgen, B.; Lubbers, N.; Isayev, O.; Roitberg, A. E. Less is more: Sampling chemical space with active learning. J. Chem. Phys. 2018, 148 (24), 241733. DOI: 10.1063/1.5023802.
3. Jinnouchi, R.; Miwa, K.; Karsai, F.; Kresse, G.; Asahi, R. On-the-Fly Active Learning of Interatomic Potentials for Large-Scale Atomistic Simulations. J. Phys. Chem. Lett. 2020, 11 (17), 6946–6955. DOI: 10.1021/acs.jpclett.0c01061.
4. Schran, C., Brezina, K., & Marsalek, O. (2020). Committee neural network potentials control generalization errors and enable active learning. arXiv preprint arXiv:2006.01541.

Answer 2

There are several tricks to improve the prediction that your neural network or regressor/classifier makes :

1. To select the appropriate features from the input space. Say if youre input space has around 100 features and you want to determine 10 features that affects youre output the most. This can be done by

   i) dimensionality reduction via feature extraction: In this, you would be taking combinations of existing features so that it best represents your data.

   eg: Principle Component Analysis, Linear Discriminant Analysis .. etc

   ii) dimensionality reduction via feature elimination: This is more simple in terms of implementation and the features do retain their physical interpretability. Here you would drop the least important features so as to keep the most important ones :

   eg: Lasso Regression.

2. To apply regularization: This involves constraining your weights to a specific range provided you know what that range is. This can also be done automatically by using a constraint parameter and tuning it via trial and error on your validation set.

   eg: Ridge regression, Ridge classifier

3. To apply drop-out: Choosing to switch on/off certain nodes with a fixed probability will help in reducing overfitting.

eg: dropout-regularization.

And as an overall idea, the lack of generalization of you're neural network is mainly due to poor hyperparameter optimization. That's why for training and tuning the network, you would use the training and validation set respectively. After which the network is tested on the testing set.

Hope this gives an insight :)