I have been using molecular fingerprints like ECFP (extended connectivity fingerprint), APFP (atom-pair fingerprint) etc. in my research to predict spectral properties of organic molecules with machine learning (basically a QSAR project). Now, I have got very good results from the machine learning models, but I want to understand what exactly the model is learning i.e. some kind of interpretable chemical information which can be understood by an organic chemist for example.
However, I am not sure how to do that, because from the trained model, all I can possibly get are importances or effects of each position of the fingerprint. But then I am lost because fingerprints like ECFP are hashed fingerprints, so I do not know what the number in each position actually means. Is there any way to link hashed fingerprints to some chemical structure or feature that is easy to understand?
I only found this one paper that attempts to reverse-engineer ECFP with neural networks, but I am looking for something simpler.