I am aware of software tools such as atomate, but that is quite specific to crystals. There are other equally specific workflows - I think we all have our own...
What I am interested in is making my own database that stores solid, liquid and gas state properties. I don't need help with the workflow, or organizing the workflow, but I will say I use python to run all of the workflow.
What I am after right now, is how do I save all of the outputs in a proper database form? Should I be using pandas and saving in an excel sheet, or saving in JSON?
The general scheme is that I will be calculating properties for many molecules, for instance, crystal structures or liquid state free energies of solvation. Each molecule has several properties, and these will also be added too. Some properties are scalar i.e, free energy of solvation, but others may be lattice energy using B3LYP, or PBE0 etc, so I need to be able to store almost a dictionary, just for this one property.
So I am a bit lost about an efficient means for storing all of this in a robust manner... JSON, pickle file, csv... store each molecule/structure as an independent file, or all together in one big file...
Sometimes I will just want to look up the properties for 1 molecule, othertimes for an array. Sometimes I will want to perform calculations using an array, so it may be a nuisance to load 1000 individual files etc.
For instance I see that atomate makes use of MongoDB as its database... is this a good route?