I am aware of software tools such as atomate, but that is quite specific to crystals. There are other equally specific workflows - I think we all have our own...

What I am interested in is making my own database that stores solid, liquid and gas state properties. I don't need help with the workflow, or organizing the workflow, but I will say I use python to run all of the workflow.

What I am after right now, is how do I save all of the outputs in a proper database form? Should I be using pandas and saving in an excel sheet, or saving in JSON?

The general scheme is that I will be calculating properties for many molecules, for instance, crystal structures or liquid state free energies of solvation. Each molecule has several properties, and these will also be added too. Some properties are scalar i.e, free energy of solvation, but others may be lattice energy using B3LYP, or PBE0 etc, so I need to be able to store almost a dictionary, just for this one property.

So I am a bit lost about an efficient means for storing all of this in a robust manner... JSON, pickle file, csv... store each molecule/structure as an independent file, or all together in one big file...

Sometimes I will just want to look up the properties for 1 molecule, othertimes for an array. Sometimes I will want to perform calculations using an array, so it may be a nuisance to load 1000 individual files etc.

For instance I see that atomate makes use of MongoDB as its database... is this a good route?

  • 3
    $\begingroup$ This is probably better for stackoverflow but we can leave it for now. I think someone will come along $\endgroup$ Jun 4, 2021 at 14:09
  • 3
    $\begingroup$ I think there are people in modeling who have had to address this problem in one way or another. It might be worth looking into how NIST organizes their Computational Chem Database. If the question doesn't get any traction here, I think Software Engineering SE would be your next best bet. $\endgroup$
    – Tyberius
    Jun 5, 2021 at 18:38
  • 2
    $\begingroup$ Getting to the actual question itself, I think the structure of your database will depend on the relative importance you give to accessing several properties of one molecule at a time versus a single property for many molecules at once. The same goes for whether you are more interested in comparing amongst different basis sets/functionals or same level of theory with different molecules. The best hierarchy for arranging the data is going to depend on what use cases you find most important. $\endgroup$
    – Tyberius
    Jun 5, 2021 at 18:42
  • 3
    $\begingroup$ JSON, pickle, csv, these are not databases, but file formats. There are different types of databases, but MongoDB is a relatively flexible solution if you are not sure which to choose (eg Materials Project is based on Mongo). Databases have their own query language, but the most popular ones has a Python interface, too, eg pyMongo for MongoDB. $\endgroup$
    – Greg
    Jun 28, 2021 at 4:10
  • 1
    $\begingroup$ Possibly helpful recent paper. Not the same features that you are interested in, but seems to be some discussion about what makes for a good format for allowing quick, interactive access. $\endgroup$
    – Tyberius
    Dec 6, 2021 at 1:45


Browse other questions tagged .