16
$\begingroup$

Doing matter modeling generates vast amounts of data. We might most probably be working on different projects simultaneously. We definitely will have to revisit old data for revisions, post-processing, further studies, etc. Also, we might have to go through many failed calculations to reach the final results. It is always a good practice to keep all our data even after publication. However, its quite easy to lose track of our calculations (especially looking at old calculations) unless we keep them systematically.

In my case, I have been using Medea-VASP, a propriety software with a tool called JobServer which efficiently manage the calculations and data. Here is a screenshot:

Enter image description here

How should I handle a huge number of calculations (data) efficiently and systematically while using matter modeling code?

Improve this question
$\endgroup$
2
  • $\begingroup$ Are you just looking for more tools like Medea-VASP? For example in one-topic-per-answer format? $\endgroup$
    – Nike Dattani
    May 27, 2021 at 22:40
  • $\begingroup$ Yes and No. I was wondering how people manage their vast amount of data. How to manage my data in environment without Medea VASP using tools codes like Quantum ESPRESSO and ABINIT $\endgroup$
    – Thomas ♦
    May 28, 2021 at 1:47

2 Answers 2

Reset to default
16
$\begingroup$

Several computational workflow managers try to address exactly this problem in materials modeling. Besides helping orchestrating complex simulation workflows, they keep track of all the simulations (and how they are related) in a database.

The Materials Project team uses the Fireworks package. The atomate package provides a high-level interface for the most common tasks.

Another widely used workflow manager is AiiDA. AiiDA focuses on keeping track of the full provenance of your simulations. That is, it will allow you to created directed acyclical graphs that go from some input like a crystal structure in a database to some complex properties like thermoelectric coefficients. You can find some examples for these graphs in the Materials Cloud. The power is that you can then write queries to find specific parts of the database, for example, all calculations with a specific functional, a bandgap > 3.2 eV, and that were performed in a certain time range.

The awesome materials informatics list list some materials specific solutions, the wiki of the common workflow language lists some more general solutions.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks @Kevin Jablonka. I agree atomate.org or AiiDA are the way to go. I'm commenting here as an atomate developer -- atomate is particularly nice if you're using VASP, since we have extensive VASP integration. The key feature for organization is a "drone" which can ingest a calculation directory and output a single JSON document which can be stored and queried in a database (atomate does this automatically, as well as orchestrating the running of calculations). We do all of our analysis at Materials Project in this way and it's proven very productive so far. $\endgroup$
    – Matt Horton
    Jun 3, 2021 at 21:24
10
$\begingroup$

The answer by Kevin Jablonka is much more specific to matter modeling, but I wanted to point out that the screenshot example from Medea-VASP shown in the question, can be imitated almost exactly with most general schedulers (not specific to VASP) such as SLURM, PBS, LoadLeveler, Oracle Engine Grid, etc.

The squeue command in SLURM will give output very similar to Medea-VASP, with the following correspondence table:

Medea-VASP SLURM
Job JOBID
User USER
Queue PARTITION
Name NAME
Status ST

Here is an example of SLURM output (more information such as time started, time ended, etc. can be obtained by adding the appropriate options for squeue):

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ To use this, we have to install and use a queued system right? What about bot using any of them? $\endgroup$
    – Camps ♦
    Jun 4, 2021 at 13:07
  • $\begingroup$ Yes, a system called a "scheduler" was used for the screenshot. $\endgroup$
    – Nike Dattani
    Jun 4, 2021 at 14:09

You must log in to answer this question.