6
$\begingroup$

In computational material science, we need workflows for optimization surrogate models which requires high computation resources. I am actually concerned with why material science community is using FireWorks compared to highly supported workflow management package such as TensorFlow?

$\endgroup$
1
  • 3
    $\begingroup$ What is your question? $\endgroup$
    – Cody Aldaz
    Sep 11 at 23:12
7
$\begingroup$

Tensorflow and FireWorks are different kinds of software. The "workflow management" features of Tensorflow are primarily designed to manage running Tensorflow itself, while FireWorks and related tools are designed to manage running other software.

Tensorflow is a library for machine learning. It can be used to develop, train and evaluate machine learning models and provides some tooling to build model and data pipelines. Some comparable software to Tensorflow are PyTorch, Flux.jl, and Aesara (formerly Theano). Environments like Wolfram Mathematica and MATLAB also have comparable machine learning tools.

On the other hand, FireWorks is a workflow management software. Some comparable software tools that are also general, not domain-specific are Luigi and Apache Airflow. These workflow management tools typically allow you to define and execute a directed graph of tasks with dependencies. The tasks can be anything from running code, shell scripts, running tasks that modify the workflow graph, and more. Workflow management tools are not used to build mathematical models, like the surrogate models you mentioned in your question.

In matter modeling, workflow management tools are used to orchestrate running a series of calculations in a specific order, enable passing relevant information between jobs, and to provide a high-level interface for managing job state. Some more matter modeling-specific examples are discussed here.

$\endgroup$
5
  • $\begingroup$ quite comprehensive, but can you please clarify Tensorflow are primarily designed to manage running Tensorflow itself, while FireWorks and related tools are designed to manage running other software. As per my understanding Tensorflow deals with building and executing a computational graph "workflow". Where in FireWorks a single python program,"Firework". While FireWorks manage multiple individual Firework. So how exactly do they differ? If you can elaborate on that as well that shall be great! $\endgroup$
    – gfdsal
    Sep 12 at 0:29
  • $\begingroup$ In workflow management software like FireWorks, you typically run third party software. FireWorks is a "single Python program", but it's purpose is to run other programs. For example, FireWorks is used in atomate to manage running VASP, an external DFT software. An individual Firework is the abstraction for this. The ScriptTask object is a Firework designed for running a program available on the command line. $\endgroup$ Sep 12 at 1:05
  • $\begingroup$ ok its making sense now. Did you mean "Firework" not "Fireworks" is a single python program? $\endgroup$
    – gfdsal
    Sep 12 at 1:18
  • 2
    $\begingroup$ By third party software I mean any software. That can include other Python libraries, non-Python software, command line scripts, ..., anything. That is the fundamental difference between workflow management software and something like Tensorflow's concept of workflows. I suggest also checking out the Luigi documentation if you are having trouble understanding the concept of workflows in the context of job management software. Luigi is more widely used and the documentation reflects a lot more use cases. $\endgroup$ Sep 12 at 1:26
  • 4
    $\begingroup$ Just in case it's useful, the common workflow language guys maintain a very extensive list of workflow management systems: s.apache.org/existing-workflow-systems (approaching 300 at the time of writing) $\endgroup$ Sep 12 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.