14
$\begingroup$

I am trying to make plots based to VASP simulation results, but I am struggling with how to download so much data from the server. A lot of examples on the internet are taught using the local desktop environment, which has a GUI like GNOME. They could use Python to make plots easily, but when I need to run some serious simulation, I would like to do it on an HPC cluster, which doesn't have a GUI.

Do I have to download all the data to my laptop, or are there some other solutions?

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ If you are looking for simple GUI to interface with your HPC cluster a simple solution could be to use WebMO(webmo.net). Its pretty good for molecular systems and can be interfaced with many ab-initio software that are installed on your cluster. (I never tried it for materials extensively but I guess it should work with VASP and Espresso) $\endgroup$
    – user784
    Oct 13, 2021 at 4:53
  • 2
    $\begingroup$ WebMO helps with managing jobs you submit and to a good extent with visualization. If you have output in text files(or .dat files), you could use gnuplot for plotting. To post process your data, you could use python(jupyter notebooks hosted on your server itself, more power!) $\endgroup$
    – user784
    Oct 13, 2021 at 4:57
  • 1
    $\begingroup$ If you are not comfortable with terminal, may be you could try using VSCode and ssh to your cluster. That way you could open and see your text file easily in the editor. Also, you could use it also for writing jupyter notebook or any other post processing tools. $\endgroup$
    – user784
    Oct 13, 2021 at 5:00
  • 1
    $\begingroup$ @EverydayFoolish why don't you write an answer? $\endgroup$
    – Nike Dattani
    Oct 13, 2021 at 5:30
  • 2
    $\begingroup$ @EverydayFoolish I agree with Nike, these comments are probably better collected into an answer. $\endgroup$
    – Tyberius ♦
    Oct 13, 2021 at 14:18

3 Answers 3

Reset to default
15
$\begingroup$

I prefer to just download the data from the remote disks and then do the plots on my own local machine, which has all of my favorite tools for post-processing my data.

I understand that this can be inconvenient though. It is indeed possible to run a GUI on a remote supercomputer though, if you first use ssh -X or ssh -Y when signing into the remote machine (The difference between the -X and -Y options is the degree of trust you give the X11 clients). The reason I don't usually do this, is because viewing on my local computer, a GUI that's running on a remote machine, can sometimes be painfully slow, but this will depend on many factors (speeds of the internet connections involved, age of the hardware involved, etc.), so you might at least like to try it once. From a Windows computer, you can run an X11 server such as Xming.

Personally though, I watch my data as it's being calculated, using the awk command; and if I want to make a publication-quality plot, I download the data to my local machine and do all post-processing and/or plotting offline. If you post the exact output from VASP and what precise numbers you want in your plot/visualization (with a question title like "how can I visualize the following data from the command line?"), I can try to give you an awk command that would work for your case.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Could you tell me more about how to use the awk to watch the data as it is being calculated, I am not sure how to do it? And is it necessary that I tar all the data into a single file before I download them? $\endgroup$
    – Jack
    Oct 11, 2021 at 18:18
  • 1
    $\begingroup$ @Jack It's not necessary to tar all the data into one file, but it can have some benefits (speed of file transfer often being one of them). If you ask a new question about using awk command to watch data, and you provide your specific output file and which numbers in it that you want to watch, I can try to help you. Today is a holiday in my country (Thanksgiving in Canada) so I might not be able to reply immediately, but I'll try to reply as soon as I can. $\endgroup$
    – Nike Dattani
    Oct 11, 2021 at 18:22
  • 1
    $\begingroup$ Thanks a lot for the help. I will also try to organize my need and make it neat. I really appreciate your kindly help, Happy Thanksgiving! $\endgroup$
    – Jack
    Oct 11, 2021 at 18:28
  • 2
    $\begingroup$ @Jack I watch my data during calculation using tail -f yourlogfilename.log $\endgroup$
    – Vasista
    Oct 12, 2021 at 16:00
  • $\begingroup$ @NikeDattani In some cases downloading output files could become cumbersome especially, if you HPC admin has setup some security measures that might prevent upload and download of heavy files from the remote instance. $\endgroup$
    – user784
    Oct 13, 2021 at 5:02
13
$\begingroup$

It would probably be easier to use some software such as WinSCP. It allows you to select the location on your remote system and synchronise with the local system by matching either the time stamp or size (and probably other properties). Also if you are using Linux, read the answers on this Stack Overflow question to get an idea of alternatives. This way you can visually manipulate the required files in a manner much similar to what is done in a tutorial.

Note: If possible, ask the system administrator to install xrdp (and other required packages which I am sure he would know) which is similar to Windows RDP and allows you to connect to a GUI instance of the server.

Improve this answer
$\endgroup$
5
$\begingroup$

To address your specific question: No, you don't always have to download everything locally. VASP creates a lot of different output files and sometimes (most of the times?) you only need some of those. So you can delete large output files you don't need in your study right after the calculation on the cluster and then download only the necessary files.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .