Extracting data from a file, using python or bash. Requesting help

Question

I have some directories, named ‘abc-000, abc-001, abc-002, …… abc-100’. The path of the directory can be, for example, \home\abc-000 and so on.

Inside each directory, there is a file named ‘OUTCAR’. The ‘OUTCAR’ file consists of many lines including the following:

It's noted that, this paragraph appears 5 times with same headings (but with different values) in the same 'OUTCAR' file, and we need to extract data from the final (last) paragraph only.

Now, the 9 elements of the stress matrix can be represented by the following symbols:

I need to use a python or bash code that will read each of those ‘OUTCAR’ files, and then it will save the data from those ‘OUTCAR’ files in one single text or excel file (named: stress.txt or stress.xlsx) in the following way:

abc-000    ii  ij   ik  ji  jj   jk   ki  kj   kk   ForceTotal    StressTotal

abc-001    ii  ij   ik  ji  jj   jk   ki  kj   kk   ForceTotal    StressTotal
…………………………………………………….…………………………………………………………………………………………….……………………………………………………
…………………………………………………….…………………………………………………………………………………………….……………………………………………………
abc-100    ii  ij   ik  ji  jj   jk   ki  kj   kk   ForceTotal    StressTotal

As I am new in coding, can anyone please suggest me how to do it? Any kind of help is greatly appreciated.

View and/or download link of the OUTCAR file: click here

Answer 1

I have condensed the bash script by using arrays.

for x in 'abc-*/';
do
read a[{1..9}] <<<$(cat ${x%?} | grep "stress matrix" -A6 | tail -6 | head -3 | xargs)
read b[{1,2}] <<<$(cat ${x%?} | grep "stress matrix" -A6 | tail -2 | awk '{print $6}'| xargs)
echo ${x%?} ${a[*]} ${b[*]} >>test.csv
done

test.csv shows the same results

abc-000 -467.71245 24.34138 7.71803 24.34138 -314.11534 -3.45455 7.71803 -3.45455 -371.79956 0.039198 676.007623

Answer 2

This is a basic bash script that can be used. One can make this code better too

for fold in `ls -d test-*/`; 
do
line_no=`grep -n 'stress matrix after NEB' OUTCAR|tail -n 1|cut -d ':' -f1`
sig11=`sed -n "$((${line_no}+1))p" OUTCAR|awk '{print $1}'`
sig12=`sed -n "$((${line_no}+1))p" OUTCAR|awk '{print $2}'`
sig13=`sed -n "$((${line_no}+1))p" OUTCAR|awk '{print $3}'`
sig21=`sed -n "$((${line_no}+2))p" OUTCAR|awk '{print $1}'`
sig22=`sed -n "$((${line_no}+2))p" OUTCAR|awk '{print $2}'`
sig23=`sed -n "$((${line_no}+2))p" OUTCAR|awk '{print $3}'`
sig31=`sed -n "$((${line_no}+3))p" OUTCAR|awk '{print $1}'`
sig32=`sed -n "$((${line_no}+3))p" OUTCAR|awk '{print $2}'`
sig33=`sed -n "$((${line_no}+3))p" OUTCAR|awk '{print $3}'`
total_force=`grep 'FORCE total' OUTCAR|tail -n 1|awk '{print $6}'`
total_stress=`grep 'Stress total' OUTCAR|tail -n 1|awk '{print $6}'`

echo ${fold%?} ${sig11} ${sig12} ${sig13} ${sig21} ${sig22} ${sig23} ${sig31} ${sig32} ${sig33} ${total_force} ${total_stress}>> result.csv
done

results.csv

test-000 -467.71245 24.34138 7.71803 24.34138 -314.11534 -3.45455 7.71803 -3.45455 -371.79956 0.039198 676.007623

Answer 3

Python3 alternative

You can save this to a .py file. I wrote and tested it under Python 3.10.6:

'''
Seeks a folder and all subfolders it contains for files named 'OUTCAR', extracting data given by regexp pattern. Saves
data to CSV spreadsheet. Assuming this program is saved to a file named outcar2csv.py, is invoked from the linux command
line like this:

python3 outcar2csv.py data_folder_path yourfile

Where data_folder_path is the path to the folder to be recursively searched for files named 'OUTCAR', and yourfile is
the name of the csv file to be created to save the extracted data. This file is created in the same place where
outcar2csv.py is located.
'''

import os
import re
import mmap
import csv
import time
import sys

def process_file(file, tags, pattern):
    '''
    Function seeks regexp pattern in given file, returning data from the last match found as a
    dictionary of tag : data.
    '''
    with open(file, 'r') as f:
        m = mmap.mmap(f.fileno(), 0, access = mmap.ACCESS_READ)
        last_match = pattern.findall(m)[-1]
        return {tag : data.decode() for tag, data in zip(tags, last_match)}

def row_saver(destination_file, tags):
    '''
    Generator that saves data one row a time in a CSV spreadsheet, where each row is
    passed as a dictionary compatible with DictWriter object from Python csv module.
    '''
    with open(destination_file, 'w', newline = '') as csvfile:
        fieldnames = ['folder']
        fieldnames.extend(tags)
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect='unix')
        writer.writeheader()
        while True:
            writer.writerow((yield))

def main():
    top_folder, destination_file = sys.argv[1:]
    
    t0 = time.time()

    pattern = re.compile(b"""
    \s stress \s matrix \s after \s NEB \s project \s \(eV\)
    \s+ (-?\d+\.\d+) \s+ (-?\d+\.\d+) \s+ (-?\d+\.\d+)
    \s+ (-?\d+\.\d+) \s+ (-?\d+\.\d+) \s+ (-?\d+\.\d+)
    \s+ (-?\d+\.\d+) \s+ (-?\d+\.\d+) \s+ (-?\d+\.\d+)
    \s+ FORCES: \s max \s atom, \s RMS .*
    \s+ FORCE \s total \s and \s by \s dimension \s+ (-?\d+\.\d+) .*
    \s+ Stress \s total \s and \s by \s dimension \s+ (-?\d+\.\d+) .*
    """, re.VERBOSE)

    tags = ['ii', 'ij', 'ik',
            'ji', 'jj', 'jk',
            'ki', 'kj', 'kk',
            'ForceTotal', 'StressTotal']

    saver = row_saver(destination_file, tags)
    saver.send(None)

    how_many = 0
    for dirpath, dirnames, filenames in os.walk(top_folder):
        for name in filenames:
            if name == 'OUTCAR':
                how_many += 1
                file = os.path.join(dirpath, name)
                row = {'folder': os.path.split(dirpath)[-1]}
                row.update(process_file(file, tags, pattern))
                saver.send(row)

    print(how_many, 'files processed in', round(time.time() - t0, 3), 'seconds.')

if __name__ == '__main__':
    main()

I created some mock data by modification of the OUTCAR file given, for testing it. The script can be used as exemplified below:

user@machine:~$ ls Downloads/mattermodelling
mockdata  outcar2csv.py
user@machine:~$ cd Downloads/mattermodelling
user@machine:~/Downloads/mattermodelling$ ls mockdata/
abc-000  abc-002  abc-004  abc-006  abc-008  abc-010  abc-012  abc-014  abc-016  abc-018
abc-001  abc-003  abc-005  abc-007  abc-009  abc-011  abc-013  abc-015  abc-017  abc-019
user@machine:~/Downloads/mattermodelling$ python3 outcar2csv.py ~/Downloads/mattermodelling/mockdata/ spreadsheet.csv
20 files processed in 0.377 seconds.
user@machine:~/Downloads/mattermodelling$ ls
mockdata  outcar2csv.py  spreadsheet.csv
user@machine:~/Downloads/mattermodelling$ cat spreadsheet.csv 
"folder","ii","ij","ik","ji","jj","jk","ki","kj","kk","ForceTotal","StressTotal"
"abc-007","-862.37367","50.67924","2.91086","76.73190","-612.12661","-4.15208","7.69324","-6.63256","-243.32872","3.217959","318.071499"
"abc-008","-491.44190","89.30117","6.58274","19.24851","-816.34700","-6.96243","1.26080","-2.61234","-743.36987","5.135049","572.092093"
"abc-016","-387.31784","93.66937","6.30587","52.19730","-605.28808","-2.26975","3.81523","-4.72937","-675.31002","5.550989","057.487611"
"abc-014","-809.19049","19.72927","3.31798","06.07626","-868.67673","-9.00880","8.43166","-1.77182","-047.79519","9.173462","527.617069"
"abc-001","-101.10447","65.32618","0.96934","10.07195","-343.88294","-0.12684","1.07403","-6.83529","-223.11711","1.132396","726.216061"
"abc-005","-458.47658","91.04737","9.53896","51.30511","-523.23447","-1.07006","0.63547","-9.55906","-425.59611","0.189480","709.950391"
"abc-009","-009.53367","71.75084","4.66395","11.19572","-466.63143","-2.06748","8.65482","-2.91662","-765.26647","5.337915","378.528833"
"abc-011","-089.14072","33.43878","6.52117","05.15536","-329.99575","-9.70646","9.67780","-6.04529","-632.04250","0.986787","539.275902"
"abc-003","-226.22536","85.58539","5.83980","27.83603","-170.95152","-5.91629","6.88458","-9.64086","-432.43266","9.142282","639.281772"
"abc-004","-426.08425","31.80692","1.78031","24.45488","-018.67193","-7.11674","3.13194","-9.94171","-751.95429","7.610343","967.272264"
"abc-015","-676.72436","21.50989","1.36015","04.46427","-937.33530","-4.04833","8.32037","-1.46997","-212.69520","8.042924","133.224298"
"abc-019","-230.94008","12.00516","7.90733","29.98929","-777.89241","-0.80954","5.72769","-6.88261","-671.07960","6.707047","524.685876"
"abc-000","-688.85233","72.06209","0.19069","50.47013","-548.46840","-7.30824","5.13827","-0.69694","-401.06181","7.933544","001.298579"
"abc-018","-656.39700","23.52044","3.80865","31.69298","-436.68556","-9.85304","2.57257","-0.40098","-006.07856","8.912860","482.683707"
"abc-006","-885.79980","39.81359","5.79319","30.19686","-484.66123","-2.12213","4.80344","-5.07017","-203.31291","0.387975","415.977214"
"abc-010","-533.00892","01.85543","5.31156","92.66351","-685.26155","-2.79869","5.84854","-3.74227","-366.89325","8.122444","167.212737"
"abc-002","-885.29241","08.40123","3.89631","30.16682","-058.96081","-2.43187","0.28793","-9.48565","-317.18443","5.285536","932.580443"
"abc-013","-574.77203","38.14837","6.74880","98.88899","-500.02623","-5.83326","9.87984","-3.23035","-892.95830","3.727332","414.234494"
"abc-012","-663.11922","47.40477","5.91988","23.31203","-854.46445","-2.66353","5.09891","-4.63409","-334.80885","3.028020","121.844300"
"abc-017","-012.63384","88.15725","9.74615","60.62091","-916.43703","-0.54265","5.07418","-9.11345","-438.85608","5.319353","406.512338"
user@machine:~/Downloads/mattermodelling$