Mercurial > hg > cc > cirrus_home
comparison bin/clmp.sh @ 163:ef961d91eea5
previous approach to lang/field extraction
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Mon, 18 Jul 2022 18:16:27 +0100 |
| parents | |
| children |
comparison
equal
deleted
inserted
replaced
| 162:e82981075b4a | 163:ef961d91eea5 |
|---|---|
| 1 #!/bin/bash | |
| 2 # Invoke this as e.g. sbatch --time=5:00:00 --exclusive -N 5 --ntasks-per-node 2 -c 1 masterJob.sh clmp CC-MAIN-2019-35 cdx_tab1 | |
| 3 # run clm.py in parallel | |
| 4 n=$SLURM_NTASKS | |
| 5 c=$SLURM_CPUS_PER_TASK | |
| 6 nodename=$SLURMD_NODENAME | |
| 7 local=$SLURM_LOCALID | |
| 8 node=$SLURM_NODEID | |
| 9 echo $(date) $nodename:$node start | |
| 10 | |
| 11 type parallel | |
| 12 #module load gnu-parallel | |
| 13 | |
| 14 export PYTHONPATH=$PYTHONPATH:$HOME/lib/python | |
| 15 parallel --will-cite -j $c doCLM.sh "$1" '{}' < $2/$node.txt | |
| 16 | |
| 17 echo $(date) $nodename:$node end |