Mercurial > hg > cc > cirrus_home
comparison bin/cdx_tab.sh @ 163:ef961d91eea5
previous approach to lang/field extraction
| author | Henry S. Thompson <ht@inf.ed.ac.uk> |
|---|---|
| date | Mon, 18 Jul 2022 18:16:27 +0100 |
| parents | |
| children |
comparison
equal
deleted
inserted
replaced
| 162:e82981075b4a | 163:ef961d91eea5 |
|---|---|
| 1 #!/bin/bash | |
| 2 # Invoke this as e.g. sbatch -n 30 -c 10 masterJob.sh cdx_segment CC-MAIN-2019-35 | |
| 3 # run cdx_segment.py in parallel | |
| 4 n=$SLURM_NTASKS | |
| 5 c=$SLURM_CPUS_PER_TASK | |
| 6 node=$SLURMD_NODENAME | |
| 7 local=$SLURM_LOCALID | |
| 8 proc=$SLURM_PROCID | |
| 9 echo $(date) $node:$proc start | |
| 10 | |
| 11 type parallel | |
| 12 #module load gnu-parallel | |
| 13 | |
| 14 PYTHONPATH=$PYTHONPATH:$HOME/lib/python | |
| 15 parallel --will-cite -j $c doC2T.sh "$1" '{}' < cdx_tab/$proc.txt | |
| 16 | |
| 17 echo $(date) $proc end |