Mercurial > hg > cc > cirrus_home

annotate bin/ezip.sh @ 143:ddff993994be

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
too clever by half, keys won't work in parallel for e.g. media types
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Wed, 20 Oct 2021 15:47:55 +0000
parents 464d2dfb99c9
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
88
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/bin/bash
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 # Invoke this as e.g. sbatch -n 4 -c 32 --exclusive masterJob.sh ezip
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 # run pigz -p 8 on extract_...tar files
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4 n=$SLURM_NTASKS
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 c=$SLURM_CPUS_PER_TASK
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 node=$SLURMD_NODENAME
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 local=$SLURM_LOCALID
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
8 proc=$SLURM_PROCID
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 echo $(date) $node:$proc start
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11 module load gnu-parallel
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
13 parallel --will-cite -j $c -n 1 'pigz -p 8 {}/orig/extracts/*.tar' < ezip/$proc.txt
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
15 echo $(date) $proc end
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16