Mercurial > hg > cc > cirrus_home

annotate bin/s2t.sh @ 190:f2bf736c2d40

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
working, with issues: 1) warc.export.cdx.path is not set 2) mime property isn't getting through 3) lastMod isn't showing up
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Tue, 24 Sep 2024 17:08:05 +0100
parents acae526510e2
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
159
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
1 #!/bin/bash
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
2 # Invoke this as e.g. sbatch -N 4 --ntasks=3 -c 5 s2t.sh \
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
3 # CC-MAIN-2019-35 cdx_counts 0 299
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
4
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
5 #SBATCH --time=01:00:00
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
6 #SBATCH --partition=standard
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
7 #SBATCH --qos=standard
185
acae526510e2 too many overdue updates to break down
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 169
diff changeset
8 #SBATCH --account=ec184-hst
159
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
9 #SBATCH --job-name s2t
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
10
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
11 export W=/work/dc007/dc007
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
12
169
a8e311034d43 new-style log notice
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 159
diff changeset
13 echo $(date) Launching $SLURM_JOB_NUM_NODES nodes for s2t "$@" from $(hostname) to do $SLURM_NTASKS tasks, $SLURM_CPUS_PER_TASK cpus each
159
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
14
185
acae526510e2 too many overdue updates to break down
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 169
diff changeset
15 srun -c $SLURM_CPUS_PER_TASK $W/shared/bin/csing $W/hst/bin/_s2t.sh "$@"
159
c3c3dd60b8a8 demo of slurm usage using cdx2tsv.py
Henry S. Thompson <ht@inf.ed.ac.uk>
parents:
diff changeset
16
185
acae526510e2 too many overdue updates to break down
Henry S. Thompson <ht@inf.ed.ac.uk>
parents: 169
diff changeset
17