Mercurial > hg > cc > azure
comparison workers/bin/_fixAndMerge.sh @ 27:dd19cf97b6dd
attempt to fix robustness pblms
| author | Henry S. Thompson <ht@markup.co.uk> |
|---|---|
| date | Sat, 10 Nov 2018 13:20:56 +0000 |
| parents | 1b9329f6b5e1 |
| children | 2bde174fe9e2 |
comparison
equal
deleted
inserted
replaced
| 26:58d46e6983fa | 27:dd19cf97b6dd |
|---|---|
| 1 #!/bin/bash | 1 #!/bin/bash |
| 2 # Usage: [cat remote filenames] | _fixAndMerge.sh id home pause | 2 # Usage: [cat remote filenames] | _fixAndMerge.sh id home pause log |
| 3 id=$1 | 3 id=$1 |
| 4 home=$2 | 4 home=$2 |
| 5 pause=$3 | 5 pause=$3 |
| 6 log=$4 | |
| 6 # Don't all start at once | 7 # Don't all start at once |
| 7 sleep $pause | 8 sleep $pause |
| 8 ssh $home "xargs cat" | fixDates.py | 9 echo \#.$id $(date) starting >> $log |
| 10 cat > /var/data/in$id | |
| 11 echo \#.$id $(date) got list >> $log | |
| 12 until ( rm /var/data/d$id ; xargs -n 100 ssh $home "xargs gzip -c" < /var/data/in$id | gunzip -c >> /var/data/d$id ) do echo retrying 1>&2; sleep $pause; done | |
| 13 echo \#.$id $(date) got data >> $log | |
| 14 fixDates.py < /var/data/d$id | |
| 15 echo \#.$id $(date) done >> $log | |
| 16 |