Google

Edit File: parallel_design.7

.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.40) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "PARALLEL_DESIGN 7" .TH PARALLEL_DESIGN 7 "2020-09-19" "20200822" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh options as wrapper scripts .SH "Design of GNU Parallel" .IX Header "Design of GNU Parallel" This document describes design decisions made in the development of \&\s-1GNU\s0 \fBparallel\fR and the reasoning behind them. It will give an overview of why some of the code looks the way it does, and will help new maintainers understand the code better. .SS "One file program" .IX Subsection "One file program" \&\s-1GNU\s0 \fBparallel\fR is a Perl script in a single file. It is object oriented, but contrary to normal Perl scripts each class is not in its own file. This is due to user experience: The goal is that in a pinch the user will be able to get \s-1GNU\s0 \fBparallel\fR working simply by copying a single file: No need to mess around with environment variables like \&\s-1PERL5LIB.\s0 .SS "Choice of programming language" .IX Subsection "Choice of programming language" \&\s-1GNU\s0 \fBparallel\fR is designed to be able to run on old systems. That means that it cannot depend on a compiler being installed \- and especially not a compiler for a language that is younger than 20 years old. .PP The goal is that you can use \s-1GNU\s0 \fBparallel\fR on any system, even if you are not allowed to install additional software. .PP Of all the systems I have experienced, I have yet to see a system that had \s-1GCC\s0 installed that did not have Perl. The same goes for Rust, Go, Haskell, and other younger languages. I have, however, seen systems with Perl without any of the mentioned compilers. .PP Most modern systems also have either Python2 or Python3 installed, but you still cannot be certain which version, and since Python2 cannot run under Python3, Python is not an option. .PP Perl has the added benefit that implementing the {= perlexpr =} replacement string was fairly easy. .SS "Old Perl style" .IX Subsection "Old Perl style" \&\s-1GNU\s0 \fBparallel\fR uses some old, deprecated constructs. This is due to a goal of being able to run on old installations. Currently the target is CentOS 3.9 and Perl 5.8.0. .SS "Scalability up and down" .IX Subsection "Scalability up and down" The smallest system \s-1GNU\s0 \fBparallel\fR is tested on is a 32 \s-1MB ASUS\s0 WL500gP. The largest is a 2 \s-1TB\s0 128\-core machine. It scales up to around 100 machines \- depending on the duration of each job. .SS "Exponentially back off" .IX Subsection "Exponentially back off" \&\s-1GNU\s0 \fBparallel\fR busy waits. This is because the reason why a job is not started may be due to load average (when using \fB\-\-load\fR), and thus it will not make sense to wait for a job to finish. Instead the load average must be checked again. Load average is not the only reason: \fB\-\-timeout\fR has a similar problem. .PP To not burn up too much \s-1CPU GNU\s0 \fBparallel\fR sleeps exponentially longer and longer if nothing happens, maxing out at 1 second. .SS "Shell compatibility" .IX Subsection "Shell compatibility" It is a goal to have \s-1GNU\s0 \fBparallel\fR work equally well in any shell. However, in practice \s-1GNU\s0 \fBparallel\fR is being developed in \&\fBbash\fR and thus testing in other shells is limited to reported bugs. .PP When an incompatibility is found there is often not an easy fix: Fixing the problem in \fBcsh\fR often breaks it in \fBbash\fR. In these cases the fix is often to use a small Perl script and call that. .SS "env_parallel" .IX Subsection "env_parallel" \&\fBenv_parallel\fR is a dummy shell script that will run if \&\fBenv_parallel\fR is not an alias or a function and tell the user how to activate the alias/function for the supported shells. .PP The alias or function will copy the current environment and run the command with \s-1GNU\s0 \fBparallel\fR in the copy of the environment. .PP The problem is that you cannot access all of the current environment inside Perl. E.g. aliases, functions and unexported shell variables. .PP The idea is therefore to take the environment and put it in \&\fB\f(CB$PARALLEL_ENV\fB\fR which \s-1GNU\s0 \fBparallel\fR prepends to every command. .PP The only way to have access to the environment is directly from the shell, so the program must be written in a shell script that will be sourced and there has to deal with the dialect of the relevant shell. .PP \fIenv_parallel.*\fR .IX Subsection "env_parallel.*" .PP These are the files that implements the alias or function \&\fBenv_parallel\fR for a given shell. It could be argued that these should be put in some obscure place under /usr/lib, but by putting them in your path it becomes trivial to find the path to them and \&\fBsource\fR them: .PP .Vb 1 \& source \`which env_parallel.foo\` .Ve .PP The beauty is that they can be put anywhere in the path without the user having to know the location. So if the user's path includes /afs/bin/i386_fc5 or /usr/pkg/parallel/bin or /usr/local/parallel/20161222/sunos5.6/bin the files can be put in the dir that makes most sense for the sysadmin. .PP \fIenv_parallel.bash / env_parallel.sh / env_parallel.ash / env_parallel.dash / env_parallel.zsh / env_parallel.ksh / env_parallel.mksh\fR .IX Subsection "env_parallel.bash / env_parallel.sh / env_parallel.ash / env_parallel.dash / env_parallel.zsh / env_parallel.ksh / env_parallel.mksh" .PP \&\fBenv_parallel.(bash|sh|ash|dash|ksh|mksh|zsh)\fR defines the function \&\fBenv_parallel\fR. It uses \fBalias\fR and \fBtypeset\fR to dump the configuration (with a few exceptions) into \fB\f(CB$PARALLEL_ENV\fB\fR before running \s-1GNU\s0 \fBparallel\fR. .PP After \s-1GNU\s0 \fBparallel\fR is finished, \fB\f(CB$PARALLEL_ENV\fB\fR is deleted. .PP \fIenv_parallel.csh\fR .IX Subsection "env_parallel.csh" .PP \&\fBenv_parallel.csh\fR has two purposes: If \fBenv_parallel\fR is not an alias: make it into an alias that sets \fB\f(CB$PARALLEL\fB\fR with arguments and calls \fBenv_parallel.csh\fR. .PP If \fBenv_parallel\fR is an alias, then \fBenv_parallel.csh\fR uses \&\fB\f(CB$PARALLEL\fB\fR as the arguments for \s-1GNU\s0 \fBparallel\fR. .PP It exports the environment by writing a variable definition to a file for each variable. The definitions of aliases are appended to this file. Finally the file is put into \fB\f(CB$PARALLEL_ENV\fB\fR. .PP \&\s-1GNU\s0 \fBparallel\fR is then run and \fB\f(CB$PARALLEL_ENV\fB\fR is deleted. .PP \fIenv_parallel.fish\fR .IX Subsection "env_parallel.fish" .PP First all functions definitions are generated using a loop and \&\fBfunctions\fR. .PP Dumping the scalar variable definitions is harder. .PP \&\fBfish\fR can represent non-printable characters in (at least) 2 ways. To avoid problems all scalars are converted to \eXX quoting. .PP Then commands to generate the definitions are made and separated by \&\s-1NUL.\s0 .PP This is then piped into a Perl script that quotes all values. List elements will be appended using two spaces. .PP Finally \en is converted into \e1 because \fBfish\fR variables cannot contain \en. \s-1GNU\s0 \fBparallel\fR will later convert all \e1 from \&\fB\f(CB$PARALLEL_ENV\fB\fR into \en. .PP This is then all saved in \fB\f(CB$PARALLEL_ENV\fB\fR. .PP \&\s-1GNU\s0 \fBparallel\fR is called, and \fB\f(CB$PARALLEL_ENV\fB\fR is deleted. .SS "parset (supported in sh, ash, dash, bash, zsh, ksh, mksh)" .IX Subsection "parset (supported in sh, ash, dash, bash, zsh, ksh, mksh)" \&\fBparset\fR is a shell function. This is the reason why \fBparset\fR can set variables: It runs in the shell which is calling it. .PP It is also the reason why \fBparset\fR does not work, when data is piped into it: \fB... | parset ...\fR makes \fBparset\fR start in a subshell, and any changes in environment can therefore not make it back to the calling shell. .SS "Job slots" .IX Subsection "Job slots" The easiest way to explain what \s-1GNU\s0 \fBparallel\fR does is to assume that there are a number of job slots, and when a slot becomes available a job from the queue will be run in that slot. But originally \s-1GNU\s0 \&\fBparallel\fR did not model job slots in the code. Job slots have been added to make it possible to use \fB{%}\fR as a replacement string. .PP While the job sequence number can be computed in advance, the job slot can only be computed the moment a slot becomes available. So it has been implemented as a stack with lazy evaluation: Draw one from an empty stack and the stack is extended by one. When a job is done, push the available job slot back on the stack. .PP This implementation also means that if you re-run the same jobs, you cannot assume jobs will get the same slots. And if you use remote executions, you cannot assume that a given job slot will remain on the same remote server. This goes double since number of job slots can be adjusted on the fly (by giving \fB\-\-jobs\fR a file name). .SS "Rsync protocol version" .IX Subsection "Rsync protocol version" \&\fBrsync\fR 3.1.x uses protocol 31 which is unsupported by version 2.5.7. That means that you cannot push a file to a remote system using \&\fBrsync\fR protocol 31, if the remote system uses 2.5.7. \fBrsync\fR does not automatically downgrade to protocol 30. .PP \&\s-1GNU\s0 \fBparallel\fR does not require protocol 31, so if the \fBrsync\fR version is >= 3.1.0 then \fB\-\-protocol 30\fR is added to force newer \&\fBrsync\fRs to talk to version 2.5.7. .SS "Compression" .IX Subsection "Compression" \&\s-1GNU\s0 \fBparallel\fR buffers output in temporary files. \fB\-\-compress\fR compresses the buffered data. This is a bit tricky because there should be no files to clean up if \s-1GNU\s0 \fBparallel\fR is killed by a power outage. .PP \&\s-1GNU\s0 \fBparallel\fR first selects a compression program. If the user has not selected one, the first of these that is in \f(CW$PATH\fR is used: \fBpzstd lbzip2 pbzip2 zstd pixz lz4 pigz lzop plzip lzip gzip lrz pxz bzip2 lzma xz clzip\fR. They are sorted by speed on a 128 core machine. .PP Schematically the setup is as follows: .PP .Vb 2 \& command started by parallel | compress > tmpfile \& cattail tmpfile | uncompress | parallel which reads the output .Ve .PP The setup is duplicated for both standard output (stdout) and standard error (stderr). .PP \&\s-1GNU\s0 \fBparallel\fR pipes output from the command run into the compression program which saves to a tmpfile. \s-1GNU\s0 \fBparallel\fR records the pid of the compress program. At the same time a small Perl script (called \&\fBcattail\fR above) is started: It basically does \fBcat\fR followed by \&\fBtail \-f\fR, but it also removes the tmpfile as soon as the first byte is read, and it continuously checks if the pid of the compression program is dead. If the compress program is dead, \fBcattail\fR reads the rest of tmpfile and exits. .PP As most compression programs write out a header when they start, the tmpfile in practice is removed by \fBcattail\fR after around 40 ms. .SS "Wrapping" .IX Subsection "Wrapping" The command given by the user can be wrapped in multiple templates. Templates can be wrapped in other templates. .IP "\fB\f(CB$COMMAND\fB\fR" 15 .IX Item "$COMMAND" the command to run. .IP "\fB\f(CB$INPUT\fB\fR" 15 .IX Item "$INPUT" the input to run. .IP "\fB\f(CB$SHELL\fB\fR" 15 .IX Item "$SHELL" the shell that started \s-1GNU\s0 Parallel. .IP "\fB\f(CB$SSHLOGIN\fB\fR" 15 .IX Item "$SSHLOGIN" the sshlogin. .IP "\fB\f(CB$WORKDIR\fB\fR" 15 .IX Item "$WORKDIR" the working dir. .IP "\fB\f(CB$FILE\fB\fR" 15 .IX Item "$FILE" the file to read parts from. .IP "\fB\f(CB$STARTPOS\fB\fR" 15 .IX Item "$STARTPOS" the first byte position to read from \fB\f(CB$FILE\fB\fR. .IP "\fB\f(CB$LENGTH\fB\fR" 15 .IX Item "$LENGTH" the number of bytes to read from \fB\f(CB$FILE\fB\fR. .IP "\-\-shellquote" 15 .IX Item "--shellquote" echo \fIDouble quoted \f(CI$INPUT\fI\fR .IP "\-\-nice \fIpri\fR" 15 .IX Item "--nice pri" Remote: See \fBThe remote system wrapper\fR. .Sp Local: \fBsetpriority(0,0,$nice)\fR .IP "\-\-cat" 15 .IX Item "--cat" .Vb 6 \& cat > {}; $COMMAND {}; \& perl \-e \*(Aq$bash = shift; \& $csh = shift; \& for(@ARGV) { unlink;rmdir; } \& if($bash =~ s/h//) { exit $bash; } \& exit $csh;\*(Aq "$?h" "$status" {}; .Ve .Sp {} is set to \fB\f(CB$PARALLEL_TMP\fB\fR which is a tmpfile. The Perl script saves the exit value, unlinks the tmpfile, and returns the exit value \&\- no matter if the shell is \fBbash\fR/\fBksh\fR/\fBzsh\fR (using $?) or \&\fB*csh\fR/\fBfish\fR (using \f(CW$status\fR). .IP "\-\-fifo" 15 .IX Item "--fifo" .Vb 10 \& perl \-e \*(Aq($s,$c,$f) = @ARGV; \& # mkfifo $PARALLEL_TMP \& system "mkfifo", $f; \& # spawn $shell \-c $command & \& $pid = fork || exec $s, "\-c", $c; \& open($o,">",$f) || die $!; \& # cat > $PARALLEL_TMP \& while(sysread(STDIN,$buf,131072)){ \& syswrite $o, $buf; \& } \& close $o; \& # waitpid to get the exit code from $command \& waitpid $pid,0; \& # Cleanup \& unlink $f; \& exit $?/256;\*(Aq $SHELL \-c $COMMAND $PARALLEL_TMP .Ve .Sp This is an elaborate way of: mkfifo {}; run \fB\f(CB$COMMAND\fB\fR in the background using \fB\f(CB$SHELL\fB\fR; copying \s-1STDIN\s0 to {}; waiting for background to complete; remove {} and exit with the exit code from \fB\f(CB$COMMAND\fB\fR. .Sp It is made this way to be compatible with \fB*csh\fR/\fBfish\fR. .IP "\-\-pipepart" 15 .IX Item "--pipepart" .Vb 10 \& < $FILE perl \-e \*(Aqwhile(@ARGV) { \& sysseek(STDIN,shift,0) || die; \& $left = shift; \& while($read = \& sysread(STDIN,$buf, \& ($left > 131072 ? 131072 : $left))){ \& $left \-= $read; \& syswrite(STDOUT,$buf); \& } \& }\*(Aq $STARTPOS $LENGTH .Ve .Sp This will read \fB\f(CB$LENGTH\fB\fR bytes from \fB\f(CB$FILE\fB\fR starting at \fB\f(CB$STARTPOS\fB\fR and send it to \s-1STDOUT.\s0 .ie n .IP "\-\-sshlogin $SSHLOGIN" 15 .el .IP "\-\-sshlogin \f(CW$SSHLOGIN\fR" 15 .IX Item "--sshlogin $SSHLOGIN" .Vb 1 \& ssh $SSHLOGIN "$COMMAND" .Ve .IP "\-\-transfer" 15 .IX Item "--transfer" .Vb 4 \& ssh $SSHLOGIN mkdir \-p ./$WORKDIR; \& rsync \-\-protocol 30 \-rlDzR \e \& \-essh ./{} $SSHLOGIN:./$WORKDIR; \& ssh $SSHLOGIN "$COMMAND" .Ve .Sp Read about \fB\-\-protocol 30\fR in the section \fBRsync protocol version\fR. .IP "\-\-transferfile \fIfile\fR" 15 .IX Item "--transferfile file" <<todo>> .IP "\-\-basefile" 15 .IX Item "--basefile" <<todo>> .IP "\-\-return \fIfile\fR" 15 .IX Item "--return file" .Vb 5 \& $COMMAND; _EXIT_status=$?; mkdir \-p $WORKDIR; \& rsync \-\-protocol 30 \e \& \-\-rsync\-path=cd\e ./$WORKDIR\e;\e rsync \e \& \-rlDzR \-essh $SSHLOGIN:./$FILE ./$WORKDIR; \& exit $_EXIT_status; .Ve .Sp The \fB\-\-rsync\-path=cd ...\fR is needed because old versions of \fBrsync\fR do not support \fB\-\-no\-implied\-dirs\fR. .Sp The \fB\f(CB$_EXIT_status\fB\fR trick is to postpone the exit value. This makes it incompatible with \fB*csh\fR and should be fixed in the future. Maybe a wrapping 'sh \-c' is enough? .IP "\-\-cleanup" 15 .IX Item "--cleanup" \&\f(CW$RETURN\fR is the wrapper from \fB\-\-return\fR .Sp .Vb 4 \& $COMMAND; _EXIT_status=$?; $RETURN; \& ssh $SSHLOGIN \e(rm\e \-f\e ./$WORKDIR/{}\e;\e \& rmdir\e ./$WORKDIR\e \e>\e&/dev/null\e;\e); \& exit $_EXIT_status; .Ve .Sp \&\fB\f(CB$_EXIT_status\fB\fR: see \fB\-\-return\fR above. .IP "\-\-pipe" 15 .IX Item "--pipe" .Vb 10 \& perl \-e \*(Aqif(sysread(STDIN, $buf, 1)) { \& open($fh, "|\-", "@ARGV") || die; \& syswrite($fh, $buf); \& # Align up to 128k block \& if($read = sysread(STDIN, $buf, 131071)) { \& syswrite($fh, $buf); \& } \& while($read = sysread(STDIN, $buf, 131072)) { \& syswrite($fh, $buf); \& } \& close $fh; \& exit ($?&127 ? 128+($?&127) : 1+$?>>8) \& }\*(Aq $SHELL \-c $COMMAND .Ve .Sp This small wrapper makes sure that \fB\f(CB$COMMAND\fB\fR will never be run if there is no data. .IP "\-\-tmux" 15 .IX Item "--tmux" <<\s-1TODO\s0 Fixup with '\-quoting>> mkfifo /tmp/tmx3cMEV && sh \-c 'tmux \-S /tmp/tmsaKpv1 new-session \-s p334310 \-d \*(L"sleep .2\*(R" >/dev/null 2>&1'; tmux \-S /tmp/tmsaKpv1 new-window \-t p334310 \-n wc\e 10 \e(wc\e 10\e)\e;\e perl\e \-e\e \e'while\e(\e$t++\e<3\e)\e{\e print\e \e$ARGV\e[0\e],\e\*(L"\e\en\e\*(R"\e \e}\e'\e \e$\e?h/\e$status\e \e>\e>\e /tmp/tmx3cMEV\e&echo\e wc\e\e\e 10\e;\e echo\e \eJob\e finished\e at:\e \e`date\e`\e;sleep\e 10; exec perl \-e '$/=\*(L"/\*(R";$_=<>;$c=<>;unlink \f(CW$ARGV\fR; /(\ed+)h/ and exit($1);exit$c' /tmp/tmx3cMEV .Sp mkfifo \fItmpfile.tmx\fR; tmux \-S <tmpfile.tms> new-session \-s p\fI\s-1PID\s0\fR \-d 'sleep .2' >&/dev/null; tmux \-S <tmpfile.tms> new-window \-t p\fI\s-1PID\s0\fR \-n <<shell quoted input>> \e(<<shell quoted input>>\e)\e;\e perl\e \-e\e \e'while\e(\e$t++\e<3\e)\e{\e print\e \e$ARGV\e[0\e],\e\*(L"\e\en\e\*(R"\e \e}\e'\e \e$\e?h/\e$status\e \e>\e>\e \fItmpfile.tmx\fR\e&echo\e <<shell double quoted input>>\e;echo\e \eJob\e finished\e at:\e \e`date\e`\e;sleep\e 10; exec perl \-e '$/=\*(L"/\*(R";$_=<>;$c=<>;unlink \f(CW$ARGV\fR; /(\ed+)h/ and exit($1);exit$c' \fItmpfile.tmx\fR .Sp First a \s-1FIFO\s0 is made (.tmx). It is used for communicating exit value. Next a new tmux session is made. This may fail if there is already a session, so the output is ignored. If all job slots finish at the same time, then \fBtmux\fR will close the session. A temporary socket is made (.tms) to avoid a race condition in \fBtmux\fR. It is cleaned up when \s-1GNU\s0 \fBparallel\fR finishes. .Sp The input is used as the name of the windows in \fBtmux\fR. When the job inside \fBtmux\fR finishes, the exit value is printed to the \s-1FIFO\s0 (.tmx). This \s-1FIFO\s0 is opened by \fBperl\fR outside \fBtmux\fR, and \fBperl\fR then removes the \s-1FIFO.\s0 \fBPerl\fR blocks until the first value is read from the \s-1FIFO,\s0 and this value is used as exit value. .Sp To make it compatible with \fBcsh\fR and \fBbash\fR the exit value is printed as: $?h/$status and this is parsed by \fBperl\fR. .Sp There is a bug that makes it necessary to print the exit value 3 times. .Sp Another bug in \fBtmux\fR requires the length of the tmux title and command to not have certain limits. When inside these limits, 75 '\e ' are added to the title to force it to be outside the limits. .Sp You can map the bad limits using: .Sp .Vb 4 \& perl \-e \*(Aqsub r { int(rand(shift)).($_[0] && "\et".r(@_)) } print map { r(@ARGV)."\en" } 1..10000\*(Aq 1600 1500 90 | \& perl \-ane \*(Aq$F[0]+$F[1]+$F[2] < 2037 and print \*(Aq | \& parallel \-\-colsep \*(Aq\et\*(Aq \-\-tagstring \*(Aq{1}\et{2}\et{3}\*(Aq tmux \-S /tmp/p{%}\-\*(Aq{=3 $_="O"x$_ =}\*(Aq \e \& new\-session \-d \-n \*(Aq{=1 $_="O"x$_ =}\*(Aq true\*(Aq\e {=2 $_="O"x$_ =};echo $?;rm \-f /tmp/p{%}\-O*\*(Aq \& \& perl \-e \*(Aqsub r { int(rand(shift)).($_[0] && "\et".r(@_)) } print map { r(@ARGV)."\en" } 1..10000\*(Aq 17000 17000 90 | \& parallel \-\-colsep \*(Aq\et\*(Aq \-\-tagstring \*(Aq{1}\et{2}\et{3}\*(Aq \e \& tmux \-S /tmp/p{%}\-\*(Aq{=3 $_="O"x$_ =}\*(Aq new\-session \-d \-n \*(Aq{=1 $_="O"x$_ =}\*(Aq true\*(Aq\e {=2 $_="O"x$_ =};echo $?;rm /tmp/p{%}\-O*\*(Aq \& > value.csv 2>/dev/null \& \& R \-e \*(Aqa<\-read.table("value.csv");X11();plot(a[,1],a[,2],col=a[,4]+5,cex=0.1);Sys.sleep(1000)\*(Aq .Ve .Sp For \fBtmux 1.8\fR 17000 can be lowered to 2100. .Sp The interesting areas are title 0..1000 with (title + whole command) in 996..1127 and 9331..9636. .PP The ordering of the wrapping is important: .IP "\(bu" 5 \&\f(CW$PARALLEL_ENV\fR which is set in env_parallel.* must be prepended to the command first, as the command may contain exported variables or functions. .IP "\(bu" 5 \&\fB\-\-nice\fR/\fB\-\-cat\fR/\fB\-\-fifo\fR should be done on the remote machine .IP "\(bu" 5 \&\fB\-\-pipepart\fR/\fB\-\-pipe\fR should be done on the local machine inside \fB\-\-tmux\fR .SS "Convenience options \-\-nice \-\-basefile \-\-transfer \-\-return \-\-cleanup \-\-tmux \-\-group \-\-compress \-\-cat \-\-fifo \-\-workdir \-\-tag \-\-tagstring" .IX Subsection "Convenience options --nice --basefile --transfer --return --cleanup --tmux --group --compress --cat --fifo --workdir --tag --tagstring" These are all convenience options that make it easier to do a task. But more importantly: They are tested to work on corner cases, too. Take \fB\-\-nice\fR as an example: .PP .Vb 1 \& nice parallel command ... .Ve .PP will work just fine. But when run remotely, you need to move the nice command so it is being run on the server: .PP .Vb 1 \& parallel \-S server nice command ... .Ve .PP And this will again work just fine, as long as you are running a single command. When you are running a composed command you need nice to apply to the whole command, and it gets harder still: .PP .Vb 1 \& parallel \-S server \-q nice bash \-c \*(Aqcommand1 ...; cmd2 | cmd3\*(Aq .Ve .PP It is not impossible, but by using \fB\-\-nice\fR \s-1GNU\s0 \fBparallel\fR will do the right thing for you. Similarly when transferring files: It starts to get hard when the file names contain space, :, `, *, or other special characters. .PP To run the commands in a \fBtmux\fR session you basically just need to quote the command. For simple commands that is easy, but when commands contain special characters, it gets much harder to get right. .PP \&\fB\-\-compress\fR not only compresses standard output (stdout) but also standard error (stderr); and it does so into files, that are open but deleted, so a crash will not leave these files around. .PP \&\fB\-\-cat\fR and \fB\-\-fifo\fR are easy to do by hand, until you want to clean up the tmpfile and keep the exit code of the command. .PP The real killer comes when you try to combine several of these: Doing that correctly for all corner cases is next to impossible to do by hand. .SS "\-\-shard" .IX Subsection "--shard" The simple way to implement sharding would be to: .IP "1." 5 start n jobs, .IP "2." 5 split each line into columns, .IP "3." 5 select the data from the relevant column .IP "4." 5 compute a hash value from the data .IP "5." 5 take the modulo n of the hash value .IP "6." 5 pass the full line to the jobslot that has the computed value .PP Unfortunately Perl is rather slow at computing the hash value (and somewhat slow at splitting into columns). .PP One solution is to use a compiled language for the splitting and hashing, but that would go against the design criteria of not depending on a compiler. .PP Luckily those tasks can be parallelized. So \s-1GNU\s0 \fBparallel\fR starts n sharders that do step 2\-6, and passes blocks of 100k to each of those in a round robin manner. To make sure these sharders compute the hash the same way, \f(CW$PERL_HASH_SEED\fR is set to the same value for all sharders. .PP Running n sharders poses a new problem: Instead of having n outputs (one for each computed value) you now have n outputs for each of the n values, so in total n*n outputs; and you need to merge these n*n outputs together into n outputs. .PP This can be done by simply running 'parallel \-j0 \-\-lb cat ::: outputs_for_one_value', but that is rather inefficient, as it spawns a process for each file. Instead the core code from 'parcat' is run, which is also a bit faster. .PP All the sharders and parcats communicate through named pipes that are unlinked as soon as they are opened. .SS "Shell shock" .IX Subsection "Shell shock" The shell shock bug in \fBbash\fR did not affect \s-1GNU\s0 \fBparallel\fR, but the solutions did. \fBbash\fR first introduced functions in variables named: \&\fI\f(BIBASH_FUNC_myfunc()\fI\fR and later changed that to \fIBASH_FUNC_myfunc%%\fR. When transferring functions \s-1GNU\s0 \fBparallel\fR reads off the function and changes that into a function definition, which is copied to the remote system and executed before the actual command is executed. Therefore \s-1GNU\s0 \fBparallel\fR needs to know how to read the function. .PP From version 20150122 \s-1GNU\s0 \fBparallel\fR tries both the ()\-version and the %%\-version, and the function definition works on both pre\- and post-shell shock versions of \fBbash\fR. .SS "The remote system wrapper" .IX Subsection "The remote system wrapper" The remote system wrapper does some initialization before starting the command on the remote system. .PP \fICtrl-C and standard error (stderr)\fR .IX Subsection "Ctrl-C and standard error (stderr)" .PP If the user presses Ctrl-C the user expects jobs to stop. This works out of the box if the jobs are run locally. Unfortunately it is not so simple if the jobs are run remotely. .PP If remote jobs are run in a tty using \fBssh \-tt\fR, then Ctrl-C works, but all output to standard error (stderr) is sent to standard output (stdout). This is not what the user expects. .PP If remote jobs are run without a tty using \fBssh\fR (without \fB\-tt\fR), then output to standard error (stderr) is kept on stderr, but Ctrl-C does not kill remote jobs. This is not what the user expects. .PP So what is needed is a way to have both. It seems the reason why Ctrl-C does not kill the remote jobs is because the shell does not propagate the hang-up signal from \fBsshd\fR. But when \fBsshd\fR dies, the parent of the login shell becomes \fBinit\fR (process id 1). So by exec'ing a Perl wrapper to monitor the parent pid and kill the child if the parent pid becomes 1, then Ctrl-C works and stderr is kept on stderr. .PP Ctrl-C does, however, kill the ssh connection, so any output from a remote dying process is lost. .PP To be able to kill all (grand)*children a new process group is started. .PP \fI\-\-nice\fR .IX Subsection "--nice" .PP \&\fBnice\fRing the remote process is done by \fBsetpriority(0,0,$nice)\fR. A few old systems do not implement this and \fB\-\-nice\fR is unsupported on those. .PP \fISetting \f(CI$PARALLEL_TMP\fI\fR .IX Subsection "Setting $PARALLEL_TMP" .PP \&\fB\f(CB$PARALLEL_TMP\fB\fR is used by \fB\-\-fifo\fR and \fB\-\-cat\fR and must point to a non-exitent file in \fB\f(CB$TMPDIR\fB\fR. This file name is computed on the remote system. .PP \fIThe wrapper\fR .IX Subsection "The wrapper" .PP The wrapper looks like this: .PP .Vb 10 \& $shell = $PARALLEL_SHELL || $SHELL; \& $tmpdir = $TMPDIR; \& $nice = $opt::nice; \& # Set $PARALLEL_TMP to a non\-existent file name in $TMPDIR \& do { \& $ENV{PARALLEL_TMP} = $tmpdir."/par". \& join"", map { (0..9,"a".."z","A".."Z")[rand(62)] } (1..5); \& } while(\-e $ENV{PARALLEL_TMP}); \& $SIG{CHLD} = sub { $done = 1; }; \& $pid = fork; \& unless($pid) { \& # Make own process group to be able to kill HUP it later \& setpgrp; \& eval { setpriority(0,0,$nice) }; \& exec $shell, "\-c", ($bashfunc."@ARGV"); \& die "exec: $!\en"; \& } \& do { \& # Parent is not init (ppid=1), so sshd is alive \& # Exponential sleep up to 1 sec \& $s = $s < 1 ? 0.001 + $s * 1.03 : $s; \& select(undef, undef, undef, $s); \& } until ($done || getppid == 1); \& # Kill HUP the process group if job not done \& kill(SIGHUP, \-${pid}) unless $done; \& wait; \& exit ($?&127 ? 128+($?&127) : 1+$?>>8) .Ve .SS "Transferring of variables and functions" .IX Subsection "Transferring of variables and functions" Transferring of variables and functions given by \fB\-\-env\fR is done by running a Perl script remotely that calls the actual command. The Perl script sets \fB\f(CB$ENV\fB{\fR\fIvariable\fR\fB}\fR to the correct value before exec'ing a shell that runs the function definition followed by the actual command. .PP The function \fBenv_parallel\fR copies the full current environment into the environment variable \fB\s-1PARALLEL_ENV\s0\fR. This variable is picked up by \s-1GNU\s0 \fBparallel\fR and used to create the Perl script mentioned above. .SS "Base64 encoded bzip2" .IX Subsection "Base64 encoded bzip2" \&\fBcsh\fR limits words of commands to 1024 chars. This is often too little when \s-1GNU\s0 \fBparallel\fR encodes environment variables and wraps the command with different templates. All of these are combined and quoted into one single word, which often is longer than 1024 chars. .PP When the line to run is > 1000 chars, \s-1GNU\s0 \fBparallel\fR therefore encodes the line to run. The encoding \fBbzip2\fRs the line to run, converts this to base64, splits the base64 into 1000 char blocks (so \fBcsh\fR does not fail), and prepends it with this Perl script that decodes, decompresses and \fBeval\fRs the line. .PP .Vb 2 \& @GNU_Parallel=("use","IPC::Open3;","use","MIME::Base64"); \& eval "@GNU_Parallel"; \& \& $SIG{CHLD}="IGNORE"; \& # Search for bzip2. Not found => use default path \& my $zip = (grep { \-x $_ } "/usr/local/bin/bzip2")[0] || "bzip2"; \& # $in = stdin on $zip, $out = stdout from $zip \& my($in, $out,$eval); \& open3($in,$out,">&STDERR",$zip,"\-dc"); \& if(my $perlpid = fork) { \& close $in; \& $eval = join "", <$out>; \& close $out; \& } else { \& close $out; \& # Pipe decoded base64 into \*(Aqbzip2 \-dc\*(Aq \& print $in (decode_base64(join"",@ARGV)); \& close $in; \& exit; \& } \& wait; \& eval $eval; .Ve .PP Perl and \fBbzip2\fR must be installed on the remote system, but a small test showed that \fBbzip2\fR is installed by default on all platforms that runs \s-1GNU\s0 \fBparallel\fR, so this is not a big problem. .PP The added bonus of this is that much bigger environments can now be transferred as they will be below \fBbash\fR's limit of 131072 chars. .SS "Which shell to use" .IX Subsection "Which shell to use" Different shells behave differently. A command that works in \fBtcsh\fR may not work in \fBbash\fR. It is therefore important that the correct shell is used when \s-1GNU\s0 \fBparallel\fR executes commands. .PP \&\s-1GNU\s0 \fBparallel\fR tries hard to use the right shell. If \s-1GNU\s0 \fBparallel\fR is called from \fBtcsh\fR it will use \fBtcsh\fR. If it is called from \&\fBbash\fR it will use \fBbash\fR. It does this by looking at the (grand)*parent process: If the (grand)*parent process is a shell, use this shell; otherwise look at the parent of this (grand)*parent. If none of the (grand)*parents are shells, then \f(CW$SHELL\fR is used. .PP This will do the right thing if called from: .IP "\(bu" 2 an interactive shell .IP "\(bu" 2 a shell script .IP "\(bu" 2 a Perl script in `` or using \fBsystem\fR if called as a single string. .PP While these cover most cases, there are situations where it will fail: .IP "\(bu" 2 When run using \fBexec\fR. .IP "\(bu" 2 When run as the last command using \fB\-c\fR from another shell (because some shells use \fBexec\fR): .Sp .Vb 2 \& zsh% bash \-c "parallel \*(Aqecho {} is not run in bash; \e \& set | grep BASH_VERSION\*(Aq ::: This" .Ve .Sp You can work around that by appending '&& true': .Sp .Vb 2 \& zsh% bash \-c "parallel \*(Aqecho {} is run in bash; \e \& set | grep BASH_VERSION\*(Aq ::: This && true" .Ve .IP "\(bu" 2 When run in a Perl script using \fBsystem\fR with parallel as the first string: .Sp .Vb 1 \& #!/usr/bin/perl \& \& system("parallel",\*(Aqsetenv a {}; echo $a\*(Aq,":::",2); .Ve .Sp Here it depends on which shell is used to call the Perl script. If the Perl script is called from \fBtcsh\fR it will work just fine, but if it is called from \fBbash\fR it will fail, because the command \fBsetenv\fR is not known to \fBbash\fR. .PP If \s-1GNU\s0 \fBparallel\fR guesses wrong in these situation, set the shell using \&\fB\f(CB$PARALLEL_SHELL\fB\fR. .SS "Always running commands in a shell" .IX Subsection "Always running commands in a shell" If the command is a simple command with no redirection and setting of variables, the command \fIcould\fR be run without spawning a shell. E.g. this simple \fBgrep\fR matching either 'ls ' or ' wc >> c': .PP .Vb 1 \& parallel "grep \-E \*(Aqls | wc >> c\*(Aq {}" ::: foo .Ve .PP could be run as: .PP .Vb 1 \& system("grep","\-E","ls | wc >> c","foo"); .Ve .PP However, as soon as the command is a bit more complex a shell \fImust\fR be spawned: .PP .Vb 2 \& parallel "grep \-E \*(Aqls | wc >> c\*(Aq {} | wc >> c" ::: foo \& parallel "LANG=C grep \-E \*(Aqls | wc >> c\*(Aq {}" ::: foo .Ve .PP It is impossible to tell the difference between these without parsing the string (is the \fB|\fR a pipe in shell or an alternation in a \fBgrep\fR regexp? Is \fBLANG=C\fR a command in \fBcsh\fR or setting a variable in \&\fBbash\fR? Is \fB\fR>> redirection or part of a regexp?). .PP On top of this wrapper scripts will often require a shell to be spawned. .PP The downside is that you need to quote special shell chars twice: .PP .Vb 6 \& parallel echo \*(Aq*\*(Aq ::: This will expand the asterisk \& parallel echo "\*(Aq*\*(Aq" ::: This will not \& parallel "echo \*(Aq*\*(Aq" ::: This will not \& parallel echo \*(Aq\e*\*(Aq ::: This will not \& parallel echo \e\*(Aq\*(Aq*\*(Aq\e\*(Aq ::: This will not \& parallel \-q echo \*(Aq*\*(Aq ::: This will not .Ve .PP \&\fB\-q\fR will quote all special chars, thus redirection will not work: this prints '* > out.1' and \fIdoes not\fR save '*' into the file out.1: .PP .Vb 1 \& parallel \-q echo "*" ">" out.{} ::: 1 .Ve .PP \&\s-1GNU\s0 \fBparallel\fR tries to live up to Principle Of Least Astonishment (\s-1POLA\s0), and the requirement of using \fB\-q\fR is hard to understand, when you do not see the whole picture. .SS "Quoting" .IX Subsection "Quoting" Quoting depends on the shell. For most shells '\-quoting is used for strings containing special characters. .PP For \fBtcsh\fR/\fBcsh\fR newline is quoted as \e followed by newline. Other special characters are also \e\-quoted. .PP For \fBrc\fR everything is quoted using '. .SS "\-\-pipepart vs. \-\-pipe" .IX Subsection "--pipepart vs. --pipe" While \fB\-\-pipe\fR and \fB\-\-pipepart\fR look much the same to the user, they are implemented very differently. .PP With \fB\-\-pipe\fR \s-1GNU\s0 \fBparallel\fR reads the blocks from standard input (stdin), which is then given to the command on standard input (stdin); so every block is being processed by \s-1GNU\s0 \fBparallel\fR itself. This is the reason why \fB\-\-pipe\fR maxes out at around 500 MB/sec. .PP \&\fB\-\-pipepart\fR, on the other hand, first identifies at which byte positions blocks start and how long they are. It does that by seeking into the file by the size of a block and then reading until it meets end of a block. The seeking explains why \s-1GNU\s0 \fBparallel\fR does not know the line number and why \fB\-L/\-l\fR and \fB\-N\fR do not work. .PP With a reasonable block and file size this seeking is more than 1000 time faster than reading the full file. The byte positions are then given to a small script that reads from position X to Y and sends output to standard output (stdout). This small script is prepended to the command and the full command is executed just as if \s-1GNU\s0 \&\fBparallel\fR had been in its normal mode. The script looks like this: .PP .Vb 8 \& < file perl \-e \*(Aqwhile(@ARGV) { \& sysseek(STDIN,shift,0) || die; \& $left = shift; \& while($read = sysread(STDIN,$buf, \& ($left > 131072 ? 131072 : $left))){ \& $left \-= $read; syswrite(STDOUT,$buf); \& } \& }\*(Aq startbyte length_in_bytes .Ve .PP It delivers 1 GB/s per core. .PP Instead of the script \fBdd\fR was tried, but many versions of \fBdd\fR do not support reading from one byte to another and might cause partial data. See this for a surprising example: .PP .Vb 1 \& yes | dd bs=1024k count=10 | wc .Ve .SS "\-\-block\-size adjustment" .IX Subsection "--block-size adjustment" Every time \s-1GNU\s0 \fBparallel\fR detects a record bigger than \&\fB\-\-block\-size\fR it increases the block size by 30%. A small \&\fB\-\-block\-size\fR gives very poor performance; by exponentially increasing the block size performance will not suffer. .PP \&\s-1GNU\s0 \fBparallel\fR will waste \s-1CPU\s0 power if \fB\-\-block\-size\fR does not contain a full record, because it tries to find a full record and will fail to do so. The recommendation is therefore to use a \&\fB\-\-block\-size\fR > 2 records, so you always get at least one full record when you read one block. .PP If you use \fB\-N\fR then \fB\-\-block\-size\fR should be big enough to contain N+1 records. .SS "Automatic \-\-block\-size computation" .IX Subsection "Automatic --block-size computation" With \fB\-\-pipepart\fR \s-1GNU\s0 \fBparallel\fR can compute the \fB\-\-block\-size\fR automatically. A \fB\-\-block\-size\fR of \fB\-1\fR will use a block size so that each jobslot will receive approximately 1 block. \fB\-\-block \-2\fR will pass 2 blocks to each jobslot and \fB\-\f(BIn\fB\fR will pass \fIn\fR blocks to each jobslot. .PP This can be done because \fB\-\-pipepart\fR reads from files, and we can compute the total size of the input. .SS "\-\-jobs and \-\-onall" .IX Subsection "--jobs and --onall" When running the same commands on many servers what should \fB\-\-jobs\fR signify? Is it the number of servers to run on in parallel? Is it the number of jobs run in parallel on each server? .PP \&\s-1GNU\s0 \fBparallel\fR lets \fB\-\-jobs\fR represent the number of servers to run on in parallel. This is to make it possible to run a sequence of commands (that cannot be parallelized) on each server, but run the same sequence on multiple servers. .SS "\-\-shuf" .IX Subsection "--shuf" When using \fB\-\-shuf\fR to shuffle the jobs, all jobs are read, then they are shuffled, and finally executed. When using \s-1SQL\s0 this makes the \&\fB\-\-sqlmaster\fR be the part that shuffles the jobs. The \fB\-\-sqlworker\fRs simply executes according to Seq number. .SS "\-\-csv" .IX Subsection "--csv" \&\fB\-\-pipepart\fR is incompatible with \fB\-\-csv\fR because you can have records like: .PP .Vb 7 \& a,b,c \& a," \& a,b,c \& a,b,c \& a,b,c \& ",c \& a,b,c .Ve .PP Here the second record contains a multi-line field that looks like records. Since \fB\-\-pipepart\fR does not read then whole file when searching for record endings, it may start reading in this multi-line field, which would be wrong. .SS "Buffering on disk" .IX Subsection "Buffering on disk" \&\s-1GNU\s0 \fBparallel\fR buffers output, because if output is not buffered you have to be ridiculously careful on sizes to avoid mixing of outputs (see excellent example on https://catern.com/posts/pipes.html). .PP \&\s-1GNU\s0 \fBparallel\fR buffers on disk in \f(CW$TMPDIR\fR using files, that are removed as soon as they are created, but which are kept open. So even if \s-1GNU\s0 \fBparallel\fR is killed by a power outage, there will be no files to clean up afterwards. Another advantage is that the file system is aware that these files will be lost in case of a crash, so it does not need to sync them to disk. .PP It gives the odd situation that a disk can be fully used, but there are no visible files on it. .PP \fIPartly buffering in memory\fR .IX Subsection "Partly buffering in memory" .PP When using output formats \s-1SQL\s0 and \s-1CSV\s0 then \s-1GNU\s0 Parallel has to read the whole output into memory. When run normally it will only read the output from a single job. But when using \fB\-\-linebuffer\fR every line printed will also be buffered in memory \- for all jobs currently running. .PP If memory is tight, then do not use the output format \s-1SQL/CSV\s0 with \&\fB\-\-linebuffer\fR. .PP \fIComparing to buffering in memory\fR .IX Subsection "Comparing to buffering in memory" .PP \&\fBgargs\fR is a parallelizing tool that buffers in memory. It is therefore a useful way of comparing the advantages and disadvantages of buffering in memory to buffering on disk. .PP On an system with 6 \s-1GB RAM\s0 free and 6 \s-1GB\s0 free swap these were tested with different sizes: .PP .Vb 2 \& echo /dev/zero | gargs "head \-c $size {}" >/dev/null \& echo /dev/zero | parallel "head \-c $size {}" >/dev/null .Ve .PP The results are here: .PP .Vb 10 \& JobRuntime Command \& 0.344 parallel_test 1M \& 0.362 parallel_test 10M \& 0.640 parallel_test 100M \& 9.818 parallel_test 1000M \& 23.888 parallel_test 2000M \& 30.217 parallel_test 2500M \& 30.963 parallel_test 2750M \& 34.648 parallel_test 3000M \& 43.302 parallel_test 4000M \& 55.167 parallel_test 5000M \& 67.493 parallel_test 6000M \& 178.654 parallel_test 7000M \& 204.138 parallel_test 8000M \& 230.052 parallel_test 9000M \& 255.639 parallel_test 10000M \& 757.981 parallel_test 30000M \& 0.537 gargs_test 1M \& 0.292 gargs_test 10M \& 0.398 gargs_test 100M \& 3.456 gargs_test 1000M \& 8.577 gargs_test 2000M \& 22.705 gargs_test 2500M \& 123.076 gargs_test 2750M \& 89.866 gargs_test 3000M \& 291.798 gargs_test 4000M .Ve .PP \&\s-1GNU\s0 \fBparallel\fR is pretty much limited by the speed of the disk: Up to 6 \s-1GB\s0 data is written to disk but cached, so reading is fast. Above 6 \&\s-1GB\s0 data are both written and read from disk. When the 30000MB job is running, the disk system is slow, but usable: If you are not using the disk, you almost do not feel it. .PP \&\fBgargs\fR has a speed advantage up until 2500M where it hits a wall. Then the system starts swapping like crazy and is completely unusable. At 5000M it goes out of memory. .PP You can make \s-1GNU\s0 \fBparallel\fR behave similar to \fBgargs\fR if you point \&\f(CW$TMPDIR\fR to a tmpfs-filesystem: It will be faster for small outputs, but may kill your system for larger outputs and cause you to lose output. .SS "Disk full" .IX Subsection "Disk full" \&\s-1GNU\s0 \fBparallel\fR buffers on disk. If the disk is full, data may be lost. To check if the disk is full \s-1GNU\s0 \fBparallel\fR writes a 8193 byte file every second. If this file is written successfully, it is removed immediately. If it is not written successfully, the disk is full. The size 8193 was chosen because 8192 gave wrong result on some file systems, whereas 8193 did the correct thing on all tested filesystems. .SS "Memory usage" .IX Subsection "Memory usage" Normally \s-1GNU\s0 \fBparallel\fR will use around 17 \s-1MB RAM\s0 constantly \- no matter how many jobs or how much output there is. There are a few things that cause the memory usage to rise: .IP "\(bu" 3 Multiple input sources. \s-1GNU\s0 \fBparallel\fR reads an input source only once. This is by design, as an input source can be a stream (e.g. \s-1FIFO,\s0 pipe, standard input (stdin)) which cannot be rewound and read again. When reading a single input source, the memory is freed as soon as the job is done \- thus keeping the memory usage constant. .Sp But when reading multiple input sources \s-1GNU\s0 \fBparallel\fR keeps the already read values for generating all combinations with other input sources. .IP "\(bu" 3 Computing the number of jobs. \fB\-\-bar\fR, \fB\-\-eta\fR, and \fB\-\-halt xx%\fR use \fB\fBtotal_jobs()\fB\fR to compute the total number of jobs. It does this by generating the data structures for all jobs. All these job data structures will be stored in memory and take up around 400 bytes/job. .IP "\(bu" 3 Buffering a full line. \fB\-\-linebuffer\fR will read a full line per running job. A very long output line (say 1 \s-1GB\s0 without \en) will increase \s-1RAM\s0 usage temporarily: From when the beginning of the line is read till the line is printed. .IP "\(bu" 3 Buffering the full output of a single job. This happens when using \&\fB\-\-results *.csv/*.tsv\fR or \fB\-\-sql*\fR. Here \s-1GNU\s0 \fBparallel\fR will read the whole output of a single job and save it as csv/tsv or \s-1SQL.\s0 .SS "Argument separators ::: :::: :::+ ::::+" .IX Subsection "Argument separators ::: :::: :::+ ::::+" The argument separator \fB:::\fR was chosen because I have never seen \fB:::\fR used in any command. The natural choice \fB\-\-\fR would be a bad idea since it is not unlikely that the template command will contain \fB\-\-\fR. I have seen \fB::\fR used in programming languanges to separate classes, and I did not want the user to be confused that the separator had anything to do with classes. .PP \&\fB:::\fR also makes a visual separation, which is good if there are multiple \fB:::\fR. .PP When \fB:::\fR was chosen, \fB::::\fR came as a fairly natural extension. .PP Linking input sources meant having to decide for some way to indicate linking of \fB:::\fR and \fB::::\fR. \fB:::+\fR and \fB::::+\fR was chosen, so that they were similar to \fB:::\fR and \fB::::\fR. .SS "Perl replacement strings, {= =}, and \-\-rpl" .IX Subsection "Perl replacement strings, {= =}, and --rpl" The shorthands for replacement strings make a command look more cryptic. Different users will need different replacement strings. Instead of inventing more shorthands you get more flexible replacement strings if they can be programmed by the user. .PP The language Perl was chosen because \s-1GNU\s0 \fBparallel\fR is written in Perl and it was easy and reasonably fast to run the code given by the user. .PP If a user needs the same programmed replacement string again and again, the user may want to make his own shorthand for it. This is what \fB\-\-rpl\fR is for. It works so well, that even \s-1GNU\s0 \fBparallel\fR's own shorthands are implemented using \fB\-\-rpl\fR. .PP In Perl code the bigrams \fB{=\fR and \fB=}\fR rarely exist. They look like a matching pair and can be entered on all keyboards. This made them good candidates for enclosing the Perl expression in the replacement strings. Another candidate ,, and ,, was rejected because they do not look like a matching pair. \fB\-\-parens\fR was made, so that the users can still use ,, and ,, if they like: \fB\-\-parens ,,,,\fR .PP Internally, however, the \fB{=\fR and \fB=}\fR are replaced by \e257< and \&\e257>. This is to make it simpler to make regular expressions. You only need to look one character ahead, and never have to look behind. .SS "Test suite" .IX Subsection "Test suite" \&\s-1GNU\s0 \fBparallel\fR uses its own testing framework. This is mostly due to historical reasons. It deals reasonably well with tests that are dependent on how long a given test runs (e.g. more than 10 secs is a pass, but less is a fail). It parallelizes most tests, but it is easy to force a test to run as the single test (which may be important for timing issues). It deals reasonably well with tests that fail intermittently. It detects which tests failed and pushes these to the top, so when running the test suite again, the tests that failed most recently are run first. .PP If \s-1GNU\s0 \fBparallel\fR should adopt a real testing framework then those elements would be important. .PP Since many tests are dependent on which hardware it is running on, these tests break when run on a different hardware than what the test was written for. .PP When most bugs are fixed a test is added, so this bug will not reappear. It is, however, sometimes hard to create the environment in which the bug shows up \- especially if the bug only shows up sometimes. One of the harder problems was to make a machine start swapping without forcing it to its knees. .SS "Median run time" .IX Subsection "Median run time" Using a percentage for \fB\-\-timeout\fR causes \s-1GNU\s0 \fBparallel\fR to compute the median run time of a job. The median is a better indicator of the expected run time than average, because there will often be outliers taking way longer than the normal run time. .PP To avoid keeping all run times in memory, an implementation of remedian was made (Rousseeuw et al). .SS "Error messages and warnings" .IX Subsection "Error messages and warnings" Error messages like: \s-1ERROR,\s0 Not found, and 42 are not very helpful. \s-1GNU\s0 \fBparallel\fR strives to inform the user: .IP "\(bu" 2 What went wrong? .IP "\(bu" 2 Why did it go wrong? .IP "\(bu" 2 What can be done about it? .PP Unfortunately it is not always possible to predict the root cause of the error. .SS "Determine number of CPUs" .IX Subsection "Determine number of CPUs" CPUs is an ambiguous term. It can mean the number of socket filled (i.e. the number of physical chips). It can mean the number of cores (i.e. the number of physical compute cores). It can mean the number of hyperthreaded cores (i.e. the number of virtual cores \- with some of them possibly being hyperthreaded). .PP On ark.intel.com Intel uses the terms \fIcores\fR and \fIthreads\fR for number of physical cores and the number of hyperthreaded cores respectively. .PP \&\s-1GNU\s0 \fBparallel\fR uses uses \fICPUs\fR as the number of compute units and the terms \fIsockets\fR, \fIcores\fR, and \fIthreads\fR to specify how the number of compute units is calculated. .SS "Computation of load" .IX Subsection "Computation of load" Contrary to the obvious \fB\-\-load\fR does not use load average. This is due to load average rising too slowly. Instead it uses \fBps\fR to list the number of threads in running or blocked state (state D, O or R). This gives an instant load. .PP As remote calculation of load can be slow, a process is spawned to run \&\fBps\fR and put the result in a file, which is then used next time. .SS "Killing jobs" .IX Subsection "Killing jobs" \&\s-1GNU\s0 \fBparallel\fR kills jobs. It can be due to \fB\-\-memfree\fR, \fB\-\-halt\fR, or when \s-1GNU\s0 \fBparallel\fR meets a condition from which it cannot recover. Every job is started as its own process group. This way any (grand)*children will get killed, too. The process group is killed with the specification mentioned in \fB\-\-termseq\fR. .SS "\s-1SQL\s0 interface" .IX Subsection "SQL interface" \&\s-1GNU\s0 \fBparallel\fR uses the \s-1DBURL\s0 from \s-1GNU\s0 \fBsql\fR to give database software, username, password, host, port, database, and table in a single string. .PP The \s-1DBURL\s0 must point to a table name. The table will be dropped and created. The reason for not reusing an existing table is that the user may have added more input sources which would require more columns in the table. By prepending '+' to the \s-1DBURL\s0 the table will not be dropped. .PP The table columns are similar to joblog with the addition of \fBV1\fR \&.. \fBVn\fR which are values from the input sources, and Stdout and Stderr which are the output from standard output and standard error, respectively. .PP The Signal column has been renamed to _Signal due to Signal being a reserved word in MySQL. .SS "Logo" .IX Subsection "Logo" The logo is inspired by the Cafe Wall illusion. The font is DejaVu Sans. .SS "Citation notice" .IX Subsection "Citation notice" Funding a free software project is hard. \s-1GNU\s0 \fBparallel\fR is no exception. On top of that it seems the less visible a project is, the harder it is to get funding. And the nature of \s-1GNU\s0 \fBparallel\fR is that it will never be seen by \*(L"the guy with the checkbook\*(R", but only by the people doing the actual work. .PP This problem has been covered by others \- though no solution has been found: https://www.slideshare.net/NadiaEghbal/consider\-the\-maintainer https://www.numfocus.org/blog/why\-is\-numpy\-only\-now\-getting\-funded/ .PP Before implementing the citation notice it was discussed with the users: https://lists.gnu.org/archive/html/parallel/2013\-11/msg00006.html .PP Having to spend 10 seconds on running \fBparallel \-\-citation\fR once is no doubt not an ideal solution, but no one has so far come up with an ideal solution \- neither for funding \s-1GNU\s0 \fBparallel\fR nor other free software. .PP If you believe you have the perfect solution, you should try it out, and if it works, you should post it on the email list. Ideas that will cost work and which have not been tested are, however, unlikely to be prioritized. .PP Running \fBparallel \-\-citation\fR one single time takes less than 10 seconds, and will silence the citation notice for future runs. This is comparable to graphical tools where you have to click a checkbox saying \*(L"Do not show this again\*(R". But if that is too much trouble for you, why not use one of the alternatives instead? See a list in: \&\fBman parallel_alternatives\fR. .PP As the request for citation is not a legal requirement this is acceptable under GPLv3 and cleared with Richard M. Stallman himself. Thus it does not fall under this: https://www.gnu.org/licenses/gpl\-faq.en.html#RequireCitation .SH "Ideas for new design" .IX Header "Ideas for new design" .SS "Multiple processes working together" .IX Subsection "Multiple processes working together" Open3 is slow. Printing is slow. It would be good if they did not tie up resources, but were run in separate threads. .SS "\-\-rrs on remote using a perl wrapper" .IX Subsection "--rrs on remote using a perl wrapper" \&... | perl \-pe '$/=$recend$recstart;BEGIN{ if(substr($_) eq \f(CW$recstart\fR) substr($_)="\*(L" } eof and substr($_) eq \f(CW$recend\fR) substr($_)=\*(R"" .PP It ought to be possible to write a filter that removed rec sep on the fly instead of inside \s-1GNU\s0 \fBparallel\fR. This could then use more cpus. .PP Will that require 2x record size memory? .PP Will that require 2x block size memory? .SH "Historical decisions" .IX Header "Historical decisions" These decisions were relevant for earlier versions of \s-1GNU\s0 \fBparallel\fR, but not the current version. They are kept here as historical record. .SS "\-\-tollef" .IX Subsection "--tollef" You can read about the history of \s-1GNU\s0 \fBparallel\fR on https://www.gnu.org/software/parallel/history.html .PP \&\fB\-\-tollef\fR was included to make \s-1GNU\s0 \fBparallel\fR switch compatible with the parallel from moreutils (which is made by Tollef Fog Heen). This was done so that users of that parallel easily could port their use to \s-1GNU\s0 \fBparallel\fR: Simply set \fBPARALLEL=\*(L"\-\-tollef\*(R"\fR and that would be it. .PP But several distributions chose to make \fB\-\-tollef\fR global (by putting it into /etc/parallel/config) without making the users aware of this, and that caused much confusion when people tried out the examples from \&\s-1GNU\s0 \fBparallel\fR's man page and these did not work. The users became frustrated because the distribution did not make it clear to them that it has made \fB\-\-tollef\fR global. .PP So to lessen the frustration and the resulting support, \fB\-\-tollef\fR was obsoleted 20130222 and removed one year later. .SS "Transferring of variables and functions" .IX Subsection "Transferring of variables and functions" Until 20150122 variables and functions were transferred by looking at \&\f(CW$SHELL\fR to see whether the shell was a \fB*csh\fR shell. If so the variables would be set using \fBsetenv\fR. Otherwise they would be set using \fB=\fR. This caused the content of the variable to be repeated: .PP echo \f(CW$SHELL\fR | grep \*(L"/t\e{0,1\e}csh\*(R" > /dev/null && setenv \s-1VAR\s0 foo || export VAR=foo