bash - Can GNU Parallel execute more parallel processes? -
can example execute:
parallel -j 200 < list0
where "list" has:
nice -n -20 parallel -j 100 < list2 nice -n -20 parallel -j 100 < list1
would feasible/possible?
the number of processes limited. in practice, don't run more few dozens of processes in parallel (at least on personal computer; on million dollars machines limit bigger).
read fork(2), execve(2), setrlimit(2) man pages, understand when , how can fail.
and before reaching limit fork
fail, you'll reach threshold doing more parallel processes slow down entire computation, since scheduler inside kernel overstressed.
if have access network of computers (perhaps 2 of them) consider mpi. if can recode of applications read pthreads (e.g. this tutorial).
Comments
Post a Comment