Ubuntu 18.04: GNU parallel can't find and use the full number of cores on a local system

1

I am using GNU parallel (version 20200522) on Ubuntu Linux 18.04.4 and running jobs on all cores of a local server minus 2 cores, that is I am using the -j-2 parameter.

find /folder/ -type f -iname "*.pdf" | parallel -j-2 --nice 2 "script.sh {1} {1/.}; mv -f -v {1} /folder2/; mv -f {1/.}.txt /folder3/" :::: -

However, the program shows

Error: Cannot run any jobs.

I tried using the -j100% parameter and I have seen that it uses just 1 core(job), and I deduce that, for GNU parallel, 100% of the available cores on this system is just one core.

If I use the -j5 parameter (which does not imply autodetection of the total number of cores), everything is alright, parallel launches 5 jobs and uses 5 cores.

The interesting part is that the file /root/.parallel/tmp/sshlogin/MACHINE_NAME/cpuspec contains the following:

1
6
6

which means, I think, that GNU parallel should see 6 available cores.

I have tried deleting the cpuspec file and running parallel again to redetect the total number of cores, but the cpuspec file and the behavior of the program remain the same.

On different systems, deleting the cpuspec file solved all issues, but on this particular system it is not working. The virtual machine is copied from another server with a different configuration, that is why I need deleting the cpuspec file.

What should I do to get GNU parallel to correctly detect the number of cores on the system, so that I can use the -j-2 parameter?

Update 21.07: After deleting once again the folder with the cpuspec file, running the parallel --number-of-sockets/cores/threads commands and using just once the -S 6/: parameter, the problem seems to have resolved itself. Now GNU parallel correctly detects the number of cores and the -j-2 parameters works. I am not sure what good things happened, but I am not able to reproduce the bug anymore. Ole, thank you for your answer. If I meet the bug again or if I am able to reproduce it, I will post it here.

And here is the output to the commands:

parallel --number-of-sockets
1
parallel --number-of-cores
6
parallel --number-of-threads
6

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
stepping        : 2
microcode       : 0xffffffff
cpu MHz         : 2397.218
cache size      : 15360 KB
physical id     : 0
siblings        : 6
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm pti fsgsbase bmi1 avx2 smep bmi2 erms xsaveopt
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips        : 4794.43
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

And it repeats itself for 5 more cores.
gnu-parallel
asked on Stack Overflow Jun 4, 2020 by CosminM • edited Jul 21, 2020 by CosminM

1 Answer

1

You may have found a bug. Please post the output of:

cat /proc/cpuinfo
parallel --number-of-sockets
parallel --number-of-cores
parallel --number-of-threads

Also see if you can make an MCVE.

As a workaround you can use -S 6/: to force GNU Parallel to detect 6 cores on your system.

find /folder/ -type f -iname "*.pdf" |
  parallel -S 6/: -j-2 --nice 2 "script.sh {1} {1/.}; mv -f -v {1} /folder2/; mv -f {1/.}.txt /folder3/"

(Also :::: - can be left out completely: If there is no ::: :::: then GNU Parallel reads from stdin).

answered on Stack Overflow Jun 22, 2020 by Ole Tange

User contributions licensed under CC BY-SA 3.0