linux - Run several jobs parallelly and Efficiently -

-

os: cent-os

i have 30,000 jobs(or scripts) run. each job takes 3-5 min. have 48 cpus(nproc = 48). can use 40 cpus run 40 jobs parallelly. please suggest script or tools can handle 30,000 jobs running each 40 jobs parallely.

what had done:

  • i created 40 different folders , executed jobs parallely creating shell script each directory.

  • i want know better ways handle kind of jobs next time.

as mark setchell says: gnu parallel.

find scripts/ -type f | parallel

if insists on keeping 8 cpus free:

find scripts/ -type f | parallel -j-8

but more efficient use nice give 48 cores when no 1 else needs them:

find scripts/ -type f | nice -n 15 parallel

to learn more:


Comments

Post a Comment

Popular posts from this blog

c# - How to get the current UAC mode -

-
can check current uac type set in device before running application through c#. i got know approach check whether current user administrator or not. but, need know uac type set (i.e. default/never notify) internal logic. is possible? please help. this article self-elevation , checking level program runs in. due security reasons i'm afraid won't able retrieve uac level set. http://code.msdn.microsoft.com/windowsdesktop/csuacselfelevation-644673d3
Read more

postgresql - Lazarus + Postgres: incomplete startup packet -

-
i've been building lazarus pascal program using postgresql on back-end. used work fine following lines opening db. dbconn:= tpqconnection.create(nil); dbconn.hostname := 'localhost'; dbconn.databasename:= 'dbhrs'; dbconn.username:='mizk'; dbconn.password:='123'; dbconn.open; if dbconn.connected openhrsdb := true else openhrsdb := false; but moment change localhost server's ip (on lan), program stops. no errors or warnings. have no clue happening. here related details may narrow down problem. the program i'm running workstation on same lan server. both machines run ubuntu 12.04 desktop i can access postgres db on server using pgadmin iii, guess it's not firewall issue. i have allowed postgresql ufw , ufw disabled. strangely, when firewall disabled, can ssh server terminal using workstation. when it's enabled, can't my biggest concern right why pascal program not working when i...
Read more

javascript - Ajax jqXHR.status==0 fix error -

-
$.ajax({ url: urlstring, datatype: "json", type: "get", success: function (data) { alert(data); }, error: function (jqxhr, exception) { if (jqxhr.status === 0) { alert('not connect.\n verify network.'); } else if (jqxhr.status == 404) { alert('requested page not found. [404]'); } else if (jqxhr.status == 500) { alert('internal server error [500].'); } else if (exception === 'parsererror') { alert('requested json parse failed.'); } else if (exception === 'timeout') { alert('time out error.'); } else if (exception === 'abort') { alert('ajax request aborted.'); } else { alert('uncaught error.\n' + jqxhr.responsete...
Read more