How to use Hadoop streaming input parameter for matlab shell script -

-

actually want execute matlab code in hadoop streaming. doubt how use hadoop streaming input parameter value matlab input. example ,

this matlab file imreadtest.m (simple coding)

rgbimage = imread('/usr/new.jpg'); imwrite(rgbimage,'/usr/ot/testedimage1.jpg');

my shell script 

#!/bin/sh matlabbg imreadtest.m -nodisplay

normally works in ubuntu. (not in hadoop). have stored these 2 files in hdfs using hue. matlab script looks (imrtest.m)

rgbimage = imread(stdin); imwrite(rgbimage,stdout);

my shell script (imrtest.sh). 

#!/bin/sh matlabbg imrtest.m -nodisplay

i have tried execute in hadoop streaming 

 hadoop@xxx:/usr/local/master/hadoop$ $hadoop_home/bin/hadoop jar $hadoop_home/hadoop-streaming.jar -mapper /usr/ot/imrtest.sh -file /usr/ot/imrtest.sh -input /usr/ot/testedimage.jpg -output /usr/ot/opt

but got error this

packagejobjar: [/usr/ot/imrtest.sh, /usr/local/master/temp/hadoop- unjar4018041785380098978/] [] /tmp/streamjob7077345699332124679.jar tmpdir=null 14/03/06 15:51:41 warn snappy.loadsnappy: snappy native library available 14/03/06 15:51:41 info util.nativecodeloader: loaded native-hadoop library 14/03/06 15:51:41 info snappy.loadsnappy: snappy native library loaded 14/03/06 15:51:41 info mapred.fileinputformat: total input paths process : 1 14/03/06 15:51:42 info streaming.streamjob: getlocaldirs(): [/usr/local/master/temp/mapred/local] 14/03/06 15:51:42 info streaming.streamjob: running job: job_201403061205_0015 14/03/06 15:51:42 info streaming.streamjob: kill job, run: 14/03/06 15:51:42 info streaming.streamjob: /usr/local/master/hadoop/bin/hadoop job  -dmapred.job.tracker=slave3:8021 -kill job_201403061205_0015 14/03/06 15:51:42 info streaming.streamjob: tracking url: http://slave3:50030/jobdetails.jsp?jobid=job_201403061205_0015 14/03/06 15:51:43 info streaming.streamjob:  map 0%  reduce 0% 14/03/06 15:52:15 info streaming.streamjob:  map 100%  reduce 100% 14/03/06 15:52:15 info streaming.streamjob: kill job, run: 14/03/06 15:52:15 info streaming.streamjob: /usr/local/master/hadoop/bin/hadoop job  -dmapred.job.tracker=slave3:8021 -kill job_201403061205_0015 14/03/06 15:52:15 info streaming.streamjob: tracking url: http://slave3:50030/jobdetails.jsp?jobid=job_201403061205_0015 14/03/06 15:52:15 error streaming.streamjob: job not successful. error: na 14/03/06 15:52:15 info streaming.streamjob: killjob...

streaming command failed!

jobtracker error log job is

host=null user=hadoop hadoop_user=null last hadoop input: |null| last tool output: |null| date: thu mar 06 15:51:51 ist 2014 java.io.ioexception: broken pipe @ java.io.fileoutputstream.writebytes(native method) @ java.io.fileoutputstream.write(fileoutputstream.java:297) @ java.io.bufferedoutputstream.write(bufferedoutputstream.java:122) @ java.io.bufferedoutputstream.flushbuffer(bufferedoutputstream.java:82) @ java.io.bufferedoutputstream.write(bufferedoutputstream.java:126) @ java.io.dataoutputstream.write(dataoutputstream.java:107) @ org.apache.hadoop.streaming.io.textinputwriter.writeutf8(textinputwriter.java:72) @ org.apache.hadoop.streaming.io.textinputwriter.writevalue(textinputwriter.java:51) @ org.apache.hadoop.streaming.pipemapper.map(pipemapper.java:110) @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:50) @ org.apache.hadoop.streaming.pipe    java.io.ioexception: log:null        .        .        .

please suggest me how input hadoop streaming input matlab script input, output.


Comments

Post a Comment

Popular posts from this blog

c# - How to get the current UAC mode -

-
can check current uac type set in device before running application through c#. i got know approach check whether current user administrator or not. but, need know uac type set (i.e. default/never notify) internal logic. is possible? please help. this article self-elevation , checking level program runs in. due security reasons i'm afraid won't able retrieve uac level set. http://code.msdn.microsoft.com/windowsdesktop/csuacselfelevation-644673d3
Read more

postgresql - Lazarus + Postgres: incomplete startup packet -

-
i've been building lazarus pascal program using postgresql on back-end. used work fine following lines opening db. dbconn:= tpqconnection.create(nil); dbconn.hostname := 'localhost'; dbconn.databasename:= 'dbhrs'; dbconn.username:='mizk'; dbconn.password:='123'; dbconn.open; if dbconn.connected openhrsdb := true else openhrsdb := false; but moment change localhost server's ip (on lan), program stops. no errors or warnings. have no clue happening. here related details may narrow down problem. the program i'm running workstation on same lan server. both machines run ubuntu 12.04 desktop i can access postgres db on server using pgadmin iii, guess it's not firewall issue. i have allowed postgresql ufw , ufw disabled. strangely, when firewall disabled, can ssh server terminal using workstation. when it's enabled, can't my biggest concern right why pascal program not working when i...
Read more

javascript - Ajax jqXHR.status==0 fix error -

-
$.ajax({ url: urlstring, datatype: "json", type: "get", success: function (data) { alert(data); }, error: function (jqxhr, exception) { if (jqxhr.status === 0) { alert('not connect.\n verify network.'); } else if (jqxhr.status == 404) { alert('requested page not found. [404]'); } else if (jqxhr.status == 500) { alert('internal server error [500].'); } else if (exception === 'parsererror') { alert('requested json parse failed.'); } else if (exception === 'timeout') { alert('time out error.'); } else if (exception === 'abort') { alert('ajax request aborted.'); } else { alert('uncaught error.\n' + jqxhr.responsete...
Read more