configuration - Hadoop conf to determine num map tasks -

-

i have job, hadoop jobs, seems have total of 2 map tasks when running can see in hadoop interface. however, means loading data java heap space error.

i've tried setting many different conf properties in hadoop cluster make job split more tasks nothing seems have effect.

i have tried setting mapreduce.input.fileinputformat.split.maxsize, mapred.max.split.size, dfs.block.size none seem have effect.

i'm using 0.20.2-cdh3u6, , trying run job using cascading.jdbc - job failing on reading data database. think issue can resolved increasing number of splits can't work out how that!

please help! going crazy!

2013-07-23 09:12:15,747 fatal org.apache.hadoop.mapred.child: error running child : java.lang.outofmemoryerror: java heap space         @ com.mysql.jdbc.buffer.<init>(buffer.java:59)         @ com.mysql.jdbc.mysqlio.nextrow(mysqlio.java:1477)         @ com.mysql.jdbc.mysqlio.readsinglerowset(mysqlio.java:2936)         @ com.mysql.jdbc.mysqlio.getresultset(mysqlio.java:477)         @ com.mysql.jdbc.mysqlio.readresultsforqueryorupdate(mysqlio.java:2631)         @ com.mysql.jdbc.mysqlio.readallresults(mysqlio.java:1800)         @ com.mysql.jdbc.mysqlio.sqlquerydirect(mysqlio.java:2221)         @ com.mysql.jdbc.connectionimpl.execsql(connectionimpl.java:2618)         @ com.mysql.jdbc.connectionimpl.execsql(connectionimpl.java:2568)         @ com.mysql.jdbc.statementimpl.executequery(statementimpl.java:1557)         @ cascading.jdbc.db.dbinputformat$dbrecordreader.<init>(dbinputformat.java:97)         @ cascading.jdbc.db.dbinputformat.getrecordreader(dbinputformat.java:376)         @ cascading.tap.hadoop.multiinputformat$1.operate(multiinputformat.java:282)         @ cascading.tap.hadoop.multiinputformat$1.operate(multiinputformat.java:277)         @ cascading.util.util.retry(util.java:624)         @ cascading.tap.hadoop.multiinputformat.getrecordreader(multiinputformat.java:276)         @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:370)         @ org.apache.hadoop.mapred.maptask.run(maptask.java:324)         @ org.apache.hadoop.mapred.child$4.run(child.java:266)         @ java.security.accesscontroller.doprivileged(native method)         @ javax.security.auth.subject.doas(subject.java:415)         @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1278)         @ org.apache.hadoop.mapred.child.main(child.java:260)

you should @ settings of memory management io.sort.mb or mapred.cluster.map.memory.mb because heap space errors due allocation problem , not map number.

if want force map number have consider values used prior others. instance mapreduce.input.fileinputformat.split.maxsize if small generate huge amount of taks if set mapred.tasktracker.map.tasks.maximum small value.

the dfs.block.size has impact on generated map number if greater mapreduce.input.fileinputformat.split.maxsize


Comments

Post a Comment

Popular posts from this blog

javascript - Laravel datatable invalid JSON response -

-
everything work fine on code below fetch user logs: js: <script> $(document).ready(function () { var table =$('#systemlogs').datatable({ responsive: true, processing: true, serverside: true, "language": { "url": "/datatables/media/plug-in/persian.json" }, ajax: '{!! url('/admin/systemlogs/data/systemlogsdatatable') !!}', columns: [ { data: 'name', name: 'name' }, { data: 'message_text', name: 'message_text' }, { data: 'remote_addr', name: 'remote_addr' }, { data: 'log_created_at', name: 'management_logs.created_at' }, { data: 'log_updated_at', name: 'management_logs.updated_at' }, ] }); }); </script> routes: route::get('/systemlogs/data/systemlogsdatatable','systemlogscontrolle...
Read more

java - Exception in thread "main" org.springframework.context.ApplicationContextException: Unable to start embedded container; -

-
i getting error when run spring application: exception in thread "main" org.springframework.context.applicationcontextexception: unable start embedded container; nested exception org.springframework.context.applicationcontextexception: unable start embeddedwebapplicationcontext due missing embeddedservletcontainerfactory bean. @ org.springframework.boot.context.embedded.embeddedwebapplicationcontext.onrefresh(embeddedwebapplicationcontext.java:124) @ org.springframework.context.support.abstractapplicationcontext.refresh(abstractapplicationcontext.java:476) @ org.springframework.boot.context.embedded.embeddedwebapplicationcontext.refresh(embeddedwebapplicationcontext.java:109) @ org.springframework.boot.springapplication.refresh(springapplication.java:691) @ org.springframework.boot.springapplication.run(springapplication.java:320) @ org.springframework.boot.springapplication.run(springapplication.java:952) @ org.springframework.boot.springa...
Read more

sql server 2008 - My Sql Code Get An Error Of Msg 245, Level 16, State 1, Line 1 Conversion failed when converting the varchar value '8:45 AM' to data type int -

-
msg 245, level 16, state 1, line 1 conversion failed when converting varchar value '8:45 am' data type int. script declare @specimenid bigint = 0, @script varchar(max); declare @starttime int = (select convert(int,substring(start_time,0,charindex(':', start_time)) - 12) appt appt._id=112601) --select @starttime set @script = 'select distinct ptlast_name + '', '' + pt.first_name patient_name, dbo.formatdate(pt.dob,''mm/dd/yyyy'') dob, (convert(varchar, (datediff(yy, dob, appt_date) - case when (month(dob) > month(appt_date)) or (month(dob) = month(appt_date) , day(dob) > day(appt_date)) 1 else 0 end)) + ''/'' + convert(varchar, (datediff(m, dateadd(yy, (datediff(yy, dob, appt_date) - case when (month(dob) > month(appt_date)) or (month(dob) = month(appt_date) , day(dob) > day(appt_date)) 1 else 0 end), d...
Read more