hadoop - Where are logs in Spark on YARN? -


i'm new spark. can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). there no log after execution.

the following command used run spark example. logs not found in history server in normal mapreduce job.

spark_jar=./assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop2.0.0-cdh4.2.1.jar \ ./bin/spark-class org.apache.spark.deploy.yarn.client --jar ./spark-example-1.0.0.jar \ --class simpleapp --args yarn-standalone  --num-workers 3 --master-memory 1g \ --worker-memory 1g --worker-cores 1

where can find logs/stderr/stdout?

is there someplace set configuration? did find output console saying:

14/04/14 18:51:52 info client: command applicationmaster: $java_home/bin/java -server -xmx640m -djava.io.tmpdir=$pwd/tmp org.apache.spark.deploy.yarn.applicationmaster --class simpleapp --jar ./spark-example-1.0.0.jar --args 'yarn-standalone' --worker-memory 1024 --worker-cores 1 --num-workers 3 1> <log_dir>/stdout 2> <log_dir>/stderr

in line, notice 1> $log_dir/stdout 2> $log_dir/stderr

where can log_dir set?

pretty article question:

running spark on yarn - see section "debugging application". decent explanation required examples.

the thing need follow correctly working history server spark close spark context in application. otherwise, application history server not see complete , not show (despite history ui accessible not visible).


Comments

Post a Comment

Popular posts from this blog

apache - Remove .php and add trailing slash in url using htaccess not loading css -

i have add following code htaccess file removing .php extension , add trailing slash urls. worked fine reduced page loading speed , css code not loading @ all. pages ugly. please help rewriteengine on rewritebase / ## hide .php extension snippet # externally redirect /dir/foo.php /dir/foo rewritecond %{the_request} ^[a-z]{3,}\s([^.]+)\.php [nc] rewriterule ^ %1/ [r,l] # add trailing slash rewritecond %{request_filename} !-f rewritecond %{request_uri} !/$ rewriterule . %{request_uri}/ [l,r=301] # internally forward /dir/foo /dir/foo.php rewritecond %{request_filename} !-d rewritecond %{request_filename}.php -f rewriterule ^(.*?)/?$ $1.php [l] we can rewrite urls using .htaccess files. before writing in .htaccess files need verify kind of server using. can check server adding a <?php phpinfo(); ?> in php page. ensure mod_rewrite module enabled in apache server. can identified checking phpinfo included page. common methods used url rewriting follows ...
Read more

javascript - jQuery show full size image on click -

pretty self explanatory not workking . i want display full image when user click on image... yes know there tons of plugins but i want keep simple. any idea ? css : .full { display: none; width: 500px; height: 500px; margin-left: 100px; margin-top: -300px; position: fixed; background-color: #f00; z-index: 6; } .thumbnail { width: 50px; height: 50px; } html: <img class="thumbnail" src="http://4.bp.blogspot.com/-uhbdp2esxng/txneptvi92i/aaaaaaaaaky/upfqis7zto0/s1600/rihanna-wallpaper.jpg"> jquery: $(document).ready(function() { $('.thumbnail').on('click', function() { var img = $('<img />',{src:this.src,'class':'full'}); $('.full').html(img).show(); console.log('done!'); }); }); try looking how jquery works. $(document).ready(function() { $('.thumbnail').on('click', function(...
Read more