scala - Mock a TSV source with Scalding JobTest -


i'm having hard time making unit test scalding job.

my job expects file 3 fields:

  textline(args("input"))     .map('url -> ('fetchedurl,'date,'info)){   ...

naively would've expected fields got mapped ntuple, without needing further setup. test it's not , further contract needs stablished:

jobtest[com.kohls.crawler.miner]   .arg("input", "inputfile")   .arg("output", "outputfile")   .source(textline("inputfile"), list(("https://en.wikipedia.org/wiki/test" ,"mon apr 14 15:08:11 cdt 2014", "extra info")))   .sink[(string,date,array[byte])](tsv("outputfile")){ ... }

this fails cascading.tuple.fieldsresolverexception: not select fields: [{1}:'url'], from: [{2}:'offset', 'line']. guess need declare tsv fields in kind of way before feeding textline's input.

most documentation i've found spotty in regard. correct why define test?

you should use tsv instead of textline. tsv takes declared fields second input parameter. job this:

tsv(args("input"), ('fetchedurl,'date,'info), skipheader = false/true).read   .map(...)   .write(tsv(args("output"), writeheader = false/true)

and job test this:

jobtest[com.kohls.crawler.miner]   .arg("input", "inputfile")   .arg("output", "outputfile")   .source(tsv("inputfile"), list(("https://en.wikipedia.org/wiki/test" ,"mon apr 14 15:08:11 cdt 2014", "extra info")))   .sink[(string,date,array[byte])](tsv("outputfile")) { ... }   .run   .finish

Comments

Post a Comment

Popular posts from this blog

apache - Remove .php and add trailing slash in url using htaccess not loading css -

i have add following code htaccess file removing .php extension , add trailing slash urls. worked fine reduced page loading speed , css code not loading @ all. pages ugly. please help rewriteengine on rewritebase / ## hide .php extension snippet # externally redirect /dir/foo.php /dir/foo rewritecond %{the_request} ^[a-z]{3,}\s([^.]+)\.php [nc] rewriterule ^ %1/ [r,l] # add trailing slash rewritecond %{request_filename} !-f rewritecond %{request_uri} !/$ rewriterule . %{request_uri}/ [l,r=301] # internally forward /dir/foo /dir/foo.php rewritecond %{request_filename} !-d rewritecond %{request_filename}.php -f rewriterule ^(.*?)/?$ $1.php [l] we can rewrite urls using .htaccess files. before writing in .htaccess files need verify kind of server using. can check server adding a <?php phpinfo(); ?> in php page. ensure mod_rewrite module enabled in apache server. can identified checking phpinfo included page. common methods used url rewriting follows ...
Read more

javascript - jQuery show full size image on click -

pretty self explanatory not workking . i want display full image when user click on image... yes know there tons of plugins but i want keep simple. any idea ? css : .full { display: none; width: 500px; height: 500px; margin-left: 100px; margin-top: -300px; position: fixed; background-color: #f00; z-index: 6; } .thumbnail { width: 50px; height: 50px; } html: <img class="thumbnail" src="http://4.bp.blogspot.com/-uhbdp2esxng/txneptvi92i/aaaaaaaaaky/upfqis7zto0/s1600/rihanna-wallpaper.jpg"> jquery: $(document).ready(function() { $('.thumbnail').on('click', function() { var img = $('<img />',{src:this.src,'class':'full'}); $('.full').html(img).show(); console.log('done!'); }); }); try looking how jquery works. $(document).ready(function() { $('.thumbnail').on('click', function(...
Read more