sql - How do I accumulate arrays into maps? -


i have table t columns:

cookie     string   keywords   array<string>    fqdn       string   pixel      bigint

i want write like

select cookie, ???? t group cookie;

to table columns

cookie     string   keywords   map<string,int>    fqdn       map<string,int>   pixel      array<bigint>

where

  • cookie unique (guaranteed by cookie)
  • keywords counts how many times keyword appeared in arrays in original table t
  • fqdn counts how many times domain appeared in rows given cookie
  • pixel counts how many times pixel appeared in rows given cookie

you can use "vector" udf's in brickhouse ( http://github.com/klout/brickhouse ). in brickhouse, either array or map can considered "vector". array, array index considered dimension, , numeric value considered magnitude in dimension. map, consider string key "dimension" of vector in large dimensional space, , map value magnitude. ( text-analysis type problems, similar looks doing).

something following should work

select cookie,    union_vector_sum( keyword_map),    union_vector_sum( map( fqdn, 1 ) ),    collect_set( pixel) (   select cookie, fqdn, pixel,          collect( keyword, 1 ) keyword_map   t   lateral view explode( keywords ) k keyword   group cookie, fqdn, pixel ) xk group cookie;

we should have new map constructor udf takes array , single value, don't need inner explode , collect. don't think produce additional map-reduce step in form however.

there vector, , "bag of words" udfs in brickhouse now, should add more. have special requests ??


Comments

Post a Comment

Popular posts from this blog

apache - Remove .php and add trailing slash in url using htaccess not loading css -

i have add following code htaccess file removing .php extension , add trailing slash urls. worked fine reduced page loading speed , css code not loading @ all. pages ugly. please help rewriteengine on rewritebase / ## hide .php extension snippet # externally redirect /dir/foo.php /dir/foo rewritecond %{the_request} ^[a-z]{3,}\s([^.]+)\.php [nc] rewriterule ^ %1/ [r,l] # add trailing slash rewritecond %{request_filename} !-f rewritecond %{request_uri} !/$ rewriterule . %{request_uri}/ [l,r=301] # internally forward /dir/foo /dir/foo.php rewritecond %{request_filename} !-d rewritecond %{request_filename}.php -f rewriterule ^(.*?)/?$ $1.php [l] we can rewrite urls using .htaccess files. before writing in .htaccess files need verify kind of server using. can check server adding a <?php phpinfo(); ?> in php page. ensure mod_rewrite module enabled in apache server. can identified checking phpinfo included page. common methods used url rewriting follows ...
Read more

javascript - jQuery show full size image on click -

pretty self explanatory not workking . i want display full image when user click on image... yes know there tons of plugins but i want keep simple. any idea ? css : .full { display: none; width: 500px; height: 500px; margin-left: 100px; margin-top: -300px; position: fixed; background-color: #f00; z-index: 6; } .thumbnail { width: 50px; height: 50px; } html: <img class="thumbnail" src="http://4.bp.blogspot.com/-uhbdp2esxng/txneptvi92i/aaaaaaaaaky/upfqis7zto0/s1600/rihanna-wallpaper.jpg"> jquery: $(document).ready(function() { $('.thumbnail').on('click', function() { var img = $('<img />',{src:this.src,'class':'full'}); $('.full').html(img).show(); console.log('done!'); }); }); try looking how jquery works. $(document).ready(function() { $('.thumbnail').on('click', function(...
Read more