linux - How to check folder of text files for duplicate URLs -


i have folder *.txt files. want regularly check files duplicate urls.

actually, save bookmarks in these files, @ least 2 lines, such as:

www.domain.com quite popular domain name

as happens, save same url description, such as:

www.domain.com should buy domain whenever happen have enough money

all entries separated single blank lines. , urls in markdown format:

[domain.com](www.domain.com)

how crawl folder duplicate urls?

the solution found far cat in combination it's uniq pipe:

cat folder/* |sort|uniq|less > dupefree.txt

the problem is:

  1. this check full identical lines - markdown urls ignored , connected comments lost
  2. i don't want output cleaned text file need hint urls duplicates

how can proper duplicate check?

here source file made description

cat file  www.domain.com quite popular domain name  www.domain.com should buy domain whenever happen have enough money entries separated single blank lines. , urls in markdown format:  [domain.com](www.domain.com) how crawl folder duplicate urls?

using awk export duplicate domain name:

awk 'begin{fs="\n";rs=""} { if ($1~/\[/) { split($1,a,"[)(]"); domain[a[2]]++}   else {domain[$1]++} } end{ (i in domain)        if (domain[i]>1) print "duplicate domain found: ",i     }' file  duplicate domain found:  www.domain.com

Comments

Post a Comment

Popular posts from this blog

apache - Remove .php and add trailing slash in url using htaccess not loading css -

i have add following code htaccess file removing .php extension , add trailing slash urls. worked fine reduced page loading speed , css code not loading @ all. pages ugly. please help rewriteengine on rewritebase / ## hide .php extension snippet # externally redirect /dir/foo.php /dir/foo rewritecond %{the_request} ^[a-z]{3,}\s([^.]+)\.php [nc] rewriterule ^ %1/ [r,l] # add trailing slash rewritecond %{request_filename} !-f rewritecond %{request_uri} !/$ rewriterule . %{request_uri}/ [l,r=301] # internally forward /dir/foo /dir/foo.php rewritecond %{request_filename} !-d rewritecond %{request_filename}.php -f rewriterule ^(.*?)/?$ $1.php [l] we can rewrite urls using .htaccess files. before writing in .htaccess files need verify kind of server using. can check server adding a <?php phpinfo(); ?> in php page. ensure mod_rewrite module enabled in apache server. can identified checking phpinfo included page. common methods used url rewriting follows ...
Read more

javascript - jQuery show full size image on click -

pretty self explanatory not workking . i want display full image when user click on image... yes know there tons of plugins but i want keep simple. any idea ? css : .full { display: none; width: 500px; height: 500px; margin-left: 100px; margin-top: -300px; position: fixed; background-color: #f00; z-index: 6; } .thumbnail { width: 50px; height: 50px; } html: <img class="thumbnail" src="http://4.bp.blogspot.com/-uhbdp2esxng/txneptvi92i/aaaaaaaaaky/upfqis7zto0/s1600/rihanna-wallpaper.jpg"> jquery: $(document).ready(function() { $('.thumbnail').on('click', function() { var img = $('<img />',{src:this.src,'class':'full'}); $('.full').html(img).show(); console.log('done!'); }); }); try looking how jquery works. $(document).ready(function() { $('.thumbnail').on('click', function(...
Read more