python - Pythonic way to process 200 million element data set? -


i have directory of 1,000 files. each file has many lines each line ngram varying 4 - 8 bytes. i'm trying parse files distinct ngrams header row, each file, want write row has frequency of ngram sequence occurring within file.

the following code made through gathering headers, hit memory error when trying write headers csv file. running on amazon ec2 instance 30gb of ram. can provide recommendations optimizations of i'm unaware?

#note: combination of list , set used maintain order of metadata #but still performance since non-meta headers not need maintain order header_list = [] header_set = set() header_list.extend(meta_list) ngram_dir in ngram_dirs:   ngram_files = os.listdir(ngram_dir)   ngram_file in ngram_files:             open(ngram_dir+ngram_file, 'r') file:         line in file:           if not '.' in line , line.rstrip('\n') not in ignore_list:             header_set.add(line.rstrip('\n'))  header_list.extend(header_set)#memory error occurred here  outfile = open(model_dir+model_file_name, 'w') csvwriter = csv.writer(outfile) csvwriter.writerow(header_list)  #convert ngram representations vector model of frequencies ngram_dir in ngram_dirs:   ngram_files = os.listdir(ngram_dir)   ngram_file in ngram_files:             open(ngram_dir+ngram_file, 'r') file:         write_list = []         linecount = 0         header_dict = collections.ordereddict.fromkeys(header_set, 0)         while linecount < meta_fields: #meta_fields = 3           line = file.readline()           write_list.append(line.rstrip('\n'))           linecount += 1          file_counter = collections.counter(line.rstrip('\n') line in file)         header_dict.update(file_counter)         value in header_dict.itervalues():           write_list.append(value)         csvwriter.writerow(write_list)  outfile.close()

just don't extend list then. use chain itertools chain list , set instead.

instead of this:

header_list.extend(header_set)#memory error occurred here

do (assuming csvwriter.writerow accepts iterator):

headers = itertools.chain(header_list, header_set) ... csvwriter.writerow(headers)

that should @ least avoid memory issue you're seeing.


Comments

Post a Comment

Popular posts from this blog

apache - Remove .php and add trailing slash in url using htaccess not loading css -

i have add following code htaccess file removing .php extension , add trailing slash urls. worked fine reduced page loading speed , css code not loading @ all. pages ugly. please help rewriteengine on rewritebase / ## hide .php extension snippet # externally redirect /dir/foo.php /dir/foo rewritecond %{the_request} ^[a-z]{3,}\s([^.]+)\.php [nc] rewriterule ^ %1/ [r,l] # add trailing slash rewritecond %{request_filename} !-f rewritecond %{request_uri} !/$ rewriterule . %{request_uri}/ [l,r=301] # internally forward /dir/foo /dir/foo.php rewritecond %{request_filename} !-d rewritecond %{request_filename}.php -f rewriterule ^(.*?)/?$ $1.php [l] we can rewrite urls using .htaccess files. before writing in .htaccess files need verify kind of server using. can check server adding a <?php phpinfo(); ?> in php page. ensure mod_rewrite module enabled in apache server. can identified checking phpinfo included page. common methods used url rewriting follows ...
Read more

javascript - jQuery show full size image on click -

pretty self explanatory not workking . i want display full image when user click on image... yes know there tons of plugins but i want keep simple. any idea ? css : .full { display: none; width: 500px; height: 500px; margin-left: 100px; margin-top: -300px; position: fixed; background-color: #f00; z-index: 6; } .thumbnail { width: 50px; height: 50px; } html: <img class="thumbnail" src="http://4.bp.blogspot.com/-uhbdp2esxng/txneptvi92i/aaaaaaaaaky/upfqis7zto0/s1600/rihanna-wallpaper.jpg"> jquery: $(document).ready(function() { $('.thumbnail').on('click', function() { var img = $('<img />',{src:this.src,'class':'full'}); $('.full').html(img).show(); console.log('done!'); }); }); try looking how jquery works. $(document).ready(function() { $('.thumbnail').on('click', function(...
Read more