regex - Why is Scala's combinator parsing slow when parsing large files? What can I do? -

i need parse files have millions of lines. noticed combinator parser gets slower , slower parses more , more lines. problem seems in scala's "rep" or regex parsers, because behaviour occurs simple example parser shown below:

def file: parser[int] = rep(line) ^^ { 1 }  // file repetition of lines  def line: parser[int] = """(?m)^.*$""".r ^^ { 0 } // reads line , returns 0 

when try parse file millions of lines of equal length simple parser, in beginning parses 46 lines/ms. after 370000 lines, speed drops 20 lines/ms. after 840000 lines, drops 10 lines/ms. after 1790000 lines, 5 lines/ms...

my questions are:

• why happen?

• what can prevent this?

this result of change in java 7u6 doesn't have substrings part of original string. big strings copied on , over, causing lots , lots of memory churn (among other things). increase amount of stuff you've parsed (i'm assuming you're storing @ least of it), garbage collector has more , more work do, creating garbage has steeper , steeper penalty.

there ticket fix memory usage, , code zach moazeni there lets wrap strings inside construct make substrings (which can pass parser in place of strings).

this won't change overall result parsing slows down, should reduce time overall.

also, wouldn't advise making file repetition of lines. you're making parser keep track of entire file when need not. i'd feed in line @ time. (and if lines short, may not need above fix.)