Possible to set a "nice" level or a throttle of some sort?

[jagthedrummer]

First of all, thanks for this excellent library!

I'm using it in an electron app and the end user can point the app at any directory they'd like, and then the app finds and processes particular types of files in that folder using a glob pattern.

If the user points it at a particularly large folder then the glob process can take several seconds (or minutes!) and the application becomes very sluggish while this happens.

It would be awesome to be able to set a "nice" level or somehow be able to indicate that a slower walk is acceptable in exchange for not clobbering the system.

[isaacs]

Yes that would be a good idea. In extreme cases, it may even be faster to throttle, if it saves switching and memory management costs.

[isaacs]

I took some time to try to make this work, by setting a max number of readdir calls that could be in progress at any one time.

At least a simple naive approach to this sent performance going off a cliff if the max parallel number was less than a couple thousand. ("Cliff" = about a 20% reduction in performance.) Afaict, the slowdown happens just due to the added work of pushing into an array and checking the status.

Putting the throttling up higher (ie, around walkCB2 or walkCB3) isn't possible, because it's a "fan out" kind of action, so it'll deadlock if it hits the limit.

Doing this properly without a significant perf hit looks like it might be a pretty significant refactor.

[6624658763]

Información sobre ivex