Image sequence array concatenation - need optimization help!

Find out why the . goes before the /

Moderator: Paul Tuersley

Post Reply
SpencerT
Posts: 8
Joined: March 29th, 2010, 5:08 pm

Help needed!

I have a script I've been using/developing for some time now which greatly simplifies importing multiple image sequences (mainly for CG renders with render passes). Everything is working great but there's one specific area that runs really slowly when you start getting into thousands of images and I'm trying to figure out how to speed it up.

What I am trying to optimize is going through an array of potentially thousands of image files, and concatenating it down to just the first image in the sequence. For example my array might be [img.001.png, img.002.png, img.003.png, img.004.png, img.005.png] and what I want to do is remove every instance of "img.###.png" except for the first one. So after my function I would have just: [img.001.png].

Here's how I do it right now:
1) I take array item A, regex out the number sequence
2) then array item B, regex out the number sequence
3) and compare the two to see if they are identical. If they are I splice out item B

This works 100% of the time which is awesome, but is quite slow when you get above a thousand or two images... What I'm trying to do is find a way to do this "pruning" step more quickly. A friend of mine told me regex is pretty slow so maybe I need to do it without regex? I also found several native JavaScript array methods that might help such as forEach() every() and filter() but I don't see how these are going to be faster because I still need to do 2 regex evaluations, and I still need to compare the items to each other.

Any help would be immensely appreciated! And if anyone wants my script just ask, I'm definitely not planning on charging for it.

- Spencer


Here's a snippet of my code:

Code: Select all

currentFolder = new Folder ("//12.34.5.67/my folder on the network/")
var folderChildren = currentFolder.getFiles().sort();

var searcher = new RegExp("\\d{3,5}[.]");
for (var i = 0; i < folderChildren.length; i++) {
    // Go through the array and strip out all elements that are the same once their numbers have been removed with a regex
    if (i > 0) {
        currentResult = searcher.exec(folderChildren[i].name); //check if a sequence
        if (currentResult) { // it is a sequence
            // first parse out the comparison strings - current item and item before
            var testNameBefore = folderChildren[i-1].name;

            //if we have a sequence before our current item, we need to delete the numbers.
            var beforeNum = searcher.exec(testNameBefore);
            if (beforeNum) {
                testNameBefore = testNameBefore.substring(0, testNameBefore.length-8);
            }

            var testNameCurrent = folderChildren[i].name;
            testNameCurrent = folderChildren[i].name.substring(0, testNameCurrent.length-8);

            //compare to the element before it and delete if the same!!
            if (testNameBefore == testNameCurrent) {
                folderChildren.splice(i, 1);
                i--;
            }
        }
    }
}
beginUndoGroup
Posts: 81
Joined: November 27th, 2012, 6:41 am

ExtendScript doesnt like big arrays. Past 500 entries it is getting a bit slow.
What it slowing you down is probably the use of sort () and splice().
You dont need the sort() because getFiles() takes the files sorted anyway.
And to avoid the splice(), you can push a new array (the final result) instead of splicing the big array.
This should considerably fasten things up.
Another way is to use the fact that the getFiles() method accepts one argument, that can be a function (but also a string or regexp) that should return true/false, but in between you can put whatever you like:

Code: Select all

if (!currentFolder) currentFolder = Folder.selectDialog();
var sequence = {};
var sequenceFiles = [];
var searcher = /^(.+\D)(\d+)\.\w{2,4}$/;
currentFolder.getFiles(function(file){
    var sequenceName, num;
    if (file instanceof File && searcher.test(file.displayName)){
        sequenceName = RegExp.$1;
        num = +RegExp.$2;
        if (!sequence[sequenceName]){
            sequence[sequenceName] = true;
            sequenceFiles.push(file);
            };
        };
    return false;
    });
The regexp captures the name of the file (without the numbering) and the numbering (but that one is finally not used).
The sequence variable is an auxialiry object that maintains the list of names of the current sequences, and the final result is sequenceFiles (an array).

The above code took less than a second in a folder with 13000+ files (a single image sequence).
Your code gets stuck already at the sort stage (takes forever) and again at the splice stage (i cancelled task after 5 minutes).

Xavier
Post Reply