Sort an Array by the "Levenshtein Distance" with Best Performance in JavaScript

Sort an array by the Levenshtein Distance with best performance in Javascript

I wrote an inline spell checker a few years ago and implemented a Levenshtein algorithm - since it was inline and for IE8 I did quite a lot of performance optimisation.

var levDist = function(s, t) {
    var d = []; //2d matrix

    // Step 1
    var n = s.length;
    var m = t.length;

    if (n == 0) return m;
    if (m == 0) return n;

    //Create an array of arrays in javascript (a descending loop is quicker)
    for (var i = n; i >= 0; i--) d[i] = [];

    // Step 2
    for (var i = n; i >= 0; i--) d[i][0] = i;
    for (var j = m; j >= 0; j--) d[0][j] = j;

    // Step 3
    for (var i = 1; i <= n; i++) {
        var s_i = s.charAt(i - 1);

        // Step 4
        for (var j = 1; j <= m; j++) {

            //Check the jagged ld total so far
            if (i == j && d[i][j] > 4) return n;

            var t_j = t.charAt(j - 1);
            var cost = (s_i == t_j) ? 0 : 1; // Step 5

            //Calculate the minimum
            var mi = d[i - 1][j] + 1;
            var b = d[i][j - 1] + 1;
            var c = d[i - 1][j - 1] + cost;

            if (b < mi) mi = b;
            if (c < mi) mi = c;

            d[i][j] = mi; // Step 6

            //Damerau transposition
            if (i > 1 && j > 1 && s_i == t.charAt(j - 2) && s.charAt(i - 2) == t_j) {
                d[i][j] = Math.min(d[i][j], d[i - 2][j - 2] + cost);
            }
        }
    }

    // Step 7
    return d[n][m];
}

How can I sort a list using the Levenshtein distance as criterion?

You could pass a custom compare method to the lists sort-method.

The syntax is like below, where "YourComparer" is a function with your implementation for comparing two strings, and "list" is a list of strings.

list.Sort(Function(s1, s2) YourComparer(s1, s2))

"YourComparer" should return an integer value given two string, indicating if the first string should come before, after or have the same position as the second.

Below is an example implementation.

Private Function YourComparer(ByVal s1 As String, ByVal s2 As String) As Integer

    ' **** compare using your own implementation ****

    ' return less than zero if s1 should preceed s2
    ' return zero if s1 has same position in sort order
    ' return greater than zero if s1 should follow s2

    ' EXAMPLE using the length of strings to determine sort order
    ' Replace section with your own implementation
    If (s1.Length = s2.Length) Then Return 0 ' same position
    If (s1.Length > s2.Length) Then Return -1 ' should come before

    Return 1 ' should come after

End Function

What is the fastest levenshtein algorithm for high frequent use

Since you are comparing against the same word over and over, you can get a little performance improvement by using partial application and caching there:

function levenshtein(s1) {
    var row0 = [], row1 = [], s1_len = s1.length;
    if (s1_len === 0)
        return function(s2) {
            return s2.length;
        };
    return function(s2) {
        var s2_len = s2.length, s1_idx, s2_idx = 0, a, b, c, c2, i = 0;
        if (s2_len === 0)
            return s1_len;
        …
        return b;
    };
}

I also repeated myself a bit by linearizing the loop somewhat.

Not sure whether it gets lot faster, but you can omit one of the arrays - you do not need to read/write them in an alternating manner:

function levenshtein(s1) {
    var s1_len = s1.length, row = new Array(s1_len);
    if (s1_len === 0)
        return function(s2) {
            return s2.length;
        };
    return function(s2) {
        var s2_len = s2.length, s1_idx, s2_idx = 0, a, b, c, c2, i = 0;
        if (s2_len === 0)
            return s1_len;
        while (i < s1_len)
           row[i] = ++i;
        while (s2_idx < s2_len) {
            c2 = s2[s2_idx];
            a = s2_idx;
            ++s2_idx;
            b = s2_idx;
            for (s1_idx = 0; s1_idx < s1_len; ++s1_idx) {
                c = a + (s1[s1_idx] === c2 ? 0 : 1);
                a = row[s1_idx];
                b = b < a ? (b < c ? b + 1 : c) : (a < c ? a + 1 : c);
                row[s1_idx] = b;
            }
        }
        return b;
    };
}

I don't think further optimisations can be made without putting your millions of words in a dedicated data structure (e.g. a prefix trie).

Fastest general purpose Levenshtein Javascript implementation

We had a competition for fun at work about making the fastest levenshtein implementation and I came up with a faster one. First of all, I must say that it was not easy to beat your solution which was the fastest to find "out there". :)

This is tested with node.js and it my benchmarks results indicates that this implementation is ~15% faster on small texts (random words size 2-10 characters) and over twice as fast on longer texts (with lengths 30+ containing random characters)

Note: I removed array caching of all implementations

function levenshtein(s, t) {
    if (s === t) {
        return 0;
    }
    var n = s.length, m = t.length;
    if (n === 0 || m === 0) {
        return n + m;
    }
    var x = 0, y, a, b, c, d, g, h, k;
    var p = new Array(n);
    for (y = 0; y < n;) {
        p[y] = ++y;
    }

    for (; (x + 3) < m; x += 4) {
        var e1 = t.charCodeAt(x);
        var e2 = t.charCodeAt(x + 1);
        var e3 = t.charCodeAt(x + 2);
        var e4 = t.charCodeAt(x + 3);
        c = x;
        b = x + 1;
        d = x + 2;
        g = x + 3;
        h = x + 4;
        for (y = 0; y < n; y++) {
            k = s.charCodeAt(y);
            a = p[y];
            if (a < c || b < c) {
                c = (a > b ? b + 1 : a + 1);
            }
            else {
                if (e1 !== k) {
                    c++;
                }
            }

            if (c < b || d < b) {
                b = (c > d ? d + 1 : c + 1);
            }
            else {
                if (e2 !== k) {
                    b++;
                }
            }

            if (b < d || g < d) {
                d = (b > g ? g + 1 : b + 1);
            }
            else {
                if (e3 !== k) {
                    d++;
                }
            }

            if (d < g || h < g) {
                g = (d > h ? h + 1 : d + 1);
            }
            else {
                if (e4 !== k) {
                    g++;
                }
            }
            p[y] = h = g;
            g = d;
            d = b;
            b = c;
            c = a;
        }
    }

    for (; x < m;) {
        var e = t.charCodeAt(x);
        c = x;
        d = ++x;
        for (y = 0; y < n; y++) {
            a = p[y];
            if (a < c || d < c) {
                d = (a > d ? d + 1 : a + 1);
            }
            else {
                if (e !== s.charCodeAt(y)) {
                    d = c + 1;
                }
                else {
                    d = c;
                }
            }
            p[y] = d;
            c = a;
        }
        h = d;
    }

    return h;
}

On longer texts it will get almost up to 3 times the speed of your implementation if it initially cache the inner loop's s.charCodeAt(y) in an Uint32Array. Longer texts also seemed to benefit from using a Uint16Array as a the distance cost array. Here is the code for that solution

function levenshtein(s, t) {
    if (s === t) {
        return 0;
    }
    var n = s.length, m = t.length;
    if (n === 0 || m === 0) {
        return n + m;
    }
    var x = 0, y, a, b, c, d, g, h;
    var p = new Uint16Array(n);
    var u = new Uint32Array(n);
    for (y = 0; y < n;) {
        u[y] = s.charCodeAt(y);
        p[y] = ++y;
    }

    for (; (x + 3) < m; x += 4) {
        var e1 = t.charCodeAt(x);
        var e2 = t.charCodeAt(x + 1);
        var e3 = t.charCodeAt(x + 2);
        var e4 = t.charCodeAt(x + 3);
        c = x;
        b = x + 1;
        d = x + 2;
        g = x + 3;
        h = x + 4;
        for (y = 0; y < n; y++) {
            a = p[y];
            if (a < c || b < c) {
                c = (a > b ? b + 1 : a + 1);
            }
            else {
                if (e1 !== u[y]) {
                    c++;
                }
            }

            if (c < b || d < b) {
                b = (c > d ? d + 1 : c + 1);
            }
            else {
                if (e2 !== u[y]) {
                    b++;
                }
            }

            if (b < d || g < d) {
                d = (b > g ? g + 1 : b + 1);
            }
            else {
                if (e3 !== u[y]) {
                    d++;
                }
            }

            if (d < g || h < g) {
                g = (d > h ? h + 1 : d + 1);
            }
            else {
                if (e4 !== u[y]) {
                    g++;
                }
            }
            p[y] = h = g;
            g = d;
            d = b;
            b = c;
            c = a;
        }
    }

    for (; x < m;) {
        var e = t.charCodeAt(x);
        c = x;
        d = ++x;
        for (y = 0; y < n; y++) {
            a = p[y];
            if (a < c || d < c) {
                d = (a > d ? d + 1 : a + 1);
            }
            else {
                if (e !== u[y]) {
                    d = c + 1;
                }
                else {
                    d = c;
                }
            }
            p[y] = d;
            c = a;
        }
        h = d;
    }

    return h;
}

All benchmark results is from my tests and test-data might be different with your test-data.

The main 2 difference in this solution than to yours (and some other fast ones) I think is

• Not always do compare of the characters in the inner loop if not necessary.
• Sort of "Loop unrolling" in the outer loop doing 4 rows at a time in the levenshtein "matrix". This was a major performance win.

http://jsperf.com/levenshtein-distance/24

I will put this solution on github when I find the time :)

Update:
Finally, I put the solution on github
https://github.com/gustf/js-levenshtein. Its a bit modified/optimized but it is the same base algorithm.

two sets of strings, how to compare the percentage they are the same?

If the strings are always of the same length, consider calculating their Hamming distance. If they are not always of the same length, look into the Levenshtein distance.

How to efficiently find similar strings in a unique string in JavaScript?

Your problem is not the speed of the Levenshtein distance implementation. Your problem is that you have to compare each word with each other word. This means you make 13000² comparisons (and each time calculate the Levenshtein distance).

So my approach would be to try to reduce the number of comparisons.

Here are some ideas:

• words are only similar if their lengths differ less than 20% (just my estimation)
  
  → we can group by length and only compare words with other words of length ±20%

• words are only similar if they share a lot of letters
  
  → we can create a list of e.g. 3-grams (all lower case) that refer to the words they are part of.
  
  → only compare (e.g. with Levenshtein distance) a word with other words that have several 3-grams in common with it.

javascript: return a string similarity score between two strings

Here is reference to a library called jsdifflib.

Here is a demo which gives this.

---

Related Topics