Using Reduce to Remove Duplicates

Most examples I see for reduce talk about summing up numbers. Can we use reduce to do more than punching numbers? Doesn’t it sound ironic that these accumulator or sum calculator examples look more like they are increasing some values rather than reducing? I know I’m pushing the envelope here. I think removing duplicates from an array would be a more suitable example of reducing the array we are working with. Because, after all, we would be reducing it to a minimal state.

How ever you define what reduce does, here is my take on using reduce to remove duplicates from an array. It only works for simple data types of course but you can enhance it the way you see it fit. For simplicity sake, I’ll attach my solution to Array.prototype. Again, this is optional.

What’s the Magic Sauce?

The key element to our solution is the initialValue parameter as it is defined in the Javascript Reference. Think about it. We are given an array, we want to return an array with some elements removed from the original array. Well, reverse the problem, what if you started with a blank array and only added elements to that blank array if it didn’t already have the element from the original array. That’s exactly what we are going to do.

Array.prototype.removeDuplicates = function() {
  return this.reduce(
    (result, nextItem) =>
      result.includes(nextItem) ? result : result.concat(nextItem),
    []
  );
};

Look at that [] at the end of line 2. That’s the initialValue parameter I talked about. It’s an empty array waiting to be filled after each iteration of reduce. The code is actually pretty straightforward but let’s examine that ternary expression. For each item reduce goes over, it checks if it’s already added to the result array. If it is already in the result set then we assign result back to itself under the hood like result = result. If the item doesn’t exist then we concatenate the element to result which is then interpreted as result = result.concat[...]. Remember, concat doesn’t alter the array, it simply returns a new array with the concatenated element.

So, what’s with that [] then? That actually becomes result as soon as reduce starts its operation. Imagine you didn’t have that parameter available to you, then you would have to define an empty array outside the reduce call. Then, in each reduce iteration, you’d check things and put into that outside array. Instead, it’s now part of the reduce call.

Wrapping Up

I put some test code in JSBin. The code is also available at Github Gist. As a final note, you can use this technique as a separate function so it doesn’t necessarily have to be in Array.prototype but it is often handy to run it over an array and chain it as you go.

In another article, I'll present an advanced version or use of the technique I showed in this article. It deals with consecutive or rather every couple of items in an array. You may find that article complementary once you master the trick here.