MongoDB Transactions vs Two-Phase commit

In this article we are going to investigate the difference in performance between using MongoDB Transactions or a Two-phase commit for a simple bank account transfer. The benchmarks were run on a local desktop and over a limited set of accounts to ensure we can get intermittent write conflicts.

Links

Learn more about MongoDB at the following locations.

The Code

We are going to try to transfer an amount of money between two accounts and rollback if it fails. There are two implementations. The first is based on using a two-phase commit without the use of multi-document transactions. The second implementation is using the new multi-document transaction support in MongoDB 4.0.x or higher. For simplicities sake we present the code using mongo shell syntax.

For the actual benchmarking we used a kotlin/java framework that is in development that can be found at https://github.com/learn-mongodb-the-hardway/mongodb-schema-simulator.

Multi-Document Transaction

For the multi-document transaction approach we retry transactions on failure to ensure it passed. Below is some sample code of how this could be done in the mongo shell.

var db = db.getSisterDB("bank");
var session = db.getMongo().startSession();
var accounts = session.getDatabase("bank").accounts;
var transactions = session.getDatabase("bank").transactions;

// Retries a transaction commit
function retryUnknownTransactionCommit(session) {
  while(true) {
    try {
      // Attempt to commit the transaction
      session.commitTransaction();
      break;
    } catch (err) {
      if (err.errorLabels != null 
        && err.errorLabels.includes("UnknownTransactionCommitResult")) {
          // Keep retrying the transaction
          continue;
        }

      // The transaction cannot be retried, 
      // return the exception
      return err;
    }
  }
}

function executeTransaction(session, from, to, amount) {
  while (true) {
    try {
      // Start a transaction on the current session
      session.startTransaction({ 
        readConcern: { level: "snapshot" }, writeConcern: { w: "local" } 
      });

      // Debit the `from` account
      var result = accounts.updateOne(
        { name: from, amount: { $gte: amount } }, 
        { $inc: { amount: -amount } });

      // If we could not debit the account, abort the
      // transaction and throw an exception
      if (result.modifiedCount == 0) {
        session.abortTransaction();
        throw Error("failed to debit the account [" + from + "]");
      }

      // Credit the `from` account
      result = accounts.updateOne(
        { name: to }, 
        { $inc: { amount: amount } });

      // If we could not credit the account, abort the
      // transaction and throw an exception
      if (result.modifiedCount == 0) {
        session.abortTransaction();
        throw Error("failed to credit the account [" + to + "]");
      }

      // Insert a record of the transaction
      transactions.insertOne(
        { from: from, to: to, amount: amount, on: new Date() });

      // Attempt to commit the transaction
      session.commitTransaction();
      // Transaction was committed successfully break the while loop
      break;
    } catch (err) {
      // If we have no error labels rethrow the error
      if (err.errorLabels == null) {
        throw err;
      }

      // Our error contains UnknownTransactionCommitResult label
      if (err.errorLabels.includes("UnknownTransactionCommitResult")) {
        // Retry the transaction commit
        var exception = retryUnknownTransactionCommit(session, err);
        // No error, commit as successful, break the while loop
        if (exception == null) break;
        // Error has no errorLabels, rethrow the error
        if (exception.errorLabels == null) throw exception;

        // Error labels include TransientTransactionError label
        // Start while loop again, creating a new transaction
        if (err.errorLabels.includes("TransientTransactionError")) {
          continue;
        }
        
        // Rethrow the error
        throw exception;
      }

      // Error labels include TransientTransactionError label
      // Start while loop again, creating a new transaction
      if (err.errorLabels.includes("TransientTransactionError")) {
        continue;
      }

      // Rethrow the error
      throw err;
    }
  }
}  

executeTransaction(session, "Peter", "Joe", 100);

Let's break down the code in rough steps. First let's look at the executeTransaction method.

function executeTransaction(session, from, to, amount) {
  while (true) {
    try {
      // Start a transaction on the current session
      session.startTransaction({ 
        readConcern: { level: "snapshot" }, writeConcern: { w: "local" } 
      });

      // Debit the `from` account
      var result = accounts.updateOne(
        { name: from, amount: { $gte: amount } }, 
        { $inc: { amount: -amount } });

      // If we could not debit the account, abort the
      // transaction and throw an exception
      if (result.modifiedCount == 0) {
        session.abortTransaction();
        throw Error("failed to debit the account [" + from + "]");
      }

      // Credit the `from` account
      result = accounts.updateOne(
        { name: to }, 
        { $inc: { amount: amount } });

      // If we could not credit the account, abort the
      // transaction and throw an exception
      if (result.modifiedCount == 0) {
        session.abortTransaction();
        throw Error("failed to credit the account [" + to + "]");
      }

      // Insert a record of the transaction
      transactions.insertOne(
        { from: from, to: to, amount: amount, on: new Date() });

      // Attempt to commit the transaction
      session.commitTransaction();
      // Transaction was committed successfully break the while loop
      break;
    } catch (err) {
      // If we have no error labels rethrow the error
      if (err.errorLabels == null) {
        throw err;
      }

      // Error labels include TransientTransactionError label
      // Start while loop again, creating a new transaction
      if (err.errorLabels.includes("TransientTransactionError")) {
        continue;
      }

      // Our error contains the UnknownTransactionCommitResult label
      if (err.errorLabels.includes("UnknownTransactionCommitResult")) {
        // Retry the transaction commit
        var exception = retryUnknownTransactionCommit(session);
        // No error, commit as successful, break the while loop
        if (exception == null) break;
        // Error has no errorLabels, rethrow the error
        if (exception.errorLabels == null) throw exception;

        // Error labels include TransientTransactionError label
        // Start while loop again, creating a new transaction
        if (err.errorLabels.includes("TransientTransactionError")) {
          continue;
        }
        
        // Rethrow the error
        throw exception;
      }

      // Rethrow the error
      throw err;
    }
  }
}  

First we create a new transaction and then we apply the operations to transfer the money from one account to another. The first statement debits the from account.

var result = accounts.updateOne(
  { name: from, amount: { $gte: amount } }, 
  { $inc: { amount: -amount } });
if (result.modifiedCount == 0) {
  session.abortTransaction();
  throw Error("failed to debit the account [" + from + "]");
}

If the debit fails we abort the transaction and throw an error to signal the debit operation failed. Next we credit the to account.

result = accounts.updateOne(
  { name: to }, 
  { $inc: { amount: amount } });
if (result.modifiedCount == 0) {
  session.abortTransaction();
  throw Error("failed to credit the account [" + to + "]");
}

If the credit fails we abort the transaction and throw an error to signal the credit operation failed. Finally we record the transaction.

transactions.insertOne(
    { from: from, to: to, amount: amount, on: new Date() });

Once we have set up all the operations we attempt to commit the transaction.

session.commitTransaction();

If the transaction fails that's when the fun starts. Let's look at the exception handling.

} catch (err) {
  // If we have no error labels rethrow the error
  if (err.errorLabels == null) {
    throw err;
  }

  // Error labels include TransientTransactionError label
  // Start while loop again, creating a new transaction
  if (err.errorLabels.includes("TransientTransactionError")) {
    continue;
  }

  // Our error contains the UnknownTransactionCommitResult label
  if (err.errorLabels.includes("UnknownTransactionCommitResult")) {
    // Retry the transaction commit
    var exception = retryUnknownTransactionCommit(session);
    // No error, commit as successful, break the while loop
    if (exception == null) break;
    // Error has no errorLabels, rethrow the error
    if (exception.errorLabels == null) throw exception;

    // Error labels include TransientTransactionError label
    // Start while loop again, creating a new transaction
    if (err.errorLabels.includes("TransientTransactionError")) {
      continue;
    }
    
    // Rethrow the error
    throw exception;
  }

  // Rethrow the error
  throw err;
}

If we have an error object with no errorLabels we rethrow it as the transaction cannot be retried. However if we do have errorLabels we need to inspect them.

If the label TransientTransactionError is present we cannot retry the current transaction so we continue the while loop, forcing the creation of a new transaction.

However if the errorLabels contains the label UnknownTransactionCommitResult we can retry the current transaction. We do this by calling in the retryUnknownTransactionCommit function with the current session. Lets look at the function in details.

// Retries a transaction commit
function retryUnknownTransactionCommit(session) {
  while(true) {
    try {
      // Attempt to commit the transaction
      session.commitTransaction();
      break;
    } catch (err) {
      if (err.errorLabels != null 
        && err.errorLabels.includes("UnknownTransactionCommitResult")) {
          // Keep retrying the transaction
          continue;
        }

      // The transaction cannot be retried, 
      // return the exception
      return err;
    }
  }
}

While the session.commitTransaction() call returns the UnknownTransactionCommitResult we keep retrying the transaction. If the transaction commit is successful we break out of the loop returning null. If the error returned is different to UnknownTransactionCommitResult we return the error.

Returning to the point where we call the retryUnknownTransactionCommit function we see the following logic.

// Retry the transaction commit
var exception = retryUnknownTransactionCommit(session);
// No error, commit as successful, break the while loop
if (exception == null) break;
// Error has no errorLabels, rethrow the error
if (exception.errorLabels == null) throw exception;

// Error labels include TransientTransactionError label
// Start while loop again, creating a new transaction
if (err.errorLabels.includes("TransientTransactionError")) {
  continue;
}

// Rethrow the error
throw exception;

If the returned exception is null we break the while loop as our transaction was successfully committed. If the exception does not include errorLabels we rethrow the exception. On the other hand if the exception contains errorLabels and the labels include the label TransientTransactionError we cannot retry the current transaction so we continue the while loop, forcing the creation of a new transaction.

Two Phase Commit

The Two Phase commit approach takes a different approach using double bookkeeping to ensure consistent account transfers. Let's look at a self contained example below that showcases how the pattern could be implemented using the mongo shell.

var db = db.getSisterDB("bank");
db.dropDatabase();
var accounts = db.accounts;
var transactions = db.transactions;

accounts.insertOne({ _id: 1, name: "Joe Moneylender", balance: 1000, pendingTransactions:[] });
accounts.insertOne({ _id: 2, name: "Peter Bum", balance: 1000, pendingTransactions:[] });

function cancel(id) {
  transactions.updateOne(
    { _id: id }, 
    { $set: { state: "canceled" } }
  );
}

function rollback(from, to, amount, id) {
  // Reverse debit
  accounts.updateOne({
      name: from, 
      pendingTransactions: { $in: [id] }
    }, {
      $inc: { balance: amount }, 
      $pull: { pendingTransactions: id }
    });  

  // Reverse credit
  accounts.updateOne({
    name: to, 
    pendingTransactions: { $in: [id] }
  }, {
    $inc: { balance: -amount }, 
    $pull: { pendingTransactions: id }
  });  

  cancel(id);
}

function cleanup(from, to, id) {
  // Remove the transaction ids
  accounts.updateOne(
    { name: from }, 
    { $pull: { pendingTransactions: id } });
  
  // Remove the transaction ids
  accounts.updateOne(
    { name: to }, 
    { $pull: { pendingTransactions: id } });

  // Update transaction to committed
  transactions.updateOne(
    { _id: id }, 
    { $set: { state: "done" } });
}

function executeTransaction(from, to, amount) {
  var transactionId = ObjectId();

  transactions.insert({
    _id: transactionId, 
    source: from, 
    destination: to, 
    amount: amount, 
    state: "initial"
  });

  var result = transactions.updateOne(
    { _id: transactionId }, 
    { $set: { state: "pending" } }
  );

  if (result.modifiedCount == 0) {
    cancel(transactionId);
    throw Error("Failed to move transaction " + transactionId + " to pending");
  }

  // Set up pending debit
  result = accounts.updateOne({
    name: from, 
    pendingTransactions: { $ne: transactionId }, 
    balance: { $gte: amount }
  }, {
    $inc: { balance: -amount }, 
    $push: { pendingTransactions: transactionId }
  });

  if (result.modifiedCount == 0) {
    rollback(from, to, amount, transactionId);
    throw Error("Failed to debit " + from + " account");
  }

  // Setup pending credit
  result = accounts.updateOne({
    name: to, 
    pendingTransactions: { $ne: transactionId }
  }, {
    $inc: { balance: amount }, 
    $push: { pendingTransactions: transactionId }
  });

  if (result.modifiedCount == 0) {
    rollback(from, to, amount, transactionId);
    throw Error("Failed to credit " + to + " account");
  }

  // Update transaction to committed
  result = transactions.updateOne(
    { _id: transactionId }, 
    { $set: { state: "committed" } }
  );

  if (result.modifiedCount == 0) {
    rollback(from, to, amount, transactionId);
    throw Error("Failed to move transaction " + transactionId + " to committed");
  }

  // Attempt cleanup
  cleanup(from, to, transactionId);
}

executeTransaction("Joe Moneylender", "Peter Bum", 100);

Let's break down the function executeTransaction step by step and discuss what happens at each step and how to recover from an error.

transactions.insert({
  _id: transactionId, 
  source: from, 
  destination: to, 
  amount: amount, 
  state: "initial"
});

The first step inserts a new document into the transactions collection that contains the information about the transfer we are about to perform. The state of the transaction is set to initial signaling we have just started the process.

var result = transactions.updateOne(
  { _id: transactionId }, 
  { $set: { state: "pending" } }
);

if (result.modifiedCount == 0) {
  cancel();
  throw Error("Failed to move transaction " + transactionId + " to pending");
}

Next we attempt to flip the transaction to the state pending. If it fails (result.modifiedCount == 0) we attempt to cancel the transaction calling the function cancel. Let's look at the what the cancel function does.

function cancel(id) {
  transactions.updateOne(
    { _id: id }, 
    { $set: { state: "canceled" } }
  );
}

The function basically attempts to set the state of the transaction to canceled. After returning from the cancel function we throw an exception signalling the caller of the executeTransaction function that is failed.

However if we are successful at setting the transaction state to pending we can start the process of applying the transaction.

result = accounts.updateOne({
  name: from, 
  pendingTransactions: { $ne: transactionId }, 
  balance: { $gte: amount }
}, {
  $inc: { balance: -amount }, 
  $push: { pendingTransactions: transactionId }
});

if (result.modifiedCount == 0) {
  rollback(from, to, amount, transactionId);
  throw Error("Failed to debit " + from + " account");
}

We look up the from account, ensuring that the pendingTransactions array does not contain the transactionId and that the account balance is greater or equal to the amount we are going to debit. If we the document matches we are going to debit the account balance by amount and push the transactionId to the pendingTransactions array.

If no document was modified we know the update of the account failed and we need to call the rollback function to cancel the transaction before we throw an exception signaling the application that the transfer failed. Let's have a look at the rollback function.

function rollback(from, to, amount, id) {
  // Reverse debit
  accounts.updateOne({
      name: from, 
      pendingTransactions: { $in: [id] }
    }, {
      $inc: { balance: amount }, 
      $pull: { pendingTransactions: id }
    });  

  // Reverse credit
  accounts.updateOne({
    name: to, 
    pendingTransactions: { $in: [id] }
  }, {
    $inc: { balance: -amount }, 
    $pull: { pendingTransactions: id }
  });  

  cancel(id);
}

To rollback a transaction we need to reverse it on the from and to accounts. First we have to remove the transaction from the from account returning the reserved amount to the balance.

accounts.updateOne({
    name: from, 
    pendingTransactions: { $in: [id] }
  }, {
    $inc: { balance: amount }, 
    $pull: { pendingTransactions: id }
  });  

We will update the account if it contains the transaction by matching on the name and if the pendingTransactions array contains transaction id. If the document matches we will add the amount to the balance and remove the transaction id from the pendingTransaction. Next we need to reverse the transaction on the to account as well.

accounts.updateOne({
  name: to, 
  pendingTransactions: { $in: [id] }
}, {
  $inc: { balance: -amount }, 
  $pull: { pendingTransactions: id }
});  

The only difference from the from account is that we will deduct the amount from the balance of the account. Finally we call the cancel method to set the transaction state to canceled. Returning to the executeTransaction function lets look at the next statement.

result = accounts.updateOne({
  name: to, 
  pendingTransactions: { $ne: transactionId }
}, {
  $inc: { balance: amount }, 
  $push: { pendingTransactions: transactionId }
});

if (result.modifiedCount == 0) {
  rollback(from, to, amount, transactionId);
  throw Error("Failed to credit " + to + " account");
}

Just as in the case of applying the transactionId to the from account we ensure the account does not already contain the transactionId in the pendingTransaction. If it does not exist in pendingTransaction we add the amount to the balance and push the transactionId to the pendingTransactions array.

If the document fails to update, we call the rollback function as we did previously and then throw an exception to signal the application the transaction failed.

Finally we are going to flip the state of the transaction to committed.

result = transactions.updateOne(
  { _id: transactionId }, 
  { $set: { state: "committed" } }
);

if (result.modifiedCount == 0) {
  rollback(from, to, amount, transactionId);
  throw Error("Failed to move transaction " + transactionId + " to committed");
}

If it fails we call the rollback function to reverse the transaction. Finally we call the cleanup function. Let's have a look at what the function does.

function cleanup(from, to, id) {
  // Remove the transaction ids
  accounts.updateOne(
    { name: from }, 
    { $pull: { pendingTransactions: id } });
  
  // Remove the transaction ids
  accounts.updateOne(
    { name: to }, 
    { $pull: { pendingTransactions: id } });

  // Update transaction to committed
  transactions.updateOne(
    { _id: id }, 
    { $set: { state: "done" } });
}

The first update will remove the transactionId from the from account. The second update will do the same for the to account. Finally the last update will set the transaction state to done finalizing the transfer between the two accounts.

Performance Run and Analysis

Lets run two comparison benchmarks to look at two specific traffic scenarios applied to both the transaction approach as well as the Two-Phase approach.

The first scenario is one where we have a single thread performing an account transfer every millisecond for 35 seconds.

The second scenario we run the same transfer every millisecond but using five threads for 35 seconds.

We use the schema simulator tool at https://github.com/learn-mongodb-the-hardway/mongodb-schema-simulator to generate the load and record the results.

We take measurement from 5 to 35 seconds to avoid the initial period of cache warmup on MongoDB as well as Java JIT warmup.

Single Thread

For the single thread scenario we get the following results.

The graph above shows the results of the transaction approach.

The graph above shows the results of the Two-Phase approach. Lets take the key numbers and put them in a table for easing comparison.

I> mean is the geometric mean. The geometric mean is a mean or average, which indicates the central tendency or typical value of a set of numbers
I> min is the minimum value found in the set.
I> max is the maximum value found in the set.
I> pth percentile is the percentage of the data that is smaller than the value. A 95 percentile of 100 would mean 95% of the values in the set vas lower than 100.

Let's break down the numbers.

1. The mean for the transaction approach is ~2x lower.
2. The min for the transaction approach is ~2x lower.
3. The max for the transaction approach is ~2x lower.
4. The 95 percentile for the transaction approach is ~2x lower.
5. The 99 percentile for the transaction approach is ~2x lower.

Looking at this we can see that the transaction support is roughly two times faster than the Two-Phase commit approach. This makes sense, as the amount of operations we need to perform against MongoDB to finish a Two-Phase commit, is larger than what is needed for a transaction approach.

Given this we might conclude that the transaction approach is superior to the Two-Phase approach. But hold your horses. Lets see what happens when we force transaction collisions by increasing the load.

Multiple Threads

For the multiple threads scenario we get the following results.

The first graph shows an account transfer using the MongoDB 4.0.x transaction support approach.

The second graph shows an account transfer using the Two-Phase commit approach. Lets grab the main numbers and compare them.

Let's break down the numbers.

1. The mean is very similar between the two approaches.
2. The min for the transaction approach is lower.
3. The max for the transaction approach is lower.
4. The 95 percentile is higher for the transaction approach.
5. The 99 percentile is very similar between the two approaches.

We can see that Two-Phase commit have in general a better overall performance characteristic due to the 95 percentile saying 95% of operations took 19.98 ms or less compared to 30.62 ms for the transaction approach.

The reason the Two-Phase approach is now competitive with the transaction approach, is due to the transaction approach starting to experience write conflicts due to the higher concurrent load, forcing the benchmark to retry the transactions until they are successful.

Conclusion

From an initial inspection it would seem that using transactions would is still a viable way forward as the difference is not enormous between the two approaches. However before we decide, we have to take a couple of additional factors into consideration.

1. Transactions only work for replicasets as of MongoDB 4.0.x.
2. The maximum run-time for a transaction is a minute (anything over a minute gets aborted).
3. The locking of documents for transactions can cause performance bottlenecks on documents that get a lot of writes as all the transactions get serialized and have to wait their turn to be processed.

Once MongoDB supports sharded transactions your repliaset transactions might end up being distributed transactions that involve multiple shards. This introduces a large performance penalty as well as new and very complex failure modes, any which one could cause the transaction to fail. In this case its very likely the Two-Phase commit approach would significantly outperform the transaction approach.