“We believe IT decision-makers will benefit from the focused results of our report, as they seek to gain a clearer understanding of how databases can support their application needs.”
The Thumbtack benchmark report is intended to help organizations obtain an independent understanding of the strengths and weaknesses of various NoSQL database approaches to supporting applications that process huge volumes of data and need durable and consistent storage semantics. In it, Thumbtack takes a focused, tuned and optimized approach to answering specific real-world questions by evaluating four databases (Aerospike, Cassandra, Couchbase, and MongoDB) as they would be optimized for this scenario.
In particular, the report focused on how durability and consistency needs affect raw performance. The data strongly suggests that for synchronously replicated, durable data, Aerospike was able to outperform the other databases by a wide margin, whereas when those constraints are relaxed, both Aerospike and Couchbase are able to process transaction volumes approaching 1 million operations per second.
“Previous published NoSQL benchmarks typically have run out-of-the-box queries against default settings on a broad set of databases, but not all use cases are the same. That is why we have focused on testing a few NoSQL databases for their ability to address a specific class of real-world problem with big data,” said Ben Engber, Thumbtack Technology CEO and co-author of the benchmark study. “We believe IT decision-makers will benefit from the focused results of our report, as they seek to gain a clearer understanding of how databases can support their application needs.”
Thumbtack performed its tests using a modified version of the Yahoo Cloud Serving Benchmark (YCSB) from Yahoo! Research, which is rapidly becoming the standard for benchmarking NoSQL databases. However, because the YCSB hasn’t been optimized for testing at the highest volumes, Thumbtack altered and extended the YCSB tool and supporting scripts to overcome these limitations as part of this benchmarking project. In conjunction with publishing the report, Thumbtack has also made available its changes to the core YCSB framework on which the tests were run. These include various performance and functional changes that allow YCSB to create a higher load and support a variety of different kinds of replication and durability models. The updated code is available at https://github.com/thumbtack-technology/ycsb.
Today’s report is the first in a series of reports Thumbtack plans to produce to help organizations make the correct decisions in the very broadly defined NoSQL space. Upcoming reports will provide detailed analysis of aspects other than raw key-value performance, including fault-tolerance and secondary index support. Additional databases and hosting environments will also be incorporated into the analysis.
Based in Brooklyn, New York, Thumbtack Technology is a leading software development services firm that specializes in building and integrating scalable applications and systems for Fortune 500 clients and start-ups. Thumbtack is business driven, focusing on key customer pain points, tackling the most difficult problems while scaling applications to an Internet audience. To learn more, visit http://www.thumbtack.net.