Benchmarking

All of Skytable's components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable's release bundles come with the sky-bench benchmarking tool. It's important to understand how the tool works to understand what you're benchmarking and what the results mean.

The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors.

info

The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed.

Benchmark workload

The workload that the engine currently uses (as of v0.8.1) is the following:

• A model bench.bench is created with a primary key (of type string and with a column of type uint8)
• Multiple clients are created (simulating "application servers")
• Queries are run against unique rows. This means that unlike redis-benchmark every query group touches a different row, as it would generally do in the real-world
• The following queries (collectively a "query group") are run for each unique row:
  • The row is first created with an INSERT
  • All columns of the row are returned with a SELECT
  • The integer column is incremented with an UPDATE
  • The row is finally removed with a DELETE
• By default, 1,000,000 rows are created and manipulated
• The time taken for each row to be sent, read back and decoded into a readable form is recorded (time taken to parse into actual language structures such as maps and lists) towards the total time taken, once again unlike many other benchmark tools
• In total 4,000,000 queries are run (by default)

caution

The benchmark tool will create a space bench and a model bench and will completely remove the space and associated data once the benchmark is complete. Do not use this space!

Running the benchmark

Now that you know how the benchmark engine works, go ahead and benchmark for yourself.

1. Run:

   sky-bench --password <root_password>
   Note: You can ignore the --password argument if you have already set it using the SKYDB_PASSWORD environment variable
2. The benchmark engine will run the full workload (as described earlier)
3. Witness 4,000,000 queries being executed in real-time. Good luck and enjoy the results!

tip

You can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting.