Skip to main content
Version: 0.6.2

Persistence

Skytable supports the persistent storage of data, which is an inherently obvious need for any database. In this document we explore how Skytable's persistence works.

Data directory structure#

Whenever you start Skytable, it will create a number of directories under a root 'data' directory. This is what the data directory structure looks like:

|__user-files (your other files)|__data   |__data.bin   |__snapshots      |__<YYYYMMDD>-<HHMMSS>.snapshot (many)      |__remote         |___<snapshotname>.snapshot (many)

The data.bin file is primarily used for persistence while the snapshots folder contains automatically created snapshots and remotely created snapshots.

How Skytable stores data#

As soon as you start Skytable, it will look for a data.bin file in the data directory; if the data file is found and successfully read, then the previous data is moved into the in-memory table. If no file was found, then the database server will create one. Once you terminate the daemon, it will flush all data to this file. All writes are fsynced by the time they complete.

Save failure during termination#

The server would attempt to do a final save operation before it terminates and if this fails, the server would enter into a retry loop. It will try the save operation after every 10 seconds. Exponential backoff was not chosen because it could increase to extremely large values that may hurt a sysadmin's time and productivity.

You might be interested in more features like BGSAVE and snapshots that can be configured and used by users.

Automated background saving#

Skytable supports automated saving of data in the background, via BGSAVE. BGSAVE, runs every two minutes to flush all the data in the in-memory table onto disk, unless customized through the configuration file. BGSAVE is enabled by default and we don't recommend disabling it until you're sure that your hardware will never fail; it is likely that this will never be the case. First BGSAVE will create a temporary file and then flush the current in-memory table onto disk. It will then replace the old data.bin file. The daemon automatically fsyncs after every successful write (whether to the buffers or to the actual disk).

Reliability of BGSAVE#

It can happen that BGSAVE fails to flush data to the disk due to some unforeseen system issues (such as lack of empty disk space, permission errors, etc). But if we continue to accept modifications to the data, it is a bad idea because this data may never get updated! This is why if BGSAVE fails, it will automatically poison the in-memory table preventing it from accepting any write/update operations. Poisioning is nothing but a global flag set in the database that indicates that the DB shouldn't accept any more updates/writes to data and in such a poisoned state, only reads are permitted.

BGSAVE Recovery#

BGSAVE will automatically try to recover on every 120s (or whatever duration was set). If the problem was corrected (say it was a permissions issue), then the database server will automatically resume writes and unpoison the database.