Skytable supports the persistent storage of data, which is an inherently obvious need for any database. In this document we explore how Skytable's persistence works.
Data directory structure
Whenever you start Skytable, it will create a number of directories under a root 'data' directory. This is what the data directory structure looks like:
|__user-files (your other files)
data.bin file is primarily used for persistence while the snapshots folder contains automatically created
snapshots and remotely created snapshots.
How Skytable stores data
As soon as you start Skytable, it will look for a
data.bin file in the data directory and then fall back to the
data.bin file in the current directory for backwards compatibility. The database daemon will then attempt to
migrate the older data into the structure noted above if required.
This backward compatibility will possibly be removed in future releases
If the data file is found and successfully read, then the previous data is moved into the in-memory table. If no file was found, then the database server will create one. Once you terminate the daemon, it will flush all data to this file. There are more features like BGSAVE and snapshots that can be configured and used by users.
Automated background saving
Skytable supports automated saving of data in the background, via
BGSAVE, runs every two minutes to flush all the data in the in-memory table onto disk, unless customized through the configuration file. BGSAVE is enabled by default and we don't recommend disabling it until you're sure that
your hardware will never fail; it is likely that this will never be the case. First BGSAVE will create a temporary
file and then flush the current in-memory table onto disk. It will then replace the old
Reliability of BGSAVE
It can happen that BGSAVE fails to flush data to the disk due to some unforeseen system issues (such as lack of empty disk space, permission errors, etc). But if we continue to accept modifications to the data, it is a bad idea because this data may never get updated! This is why if BGSAVE fails, it will automatically poison the in-memory table preventing it from accepting any write/update operations. Poisioning is nothing but a global flag set in the database that indicates that the DB shouldn't accept any more updates/writes to data and in such a poisoned state, only reads are permitted.