Persistence
Skytable supports the persistent storage of data, which is an inherently obvious need for any database. In this document we explore how Skytable's persistence works.
How Skytable stores data
As soon as you start Skytable, it will look for a PRELOAD file in the data directory. If it
isn't found — the server would expect this to be a new instance and will create the
required files for you. Skytable uses a disk storage format called Cyanstore (version 1A)
for storing your data and all files are encoded/decoded in compliance with this format.
Once you terminate the daemon, it will flush all data to disk. All writes are fsynced by the time
they complete.
Save failure during termination
The server would attempt to do a final save operation before it terminates and if this fails, the server would enter into a retry loop. It will try the save operation after every 10 seconds. Exponential backoff was not chosen because it could increase to extremely large values that may hurt a sysadmin's time and productivity.
You might be interested in more features like BGSAVE and snapshots that can be configured and used by users.
Automated background saving
Skytable supports automated saving of data in the background, via BGSAVE. BGSAVE, runs every two minutes to flush all the data in the in-memory table onto disk, unless customized. BGSAVE is enabled by default and we don't recommend disabling it until you're sure that
your hardware will never fail; it is likely that this will never be the case. First BGSAVE will create a temporary
file and then flush the current in-memory table onto disk. The daemon automatically fsyncs after every successful write (whether to the buffers or
to the actual disk).
Reliability of BGSAVE
It can happen that BGSAVE fails to flush data to the disk due to some unforeseen system issues (such as lack of empty disk space, permission errors, etc). But if we continue to accept modifications to the data, it is a bad idea because this data may never get updated! This is why if BGSAVE fails, it will automatically poison the in-memory table preventing it from accepting any write/update operations. Poisioning is nothing but a global flag set in the database that indicates that the DB shouldn't accept any more updates/writes to data and in such a poisoned state, only reads are permitted.
BGSAVE Recovery
BGSAVE will automatically try to recover on every 120s (or whatever duration was set). If the problem was corrected (say it was a permissions issue), then the database server will automatically resume writes and unpoison the database.
Avoid editing/changing/removing files from the data directory by hand. All files stored in the data directory are critical to the server and your data. If you end up corrupting any file due to a bad edit — you might end up losing access to your data!