Kyoto Cabinet is a library of routines for managing a database. The database is a simple data file containing records, each is a pair of a key and a value. Every key and value is serial bytes with variable length. Both binary data and character string can be used as a key and a value. Each key must be unique within a database. There is neither concept of data tables nor data types. Records are organized in hash table or B+ tree.
The following access methods are provided to the database: storing a record with a key and a value, deleting a record by a key, retrieving a record by a key. Moreover, traversal access to every key are provided. These access methods are similar to ones of DBM (or its followers: NDBM and GDBM) library defined in the UNIX standard. Kyoto Cabinet is an alternative for DBM because of its higher performance.
Each operation of hash database has the time complexity of "O(1)". Therefore, in theory, the performance is constant regardless of the scale of the database. In practice, the performance is determined by the speed of the main memory or the storage device. If the size of the database is less than the capacity of the main memory, the performance will seem on-memory speed which is faster than std::map of STL. Of course, the database size can be greater than the capacity of the main memory and the upper limit is 8 exabytes. Even in that case, each operation needs only one or two seeking of the storage device.
Each operation of B+ tree has the time complexity of "O(log N)". Therefore, in theory, the performance is logarithmic to the scale of the database. Although the performance of random access of B+ tree is slower than that of hash database, B+ tree supports sequential access in order of the keys, which realizes forward matching search for strings and range search for integers. The performance of sequential access is much faster than that of random access.
As the API is based on object-oriented design, hash database and B+ tree database have same methods which inherited from the upper abstract class. Prototype database by containers of STL and cache database with LRU deletion algorithm are also provided under the same base class. All databases have practical utility methods related to transaction and cursor. Programs for command line interface are also included in the package.
Kyoto Cabinet runs very fast. For example, elapsed time to store one million records is 0.9 seconds for hash database, and 1.1 seconds for B+ tree database. Moreover, the size of database of Kyoto Cabinet is very small. For example, overhead for a record is 16 bytes for hash database, and 4 bytes for B+ tree database. Furthermore, scalability of Kyoto Cabinet is great. The database size can be up to 8EB (9.22e18 bytes).
Kyoto Cabinet is written in the C++ language, and provided as API of C++, C, Java, Python, Ruby, and Perl. Kyoto Cabinet is available on platforms which have API conforming to C++03 with the TR1 library extensions. Kyoto Cabinet is a free software licensed under the GNU General Public License.
This section describes the features of Kyoto Cabinet.
The original DBM was developed by Kenneth Thompson as a part of the original AT&T UNIX. After that, a lot of followers developed such DBM-like products as NDBM, SDBM, GDBM, TDB, and BerkeleyDB. In 2003, I developed QDBM to replace GDBM for performance reason.
In 2007, Tokyo Cabinet was developed as the successor of QDBM on the following purposes. They were achieved and Tokyo Cabinet could replace conventional DBM products.
In 2009, Kyoto Cabinet was developed as another successor of QDBM. Compared with the sibling product (Tokyo Cabinet), the following advantages were pursued. However, the performance of Tokyo Cabinet is higher than Kyoto Cabinet, at least in single thread operations.
I'll maintain the both of Tokyo Cabinet and Kyoto Cabinet because their values are different.
Kyoto Cabinet uses hash algorithm to retrieve records. If a bucket array has sufficient number of elements, the time complexity of retrieval is "O(1)". That is, time required for retrieving a record is constant, regardless of the scale of a database. It is also the same about storing and deleting. Collision of hash values is managed by separate chaining. Data structure of the chains is binary search tree. Even if a bucket array has unusually scarce elements, the time complexity of retrieval is "O(log n)".
Kyoto Cabinet attains improvement in retrieval by loading RAM with the whole of the bucket array. If the bucket array is on RAM, it is possible to access a region of a target record by about one set of file operations such as `lseek', `read', and `write'. The bucket array saved in a file is not read into RAM with the `read' call but directly mapped to RAM with the `mmap' call. Therefore, preparation time on connecting to a database is very short, and two or more processes can share the same memory map.
The hash function used for hash table is MurMurHash 2.0. If the number of elements of the bucket array is about half of records stored within a database, although it depends on characteristic of the input, the probability of collision of hash values is about 55.3% (35.5% if the same, 20.4% if twice, 11.0% if four times, 5.7% if eight times). In that case, it is possible to retrieve a record by two or less sets of file operations. If it is made into a performance index, in order to handle a database containing one million of records, a bucket array with half a million of elements is required. The size of each element is 4 bytes. That is, if 2M bytes of RAM is available, a database containing one million records can be handled.
When overwriting a record with a value whose size is greater than the existing one, it is necessary to move the region to another position of the file. Because the time complexity of the operation depends on the size of the region of a record, extending values successively is inefficient. However, Kyoto Cabinet deal with this problem by alignment. If increment can be put in padding, it is not necessary to move the region.
Generally speaking, while succession of updating, fragmentation of available regions occurs, and the size of a database grows rapidly. Kyoto Cabinet deals with this problem by the free block pool and the auto defragmentation mechanism. If a record is removed or shifted to another position, the region will be treated as a free block. The free block pool manages free blocks and reuses the best fit region for a new record. The auto defragmentation is to shift records and free blocks separately. Successive free blocks coalesce into one.
Although B+ tree database is slower than hash database, it features ordering access to each record. The order can be assigned by users. Records of B+ tree are sorted and arranged in logical pages. Sparse index organized in B tree that is multiway balanced tree are maintained for each page. Thus, the time complexity of retrieval and so on is "O(log n)". Cursor is provided to access each record in order. The cursor can jump to a position specified by a key and can step forward or backward from the current position. Because each page is arranged as double linked list, the time complexity of stepping cursor is "O(1)".
B+ tree database is implemented, based on the above hash database. Because each page of B+ tree is stored as each record of hash database, B+ tree database inherits efficiency of storage management of hash database. Because the header of each record is smaller and alignment of each page is adjusted according to the page size, in most cases, the size of database file is cut by half compared to one of hash database.
Although operations of many pages are required to update B+ tree, Kyoto Cabinet expedites the process by the page cache and reducing file operations. The page cache is implemented with double layered LRU list, which realizes that frequently accessed pages are cached in the "hot" list and recently accessed pages are cached in the "warm" LRU list. If the page cache works efficiently and the whole of the sparse index is cached on memory, it is possible to retrieve a record by one or less set of file operations.
Each page of B+ tree can be stored with compressed. The default compression method is "Deflate" by ZLIB. Because records in a page has similar patterns, high efficiency of compression is expected due to the Lempel-Ziv algorithm. In case of handling text data, the size of a database is reduced to about 50% or less. If the scale of a database is large and disk I/O is the bottleneck, featuring compression makes the processing speed improved to a large extent. Moreover, you can specify such external compression algorithms as LZO and LZMA.
Kyoto Cabinet features transaction mechanisms. It is possible to commit a series of operations between the beginning and the end of the transaction in a lump, or to abort the transaction and perform rollback to the state before the transaction. Two isolation levels are supported; "serializable" and "read uncommitted". Durability is secured by write ahead logging and shadow paging.
Auto transaction and auto recovery mechanisms are also supported. If the auto transaction option is specified when opening the database, every updating operation is guarded by transaction which is committed implicitly. Therefore, durability can be assured without explicit transaction operations. The auto recovery mechanism works after the database is crashed outside transaction. If inconsistency of the database is detected when opening the database, all regions are scanned as with "fsck" and the database is reconstructed with surviving records implicitly.
Kyoto Cabinet provides two modes to connect to a database: "reader" and "writer". A reader can perform retrieving but neither storing nor deleting. A writer can perform all access methods. Exclusion control between processes is performed when connecting to a database by file locking. While a writer is connected to a database, neither readers nor writers can be connected. While a reader is connected to a database, other readers can be connect, but writers can not. According to this mechanism, data consistency is guaranteed with simultaneous connections in multitasking environment.
Functions of API are reentrant and available in multi-thread environment. Different database objects can be operated in parallel entirely. For simultaneous operations against the same database object, rwlock (reader-writer lock) is used for exclusion control. That is, while a writing thread is operating an object, other reading threads and writing threads are blocked. However, while a reading thread is operating an object, reading threads are not blocked. Locking granularity depends on data structures. Hash database uses record locking. B+ tree database uses page locking.
In order to improve performance and concurrency, Kyoto Cabinet uses such atomic operations built in popular CPUs as atomic-increment and CAS (compare-and-swap). Lock primitives provided by the native environment such as the POSIX thread package are alternated by own primitives using CAS.
Kyoto Cabinet provides simple APIs based on object-oriented design. Every operation for database is encapsulated and published as lucid methods as `open', `close', `set', `remove', `get', and so on. The classes of hash database and B+ tree database are derived class of the common abstract class which defines the interface. Porting an application from one database to another is easy. Moreover, the polymorphic database API is provided to assign a database in run-time.
Kyoto Cabinet supports the "visitor" pattern. You can define arbitrary database operations with call back functions. The visitor class encapsulates that call back functions and their state data. The database class has the "accept" method, which accepts an instance of the visitor class and calls its functions with a record data. The return value of the call back function is reflected as the new state of the record.
While the core API is provided for C++, bindings for other languages such as C, Java, Python, Ruby, and Perl are also provided. Command line interfaces are also provided corresponding to each API. They are useful for prototyping, test, and debugging.
This section describes how to install Kyoto Cabinet with the source package. As for a binary package, see its installation manual.
Kyoto Cabinet is available on UNIX-like systems. At least, the following environments are supported.
gcc (GNU Compiler Collection) 4.2 or later and make (GNU Make) are required to install Kyoto Cabinet with the source package. They are installed by default on Linux, FreeBSD and so on.
As Kyoto Cabinet depends on the following libraries, install them beforehand.
When an archive file of Kyoto Cabinet is extracted, change the current working directory to the generated directory and perform installation.
Run the configuration script.
$ ./configure
Build programs.
$ make
Perform self-diagnostic test.
$ make check
Install programs. This operation must be carried out by the root user.
# make install
When a series of work finishes, the following files will be installed.
/usr/local/include/kccommon.h /usr/local/include/kcutil.h /usr/local/include/kcdb.h /usr/local/include/kcthread.h /usr/local/include/kcfile.h /usr/local/include/kccompress.h /usr/local/include/kccompare.h /usr/local/include/kcmap.h /usr/local/include/kcprotodb.h /usr/local/include/kccachedb.h /usr/local/include/kchashdb.h /usr/local/include/kctreedb.h /usr/local/include/kcpolydb.h /usr/local/include/kclangc.h /usr/local/lib/libkyotocabinet.a /usr/local/lib/libkyotocabinet.so.x.y.z /usr/local/lib/libkyotocabinet.so.x /usr/local/lib/libkyotocabinet.so /usr/local/lib/pkgconfig/kyotocabinet.pc /usr/local/bin/kcutiltest /usr/local/bin/kcutilcodec /usr/local/bin/kcprototest /usr/local/bin/kccachetest /usr/local/bin/kchashtest /usr/local/bin/kchashmgr /usr/local/bin/kctreetest /usr/local/bin/kctreemgr /usr/local/bin/kcpolytest /usr/local/bin/kcpolymgr /usr/local/bin/kclanctest /usr/local/share/kyotocabinet/... /usr/local/man/man1/...
The following options can be specified with `./configure'.
`--prefix' and other options are also available as with usual UNIX software packages. If you want to install Kyoto Cabinet under `/usr' not `/usr/local', specify `--prefix=/usr'. As well, the library search path does not include `/usr/local/lib', it is necessary to set the environment variable `LD_LIBRARY_PATH' to include `/usr/local/lib' before running applications of Kyoto Cabinet.
Kyoto Cabinet provides API of the C++ language and it is available by programs conforming to the C++03 standard. As the header files of Kyoto Cabinet are provided as `kcutil.h', `kchashdb.h', and so on, applications should include one or more of them accordingly to use the API. As the library is provided as `libkyotocabinet.a' and `libkyotocabinet.so' and they depends `libz.so', `libstdc++.so', `librt.so', `libpthread.so', `libm.so', and `libc.so', linker options corresponding to them are required by the build command. The typical build command is the following.
$ g++ -I/usr/local/include example.cc -o example \ -L/usr/local/lib -lkyotocabinet -lz -lstdc++ -lrt -lpthread -lm -lc
Microsoft Visual Studio (Visual C++) is required to build Kyoto Cabinet on Windows. The building configuration is described in the file `VCmakefile', which should be edited according to your environment. Then, perform the following command in a command prompt window.
> nmake -f VCmakefile
If you want, perform self-diagnostic test.
> nmake -f VCmakefile check
If all building processes finish successfully, the static library `kyotocabinet.lib' and some executable files are generated. As for now, neither DLL nor installation tool is provided. Please, install the header files, the library file, and the executable files by yourself.
If the header files and the library file are in the current directory, you can build an application program by the following command.
> cl /I. example.cc kyotocabinet.lib
By default, the library is built with linking to `LIBCMT.LIB' by the `/MT' option. If you want to use `MSVCRT.LIB', which is required by the MFC, rebuild the library with the `/MD' option and set the same option when building applications. If your environment is 64-bit version, add the `/D_SYS_WIN64_' option to compiler options to improve performance.
This section describes how to use Kyoto Cabinet with the command line utilities and some sample application programs.
To begin with, let's build a file hash database with the command line utility `kchashmgr'. The database stores a list of staffs of a company. Each record is composed of the key and the value. The key is the staff ID and the value is person's name.
The database must be created just one time before any database operation. Let's create a database file "casket.kch" with the default configuration.
$ kchashmgr create staffs.kch
Register some staffs into the database.
$ kchashmgr set staffs.kch 1001 "George Washington" $ kchashmgr set staffs.kch 1002 "John Adams" $ kchashmgr set staffs.kch 1003 "Thomas Jefferson" $ kchashmgr set staffs.kch 1004 "James Madison"
Check the current contents.
$ kchashmgr list -pv staffs.kch 1001 George Washington 1002 John Adams 1003 Thomas Jefferson 1004 James Madison
To retrieve the value of a record, search the database with the key.
$ kchashmgr get staffs.kch 1003 Thomas Jefferson
Of course, you can remove a record with the key.
$ kchashmgr remove staffs.kch 1003
That's all for the fundamental operations. The DBM family have been improving performance thanks to discarding the functionality.
Next, let's write a sample application program handling a file hash database. See the following source code.
#include <kchashdb.h>
using namespace std;
using namespace kyotocabinet;
// main routine
int main(int argc, char** argv) {
// create the database object
HashDB db;
// open the database
if (!db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE)) {
cerr << "open error: " << db.error().name() << endl;
}
// store records
if (!db.set("foo", "hop") ||
!db.set("bar", "step") ||
!db.set("baz", "jump")) {
cerr << "set error: " << db.error().name() << endl;
}
// retrieve a record
string* value = db.get("foo");
if (value) {
cout << *value << endl;
delete value;
} else {
cerr << "get error: " << db.error().name() << endl;
}
// traverse records
DB::Cursor* cur = db.cursor();
cur->jump();
pair<string, string>* rec;
while ((rec = cur->get_pair(true)) != NULL) {
cout << rec->first << ":" << rec->second << endl;
delete rec;
}
delete cur;
// close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
}
return 0;
}
Save the above code as a file "example.cc". Then, perform the following command line. The command `kcutilcodec conf' prints the building configuration.
$ g++ `kcutilcodec conf -i` -o example example.cc `kcutilcodec conf -l`
Execute the application program built by the above.
$ ./example hop foo:hop bar:step baz:jump
The API of the file hash database is defined in the header `kchash.h'. So, include the header near the front of a source file. All symbols of Kyoto Cabinet are packaged in the name space `kyotocabinet'. You can use them without any prefix by importing the name space.
#include <kchashdb.h> using namespace kyotocabinet;
The class `HashDB' contains all functionality of the file hash database and each instance expresses a file hash database file.
HashDB db;
Each database file must be opened by the `open' method before any database operation. The flag `HashDB::OWRITER' means the process will update the database. The flag `HashDB::OCREATE' means a database file will be created if it does not exist yet.
db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE);
Every opened database must be closed by the `close' method when it is no longer in use. Closing the database is very important to avoid data corruption and memory leak.
db.close();
To store a record, use the `set' method with the key and the value.
db.put("foo", "hop");
To retrieve the value of a record, use the `get' method with the key. The return value is NULL if no record corresponds to the key. On success, the return value is the pointer to a dynamic object and it should be deleted explicitly after the use.
string* value = db.get("foo", "hop");
if (value) {
cout << *value << endl;
delete value;
}
Except for `set' and `get', there are other methods; `add', `append', `remove', `increment', and `cas'. Each method has two versions; for `std::string' parameters and for `char*' and `size_t' parameters.
Traversing records is a bit complicated task. It needs a cursor object, which expresses the current position in the sequence of all records in the database. Each cursor is created by the `cursor' method of the database object. Each cursor should be initialized by `jump' method before actual record operations.
DB::Cursor* cur = db.cursor(); cur->jump();
The cursor class has such methods against the record at the current position as `set_value', `remove', `get_key', `get_value', and `get_pair'. Most methods have an optional stepping parameter to shift the current position to the next record atomically. Therefore, iterating such methods with the stepping parameter results in that all records are visited.
pair<string, string>* rec;
while ((rec = cur.get_pair(true)) != NULL) {
cout << rec->first << ":" << rec->second << endl;
delete rec;
}
Every operation against a record can be abstracted by the "visitor" pattern. A visitor object specifies call back methods which receives the state of a record and returns the new state. Let's see the next sample using the visitor pattern.
#include <kchashdb.h>
using namespace std;
using namespace kyotocabinet;
// main routine
int main(int argc, char** argv) {
// create the database object
HashDB db;
// open the database
if (!db.open("casket.kch", HashDB::OREADER)) {
cerr << "open error: " << db.error().name() << endl;
}
// define the visitor
class VisitorImpl : public DB::Visitor {
// call back function for an existing record
const char* visit_full(const char* kbuf, size_t ksiz,
const char* vbuf, size_t vsiz, size_t *sp) {
cout << string(kbuf, ksiz) << ":" << string(vbuf, vsiz) << endl;
return NOP;
}
// call back function for an empty record space
const char* visit_empty(const char* kbuf, size_t ksiz, size_t *sp) {
cerr << string(kbuf, ksiz) << " is missing" << endl;
return NOP;
}
} visitor;
// retrieve a record with visitor
if (!db.accept("foo", 3, &visitor, false) ||
!db.accept("dummy", 5, &visitor, false)) {
cerr << "accept error: " << db.error().name() << endl;
}
// traverse records with visitor
if (!db.iterate(&visitor, false)) {
cerr << "iterate error: " << db.error().name() << endl;
}
// close the database
if (!db.close()) {
cerr << "close error: " << db.error().name() << endl;
}
return 0;
}
The methods `accept' and `iterate' receive visitor objects. Each visitor object must be implement the interface `DB::Visitor'. The method `visit_full' is called for an existing record. The parameters specify the pointer to key buffer, the length of the key buffer, the pointer to the value buffer, the length of the value buffer, and the pointer to the variable to notice the length of the return value. The return value is `NOP' for no update, `REMOVE' to remove the record, or otherwise the pointer to the buffer contains arbitrary byte pattern to update the value. The method `visit_empty' is called for an empty record space. The specification is the same to `visit_full' except that it does not receive the record value.
As it is, almost all built-in operations like `set', `remove', and `get' are implemented with the `accept' method. The following code is to store a record.
class Setter : public DB::Visitor {
public:
Setter(const char* vbuf, size_t vsiz) : vbuf_(vbuf), vsiz_(vsiz) {}
private:
const char* visit_full(const char* kbuf, size_t ksiz,
const char* vbuf, size_t vsiz, size_t* sp) {
*sp = vsiz_;
return vbuf_;
}
const char* visit_empty(const char* kbuf, size_t ksiz, size_t* sp) {
*sp = vsiz_;
return vbuf_;
}
const char* vbuf_;
size_t vsiz_;
};
Setter setter("foo", 3);
accept(kbuf, ksiz, &setter, true);
As Kyoto Cabinet provides various database classes, they have the common base class, which defines the common interface of all database classes. That is, you can use the following classes in the same way.
| class | file | description |
|---|---|---|
ProtoHashDB |
kcprotodb.h |
prototype hash database.
on-memory database implemented with std::unorderd_map of STL.
|
ProtoTreeDB |
kcprotodb.h |
prototype tree database.
on-memory database implemented with std::map of STL.
|
CacheDB |
kccachedb.h |
cache database.
on-memory database featuring LRU deletion.
|
HashDB |
kchashdb.h |
file hash database.
file database of hash table; typical DBM.
|
TreeDB |
kctreedb.h |
file tree database.
file database of B+ tree; DBM with order.
|
PolyDB |
kcpolydb.h |
polymorphic database.
dynamic binding of the above databases.
|
Each database has different features in persistence, algorithm, time complexity, record sequence, and lock model for concurrency.
ProtoHashDB |
ProtoTreeDB |
CacheDB |
HashDB |
TreeDB |
|
| persistence | volatile | volatile | volatile | persistent | persistent |
| algorithm | hash table | red black tree | hash table | hash table | B+ tree |
| complexity | O(1) | O(log N) | O(1) | O(1) | O(log N) |
| sequence | undefined | lexical order | undefined | undefined | custom order |
| lock unit | whole (rwlock) | whole (rwlock) | record (mutex) | record (rwlock) | page (rwlock) |
The C language binding is also provided as a wrapper of the polymorphic database API. Include the header file `kclangc.h' and use the pointer to `KCDB' as a database object.
Please see the the API documents for detail. Writing your own sample application is the best way to learn this library.
This section describes how to use command line utilities. They are useful to manage database contents and to test the library and its applications.
The command `kcutiltest' is a utility for facility test and performance test of the utility functions. This command is used in the following format. `rnum' specifies the number of iterations. `path' specifies the path of a file.
kcutiltest mutex [-th num] [-iv num] rnumkcutiltest file [-th num] [-rnd] [-msiz num] path rnumkcutiltest map [-rnd] rnumkcutiltest misc rnumOptions feature the following.
This command returns 0 on success, another on failure.
The command `kcutilcodec' is a tool to use encoding and decoding features, or to show the configuration. This command is used in the following format.
kcutilcodec conf [-v|-i|-l|-p]Options feature the following.
This command returns 0 on success, another on failure.
The command `kcprototest' is a utility for facility test and performance test of the prototype database. This command is used in the following format. `rnum' specifies the number of iterations.
kcprototest order [-tree] [-th num] [-rnd] [-etc] [-tran] rnumkcprototest queue [-tree] [-th num] [-it num] [-rnd] rnumkcprototest wicked [-tree] [-th num] [-it num] rnumkcprototest tran [-tree] [-th num] [-it num] rnumOptions feature the following.
This command returns 0 on success, another on failure.
The command `kccachetest' is a utility for facility test and performance test of the cache database. This command is used in the following format. `rnum' specifies the number of iterations.
kccachetest order [-th num] [-rnd] [-etc] [-tran] [-bnum num] [-capcnt num] [-capsiz num] rnumkccachetest queue [-th num] [-it num] [-rnd] [-bnum num] [-capcnt num] [-capsiz num] rnumkccachetest wicked [-th num] [-it num] [-bnum num] [-capcnt num] [-capsiz num] rnumkccachetest tran [-th num] [-it num] [-bnum num] [-capcnt num] [-capsiz num] rnumOptions feature the following.
This command returns 0 on success, another on failure.
The command `kchashtest' is a utility for facility test and performance test of the file hash database. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.
kchashtest order [-th num] [-rnd] [-set|-get|-getw|-rem|-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnumkchashtest queue [-th num] [-it num] [-rnd] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnumkchashtest wicked [-th num] [-it num] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnumkchashtest tran [-th num] [-it num] [-hard] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnumOptions feature the following.
This command returns 0 on success, another on failure.
The command `kchashmgr' is a utility for test and debugging of the file hash database and its applications. `path' specifies the path of a database file. `key' specifies the key of a record. `value' specifies the value of a record. `file' specifies the input file.
kchashmgr create [-otr] [-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] pathkchashmgr inform [-onl|-otl|-onr] [-st] pathkchashmgr set [-onl|-otl|-onr] [-add|-app|-inci|-incd] [-sx] path key valuekchashmgr remove [-onl|-otl|-onr] [-sx] path keykchashmgr get [-onl|-otl|-onr] [-sx] [-px] [-pz] path keykchashmgr list [-onl|-otl|-onr] [-max num] [-sx] [-pv] [-px] path [key]kchashmgr import [-onl|-otl|-onr] [-sx] path [file]kchashmgr dump [-onl|-otl|-onr] path [file]kchashmgr load [-otr] [-onl|-otl|-onr] path [file]kchashmgr defrag [-onl|-otl|-onr] pathkchashmgr check [-onl|-otl|-onr] pathOptions feature the following.
This command returns 0 on success, another on failure.
The command `kctreetest' is a utility for facility test and performance test of the file tree database. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.
kctreetest order [-th num] [-rnd] [-set|-get|-getw|-rem|-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnumkctreetest queue [-th num] [-it num] [-rnd] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnumkctreetest wicked [-th num] [-it num] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnumkctreetest tran [-th num] [-it num] [-hard] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnumOptions feature the following.
This command returns 0 on success, another on failure.
The command `kctreemgr' is a utility for test and debugging of the file tree database and its applications. `path' specifies the path of a database file. `key' specifies the key of a record. `value' specifies the value of a record. `file' specifies the input file.
kctreemgr create [-otr] [-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-rcd] pathkctreemgr inform [-onl|-otl|-onr] [-st] pathkctreemgr set [-onl|-otl|-onr] [-add|-app|-inci|-incd] [-sx] path key valuekctreemgr remove [-onl|-otl|-onr] [-sx] path keykctreemgr get [-onl|-otl|-onr] [-sx] [-px] [-pz] path keykctreemgr list [-onl|-otl|-onr] [-max num] [-sx] [-pv] [-px] path [key]kctreemgr import [-onl|-otl|-onr] [-sx] path [file]kctreemgr dump [-onl|-otl|-onr] path [file]kctreemgr load [-otr] [-onl|-otl|-onr] path [file]kctreemgr defrag [-onl|-otl|-onr] pathkctreemgr check [-onl|-otl|-onr] pathOptions feature the following.
This command returns 0 on success, another on failure.
The command `kcpolytest' is a utility for facility test and performance test of the polymorphic database. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.
kcpolytest order [-th num] [-rnd] [-set|-get|-getw|-rem|-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] path rnumkcpolytest queue [-th num] [-it num] [-rnd] [-oat|-onl|-onl|-otl|-onr] path rnumkcpolytest wicked [-th num] [-it num] [-oat|-onl|-onl|-otl|-onr] path rnumkcpolytest tran [-th num] [-it num] [-hard] [-oat|-onl|-onl|-otl|-onr] path rnumkcpolytest misc pathOptions feature the following.
This command returns 0 on success, another on failure.
The command `kcpolymgr' is a utility for test and debugging of the file hash database and its applications. `path' specifies the path of a database file. `key' specifies the key of a record. `value' specifies the value of a record. `file' specifies the input file.
kcpolymgr create [-otr] [-onl|-otl|-onr] pathkcpolymgr inform [-onl|-otl|-onr] [-st] pathkcpolymgr set [-onl|-otl|-onr] [-add|-app|-inci|-incd] [-sx] path key valuekcpolymgr remove [-onl|-otl|-onr] [-sx] path keykcpolymgr get [-onl|-otl|-onr] [-sx] [-px] [-pz] path keykcpolymgr list [-onl|-otl|-onr] [-max num] [-sx] [-pv] [-px] path [key]kcpolymgr import [-onl|-otl|-onr] [-sx] path [file]kcpolymgr dump [-onl|-otl|-onr] path [file]kcpolymgr load [-otr] [-onl|-otl|-onr] path [file]kcpolymgr defrag [-onl|-otl|-onr] pathkcpolymgr check [-onl|-otl|-onr] pathOptions feature the following.
The command `kclangctest' is a utility for facility test and performance test of the C language binding. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.
kcpolytest order [-rnd] [-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] path rnumOptions feature the following.
This command returns 0 on success, another on failure.
This section describes tips and hacks to use Kyoto Cabinet.
The cache database is on-memory database featuring LRU deletion. The following tuning methods are provided.
tune_buckets : sets the number of buckets of the hash table.cap_count : sets the capacity by record number.cap_size : sets the capacity by memory usage.By default, the cache database maintains all records on memory and no record is expired. If you want to expire old records to keep the memory usage constant, call `cap_count' and/or `cap_size' to limit the capacity.
The default tuning of the bucket number is about one million. If you intend to store more records, call `tune_buckets' to set the bucket number. The suggested ratio of the bucket number is the same to the total number of records and it is okay from 50% to 400%. If the ratio decreases smaller than 50%, the time efficiency will decrease gradually.
If you want to cache ten millions of records and keep the memory usage less than 8GB, the following tuning is suggested for example.
db.tune_buckets(10LL * 1000 * 1000); db.cap_count(10LL * 1000 * 1000); db.cap_size(8LL << 30); db.open(...);
All tuning methods must be called before the database is opened.
The file hash database is file database of hash table. The following tuning methods are provided.
tune_alignment : sets the power of the alignment of record size.tune_fbp : sets the power of the capacity of the free block pool.tune_options : sets the optional features.tune_buckets : sets the number of buckets of the hash table.tune_map : sets the size of the internal memory-mapped region.tune_defrag : sets the unit step number of auto defragmentation.tune_compressor : set the data compressor.The default alignment power is 3, which means the address of each record is aligned to a multiple of 8 (1<<3) bytes. If you trust that the database is constructed at a time and not updated often, call `tune_alignment' to set the alignment power 0, which means 1 (1<<0) byte. If the typical size of each record is expected to be larger than 1KB, tune the alignment 8 or more.
The tuning of the free block pool by `tune_fbp' does not have to be modified in most cases. The default is 10, which means the capacity of the free block pool is 1024 (1<<10).
The optional features by `tune_options' is useful to reduce the size of the database fine at the expense of scalability or time efficiency. If `HashDB::TSMALL' is specified, the width of record addressing is reduced from 6 bytes to 4 bytes. As the result, the footprint for each record is reduced from 16 bytes to 12 bytes. However, it limits the maximum size of the database file up to 16GB (2GB multiplied by the alignment). If `HashDB::TLINEAR' is specified, the data structure of the collision chain of hash table is changed from binary tree to linear linked list. In that case, the footprint of each record is reduced from 16 bytes to 10 bytes although the time efficiency becomes sensitive to the number of the hash buckets. If `HashDB::TCOMPRESS' is specified, the value of each record is compressed implicitly when stored in the file. If the value is bigger than 1KB or more, compression is effective.
The default tuning of the bucket number is about one million. If you intend to store more records, call `tune_buckets' to set the bucket number. The suggested ratio of the bucket number is the twice of the total number of records and it is okay from 100% to 400%. If the ratio decreases smaller than 100%, the time efficiency will decrease gradually.
The default tuning of the size of the internal memory-mapped region is 64MB. If the database size is expected to be larger than 64MB, call `tune_map to set the map size larger than the expected size of the database. Although the capacity of the RAM on the machine limits the map size, increasing the map size is effective to improve performance.
By default, auto defragmentation is disabled. If the existing records in the database are modified (removed or modified with varying the size), fragmentation of available regions proceeds gradually. In that case, call `tune_defrag' to enable auto defragmentation and set the unit step number. The suggested unit step number is 8, which means that a set of defragmentation operations is performed each 8 updating operations. The more the unit is, space efficiency becomes higher but time efficiency becomes lower.
The default compression algorithm of the `HashDB::TLINEAR' option is "Deflate" by ZLIB. If you want to use another algorithm, call `tune_compressor' to set a functor which implements compression and decompression functions.
If you intend to store ten thousands of records and reduce the database size as possible, the following tuning is suggested for example.
db.tune_alignment(0); db.tune_buckets(10LL * 1000); db.tune_options(HashDB::TSMALL | HashDB::TLINEAR); db.tune_defrag(8); db.open(...);
If you have a monster machine and intend to store ten billion records and improve time efficiency as possible, the following tuning is suggested for example.
db.tune_buckets(20LL * 1000 * 1000 * 1000); db.tune_map(300LL << 30); db.open(...);
All tuning methods must be called before the database is opened. Because the settings of `tune_alignment', `tune_fbp', `tune_options', and `tune_buckets' are recorded as the meta data of the database, the methods are called before the database is created and they can not be modified afterward. Because other tuning parameters are not recorded in the database, they should be specified before every time opening the database.
The file tree database is file database of B+ tree. Because each node of B+ tree is serialized as a page buffer and stored as a record of the file hash database, all tuning parameters of the file hash database is inherited to the file tree database. Moreover, the following tuning methods are added.
tune_page : sets the size of each page.tune_page_cache : sets the capacity size of the page cache.tune_comparator : sets the record comparator.The tuning of the page size by `tune_page' does not have to be modified in most cases. The default is 8192, which is the twice of the typical page size of popular environments. If the size of each node exceeds the parameter, the node is divided into two.
The default tuning of the capacity size of the page cache is 64MB. If your machine has abundant RAM, call `tune_page' to load all nodes on the page cache. If the RAM is not abundant, it is better to keep the default page cache size and assign the RAM for the internal memory-mapped region by `tune_map'.
The default record comparator is the lexical ordering function. That is, records in the B+ tree are placed in the lexical order of each key. If you want to use another ordering, call `tune_comparator' to set a functor which implements the ordering function.
The default alignment of the file tree database is 256 (1<<8). The default bucket number of the file tree database is about 65536. Other default tuning parameters are the same to the file hash database. Note that the bucket number should be calculated by the number of pages. The suggested ratio of the bucket number is about 10% of the number of records. If the compression option is specified, all records in each page are compressed at once. Therefore, compression is more effective for the file tree database rather than for the file hash database.
If you intend to store ten thousands of records and reduce the database size as possible, the following tuning is suggested for example.
db.tune_buckets(1LL * 1000); db.tune_options(TreeDB::TCCOMPESS); db.tune_defrag(8); db.tune_page(32768); db.open(...);
If you have a monster machine and intend to store ten billion records and improve time efficiency as possible, the following tuning is suggested for example.
db.tune_buckets(1LL * 1000 * 1000 * 1000); db.tune_map(300LL << 30); db.tune_page_cache(2LL << 30); db.open(...);
All tuning methods must be called before the database is opened. Because the setting of `tune_page' is recorded as the meta data of the database, the methods are called before the database is created and it can not be modified afterward. Because other tuning parameters are not recorded in the database, they should be specified before every time opening the database.
If an application process which opened a database terminated without closing the database, it involves risks that some records may be missing and the database may be broken. By default, durability is settled when the database is closed properly and it is not settled for each updating operation. Kyoto Cabinet deal with the problem by transaction mechanism based on WAL (write ahead logging). Transaction is begun and committed by application explicitly. Durability during transaction is settled for each updating operation. Transaction can be aborted by application. In that case, all update operations during transaction are voided and the content of the database is rollbacked. Although transaction is very useful like this, throughput of updating operation decreases to about 50% of the default manner due to overhead of writing WAL.
db.begin_transaction();
db.set("japan", "tokyo");
db.set("korea", "seoul");
db.end_transaction();
If you can't be bothered to begin and commit transaction explicitly, use auto transaction mechanism by specifying the `FileDB::AUTOTRAN' option when opening the database. Auto transaction is begun and committed implicitly for each operation. Overhead of auto transaction is lighter than explicit transaction.
db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE | HashDB::OAUTOTRAN);
db.set("japan", "tokyo");
db.set("china", "beijing");
After all, it is important to choose the usage according to the requirements of your application.
Any hardware will break down suddenly. Especially such storage devices as HDD and SSD are fragile. Therefore, making backup files of your database file periodically is very important even if you use transaction. You can copy a database file by such popular commands as `cp' and `tar' when the database is not being updated by another process.
If an application uses multi threads and you want to make a backup file of the database in safety, use the `FileDB::copy' method, which synchronizes the database status with the database file and makes a copy file. During the copying operation, it is assured that the database file is not updated.
db.copy("backup.kch");
You may want "hot backup", which means that other threads are not blocked while a thread is creating a backup file. In that case, use the `File::synchronize' method which synchronizes the database file and calls a function defined arbitrary. The call back function can execute a "snapshot" command provided by the operating system.
class BackupImpl : public FileProcessor {
bool process(const std::string& path, int64_t size, int64_t count) {
char cmd[1024];
sprintf(cmd, "snapshot.sh %s", path.c_str());
return system(cmd) == 0;
}
} proc;
db.synchronize(&proc);
Chiefly for the cache database, the pseudo-snapshot mechanism is provided. The `FileDB::dump_snapshot' dumps all records into a stream or a file. The `FileDB::load_snapshot' method loads records from a stream or a file. Although the operations are performed atomically, they don't finish momentarily but take time in proportion of the database size with blocking the other threads. Because the format of pseudo-snapshot data is common among the all database classes, it is useful to migrate records for each other.
db.dump_snapshot("backup.kcss");
db.load_snapshot("backup.kcss");
If you don't want to let the other threads be blocked. Use the cursor mechanism and save/load records by yourself.
Kyoto Cabinet is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.
Kyoto Cabinet is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see `http://www.gnu.org/licenses/'.
Kyoto Cabinet was written by Mikio Hirabayashi. You can contact the author by e-mail to `hirarin@gmail.com'.