What is Starskey?

Starskey is a fast embedded key-value store package for GO! Starskey implements a multi-level, durable log structured merge tree.


It is inspired by the WiscKey paper and the LevelDB implementation.

Features

Getting Started

Importing


import (
    "github.com/starskey-io/starskey"
)

Opening and configuration

To open a new database you must pass a configuration pointer. This is required.


skey, err := starskey.Open(&starskey.Config{
    Permission:        0755,                   // Dir, file permission
    Directory:         "db_dir",               // Directory to store data
    FlushThreshold:    (1024 * 1024) * 24,     // 24mb Flush threshold in bytes, for production use 64mb or higher
    MaxLevel:          3,                      // Max levels number of disk levels
    SizeFactor:        10,                     // Size factor for each level.  Say 10 that's 10 * the FlushThreshold at each level. So level 1 is 10MB, level 2 is 100MB, level 3 is 1GB.
    BloomFilter:       false,                  // If you want to use bloom filters
    SuRF:              false,                  // If enabled will speed up range queries as we check if an sstable has the keys we are looking for.
    Logging:           true,                   // Enable logging to file
    Compression:       false,                  // Enable compression
    CompressionOption: starskey.NoCompression, // Or SnappyCompression, S2Compression
    // Internal options
    // Optional: &OptionalConfig{
    //    BackgroundFSync:         .. If you don't want to fsync writes to disk (default is true)
    //    BackgroundFSyncInterval: .. Interval for background fsync, if configured true (default is 256ms)
    //    TTreeMin:                .. Minimum degree of the T-Tree
    //    TTreeMax:                .. Maximum degree of the T-Tree
    //    PageSize:                .. Page size for internal pagers
    //    BloomFilterProbability:  .. Bloom filter probability
    //    },
    }) // Config cannot be nil**
    }) // Config cannot be nil**
if err != nil {
    // ..handle error
}

Mind you examples use skey as the variable name for opened starskey instance(s).

Close

To close database gracefully.


if err := skey.Close(); err != nil {
    // ..handle error
}

CRUD Operations

Putting, Getting, Deleting

Put

To write we just pass a key and a value byte array.


key := []byte("some_key")
value := []byte("some_value")

if err := skey.Put(key, value); err != nil {
    // ..handle error
}

To update a keys value you run put with a different value.

Get

To retrieve a value you pass a key byte array.


v, err := skey.Get(key)
if err != nil {
    // ..handle error
}

Delete

To remove a key-value you pass a key byte array you want to remove.


if err := skey.Delete([]byte("key")); err != nil {
    // ..handle error
}

// Delete by range
if n, err := skey.DeleteByRange([]byte("startKey"), []byte("endKey")); err != nil {
    // ..handle error
}

// Delete by filter
compareFunc := func(key []byte) bool {
    // if has prefix "c" return true
    return bytes.HasPrefix(key, []byte("c"))
}

if n, err := skey.DeleteByFilter(compareFunc); err != nil {
    // ..handle error
}

// Delete by prefix
if n, err := skey.DeleteByPrefix([]byte("key")); err != nil {
    // ..handle error
}

Transactions

Using acid transactions to group multiple operations into a single atomic transaction.


txn := skey.BeginTxn()
if txn == nil {
    // ..handle error
}

txn.Put([]byte("key"), []byte("value"))
// or txn.Delete([]byte("key"))

if err := txn.Commit(); err != nil {
    // ..handle error
}

Or


err = skey.Update(func(txn *Txn) error {
    txn.Put([]byte("key"), []byte("value")) // or txn.Delete, txn.Get
    // ..
    return nil
})
if err != nil {
    // ..handle error
}

Querying by range

You can provide a start and end key to retrieve a range of keys.


results, err := skey.Range([]byte("key900"), []byte("key980"))
if err != nil {
    // ..handle error
}

Querying by prefix

Starskey supports optimized prefix searches.


// Longest prefix search
result, n, err := skey.LongestPrefixSearch([]byte("key"))
if err != nil {
    // ..handle error
}

// Prefix search
results, err := skey.PrefixSearch([]byte("ke"))
if err != nil {
    // ..handle error
}

Querying by filter

Using filter to filter keys based on a function.


compareFunc := func(key []byte) bool {
    // if has prefix "c" return true
    return bytes.HasPrefix(key, []byte("c"))
}

results, err := skey.FilterKeys(compareFunc)
if err != nil {
    // ..handle error
}

Benchmarks & Benchmarking

There is a benchmark program comparing against Pebble and BBolt. Random writes, reads, and deletes.


Starskey Bench

Key life cycle

A key once inserted will live in the memtable until it is flushed to disk. Once flushed to disk it will live in an sstable at l1 until it is compacted. Once compacted it will be merged into a new sstable at the next level. This process is recursive until the last level. At the last level if full we merge all sstables into a new sstable.
If a key is deleted it will live on the same way until it reaches last level at which point it will be removed entirely.

Memory and disk sorting

Sorting would be lexicographical (alphabetical), meaning it will sort based on the byte-by-byte comparisons of slices. We use bytes.Compare to sort keys in memory and on disk.