Real and synthetic data sets for benchmarking key-value stores focusing on various data types and sizes

In this article, we present real and synthetic data sets for benchmarking key-values stores. Here, we focus on various data types and sizes. Key-value pairs in key-value data sets consist of the key and the value. We can construct any kinds of data as key-value data sets by assigning an arbitrary type of data as the value and a unique ID as the key. Therefore, key-value pairs are quite worthy when we deal with big data because the data types in the big data application become more various and, even sometimes, they are not known or determined. In this article, we crawl four kinds of real data sets by varying the type of data sets (i.e., variety) and generate four kinds of synthetic data sets by varying the size of data sets (i.e., volume). For real data sets, we crawl data sets with various data types from Twitter, i.e., Tweets in text, a list of hashtags, geo-location of the tweet, and the number of followers. We also present algorithms for crawling real data sets based on REST APIs and streaming APIs and for generating synthetic data sets. Using those algorithms, we can crawl any key-value pairs of data types supported by Twitter and can generate any size of synthetic data sets by extending them simply. Last, we show that the crawled and generated data sets are actually utilized for the well-known key-value stores such as Level DB of Google, RocksDB of Facebook, and Berkeley DB of Oracle. Actually, the presented real and synthetic data sets have been used for comparing the performance of them. As an example, we present an algorithm of the basic operations for the key-value stores of LevelDB.