SharkDB: an in-memory column-oriented storage for trajectory analysis

Type
Article

Authors
Zheng, Bolong
Wang, Haozhou
Zheng, Kai
Su, Han
Liu, Kuien
Shang, Shuo

KAUST Department
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Online Publication Date
2017-05-05

Print Publication Date
2018-03

Date
2017-05-05

Abstract
The last decade has witnessed the prevalence of sensor and GPS technologies that produce a high volume of trajectory data representing the motion history of moving objects. However some characteristics of trajectories such as variable lengths and asynchronous sampling rates make it difficult to fit into traditional database systems that are disk-based and tuple-oriented. Motivated by the success of column store and recent development of in-memory databases, we try to explore the potential opportunities of boosting the performance of trajectory data processing by designing a novel trajectory storage within main memory. In contrast to most existing trajectory indexing methods that keep consecutive samples of the same trajectory in the same disk page, we partition the database into frames in which the positions of all moving objects at the same time instant are stored together and aligned in main memory. We found this column-wise storage to be surprisingly well suited for in-memory computing since most frames can be stored in highly compressed form, which is pivotal for increasing the memory throughput and reducing CPU-cache miss. The independence between frames also makes them natural working units when parallelizing data processing on a multi-core environment. Lastly we run a variety of common trajectory queries on both real and synthetic datasets in order to demonstrate advantages and study the limitations of our proposed storage.

Citation
Zheng B, Wang H, Zheng K, Su H, Liu K, et al. (2017) SharkDB: an in-memory column-oriented storage for trajectory analysis. World Wide Web. Available: http://dx.doi.org/10.1007/s11280-017-0466-9.

Acknowledgements
This work is partially supported by Natural Science Foundation of China (No. 61502324 and No. 61532018).

Publisher
Springer Nature

Journal
World Wide Web

DOI
10.1007/s11280-017-0466-9

Additional Links
http://link.springer.com/article/10.1007/s11280-017-0466-9

Permanent link to this record