SharkDB: an in-memory column-oriented storage for trajectory analysis

Handle URI:
http://hdl.handle.net/10754/623667
Title:
SharkDB: an in-memory column-oriented storage for trajectory analysis
Authors:
Zheng, Bolong; Wang, Haozhou; Zheng, Kai; Su, Han; Liu, Kuien; Shang, Shuo
Abstract:
The last decade has witnessed the prevalence of sensor and GPS technologies that produce a high volume of trajectory data representing the motion history of moving objects. However some characteristics of trajectories such as variable lengths and asynchronous sampling rates make it difficult to fit into traditional database systems that are disk-based and tuple-oriented. Motivated by the success of column store and recent development of in-memory databases, we try to explore the potential opportunities of boosting the performance of trajectory data processing by designing a novel trajectory storage within main memory. In contrast to most existing trajectory indexing methods that keep consecutive samples of the same trajectory in the same disk page, we partition the database into frames in which the positions of all moving objects at the same time instant are stored together and aligned in main memory. We found this column-wise storage to be surprisingly well suited for in-memory computing since most frames can be stored in highly compressed form, which is pivotal for increasing the memory throughput and reducing CPU-cache miss. The independence between frames also makes them natural working units when parallelizing data processing on a multi-core environment. Lastly we run a variety of common trajectory queries on both real and synthetic datasets in order to demonstrate advantages and study the limitations of our proposed storage.
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Citation:
Zheng B, Wang H, Zheng K, Su H, Liu K, et al. (2017) SharkDB: an in-memory column-oriented storage for trajectory analysis. World Wide Web. Available: http://dx.doi.org/10.1007/s11280-017-0466-9.
Publisher:
Springer Nature
Journal:
World Wide Web
Issue Date:
5-May-2017
DOI:
10.1007/s11280-017-0466-9
Type:
Article
ISSN:
1386-145X; 1573-1413
Sponsors:
This work is partially supported by Natural Science Foundation of China (No. 61502324 and No. 61532018).
Additional Links:
http://link.springer.com/article/10.1007/s11280-017-0466-9
Appears in Collections:
Articles; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.authorZheng, Bolongen
dc.contributor.authorWang, Haozhouen
dc.contributor.authorZheng, Kaien
dc.contributor.authorSu, Hanen
dc.contributor.authorLiu, Kuienen
dc.contributor.authorShang, Shuoen
dc.date.accessioned2017-05-21T05:30:10Z-
dc.date.available2017-05-21T05:30:10Z-
dc.date.issued2017-05-05en
dc.identifier.citationZheng B, Wang H, Zheng K, Su H, Liu K, et al. (2017) SharkDB: an in-memory column-oriented storage for trajectory analysis. World Wide Web. Available: http://dx.doi.org/10.1007/s11280-017-0466-9.en
dc.identifier.issn1386-145Xen
dc.identifier.issn1573-1413en
dc.identifier.doi10.1007/s11280-017-0466-9en
dc.identifier.urihttp://hdl.handle.net/10754/623667-
dc.description.abstractThe last decade has witnessed the prevalence of sensor and GPS technologies that produce a high volume of trajectory data representing the motion history of moving objects. However some characteristics of trajectories such as variable lengths and asynchronous sampling rates make it difficult to fit into traditional database systems that are disk-based and tuple-oriented. Motivated by the success of column store and recent development of in-memory databases, we try to explore the potential opportunities of boosting the performance of trajectory data processing by designing a novel trajectory storage within main memory. In contrast to most existing trajectory indexing methods that keep consecutive samples of the same trajectory in the same disk page, we partition the database into frames in which the positions of all moving objects at the same time instant are stored together and aligned in main memory. We found this column-wise storage to be surprisingly well suited for in-memory computing since most frames can be stored in highly compressed form, which is pivotal for increasing the memory throughput and reducing CPU-cache miss. The independence between frames also makes them natural working units when parallelizing data processing on a multi-core environment. Lastly we run a variety of common trajectory queries on both real and synthetic datasets in order to demonstrate advantages and study the limitations of our proposed storage.en
dc.description.sponsorshipThis work is partially supported by Natural Science Foundation of China (No. 61502324 and No. 61532018).en
dc.publisherSpringer Natureen
dc.relation.urlhttp://link.springer.com/article/10.1007/s11280-017-0466-9en
dc.rightsThe final publication is available at Springer via http://dx.doi.org/10.1007/s11280-017-0466-9en
dc.subjectSpatial databaseen
dc.subjectTrajectoryen
dc.subjectIn-memoryen
dc.subjectStorageen
dc.titleSharkDB: an in-memory column-oriented storage for trajectory analysisen
dc.typeArticleen
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
dc.identifier.journalWorld Wide Weben
dc.eprint.versionPost-printen
dc.contributor.institutionThe University of Queensland, Brisbane, Australiaen
dc.contributor.institutionPivotal Incorporated, San Francisco, USAen
dc.contributor.institutionSchool of Computer Science and Techonology, Soochow University, Suzhou, Chinaen
dc.contributor.institutionBig Data Research Center, University of Electronic Science and Technology of China, Chengdu, Chinaen
kaust.authorShang, Shuoen
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.