Accelerating SPARQL queries by exploiting hash-based locality and adaptive partitioning
Type
ArticleKAUST Department
Computer Science ProgramComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Date
2016-02-08Online Publication Date
2016-02-08Print Publication Date
2016-06Permanent link to this record
http://hdl.handle.net/10754/621375
Metadata
Show full item recordAbstract
State-of-the-art distributed RDF systems partition data across multiple computer nodes (workers). Some systems perform cheap hash partitioning, which may result in expensive query evaluation. Others try to minimize inter-node communication, which requires an expensive data preprocessing phase, leading to a high startup cost. Apriori knowledge of the query workload has also been used to create partitions, which, however, are static and do not adapt to workload changes. In this paper, we propose AdPart, a distributed RDF system, which addresses the shortcomings of previous work. First, AdPart applies lightweight partitioning on the initial data, which distributes triples by hashing on their subjects; this renders its startup overhead low. At the same time, the locality-aware query optimizer of AdPart takes full advantage of the partitioning to (1) support the fully parallel processing of join patterns on subjects and (2) minimize data communication for general queries by applying hash distribution of intermediate results instead of broadcasting, wherever possible. Second, AdPart monitors the data access patterns and dynamically redistributes and replicates the instances of the most frequent ones among workers. As a result, the communication cost for future queries is drastically reduced or even eliminated. To control replication, AdPart implements an eviction policy for the redistributed patterns. Our experiments with synthetic and real data verify that AdPart: (1) starts faster than all existing systems; (2) processes thousands of queries before other systems become online; and (3) gracefully adapts to the query load, being able to evaluate queries on billion-scale RDF data in subseconds.Citation
Harbi R, Abdelaziz I, Kalnis P, Mamoulis N, Ebrahim Y, et al. (2016) Accelerating SPARQL queries by exploiting hash-based locality and adaptive partitioning. The VLDB Journal 25: 355–380. Available: http://dx.doi.org/10.1007/s00778-016-0420-y.Publisher
Springer NatureJournal
The VLDB Journalae974a485f413a2113503eed53cd6c53
10.1007/s00778-016-0420-y