A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems

dc.contributor.authorHarshvardhan,
dc.contributor.authorWest, Brandon
dc.contributor.authorFidel, Adam
dc.contributor.authorAmato, Nancy M.
dc.contributor.authorRauchwerger, Lawrence
dc.contributor.institutionDept. of Computer Science and Engineering Texas A&M University
dc.date.accessioned2016-02-25T12:29:54Z
dc.date.available2016-02-25T12:29:54Z
dc.date.issued2015-05
dc.description.abstractWith the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.
dc.description.sponsorshipWe would like to thank Glen Hordemann for help withour initial design. We would also like to thank our anony-mous reviewers. This research is supported in part byNSF awards CCF 0702765, CNS-0551685, CCF-0833199,CCF-1439145, CCF-1423111, CCF-0830753, IIS-0917266,by DOE awards DE-AC02-06CH11357, DE-NA0002376,B575363, by Samsung, IBM, Intel, and by Award KUS-C1-016-04, made by King Abdullah University of Scienceand Technology (KAUST). This research used resources ofthe National Energy Research Scientific Computing Center,which is supported by the Office of Science of the U.S. Dept.of Energy under Contract No. DE-AC02-05CH11231.
dc.identifier.citationHarshvardhan, West B, Fidel A, Amato NM, Rauchwerger L (2015) A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems. 2015 IEEE International Parallel and Distributed Processing Symposium. Available: http://dx.doi.org/10.1109/IPDPS.2015.28.
dc.identifier.doi10.1109/IPDPS.2015.28
dc.identifier.journal2015 IEEE International Parallel and Distributed Processing Symposium
dc.identifier.urihttp://hdl.handle.net/10754/597288
dc.publisherInstitute of Electrical and Electronics Engineers (IEEE)
dc.titleA Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems
dc.typeConference Paper
display.details.left<span><h5>Type</h5>Conference Paper<br><br><h5>Authors</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Harshvardhan,,equals">Harshvardhan,</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=West, Brandon,equals">West, Brandon</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Fidel, Adam,equals">Fidel, Adam</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Amato, Nancy M.,equals">Amato, Nancy M.</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Rauchwerger, Lawrence,equals">Rauchwerger, Lawrence</a><br><br><h5>KAUST Grant Number</h5>KUS-C1-016-04<br><br><h5>Date</h5>2015-05</span>
display.details.right<span><h5>Abstract</h5>With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques that can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into sub graphs that fit in RAM and uses a paging-like technique to load sub graphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.<br><br><h5>Citation</h5>Harshvardhan, West B, Fidel A, Amato NM, Rauchwerger L (2015) A Hybrid Approach to Processing Big Data Graphs on Memory-Restricted Systems. 2015 IEEE International Parallel and Distributed Processing Symposium. Available: http://dx.doi.org/10.1109/IPDPS.2015.28.<br><br><h5>Acknowledgements</h5>We would like to thank Glen Hordemann for help withour initial design. We would also like to thank our anony-mous reviewers. This research is supported in part byNSF awards CCF 0702765, CNS-0551685, CCF-0833199,CCF-1439145, CCF-1423111, CCF-0830753, IIS-0917266,by DOE awards DE-AC02-06CH11357, DE-NA0002376,B575363, by Samsung, IBM, Intel, and by Award KUS-C1-016-04, made by King Abdullah University of Scienceand Technology (KAUST). This research used resources ofthe National Energy Research Scientific Computing Center,which is supported by the Office of Science of the U.S. Dept.of Energy under Contract No. DE-AC02-05CH11231.<br><br><h5>Publisher</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.publisher=Institute of Electrical and Electronics Engineers (IEEE),equals">Institute of Electrical and Electronics Engineers (IEEE)</a><br><br><h5>Journal</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.journal=2015 IEEE International Parallel and Distributed Processing Symposium,equals">2015 IEEE International Parallel and Distributed Processing Symposium</a><br><br><h5>DOI</h5><a href="https://doi.org/10.1109/IPDPS.2015.28">10.1109/IPDPS.2015.28</a></span>
kaust.grant.numberKUS-C1-016-04
orcid.authorHarshvardhan,
orcid.authorWest, Brandon
orcid.authorFidel, Adam
orcid.authorAmato, Nancy M.
orcid.authorRauchwerger, Lawrence
Files