Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memory

Handle URI:
http://hdl.handle.net/10754/598920
Title:
Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memory
Authors:
Pearce, Roger; Gokhale, Maya; Amato, Nancy M.
Abstract:
Processing large graphs is becoming increasingly important for many domains such as social networks, bioinformatics, etc. Unfortunately, many algorithms and implementations do not scale with increasing graph sizes. As a result, researchers have attempted to meet the growing data demands using parallel and external memory techniques. We present a novel asynchronous approach to compute Breadth-First-Search (BFS), Single-Source-Shortest-Paths, and Connected Components for large graphs in shared memory. Our highly parallel asynchronous approach hides data latency due to both poor locality and delays in the underlying graph data storage. We present an experimental study applying our technique to both In-Memory and Semi-External Memory graphs utilizing multi-core processors and solid-state memory devices. Our experiments using synthetic and real-world datasets show that our asynchronous approach is able to overcome data latencies and provide significant speedup over alternative approaches. For example, on billion vertex graphs our asynchronous BFS scales up to 14x on 16-cores. © 2010 IEEE.
Citation:
Pearce R, Gokhale M, Amato NM (2010) Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memory. 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis. Available: http://dx.doi.org/10.1109/sc.2010.34.
Publisher:
IEEE
Journal:
2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
KAUST Grant Number:
KUS-C1–016-04
Issue Date:
Nov-2010
DOI:
10.1109/sc.2010.34
Type:
Conference Paper
Sponsors:
This work was partially performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52–07NA27344 (LLNL-CONF-427572). Funding partially provided by LDRD 07-ERD-063. Portions of experiments were performed at the Livermore Computing facility resources. This research supported in part by NSF awards CRI-0551685, CCF-0833199, CCF-0830753, IIS-096053, IIS-0317266, by NSF/DNDO award 2008-DN-077-ARI018–02, by the DOE NNSA under the Predictive Science Academic Alliances Program by grant DE-FC52–08NA28616, by THECB NHARP grant 000512–0097-2009, by Chevron, IBM, Intel, HP, Oracle/Sun and by King Abdullah University of Science and Technology (KAUST) Award KUS-C1–016-04 Pearce is supported in part by a Lawrence Scholar Fellowship and a Dept. of Education Graduate Fellowship (GAANN).
Appears in Collections:
Publications Acknowledging KAUST Support

Full metadata record

DC FieldValue Language
dc.contributor.authorPearce, Rogeren
dc.contributor.authorGokhale, Mayaen
dc.contributor.authorAmato, Nancy M.en
dc.date.accessioned2016-02-25T13:43:44Zen
dc.date.available2016-02-25T13:43:44Zen
dc.date.issued2010-11en
dc.identifier.citationPearce R, Gokhale M, Amato NM (2010) Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memory. 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis. Available: http://dx.doi.org/10.1109/sc.2010.34.en
dc.identifier.doi10.1109/sc.2010.34en
dc.identifier.urihttp://hdl.handle.net/10754/598920en
dc.description.abstractProcessing large graphs is becoming increasingly important for many domains such as social networks, bioinformatics, etc. Unfortunately, many algorithms and implementations do not scale with increasing graph sizes. As a result, researchers have attempted to meet the growing data demands using parallel and external memory techniques. We present a novel asynchronous approach to compute Breadth-First-Search (BFS), Single-Source-Shortest-Paths, and Connected Components for large graphs in shared memory. Our highly parallel asynchronous approach hides data latency due to both poor locality and delays in the underlying graph data storage. We present an experimental study applying our technique to both In-Memory and Semi-External Memory graphs utilizing multi-core processors and solid-state memory devices. Our experiments using synthetic and real-world datasets show that our asynchronous approach is able to overcome data latencies and provide significant speedup over alternative approaches. For example, on billion vertex graphs our asynchronous BFS scales up to 14x on 16-cores. © 2010 IEEE.en
dc.description.sponsorshipThis work was partially performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52–07NA27344 (LLNL-CONF-427572). Funding partially provided by LDRD 07-ERD-063. Portions of experiments were performed at the Livermore Computing facility resources. This research supported in part by NSF awards CRI-0551685, CCF-0833199, CCF-0830753, IIS-096053, IIS-0317266, by NSF/DNDO award 2008-DN-077-ARI018–02, by the DOE NNSA under the Predictive Science Academic Alliances Program by grant DE-FC52–08NA28616, by THECB NHARP grant 000512–0097-2009, by Chevron, IBM, Intel, HP, Oracle/Sun and by King Abdullah University of Science and Technology (KAUST) Award KUS-C1–016-04 Pearce is supported in part by a Lawrence Scholar Fellowship and a Dept. of Education Graduate Fellowship (GAANN).en
dc.publisherIEEEen
dc.titleMultithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memoryen
dc.typeConference Paperen
dc.identifier.journal2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysisen
dc.contributor.institutionTexas A and M University, College Station, United Statesen
dc.contributor.institutionLawrence Livermore National Laboratory, Livermore, United Statesen
kaust.grant.numberKUS-C1–016-04en
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.