Skip to Main Content
 

Global Search Box

 
 
 
 

Files

File List


Mansa Kedia - Thesis.pdf (1.85 MB)

ETD Abstract Container

Abstract Header

Profile, Monitor, and Introspect Spark Jobs Using OSU INAM

Kedia, Mansa
http://rave.ohiolink.edu/etdc/view?acc_num=osu160692347877477

Abstract Details

2020, Master of Science, Ohio State University, Computer Science and Engineering.
With Apache Spark gaining popularity in the Big Data domain, it is getting crucial to be able to profile Spark applications and get details about each stage to help optimize performance. Spark already exposes a suite of web UIs to help the users monitor the basic application statistics. However, this information is not sufficient. An area with a lot of potential for performance improvement is the shuffle phase. If a user can get insights about this phase, it can help them to address many sources of inefficiency by modifying some design decisions. We take up this challenge of introducing a new capability to OSU INAM that allows it to gain insights about spark based big data applications to help in performance troubleshooting and workload characterization. We present a holistic view by correlating network information and spark middleware level information as well as the data transfer that happens during the shuffle phase. To demonstrate the use of this capability, we run different types of spark applications/benchmarks that help highlight the different communication patterns.
Dhabaleswar K, Panda (Advisor)
Radu Teodorescu (Committee Member)
Hari Subramoni (Committee Member)
Aamir Shafi (Committee Member)
56 p.
Computer Science
HPC; OSU INAM; Apache Spark; RDMA-Spark

Recommended Citations

Citations

  • Kedia, M. (2020). Profile, Monitor, and Introspect Spark Jobs Using OSU INAM [Master's thesis, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu160692347877477

    APA Style (7th edition)

  • Kedia, Mansa. Profile, Monitor, and Introspect Spark Jobs Using OSU INAM. 2020. Ohio State University, Master's thesis. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu160692347877477.

    MLA Style (8th edition)

  • Kedia, Mansa. "Profile, Monitor, and Introspect Spark Jobs Using OSU INAM." Master's thesis, Ohio State University, 2020. http://rave.ohiolink.edu/etdc/view?acc_num=osu160692347877477

    Chicago Manual of Style (17th edition)

osu160692347877477
176
© 2020, all rights reserved.
This open access ETD is published by The Ohio State University and OhioLINK.