Skip to Main Content
 

Global Search Box

 
 
 
 

ETD Abstract Container

Abstract Header

Type- and Workload-Aware Scheduling of Large-Scale Wide-Area Data Transfers

Kettimuthu, Rajkumar

Abstract Details

2015, Doctor of Philosophy, Ohio State University, Computer Science and Engineering.
Data generated by experimental, simulation, and observational science is growing exponentially. The resulting datasets are often transported over wide-area networks for storage, analysis, or visualization. Analysis of wide-area transfer logs reveals that traffic is typically nonuniform. While the average load does not saturate the resources involved in wide-area data movement, bursty periods often saturate the system. Thus, efficient scheduling of data transfers can optimize the resource utilization and improve the overall performance of data transfers. Such scheduling requires the ability to control the bandwidth allocated to an individual transfer. Achieving this is difficult, however. End-to-end wide-area data transfers involve many components affecting overall transfer performance. These components include storage systems, host resources including the network interface cards, the local campus network at both the source and destination, and the wide-area network (WAN). Moreover, all these components are shared across multiple applications and multiple users. Depending on system configuration and the load, different resources can be bottlenecks at different instances. Also, the current state of the art in file transfer scheduling is best effort for all transfers. But other types of transfers exist, including near-real-time and flexible transfers. A data transfer scheduler can use this knowledge to effectively schedule different types of transfers. We investigate the file transfer scheduling problem, where transfers of different types among different endpoints must be scheduled in order to maximize pertinent metrics. First, we characterize and model file transfers using a few key parameters such as concurrency used for a transfer, known load at the source and destination, and an estimate of the unknown external load. We show the effectiveness of the models by using them to control the bandwidth allocation for file transfers. Through extensive evaluation, we find that our models can be used to obtain desired bandwidth allocations with a mean (median) error rate of 19.8% (13.8%), with 38% of the errors in our benchmark tests being less than 10% and 54% of them being less than 15%. Second, we develop a load-aware transfer scheduling algorithm, SEAL, that exploits the fact that the aggregate bandwidth obtained over a network or at a storage system tends to increase with the number of concurrent transfers—but only up to a certain limit. It uses runtime information and data-driven models to approximate system load and adapt transfer schedules and concurrency so as to maximize performance while avoiding saturation. We implement this algorithm using GridFTP as the transfer protocol and evaluate it using real transfer logs in a production WAN environment. Results show that SEAL can reduce average slowdowns and improve turnaround times by up to 25% and worst-case slowdown and turnaround times by up to 50%, compared with the best-performing baseline scheme. Third, we extend SEAL to build a type-aware algorithm, STEAL, that further leverages user-supplied categorization of transfers as either best-effort (requiring immediate processing) or batch (less time-critical). The algorithm addresses a bi-objective problem of minimizing the average slowdown for best-effort transfers and maximizing the spare bandwidth utilized for batch transfers. Results show that STEAL reduces the average slowdown of interactive transfers by 63% compared with the best-performing baseline and by 21% compared with SEAL. For batch transfers, compared with the best-performing baseline, STEAL improves by 18% the utilization of the bandwidth unused by interactive transfers. We also build another algorithm, RESEAL, on top of SEAL to handle response-critical transfers. It uses a value-based scheduling approach because such transfers have time constraints and the value provided for those transfers starts decreasing beyond a certain completion time. This algorithm considers a mix of response-critical and best-effort transfers and addresses a bi-objective problem of maximizing the aggregate value for response-critical transfers and minimizing the average slowdown for best-effort transfers. We evaluate RESEAL in a production WAN environment using real-world transfer logs and show that response-critical transfers can achieve an aggregate value of 90% of their maximum aggregate value for transfer logs with load as high as 60% with only 9% increase in slowdown for best-effort tasks.
Gagan Agrawal (Advisor)
P. Sadayappan (Advisor)
Christopher Stewart (Committee Member)
Ian Foster (Committee Member)
167 p.

Recommended Citations

Citations

  • Kettimuthu, R. (2015). Type- and Workload-Aware Scheduling of Large-Scale Wide-Area Data Transfers [Doctoral dissertation, Ohio State University]. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view?acc_num=osu1437747493

    APA Style (7th edition)

  • Kettimuthu, Rajkumar. Type- and Workload-Aware Scheduling of Large-Scale Wide-Area Data Transfers. 2015. Ohio State University, Doctoral dissertation. OhioLINK Electronic Theses and Dissertations Center, http://rave.ohiolink.edu/etdc/view?acc_num=osu1437747493.

    MLA Style (8th edition)

  • Kettimuthu, Rajkumar. "Type- and Workload-Aware Scheduling of Large-Scale Wide-Area Data Transfers." Doctoral dissertation, Ohio State University, 2015. http://rave.ohiolink.edu/etdc/view?acc_num=osu1437747493

    Chicago Manual of Style (17th edition)