A performance prediction framework for grid-based data mining applications

Leonid Glimcher, Gagan Agrawal

Research output: Contribution to conferencePaperpeer-review

7 Scopus citations

Abstract

For a grid middleware to perform resource allocation, prediction models are needed, which can determine how long an application will take for completion on a particular platform or configuration. In this paper, we take the approach that by focusing on the characteristics of the class of applications a middleware is suited for, we can develop simple performance models that can be very accurate in practice. The particular middleware we consider is FREERIDE-G (FRamework for Rapid Implementation of Datamining Engines in Grid), which supports a high-level interface for developing data mining and scientific data processing applications that involve data stored in remote repositories. The FREERIDE-G system needs detailed performance models for performing resource selection, i.e., choosing computing nodes and replica of the dataset. This paper presents and evaluates such a performance model. By exploiting the fact that the processing structure of data mining and scientific data analysis applications developed on FREERIDE-G involves generalized reductions, we are able to develop an accurate performance prediction model. We have evaluated our model using implementations of three well-known data mining algorithms and two scientific data analysis applications developed using FREERIDE-G. Results from these five applications show that we are able to accurately predict execution times for applications as we vary the number of storage nodes, number of nodes available for computation, the dataset size, the network bandwidth, and the underlying hardware.

Original languageEnglish (US)
Pages1-10
DOIs
StatePublished - 2007
Externally publishedYes
Event21st International Parallel and Distributed Processing Symposium, IPDPS 2007 - Long Beach, CA, United States
Duration: Mar 26 2007Mar 30 2007

Conference

Conference21st International Parallel and Distributed Processing Symposium, IPDPS 2007
Country/TerritoryUnited States
CityLong Beach, CA
Period3/26/073/30/07

ASJC Scopus subject areas

  • Hardware and Architecture
  • Software
  • General Mathematics

Fingerprint

Dive into the research topics of 'A performance prediction framework for grid-based data mining applications'. Together they form a unique fingerprint.

Cite this