A compression framework for multidimensional scientific datasets

Tekin Bicer, Gagan Agrawal

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations

Abstract

Scientific simulations and instruments can generate tremendous amount of data in short time periods. Since the generated data is used for inferring new knowledge, it is important to efficiently store and provide it to the scientific endeavors. Although parallel and distributed systems can help to ease the management of such data, the transmission and storage are still challenging problems. Compression is a popular approach for reducing data transfer overheads and storage requirements. However, effectively supporting compression for scientific simulation data and integrating compression algorithms with simulation applications remain a challenge. In this work, we focus on management of multidimensional scientific datasets using domain specific compression algorithms. We propose a compression framework and methodology in order to maximize the bandwidth and storage utilization. We port our framework into PnetCDF and present our preliminary experimental results.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE 27th International Parallel and Distributed Processing Symposium Workshops and PhD Forum, IPDPSW 2013
PublisherIEEE Computer Society
Pages2250-2253
Number of pages4
ISBN (Print)9780769549798
DOIs
StatePublished - 2013
Externally publishedYes
Event2013 IEEE 37th Annual Computer Software and Applications Conference, COMPSAC 2013 - Boston, MA, Japan
Duration: Jul 22 2013Jul 26 2013

Publication series

NameProceedings - IEEE 27th International Parallel and Distributed Processing Symposium Workshops and PhD Forum, IPDPSW 2013

Conference

Conference2013 IEEE 37th Annual Computer Software and Applications Conference, COMPSAC 2013
CountryJapan
CityBoston, MA
Period7/22/137/26/13

Keywords

  • Compression
  • Data management
  • Distributed data processing
  • PNetCDF

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Software
  • Theoretical Computer Science

Fingerprint Dive into the research topics of 'A compression framework for multidimensional scientific datasets'. Together they form a unique fingerprint.

Cite this