Compiling dynamic data structures in python to enable the use of multi-core and many-core libraries

Bin Ren, Gagan Agrawal

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Programmer productivity considerations are increasing the popularity of interpreted languages like Python. At the same time, for applications where performance is important, these languages clearly lack even on uniprocessors. In addition, the use of dynamic data structures in a language like Python makes it very hard to use emerging libraries for enabling the execution on multi-core and many-core architectures. This paper presents a framework for compiling Python to use multi-core and many-core libraries. The key component of our framework involves a suite of algorithms for replacing dynamic and/or nested data structures by arrays, while minimizing unnecessary data copying costs. This involves a novel use of an existing partial redundancy elimination algorithm, development of a new demand-driven interprocedural partial redundancy algorithm, a data flow formulation for determining that the contents of the data structure are of the same type, and a linearization algorithm. We have evaluated our framework using data mining and two linear algebra applications written in pure Python. The key observations were: 1) the code generated by our framework is only 10% to 20% slower compared to the hand-written C code that invokes the same libraries, 2) our optimizations turn out to be significant for improving the performance in most cases, and 3) we outperform interpreted Python and the C++ code generated by an existing tool by one to two orders of magnitude.

Original languageEnglish (US)
Title of host publicationProceedings - 2011 International Conference on Parallel Architectures and Compilation Techniques, PACT 2011
Pages68-77
Number of pages10
DOIs
StatePublished - 2011
Externally publishedYes
Event20th International Conference on Parallel Architectures and Compilation Techniques, PACT 2011 - Galveston, TX, United States
Duration: Oct 10 2011Oct 14 2011

Publication series

NameParallel Architectures and Compilation Techniques - Conference Proceedings, PACT
ISSN (Print)1089-795X

Conference

Conference20th International Conference on Parallel Architectures and Compilation Techniques, PACT 2011
CountryUnited States
CityGalveston, TX
Period10/10/1110/14/11

Keywords

  • Compilation for multi-core and many-core
  • Python
  • Redundancy elimination

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Compiling dynamic data structures in python to enable the use of multi-core and many-core libraries'. Together they form a unique fingerprint.

Cite this