Applied Parallel Programming

CSPP51085

Winter 2011

Course Description: Parallel computing allows multiple processing units to work together simultaneously on a common task. For certain types of applications, parallelization can increase execution time in proportion to the number of computers or processors used. This is a huge advantage for applications which have performance and/or memory bottlenecks, such as one typically encounters in financial modelling, physics, engineering, or other applied science domains. This is a fast-paced applied programming course aimed at students with significant development experience in either C, C++, or FORTRAN (Java, Matlab, or Python are also possible, but not ideal). No prior knowledge of parallel computing is assumed. Students should, however, have both an interest and some previous experience in either algorithmic development, numerical methods, applied mathematics, or perhaps any physics or engineering-type discipline. A brief overview of parallel computing will be presented at the outset, but the course will be less on overview of HPC architectures and much more a focus on algorithmic implementation and performance tuning. The goal of the course it to give students experience in developing efficient, scalable (distributed memory) parallel algorithms appropriate for any system running an implementation of the Message Passing Interface (MPI). Assignments will be designed with some flexibility to allow students to explore applying parallel techniques to applications in their own field of interest.

Course Format: Combination of instructor lecture, student presentations, and group discussion.

Grading: The course grade will be based entirely on the four homework assignments.

Getting Help: When possible, all technical questions should be discussed over the course listhost (see link above). Otherwise, please consult TA first and then professor if further assistance is required.

Suggested texts

• Using MPI
  by Gropp, Lusk, and Skjellum
  MIT Press
  2nd edition
• Numerical Recipes in C (or F77 or F90)
  online version here
• MPI: The Complete Reference
  Marc Snir, Steve W. Otto, Steven Huss-Lederman, David W. Walker, and Jack Dongarra.
  The MIT Press
  online version

Required Computing Resources

• An MPI implementation, such as MPICH2, available free here
• A compiler in the language of your choice (everything is available on cs cluster)
• We will get accounts on the ANL BG/P and Full Disclosure parallel machine

Useful web resources

• Argonne MPI page
• Top 500
• Supercomputing online

Cheat sheets for class

Lectures

Week 1 Jan 4

• Brief overview of supercomputing: history, vector machines, SMPs, GPGPUs, Top500, etc.
• Installing MPICH
• Running parallel jobs locally on CS cluster
• Overview of initial MPI functions: MPI_init, MPI_Comm_rank, MPI_Comm_size, MPI_Send, MPI_Recv, MPI_Reduce, MPI_Bcast
• Examples: Some basic MPI programs
• Notes
• Assignment 1

Week 2 Jan 11

• Point-to-point communication semantics in depth, blocking vs. non-blocking, send modes, etc.
• Examples of embarrasingly parallel applications: fractals, monte carlo, image processing
• Account setup on ANL cluster
• Examples: Example course programs
• Notes: Summary of lecture
• Assignment: Assignment 2

Week 3 Jan 18

• Non-blocking communication, cont.
• Consideration of solutions to simple dynamic load balancing for Mandelbrot set
• Introduction to heat equation
• Discussion of bc's, ic's
  
• Examples: Example programs
• Notes: Summary of lecture

Week 4 Jan 25

• continue discussion: solution of 2d heat equation
• MPI Collective functions
• Notes: Summary of lecture
• Examples: IDL prototype
• Homework: Assignment 3

Week 5 Feb. 1

• Prototype distributed neutral particle simulator
• MADRE Library

Week 6 Feb. 8 NO CLASS, Fusion Simulation Program workshop in Sand Diego -- class Rescheduled
Homework: Assignment 4

Week 7 Feb. 15

• MPI Collective functions, cont. (notes from Week 4)
• New application: N-body problem
• Useful techniques for hw3 -Cartesian Communicators, MPI_Send_recv_replace, Parallel grid abstraction
• Examples: Simple in-class programs
• Notes: Summary of lecture

Week 8 Feb. 22

• Tutorial on MPI User Defined Types
• Notes: Summary of lecture

Week 9 March 1

• Parallel linear systems
  • Gauss Jordan elimination in serial and parallel
  • Tri-diagonal systems in serial and parallel (cyclic reduction)
• Implicit heat equation
• Introduction to PETSc

Week 10 March 8

• GPGPU programming
• Final projects