GPU Programming: When, Why and How?

Graphical processing units (GPUs) are the workhorse of many high performance computing (HPC) systems around the world. The number of GPU-enabled supercomputers on the Top500 has been steadily increasing in recent years and this development is expected to continue. In the near future, the majority of HPC computing power available to researchers and engineers is likely to be provided by GPUs or other types of accelerators. Programming GPUs and other accelerators is thus crucial to developers of software run on HPC systems.

However, the landscape of GPU hardware, software and programming environments is complicated. Multiple vendors compete in the high-end GPU market, with each vendor providing its own software stack and development toolkits, and even beyond that, there is a proliferation of tools, languages and frameworks that can be used to write code for GPUs. It can thus be difficult for individual developers and project owners to know how to navigate across this landscape and select the most appropriate GPU programming framework for their projects based on the requirements of a given project and technical requirements of any existing code.

This material is meant to help both software developers and decision makers navigate the GPU programming landscape and make more informed decisions on which languages or frameworks to learn and use for their projects. Specifically, you will:

• Understand why and when to use GPUs.

• Become comfortable with key concepts in GPU programming.

• Acquire a comprehensive overview of different software frameworks, what levels they operate at, and which to use when.

• Learn the fundamentals in at least one framework to a level which will enable you to quickly become a productive GPU programmer.

Prerequisites

Familiarity with one or more programming languages like C/C++, Fortran, Python or Julia is recommended.

Prerequisites

• Setup

The lesson

• Why GPUs?
• The GPU hardware and software ecosystem
• What problems fit to GPU?
• GPU programming concepts
• Introduction to GPU programming models
• High-level language support
• Directive-based models
• Multiple GPU programming with MPI
• Non-portable kernel-based models
• Portable kernel-based models
• Preparing code for GPU porting
• Recommendations
• GPU programming example: stencil computation

Reference

• Quick Reference
• Glossary
• Instructor’s guide

About

• All lessons
• ENCCS

Who is the course for?

This material is most relevant to researchers and engineers who already develop software which runs on CPUs in workstations or supercomputers, but also to decision makers or project managers who don’t write code but make strategic decisions in software projects, whether it’s in academia, industry or the public sector.

About the course

This training material is the result of a multilateral effort by GPU programming experts from:

• Aalto University in Finland

• Aarhus University in Denmark

• CSC in Finland

• ENCCS in Sweden

• HPC2N centre in Umeå, Sweden

• KTH Royal Institute for Technology in Sweden

• NRIS in Norway

• Vilnius University in Lithuania and NCC Lithuania

See also

Links to additional resources and tutorials can be found in the lesson episodes.

Credits

Several sections in this lesson have been adapted from the following sources created by ENCCS and CSC, which are all distributed under a Creative Commons Attribution license (CC-BY-4.0):

• OpenMP for GPU offloading

• High Performance Data Analytics in Python

• Julia for HPC

The lesson file structure and browsing layout is inspired by and derived from work by CodeRefinery licensed under the MIT license. We have copied and adapted most of their license text.

Instructional Material

This instructional material is made available under the Creative Commons Attribution license (CC-BY-4.0). The following is a human-readable summary of (and not a substitute for) the full legal text of the CC-BY-4.0 license. You are free to:

• share - copy and redistribute the material in any medium or format

• adapt - remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow these license terms:

• Attribution - You must give appropriate credit (mentioning that your work is derived from work that is Copyright (c) ENCCS and individual contributors and, where practical, linking to https://enccs.github.io/sphinx-lesson-template), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

• No additional restrictions - You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

With the understanding that:

• You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.

• No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

Software

Except where otherwise noted, the example programs and other software provided with this repository are made available under the OSI-approved MIT license.