High Performance Data Analytics in Python

Scientists, engineers and professionals from many sectors are seeing an enormous growth in the size and number of datasets relevant to their domains. Professional titles have emerged to describe specialists working with data, such as data scientists and data engineers, but also other experts are finding it necessary to learn tools and techniques to work with big data. Typical tasks include preprocessing, analysing, modeling and visualising data.

Python is an industry-standard programming language for working with data on all levels of the data analytics pipeline. This is in large part because of the rich ecosystem of libraries ranging from generic numerical libraries to special-purpose and/or domain-specific packages, often supported by large developer communities and stable funding sources.

This lesson is meant to give an overview of working with research data in Python using general libraries for storing, processing, analysing and sharing data. The focus is on high performance. After covering tools for performant processing on single workstations the focus shifts to profiling and optimising, parallel and distributed computing.

Prerequisites

• Basic experience with Python

• Basic experience in working in a Linux-like terminal

• Some prior experience in working with large or small datasets

15 min	Motivation
60 min	Scientific data
90 min	Efficient array computing
90 min	Parallel computing
90 min	Profiling and optimizing
90 min	Performance boosting
90 min	Dask for scalable analytics

Preparation

• Installation and HPC access

The lesson

• Motivation
• Scientific data
• Efficient array computing
• Parallel computing
• Profiling and optimizing
• Performance boosting
• Dask for scalable analytics

Optional material

• Optional: more on Pandas
• GPU computing

Reference

• Instructor’s guide

Who is the course for?

This material is for all researchers and engineers who work with large or small datasets and who want to learn powerful tools and best practices for writing more performant, parallelised, robust and reproducible data analysis pipelines.

About the course

This lesson material is developed by the EuroCC National Competence Center Sweden (ENCCS) and taught in ENCCS workshops. Each lesson episode has clearly defined objectives that will be addressed and includes multiple exercises along with solutions, and is therefore also useful for self-learning. The lesson material is licensed under CC-BY-4.0 and can be reused in any form (with appropriate credit) in other courses and workshops. Instructors who wish to teach this lesson can refer to the Instructor’s guide for practical advice.

See also

Each lesson episode has a “See also” section at the end which lists recommended further learning material.

Credits

The lesson file structure and browsing layout is inspired by and derived from work by CodeRefinery licensed under the MIT license. We have copied and adapted most of their license text.

Several examples and formulations are inspired by other open source educational material, in particular:

• Python for Scientific Computing

• Data analysis workflows with R and Python

• Python in High Performance Computing

• Code examples from High Performance Python

• HPC Carpentry’s Introduction to High-Performance Computing in Python

• Python Data Science Handbook

• An Introduction to Earth and Environmental Data Science

• Parallel and GPU Programming in Python

• Python in HPC

• High-performance computing with Python

• Advanced Python for data science in biology

• Introduction to Numba

• Python for Data Analysis

• GTC2017-numba

• IPython Cookbook

• Scipy Lecture Notes

• Machine Learning and Data Science Notebooks

• Elegant SciPy

• A Comprehensive Guide to NumPy Data Types

Instructional Material

This instructional material is made available under the Creative Commons Attribution license (CC-BY-4.0). The following is a human-readable summary of (and not a substitute for) the full legal text of the CC-BY-4.0 license. You are free to:

• share - copy and redistribute the material in any medium or format

• adapt - remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow these license terms:

• Attribution - You must give appropriate credit (mentioning that your work is derived from work that is Copyright (c) HPDA-Python and individual contributors and, where practical, linking to https://enccs.se), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

• No additional restrictions - You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

With the understanding that:

• You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.

• No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

Software

Except where otherwise noted, the example programs and other software provided with this repository are made available under the OSI-approved MIT license.