Dask: Glossary

Key Points

Introduction

  • Dask has a familiar interface, similar to NumPy, Pandas, or PySpark.

  • Dasks creates task graphs which allow it to schedule independent tasks in parallel.
Python

  • A virtual environment isolates python packages allowing easy switching between different packages and versions avoiding conflicts.

  • Using a main function allows python scripts to be imported later as modules if desired and reuse the functions in other scripts.

  • srun can be used to run jobs on compute nodes.

  • The Python time module can be used to time sections of code.
Dask Delayed

  • Used the delayed function to wrap function calls and to allow dask to build the task graph.

  • Visualize the task graph by calling the visualize member function of the object returned from the last delayed function. The graph will be created in the mydask.png file.

  • To run your dask task graph call the compute function on the object returned from the last delayed function.
Real Computations
Distributed Computations
Cython
NumPy
Dask Array
Wrapping Up

Glossary

The glossary would go here, formatted as:

{:auto_ids}
key word 1
:   explanation 1

key word 2
:   explanation 2

({:auto_ids} is needed at the start so that Jekyll will automatically generate a unique ID for each item to allow other pages to hyperlink to specific glossary entries.) This renders as:

key word 1
explanation 1
key word 2
explanation 2