2021-03-09 - Shane Snyder - Darshan: Enabling IO Understanding in an Evolving HPC Landscape
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title: Darshan: Enabling Application IO Understanding in an Evolving HPC Landscape
Speaker:
Shane Snyder (Argonne National Laboratory)
Abstract:
Darshan is a lightweight, application I/O characterization tool that captures detailed statistics describing an application's I/O workload. Installed and enabled by default at many production HPC facilities (including at NERSC), Darshan has become an invaluable tool for users, system admins, and I/O researchers to investigate and tune the I/O behavior of applications. While the initial focus of Darshan was on instrumenting file-based APIs (e.g., POSIX, MPI-IO) for MPI applications, much recent work has focused on extending Darshan to new contexts that are increasingly relevant in the HPC community, including object-based storage APIs (e.g., DAOS) and non-MPI computational frameworks (e.g., Spark, TensorFlow). In this seminar, we describe how users can leverage Darshan to better understand the I/O behavior of their applications. We provide details on how users can produce Darshan instrumentation data for their applications and how to further analyze this data, focusing specifically on the Cori system at NERSC. New and upcoming features are covered that aim to extend Darshan to exciting I/O instrumentation contexts for HPC, including instrumentation modules for HDF5 and DAOS libraries, as well as support for instrumenting non-MPI applications and frameworks. We further walk through a couple of Darshan log analysis examples to help illustrate the types of I/O insights that can be attained using Darshan log data and analysis tools.
Bio:
Shane Snyder is a software engineer in the Mathematics and Computer Science Division of Argonne National Laboratory. He received his master's degree in computer engineering from Clemson University in 2013. His research interests primarily include the design of high-performance distributed storage systems and the characterization and analysis of I/O workloads on production HPC systems.
Видео 2021-03-09 - Shane Snyder - Darshan: Enabling IO Understanding in an Evolving HPC Landscape канала NERSC
Title: Darshan: Enabling Application IO Understanding in an Evolving HPC Landscape
Speaker:
Shane Snyder (Argonne National Laboratory)
Abstract:
Darshan is a lightweight, application I/O characterization tool that captures detailed statistics describing an application's I/O workload. Installed and enabled by default at many production HPC facilities (including at NERSC), Darshan has become an invaluable tool for users, system admins, and I/O researchers to investigate and tune the I/O behavior of applications. While the initial focus of Darshan was on instrumenting file-based APIs (e.g., POSIX, MPI-IO) for MPI applications, much recent work has focused on extending Darshan to new contexts that are increasingly relevant in the HPC community, including object-based storage APIs (e.g., DAOS) and non-MPI computational frameworks (e.g., Spark, TensorFlow). In this seminar, we describe how users can leverage Darshan to better understand the I/O behavior of their applications. We provide details on how users can produce Darshan instrumentation data for their applications and how to further analyze this data, focusing specifically on the Cori system at NERSC. New and upcoming features are covered that aim to extend Darshan to exciting I/O instrumentation contexts for HPC, including instrumentation modules for HDF5 and DAOS libraries, as well as support for instrumenting non-MPI applications and frameworks. We further walk through a couple of Darshan log analysis examples to help illustrate the types of I/O insights that can be attained using Darshan log data and analysis tools.
Bio:
Shane Snyder is a software engineer in the Mathematics and Computer Science Division of Argonne National Laboratory. He received his master's degree in computer engineering from Clemson University in 2013. His research interests primarily include the design of high-performance distributed storage systems and the characterization and analysis of I/O workloads on production HPC systems.
Видео 2021-03-09 - Shane Snyder - Darshan: Enabling IO Understanding in an Evolving HPC Landscape канала NERSC
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![6 Case Study 1: XGC1 (NERSC Cori KNL Training 6/2017)](https://i.ytimg.com/vi/3C7LGn6RoZA/default.jpg)
![21 - Scaling Neural Networks Training - Thorsten Kurth](https://i.ytimg.com/vi/cRjiwIi_kuc/default.jpg)
![2021-12-06 - Ryan Chard - funcX: Federated FaaS for Scientific Computing](https://i.ytimg.com/vi/vMSgcjET9Uk/default.jpg)
![2022-06-14 - Jihan Kim - Artificial Design of Porous Materials](https://i.ytimg.com/vi/X7cW-HGTdQw/default.jpg)
![2021-08-24 - Kjiersten Fagnan - JGI Computing - the future is looking cloudy](https://i.ytimg.com/vi/WRb5QDPNCrA/default.jpg)
![Jupyter demo video](https://i.ytimg.com/vi/7WAC_-jq0v0/default.jpg)
![Checkpoint/Restart MPI Applications with MANA on Cori](https://i.ytimg.com/vi/VhndHDsg_sQ/default.jpg)
![Current State of LLVM Compiler](https://i.ytimg.com/vi/Zp2cs89fzXk/default.jpg)
![Julia](https://i.ytimg.com/vi/tSV-MNTmxBE/default.jpg)
![Exploring the Dark Universe with Perlmutter](https://i.ytimg.com/vi/A-JVZRyNUZo/default.jpg)
![Application of GPUs in Proteomics and Connectomics](https://i.ytimg.com/vi/N3mMrlQsEE0/default.jpg)
![March 2022 NUG Meeting: Remote development in NERSC Shifter containers with Visual Studio Code](https://i.ytimg.com/vi/jjlTbOS9KDg/default.jpg)
![03 Codee parallelization capabilities](https://i.ytimg.com/vi/HuESp2_hecE/default.jpg)
![Current State of CUDA Compilers and SDK](https://i.ytimg.com/vi/Ff75TI7-nZY/default.jpg)
![Intro to GPU: 04 NVidia Software Stack, Part 2](https://i.ytimg.com/vi/NxB-CSWs_oU/default.jpg)
![Day 1 Intro to AI Lecture & CNN Primer & Keras 101](https://i.ytimg.com/vi/qSxvc3nKCeQ/default.jpg)
![2022-06-28 - S. Subramanian - FourCastNet: Data driven high resolution atmosphere modeling at scale](https://i.ytimg.com/vi/RIUV1vpNalI/default.jpg)
![NERSC Totalview Tutorial - Part2](https://i.ytimg.com/vi/Q6qzVOyWo4M/default.jpg)
![NERSC Users Group 2020: Breakout 3 (ERCAP and Iris Tips and Tricks)](https://i.ytimg.com/vi/T5DVx01edPc/default.jpg)
![GPU-Accelerated Quantum Circuit Simulations with QCLAB++](https://i.ytimg.com/vi/5m2MR8GpFSc/default.jpg)