Würzburg 2018 – scientific programme

Parts | Days | Selection | Search | Updates | Downloads | Help

T: Fachverband Teilchenphysik

T 87: Datenanalyse

T 87.4: Talk

Thursday, March 22, 2018, 17:15–17:30, Z6 - SR 2.005

Distributed make-like Analyses on the Grid based on Spotify's Pipelining Package luigi — •Marcel Rieger, Martin Erdmann, Benjamin Fischer, and Ralf Florian von Cube — III. Physikalisches Institut A, RWTH Aachen University

In particle physics, workflow management systems are primarily used as tailored solutions in dedicated areas such as Monte Carlo production. However, physicists performing data analyses are usually required to steer their individual workflows manually which is time- consuming and often leads to undocumented relations between particular workloads. We present the luigi analysis workflow (law) Python package which is based on the open- source pipelining package luigi, originally developed by Spotify. It entails a generic analysis design pattern with make-like execution allowing for the definition of arbitrary workloads and all dependencies between them in a scalable structure which shifts the focus from executing to defining an analysis. To cope with the sophisticated demands of end-to-end HEP analyses, it provides remote execution on WLCG infrastructure, remote file access through Grid File Access Library (GFAL2), and a software sandboxing mechanism with support for Docker and Singularity containers. The novel approach was successfully applied in a ttH cross section measurement with CMS.