Gießen 2024 – scientific programme

Parts | Days | Selection | Search | Updates | Downloads | Help

HK: Fachverband Physik der Hadronen und Kerne

HK 54: Computing II

HK 54.2: Talk

Wednesday, March 13, 2024, 17:45–18:00, HBR 19: C 103

ASAPO: A high-speed streaming framework to support an automated data-processing pipeline. — •Mikhail Karnevskiy — DESY, Hamburg, Germany

Modern high-speed and high-resolution detectors produce a significant data rate, thereby increasing the need for online data processing and data reduction. A well-developed common data processing framework, provided as a service, minimizes the cost of implementing different use cases, enables more efficient scientific work due to a higher degree of automation, and enhances the reliability of the overall system.

ASAPO is a high-performance streaming framework actively developed at DESY to enable online and offline data processing using TCP/IP and RDMA over Ethernet and Infiniband. It efficiently facilitates high-bandwidth communication between detectors, the storage system, and independent analysis processes. User-friendly interfaces are available for C/C++, Python on all major platforms. A high-level Python library reduces boilerplate code when writing independent analysis workers, which can be combined into complex pipelines. ASAPO supports users with automatic retransfer, trivial parallelization on a per-image basis, multi-module detectors, and web-based monitoring.

Several experimental facilities at Petra III already use ASAPO for various data-processing pipelines. Involved algorithms include azimuthal integration of X-ray scattering data, peak finding, and indexing of diffraction patterns.

Keywords: data processing; streaming framework; online analysis; automated analysis; data reduction