G. A. Stewart

Machine learning is an important applied research area in particle physics, beginning with applications to high-level physics analysis in the 1990s and 2000s, followed by an explosion of applications in particle and event identification and reconstruction in the 2010s. In this document we discuss promising future research and development areas in machine learning in particle physics with a roadmap for their implementation, software and hardware resource requirements, collaborative initiatives with the data science community, academia and industry, and training the particle physics community in data science. The main objective of the document is to connect and motivate these areas of research and development with the physics drivers of the High-Luminosity Large Hadron Collider and future neutrino experiments and identify the resource needs for their implementation. Additionally we identify areas where collaboration with external communities will be of great benefit.

Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. The data can be distributed across heterogeneous data centers at widely distributed locations. Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities. In this article, we detail the fundamental concepts of Rucio, describe the architecture along with implementation details, and give operational experience from production usage.

Abstract We review the main software and computing challenges for the Monte Carlo physics event generators used by the LHC experiments, in view of the High-Luminosity LHC (HL-LHC) physics programme. This paper has been prepared by the HEP Software Foundation (HSF) Physics Event Generator Working Group as an input to the LHCC review of HL-LHC computing, which has started in May 2020.

The European Particle Physics Strategy Update (EPPSU) process takes a bottom-up approach, whereby the community is first invited to submit proposals (also called inputs) for projects that it would like to see realised in the near-term, mid-term and longer-term future. National inputs as well as inputs from National Laboratories are also an important element of the process. All these inputs are then reviewed by the Physics Preparatory Group (PPG), whose role is to organize a Symposium around the submitted ideas and to prepare a community discussion on the importance and merits of the various proposals. The results of these discussions are then concisely summarised in this Briefing Book, prepared by the Conveners, assisted by Scientific Secretaries, and with further contributions provided by the Contributors listed on the title page. This constitutes the basis for the considerations of the European Strategy Group (ESG), consisting of scientific delegates from CERN Member States, Associate Member States, directors of major European laboratories, representatives of various European organizations as well as invitees from outside the European Community. The ESG has the mission to formulate the European Strategy Update for the consideration and approval of the CERN Council.

A prototype particle tracking telescope was constructed using Timepix and Medipix ASIC hybrid pixel assemblies as the six sensing planes. Each telescope plane consisted of one 1.4 cm2 assembly, providing a 256 ×256 array of 55μm square pixels. The telescope achieved a pointing resolution of 2.4μm at the position of the device under test. During a beam test in 2009 the telescope was used to evaluate in detail the performance of two Timepix hybrid pixel assemblies; a standard planar 300μm thick sensor, and 285μm thick double sided 3D sensor. This paper describes a charge calibration study of the pixel devices, which allows the true charge to be extracted, and reports on measurements of the charge collection characteristics and Landau distributions. The planar sensor achieved a best resolution of 4.0±0.1μm for angled tracks, and resolutions of between 4.4 and 11μm for perpendicular tracks, depending on the applied bias voltage. The double sided 3D sensor, which has significantly less charge sharing, was found to have an optimal resolution of 9.0±0.1μm for angled tracks, and a resolution of 16.0±0.2μm for perpendicular tracks. Based on these studies it is concluded that the Timepix ASIC shows an excellent performance when used as a device for charged particle tracking.

Rucio is the next-generation Distributed Data Management (DDM) system benefiting from recent advances in cloud and "Big Data" computing to address HEP experiments scaling requirements. Rucio is an evolution of the ATLAS DDM system Don Quijote 2 (DQ2), which has demonstrated very large scale data management capabilities with more than 140 petabytes spread worldwide across 130 sites, and accesses from 1,000 active users. However, DQ2 is reaching its limits in terms of scalability, requiring a large number of support staff to operate and being hard to extend with new technologies. Rucio will deal with these issues by relying on a conceptual data model and new technology to ensure system scalability, address new user requirements and employ new automation framework to reduce operational overheads. We present the key concepts of Rucio, including its data organization/representation and a model of how to manage central group and user activities. The Rucio design, and the technology it employs, is described, specifically looking at its RESTful architecture and the various software components it uses. We show also the performance of the system.

Research in high energy physics (HEP) requires huge amounts of computing and storage, putting strong constraints on the code speed and resource usage. To meet these requirements, a compiled high-performance language is typically used; while for physicists, who focus on the application when developing the code, better research productivity pleads for a high-level programming language. A popular approach consists of combining Python, used for the high-level interface, and C++, used for the computing intensive part of the code. A more convenient and efficient approach would be to use a language that provides both high-level programming and high-performance. The Julia programming language, developed at MIT especially to allow the use of a single language in research activities, has followed this path. In this paper the applicability of using the Julia language for HEP research is explored, covering the different aspects that are important for HEP code development: runtime performance, handling of large projects, interface with legacy code, distributed computing, training, and ease of programming. The study shows that the HEP community would benefit from a large scale adoption of this programming language. The HEP-specific foundation libraries that would need to be consolidated are identified

The Production and Distributed Analysis System (PanDA) plays a key role in the ATLAS distributed computing infrastructure. All ATLAS Monte-Carlo simulation and data reprocessing jobs pass through the PanDA system. We will describe how PanDA manages job execution on the grid using dynamic resource estimation and data replication together with intelligent brokerage in order to meet the scaling and automation requirements of ATLAS distributed computing. PanDA is also the primary ATLAS system for processing user and group analysis jobs, bringing further requirements for quick, flexible adaptation to the rapidly evolving analysis use cases of the early datataking phase, in addition to the high reliability, robustness and usability needed to provide efficient and transparent utilization of the grid for analysis users. We will describe how PanDA meets ATLAS requirements, the evolution of the system in light of operational experience, how the system has performed during the first LHC data-taking phase and plans for the future.

During the Long Shutdown of the LHC, the ATLAS collaboration overhauled its analysis model based on experience gained during Run 1. A significant component of the model is a "Derivation Framework" that takes the petabyte-scale AOD output from ATLAS reconstruction and produces samples, typically terabytes in size, targeted at specific analyses. The framework incorporates all of the functionality of the core reconstruction software, while producing outputs that are simply configured. Event selections are specified via a concise domain-specific language, including support for logical operations. The output content can be highly optimised to minimise disk requirements, while maintaining the same C++ interface. The framework includes an interface to the late-stage physics analysis tools, ensuring that the final outputs are consistent with tool requirements. Finally, the framework allows several outputs to be produced for the same input, providing the possibility to optimise configurations to computing resources.

This study uses a one-dimensional equation system and experimental techniques to provide a comprehensive description of vertical gas–solid two-phase flow. The results from non-accelerating flow experiments conducted with a riser tube of bore 192 mm and height 16.2 m using spherical glass beads of average diameter 64 μm are presented. The solids volume fraction, which was measured directly using quick-closing valves, was less than 0.01 in all cases. The frictional pressure drop was recognised to be an important component of the total pressure gradient in the riser. At low gas velocities, negative frictional pressure gradients occurred. The solids friction factor was found to be constant at high solids velocities and decrease to negative values as the solids velocity was reduced. The slip velocity was found to be always greater than the single-particle terminal velocity and to increase with decreasing gas velocity or increasing solids mass flux. This is different to that which has usually been reported in literature, and is thought to be due to the large diameter of riser used in this study. In addition, the slip velocity increased (independently of solids mass flux) with increasing solids concentration.

Detectors of High Energy Physics experiments, such as the ATLAS dectector [1] at the Large Hadron Collider [2], serve as cameras that take pictures of the particles produced in the collision events. One of the key detector technologies used for measuring the energy of particles are calorimeters. Particles will lose their energy in a cascade (called a shower) of electromagnetic and hadronic interactions with a dense absorbing material. The number of the particles produced in this showering process is subsequently measured across the sampling layers of the calorimeter. The deposition of energy in the calorimeter due to a developing shower is a stochastic process that can not be described from first principles and rather relies on a precise simulation of the detector response. It requires the modeling of particles interactions with matter at the microscopic level as implemented using the Geant4 toolkit [3]. This simulation process is inherently slow and thus presents a bottleneck in the ATLAS simulation pipeline [4]. The current work addresses this limitation. To meet the growing analysis demands, ATLAS already relies strongly on fast calorimeter simulation techniques based on thousands of individual parametrizations of the calorimeter response [5]. The algorithms currently employed for physics analyses by the ATLAS collaboration achieve a significant speedup over the full simulation of the detector response at the cost of accuracy. Current developments [6] [7] aim at improving the modeling of taus, jet-substructure-based boosted objects or wrongly identified objects in the calorimeter and will benefit from an improved detector description following data taking and a more detailed forward calorimeter geometry. Deep Learning techniques have been improving state of the art results in various science areas such as: astrophysics [8], cosmology [9] and medical imaging [10]. These techniques are able to describe complex data structures and scale well with highdimensionality problems. Generative models are powerful deep learning algorithms to map complex distributions into a lower dimensional space, to generate samples of higher dimensionality and to approximate the underlying probability densities. Among the most promising approaches are Variational Auto-Encoders [11] [12] and Generative Adversarial Networks [13]. In this context, the talk presents the first application of such models to the fast simulation of the calorimeter response in the ATLAS detector. This work [14] demonstrates the feasibility of using such algorithms for large scale high energy physics experiments in the future, and opens the possibility to complement current techniques.

By combining all the data available from the Genetic Analysis of Multiple sclerosis in EuropeanS (GAMES) project, we have been able to identify 17 microsatellite markers showing consistent evidence for apparent association. As might be expected five of these markers map within the Major Histocompatibility Complex (MHC) and are in LD with HLA-DRB1. Individual genotyping of the 12 non-MHC markers confirmed association for three of them--D11S1986, D19S552 and D20S894. Association mapping across the candidate genes implicated by these markers in 937 UK trio families revealed modestly associated haplotypes in JAG1 (p=0.019) on chromosome 20p12.2 and POU2AF1 (p=0.003) on chromosome 11q23.1.

Grids allow for collaborative e-Research to be undertaken, often across institutional and national boundaries. Typically this is through the establishment of virtual organizations (VOs) where policies on access and usage of resources across partner sites are defined and subsequently enforced. For many VOs, these agreements have been lightweight and erred on the side of flexibility with minimal constraints on the kinds of jobs a user is allowed to run or the amount of resources that can be consumed. For many new domains such as e-Health, such flexibility is simply not tenable. Instead, precise definitions of what jobs can be run, and what data can be accessed by who need to be defined and enforced by sites. The role based access control model (KBAC) provides a well researched paradigm for controlling access to large scale dynamic VOs. However, the standard RBAC model assumes a single domain with centralised role management. When RBAC is applied to VOs, it does not specify how or where roles should be defined or made known to the distributed resource sites (who are always deemed to be autonomous to make access control decisions). Two main possibilities exist based on either a centralized or decentralized approach to VO role management. We present the advantages and disadvantages of the centralized and decentralized role models and describe how we have implemented them in a range of security focused e-Research domains at the National e-Science Centre (NeSC) at the University of Glasgow.

The EDM4hep project aims to design the common event data model for the Key4hep project and is generated via the podio toolkit. We present the first version of EDM4hep and discuss some of its use cases in the Key4hep project. Additionally, we discuss recent developments in podio, like the updates of the automatic code generation and also the addition of a second I/O backend based on SIO. We compare the available backends using benchmarks based on physics use cases, before we conclude with a discussion of currently ongoing work and future developments.

This white paper presents the current status of the R&D for Analysis Facilities (AFs) and attempts to summarize the views on the future direction of these facilities. These views have been collected through the High Energy Physics (HEP) Software Foundation's (HSF) Analysis Facilities forum, established in March 2022, the Analysis Ecosystems II workshop, that took place in May 2022, and the WLCG/HSF pre-CHEP workshop, that took place in May 2023. The paper attempts to cover all the aspects of an analysis facility.

Detector studies for future experiments rely on advanced software tools to estimate performance and optimize their design and technology choices. The Key4hep project provides a flexible turnkey solution for the full experiment life-cycle based on established community tools such as ROOT, Geant4, DD4hep, Gaudi, podio and spack. Members of the CEPC, CLIC, EIC, FCC, and ILC communities have joined to develop this framework and have merged, or are in the progress of merging, their respective software environments into the Key4hep stack. These proceedings will give an overview over the recent progress in the Key4hep project: covering the developments towards adaptation of state-of-theart tools for simulation (DD4hep, Gaussino), track and calorimeter reconstruction (ACTS, CLUE), particle flow (PandoraPFA), analysis via RDataFrame, and visualization with Phoenix, as well as tools for testing and validation.

A performant and easy-to-use event data model (EDM) is a key component of any HEP software stack. The podio EDM toolkit provides a user friendly way of generating such a performant implementation in C++ from a high level description in yaml format. Finalizing a few important developments, we are in the final stretches for release v1.0 of podio, a stable release with backward compatibility for datafiles written with podio from then on. We present an overview of the podio basics, and go into slighty more technical detail on the most important topics and developments. These include: schema evolution for generated EDMs, multithreading with podio generated EDMs, the implementation of them as well as the basics of I/O. Using EDM4hep, the common and shared EDM of the Key4hep project, we highlight a few of the smaller features in action as well as some lessons learned during the development of EDM4hep and podio. Finally, we show how podio has been integrated into the Gaudi based event processing framework that is used by Key4hep, before we conclude with a brief outlook on potential developments after v1.0.

In November 2022, the HEP Software Foundation and the Institute for Research and Innovation for Software in High-Energy Physics organized a workshop on the topic of Software Citation and Recognition in HEP. The goal of the workshop was to bring together different types of stakeholders whose roles relate to software citation, and the associated credit it provides, in order to engage the community in a discussion on: the ways HEP experiments handle citation of software, recognition for software efforts that enable physics results disseminated to the public, and how the scholarly publishing ecosystem supports these activities. Reports were given from the publication board leadership of the ATLAS, CMS, and LHCb experiments and HEP open source software community organizations (ROOT, Scikit-HEP, MCnet), and perspectives were given from publishers (Elsevier, JOSS) and related tool providers (INSPIRE, Zenodo). This paper summarizes key findings and recommendations from the workshop as presented at the 26th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2023).

The Julia programming language was created 10 years ago and is now a mature and stable language with a large ecosystem including more than 8,000 third-party packages. It was designed for scientific programming to be a high-level and dynamic language as Python is, while achieving runtime performances comparable to C/C++ or even faster. With this, we ask ourselves if the Julia language and its ecosystem is ready now for its adoption by the High Energy Physics community. We will report on a number of investigations and studies of the Julia language that have been done for various representative HEP applications, ranging from computing intensive initial data processing of experimental data and simulation, to final interactive data analysis and plotting. Aspects of collaborative code development of large software within a HEP experiment has also been investigated: scalability with large development teams, continuous integration and code test, code reuse, language interoperability to enable an adiabatic migration of packages and tools, software installation and distribution, training of the community, benefit from development from industry and academia from other fields.

The evaluation of new computing languages for a large community, like HEP, involves comparison of many aspects of the languages’ behaviour, ecosystem and interactions with other languages. In this paper we compare a number of languages using a common, yet non-trivial, HEP algorithm: the anti- k T clustering algorithm used for jet finding. We compare specifically the algorithm implemented in Python (pure Python and accelerated with numpy and numba), and Julia, with respect to the reference implementation in C++, from Fastjet. As well as the speed of the implementation we describe the ergonomics of the language for the coder, as well as the efforts required to achieve the best performance, which can directly impact on code readability and sustainability.

ATLAS has recorded more than 8 petabyte(PB) of RAW data since the LHC started running at the end of 2009. Many more derived data products and complimentary simulation data have also been produced by the collaboration and, in total, 90PB are currently stored in the Worldwide LHC Computing Grid by ATLAS. All these data are managed by the ATLAS Distributed Data Management system, called Don Quijote 2 (DQ2). DQ2 has evolved rapidly to help ATLAS Computing operations manage these large quantities of data across the many grid sites at which ATLAS runs, and to help ATLAS physicists get access to these data.

PODIO is a C++ toolkit for the creation of event data models (EDMs) with a fast and efficient I/O layer. It employs plain-old-data (POD) data structures wherever possible, while avoiding deep object-hierarchies and virtual inheritance. A lightweight layer of handle classes provides the necessary highlevel interface for the physicist. PODIO creates all EDM code from simple instructive YAML files, describing the actual EDM entities. Since its original development PODIO has been very actively used for Future Circular Collider (FCC) studies. In its original version, the underlying I/O was entirely based on the automatic streaming code generated with ROOT dictionaries. Recently two additional I/O implementations have been added. One is based on HDF5 and the other uses SIO, a simple binary I/O library provided by LCIO. We briefly introduce the main features of PODIO and then report on recent developments with a focus on performance comparisons between the available I/O implementations. We conclude with presenting recent activities on porting the well-established LCIO EDM to PODIO and the recent EDM4hep project.

ATLAS's current software framework, Gaudi/Athena, has been very successful for the experiment in LHC Runs 1 and 2. However, its single threaded design has been recognized for some time to be increasingly problematic as CPUs have increased core counts and decreased available memory per core. Even the multi-process version of Athena, AthenaMP, will not scale to the range of architectures we expect to use beyond Run2.

Three-dimensional (3D) silicon sensors offer potential advantages over standard planar sensors for radiation hardness in future high energy physics experiments and reduced charge-sharing for X-ray applications, but may introduce inefficiencies due to the columnar electrodes. These inefficiencies are probed by studying variations in response across a unit pixel cell in a 55μm pitch double-sided 3D pixel sensor bump bonded to TimePix and Medipix2 readout ASICs. Two complementary characterisation techniques are discussed: the first uses a custom built telescope and a 120GeV pion beam from the Super Proton Synchrotron (SPS) at CERN; the second employs a novel technique to illuminate the sensor with a micro-focused synchrotron X-ray beam at the Diamond Light Source, UK. For a pion beam incident perpendicular to the sensor plane an overall pixel efficiency of 93.0±0.5% is measured. After a 10o rotation of the device the effect of the columnar region becomes negligible and the overall efficiency rises to 99.8±0.5%. The double-sided 3D sensor shows significantly reduced charge sharing to neighbouring pixels compared to the planar device. The charge sharing results obtained from the X-ray beam study of the 3D sensor are shown to agree with a simple simulation in which charge diffusion is neglected. The devices tested are found to be compatible with having a region in which no charge is collected centred on the electrode columns and of radius 7.6±0.6μm. Charge collection above and below the columnar electrodes in the double-sided 3D sensor is observed.

ATLAS's current software framework, Gaudi/Athena, has been very successful for the experiment in LHC Runs 1 and 2. However, its single-threaded design has been recognised for some time to be increasingly problematic as CPUs have increased core counts and decreased available memory per core. Even the multi-process version of Athena, AthenaMP, will not scale to the range of architectures we expect to use beyond Run2.

Distributed management of data is one of the most important problems facing grids. Within the Enabling Grids for Enabling eScience (EGEE) project, currently the world's largest production grid, a sophisticated hierarchy of data management and storage tools have been developed to help Virtual Organisations (VOs) with this task. In this paper we review the technologies employed for storage and data management in EGEE, and the associated Worldwide LHC Computing Grid (WLCG). We describe from low level networking and site storage technologies, through data transfer and cataloging middleware components. A particular emphasis is placed on deployment of these services in a large scale production environment. We also examine the interface between generic and VO specific data management, taking the example of the ATLAS high energy physics experiment at CERN.

The ATLAS experiment at the CERN LHC is one of the largest users of grid computing infrastructure, which is a central part of the experiment's computing operations. Considerable efforts have been made to use grid technology in the most efficient and effective way, including the use of a pilot job based workload management framework. In this model the experiment submits 'pilot' jobs to sites without payload. When these jobs begin to run they contact a central service to pick-up a real payload to execute. The first generation of pilot factories were usually specific to a single Virtual Organization (VO), and were bound to the particular architecture of that VO's distributed processing. A second generation provides factories which are more flexible, not tied to any particular VO, and provide new and improved features such as monitoring, logging, profiling, etc. In this paper we describe this key part of the ATLAS pilot architecture, a second generation pilot factory, AutoPyFactory. AutoPyFactory has a modular design and is highly configurable. It is able to send different types of pilots to sites and exploit different submission mechanisms and queue characteristics. It is tightly integrated with the PanDA job submission framework, coupling pilot flow to the amount of work the site has to run. It gathers information from many sources in order to correctly configure itself for a site and its decision logic can easily be updated. Integrated into AutoPyFactory is a flexible system for delivering both generic and specific job wrappers which can perform many useful actions before starting to run end-user scientific applications, e.g., validation of the middleware, node profiling and diagnostics, and monitoring. AutoPyFactory also has a robust monitoring system that has been invaluable in establishing a reliable pilot factory service for ATLAS.

Future HEP experiments require detailed simulation and advanced reconstruction algorithms to explore the physics reach of their proposed machines and to design, optimise, and study the detector geometry and performance. To synergize the development of the CLIC and FCC software efforts, the CERN EP R&D roadmap proposes the creation of a “Turnkey Software Stack”, which is foreseen to provide all the necessary ingredients, from simulation to analysis, for future experiments; not only CLIC and FCC, but also for proposed Super-tau-charm factories, CEPC, and ILC. The software stack will facilitate writing specific software for experiments ensuring coherency and maximising the re-use of established packages to benefit from existing solutions and community developments, for example, ROOT, Geant4, DD4hep, Gaudi and podio. As a showcase for the software stack, the existing CLIC reconstruction software, written for iLCSoft, is being to be ported to Gaudi. In parallel, the back-end of the LCIO event data model can be replaced by an implementation in podio. These changes will enable the sharing of the algorithms with other users of the software stack. We will present the current status and plans of the turnkey software stack, with a focus of the adaptation of the CLIC reconstruction chain to Gaudi and podio, and detail the plans for future developments to generalise their applicability to FCC and beyond.

The ATLAS Distributed Data Management system stores more than 150PB of physics data across 120 sites globally. To cope with the anticipated ATLAS workload of the coming decade, Rucio, the next-generation data management system has been developed. Replica management, as one of the key aspects of the system, has to satisfy critical performance requirements in order to keep pace with the experiment's high rate of continual data generation. The challenge lies in meeting these performance objectives while still giving the system users and applications a powerful toolkit to control their data workflows. In this work we present the concept, design and implementation of the replica management in Rucio. We will specifically introduce the workflows behind replication rules, their formal language definition, weighting and site selection. Furthermore we will present the subscription component, which offers functionality for users to proclaim interest in data that has not been created yet. This contribution describes the concept and the architecture behind those components and will show the benefits made by this system.

The detailed analysis of a ground-breaking sample of 100,000 n-Channel MOSFETs, simulated with the Glasgow 3D device simulator, has allowed the distribution of random discrete dopant induced threshold voltage fluctuations to be constructed based on underlying physical processes. The construction may also be statistically enhanced, allowing a significant reduction in the computational effort necessary to accurately model random discrete dopant induced variability.

During the current LHC shutdown period the ATLAS experiment will upgrade the Trigger and Data Acquisition system to include a hardware tracker coprocessor: the Fast TracKer (FTK). The FTK receives data from the 80 million of channels of the ATLAS silicon detector, identifying charged tracks and reconstructing their parameters at a rate of up to 100 KHz and within 100 microseconds. To achieve this performance, the FTK system identifies candidate tracks utilizing the computing power of a custom ASIC chip with associative memory (AM) designed to perform "pattern matching" at very high speed; track parameters are then calculated using modern FPGAs. A detailed simulation of this massive system has been developed with the goal of supporting the hardware design and studying its impact in the ATLAS online event selection at high LHC luminosities. We present the issues related to emulating FTK on a general-purpose CPU platform, using ATLAS computing Grid resources, and the solutions developed in order to mitigate these problems and allow the studies required to support the system design, construction and installation.

Reduced edge or ``edgeless'' detector design offers seamless tileability of sensors for a wide range of applications from particle physics to synchrotron and free election laser (FEL) facilities and medical imaging. Combined with through-silicon-via (TSV) technology, this would allow reduced material trackers for particle physics and an increase in the active area for synchrotron and FEL pixel detector systems. In order to quantify the performance of different edgeless fabrication methods, 2 edgeless detectors were characterized at the Diamond Light Source using an 11 μm FWHM 15 keV micro-focused X-ray beam. The devices under test were: a 150 μm thick silicon active edge pixel sensor fabricated at VTT and bump-bonded to a Medipix2 ROIC; and a 300 μm thick silicon strip sensor fabricated at CIS with edge reduction performed by SCIPP and the NRL and wire bonded to an ALiBaVa readout system. Sub-pixel resolution of the 55 μm active edge pixels was achieved. Further scans showed no drop in charge collection recorded between the centre and edge pixels, with a maximum deviation of 5% in charge collection between scanned edge pixels. Scans across the cleaved and standard guard ring edges of the strip detector also show no reduction in charge collection. These results indicate techniques such as the scribe, cleave and passivate (SCP) and active edge processes offer real potential for reduced edge, tiled sensors for imaging detection applications.

The ATLAS experiment has successfully used its Gaudi/Athena software framework for data taking and analysis during the first LHC run, with billions of events successfully processed. However, the design of Gaudi/Athena dates from early 2000 and the software and the physics code has been written using a single threaded, serial design. This programming model has increasing difficulty in exploiting the potential of current CPUs, which offer their best performance only through taking full advantage of multiple cores and wide vector registers. Future CPU evolution will intensify this trend, with core counts increasing and memory per core falling. With current memory consumption for 64 bit ATLAS reconstruction in a high luminosity environment approaching 4GB, it will become impossible to fully occupy all cores in a machine without exhausting available memory. However, since maximizing performance per watt will be a key metric, a mechanism must be found to use all cores as efficiently as possible.

At first glance, apprenticeship training appears to be a perfect educational solution for many Canadians, providing a clear pathway into the labour market. Moreover, its heavy emphasis upon on-the-job training makes it a potentially attractive option for individuals who are not inclined toward the classroom and lecture hall-based instruction of traditional university and college programs.

This paper describes a popularity prediction tool for data-intensive data management systems, such as ATLAS distributed data management (DDM). It is fed by the DDM popularity system, which produces historical reports about ATLAS data usage, providing information about files, datasets, users and sites where data was accessed. The tool described in this contribution uses this historical information to make a prediction about the future popularity of data. It finds trends in the usage of data using a set of neural networks and a set of input parameters and predicts the number of accesses in the near term future. This information can then be used in a second step to improve the distribution of replicas at sites, taking into account the cost of creating new replicas (bandwidth and load on the storage system) compared to gain of having new ones (faster access of data for analysis). To evaluate the benefit of the redistribution a grid simulator is introduced that is able replay real workload on different data distributions. This article describes the popularity prediction method and the simulator that is used to evaluate the redistribution.

The need to run complex workflows for a high energy physics experiment such as ATLAS has always been present. However, as computing resources have become even more constrained, compared to the wealth of data generated by the LHC, the need to use resources efficiently and manage complex workflows within a single grid job have increased.

This paper describes a monitoring framework for large scale data management systems with frequent data access. This framework allows large data management systems to generate meaningful information from collected tracing data and to be queried on demand for specific user usage patterns in respect to source and destination locations, period intervals, and other searchable parameters. The feasibility of such a system at the petabyte scale is demonstrated by describing the implementation and operational experience of a real world management information system for the ATLAS experiment employing the proposed framework. Our observations suggest that the proposed user monitoring framework is capable of scaling to meet the needs of very large data management systems.

Realizing the physics programs of the planned and upgraded high-energy physics (HEP) experiments over the next 10 years will require the HEP community to address a number of challenges in the area of software and computing. For this reason, the HEP software community has engaged in a planning process over the past two years, with the objective of identifying and prioritizing the research and development required to enable the next generation of HEP detectors to fulfill their full physics potential. The aim is to produce a Community White Paper which will describe the community strategy and a roadmap for software and computing research and development in HEP for the 2020s. The topics of event reconstruction and software triggers were considered by a joint working group and are summarized together in this document.

Common and community software packages, such as ROOT, Geant4 and event generators have been a key part of the LHC's success so far and continued development and optimisation will be critical in the future. The challenges are driven by an ambitious physics programme, notably the LHC accelerator upgrade to high-luminosity, HL-LHC, and the corresponding detector upgrades of ATLAS and CMS. In this document we address the issues for software that is used in multiple experiments (usually even more widely than ATLAS and CMS) and maintained by teams of developers who are either not linked to a particular experiment or who contribute to common software within the context of their experiment activity. We also give space to general considerations for future software and projects that tackle upcoming challenges, no matter who writes it, which is an area where community convergence on best practice is extremely useful.

The ART (ATLAS Release Tester) system is designed to run test jobs on the Grid after a nightly release of the ATLAS offline software has been built. The choice was taken to exploit the Grid as a backend as it offers a huge resource pool, suitable for a deep set of integration tests, and running the tests could be delegated to the highly scalable ATLAS production system (PanDA). The challenge of enabling the Grid as a test environment is met through the use of the CVMFS file system for the software and input data files. Test jobs are submitted to the Grid by the GitLab Continuous Integration (gitlab-ci) system, which itself is triggered at end of a release build. Jobs can be adorned with special headers that inform the system how to run the specific test, allowing many options to be customised. The gitlab-ci system waits for exit status and output files are copied back from the Grid to an EOS area accessible by users. All gitlab-ci jobs run in ART’s virtual machines, using docker images for their ATLAS setup. ART jobs can be tracked by using the PanDA system. ART can also be used to run short test jobs locally. It uses the same ART command-line interface, where the backend is replaced to access a local machine for job submission rather than the Grid. This allows developers to ensure their tests work correctly before adding them to the system. In both the Grid and local machine options, running and result copying are completely parallelized. ART is written in python, complete with its own local and Grid tests to give approximately 90% code coverage of the ART tool itself. ART has been in production for one year and fully replaces and augments the former ATLAS testing system.

These proceedings give a summary of the many software upgrade projects undertaken to prepare ATLAS for the challenges of Run-2 of the LHC. Those projects include a significant reduction of the CPU time required for reconstruction of real data with high average pile-up event rates compared to 2012. This is required to meet the challenges of the expected increase in pileup and the higher data taking rate of up to 1 kHz. By far the most ambitious project is the implementation of a completely new analysis model, based on a new ROOT readable reconstruction format, xAOD. The new model also includes a reduction framework based on a train model to centrally produce skimmed data samples and an analysis framework. These proceedings close with a brief overview of future software projects and plans that will lead up to the coming Long Shutdown 2 as the next major ATLAS software upgrade phase.

This is a post print of a paper from UK e-Science All Hands Meeting 10th-13th September 2007 published by National e-Science Centre. http://www.allhands.org.uk/2007/.

The progressive scaling of complementary metal oxide semiconductor (CMOS) transistors drives the success of the global semiconductor industry. Detailed knowledge of transistor behaviour is necessary to overcome the many fundamental challenges faced by chip and systems designers. Grid technology has enabled the unavoidable statistical variations introduced by scaling to be examined in unprecedented detail. Over 200 000 transistors have been simulated, the results of which provide detailed insight into underlying physical processes. This paper outlines recent scientific results of the nanoCMOS project and describes the way in which the scientific goals have been reflected in the grid-based e-Infrastructure.

The UK Engineering and Physical Sciences Research Council (EPSRC) funded project ¿Meeting the Design Challenges of nanoCMOS Electronics¿ (nanoCMOS) is developing a research infrastructure for collaborative electronics research across multiple institutions in the UK with especially strong industrial and commercial involvement. Unlike other domains, the electronics industry is driven by the necessity of protecting the intellectual property of the data, designs and software associated with next generation electronics devices and therefore requires fine-grained security. Similarly, the project also demands seamless access to large scale high performance compute resources for atomic scale device simulations and the capability to manage the hundreds of thousands of files and the metadata associated with these simulations. Within this context, the project has explored a wide range of authentication and authorization infrastructures facilitating compute resource access and providing fine-grained security over numerous distributed file stores and files. We conclude that no single security solution meets the needs of the project. This paper describes the experiences of applying X.509-based certificates and public key infrastructures, VOMS, PERMIS, Kerberos and the Internet2 Shibboleth technologies for nanoCMOS security. We outline how we are integrating these solutions to provide a complete end-to-end security framework meeting the demands of the nanoCMOS electronics domain.

High stopping power is one of the most important figures of merit for X-ray detectors. CdTe is a promising material but suffers from: material defects, non-ideal charge transport and long range X-ray fluorescence. Those factors reduce the image quality and deteriorate spectral information. In this project we used a monochromatic pencil beam collimated through a 20μm pinhole to measure the detector spectral response in dependance on the depth of interaction. The sensor was a 1mm thick CdTe detector with a pixel pitch of 110μm, bump bonded to a Timepix readout chip operating in Time-Over-Threshold mode. The measurements were carried out at the Extreme Conditions beamline I15 of the Diamond Light Source. The beam was entering the sensor at an angle of \texttildelow20 degrees to the surface and then passed through \texttildelow25 pixels before leaving through the bottom of the sensor. The photon energy was tuned to 77keV giving a variation in the beam intensity of about three orders of magnitude along the beam path. Spectra in Time-over-Threshold (ToT) mode were recorded showing each individual interaction. The bias voltage was varied between -30V and -300V to investigate how the electric field affected the spectral information. For this setup it is worth noticing the large impact of fluorescence. At -300V the photo peak and escape peak are of similar height. For high bias voltages the spectra remains clear throughout the whole depth but for lower voltages as -50V, only the bottom part of the sensor carries spectral information. This is an effect of the low hole mobility and the longer range the electrons have to travel in a low field.

We performed edge and surface Transient Current Technique (TCT) measurements of short, double sided 3D silicon strip detectors. Double sided 3D devices are a useful counterpart to traditional planar devices for use in the highest radiation environments. The TCT technique allows the electric field in the 3D devices to be probed in a way not possible before. The TCT technique uses the current waveform produced by the detector in response to a near delta function point laser pulse (illumination). The waveforms are recorded as a function of illumination position over the surface of the device under test as a function of detector bias.

This paper presents a system to predict future data popularity for data-intensive systems, such as the ATLAS distributed data management (DDM).Using these predictions it is possible to improve the distribution of data, helping to reduce waiting times for jobs using this data.This system is based on a tracer infrastructure that is able to monitor and store historical data accesses, which is then used to create popularity reports.These reports provide a summary of data accesses in the past, including information about the accessed files, the involved users and the sites.From this past accesses information it is possible to make near-term forecasts of data popularity.The prediction system introduced in this paper makes use of both simple prediction methods, as well as predictions made by neural networks.The best prediction method is dependent on the type of data and the access information is carefully filtered for use in either system.The second part of the paper introduces a system that effectively places data based on the predictions.This is a two phase process: In the first phase space is freed by removing unpopular replicas; in the second new replicas for popular datasets are created.The process of creating new replicas is limited by certain constraints: there is only a limited amount of space available and the creation of replicas involve transfers that use bandwidth.Furthermore, the benefits of each replica is different.The goal is to maximise the global benefit while respecting the constraints.The final part shows the evaluation of this method using a grid simulator.The simulator is able to replay workload on different data distributions while measuring the job waiting time.We show how job waiting time can be reduced based on accurate predictions about future accesses.

In the title compound, C(7)H(5)FO(2)·C(7)H(7)NO, a moderate-strength hydrogen bond is formed between the carboxyl group of one mol-ecule and the pyridine N atom of the other. The benzoic acid mol-ecule is observed to be disordered over two positions with the second orientation only 4% occupied. This disorder is also reflected in the presence of diffuse scattering in the diffraction pattern.

The High Energy Phyiscs community has developed and needs to maintain many tens of millions of lines of code and to integrate effectively the work of thousands of developers across large collaborations. Software needs to be built, validated, and deployed across hundreds of sites. Software also has a lifetime of many years, frequently beyond that of the original developer, it must be developed with sustainability in mind. Adequate recognition of software development as a critical task in the HEP community needs to be fostered and an appropriate publication and citation strategy needs to be developed. As part of the HEP Softare Foundation's Community White Paper process a working group on Software Development, Deployment and Validation was formed to examine all of these issues, identify best practice and to formulare recommendations for the next decade. Its report is presented here.

When operational, the Large Hadron Collider experiments at CERN will collect tens of petabytes of physics data per year. The worldwide LHC computing grid (WLCG) will distribute this data to over two hundred Tier-1 and Tier-2 computing centres, enabling particle physicists around the globe to access the data for analysis.

A method is proposed to reduce the power used in the motion estimation stage of an FPGA based H.264 video encoder. Distinguishing it from other algorithms is its use of information generated during the rest of the encoding process, specifically, the intra prediction stage. Using the results of the intra prediction stage, a simple algorithm determines the direction with which to propagate reference data through the systolic array, used for motion estimation, in order to minimize the array's switching activity. Results are given showing that this method can reduce the switching activity, and hence power, in the array by up-to 10%. The reduction achievable is, however, conditional on reducing the bit-width of the inputs to the motion estimation process.

Computing resources in HEP are increasingly delivered utilising grid technologies, which presents new challenges in terms of monitoring. Monitoring involves the flow of information between different communities: the various resource-providers and the different user communities. The challenge is providing information so everyone can find what they need: from the local site administrators, regional operational centres through to end-users.

In a complex multi-developer, multi-package software environment, such as the ATLAS offline framework Athena, tracking the performance of the code can be a non-trivial task in itself. In this paper we describe improvements in the instrumentation of ATLAS offline software that have given considerable insight into the performance of the code and helped to guide the optimization work. The first tool we used to instrument the code is PAPI, which is a programing interface for accessing hardware performance counters. PAPI events can count floating point operations, cycles, instructions and cache accesses. Triggering PAPI to start/stop counting for each algorithm and processed event results in a good understanding of the algorithm level performance of ATLAS code. Further data can be obtained using Pin, a dynamic binary instrumentation tool. Pin tools can be used to obtain similar statistics as PAPI, but advantageously without requiring recompilation of the code. Fine grained routine and instruction level instrumentation is also possible. Pin tools can additionally interrogate the arguments to functions, like those in linear algebra libraries, so that a detailed usage profile can be obtained. These tools have characterized the extensive use of vector and matrix operations in ATLAS tracking. Currently, CLHEP is used here, which is not an optimal choice. To help evaluate replacement libraries a testbed has been setup allowing comparison of the performance of different linear algebra libraries (including CLHEP, Eigen and SMatrix/SVector). Results are then presented via the ATLAS Performance Management Board framework, which runs daily with the current development branch of the code and monitors reconstruction and Monte-Carlo jobs. This framework analyses the CPU and memory performance of algorithms and an overview of results are presented on a web page. These tools have provided the insight necessary to plan and implement performance enhancements in ATLAS code by identifying the most common operations, with the call parameters well understood, and allowing improvements to be quantified in detail.

The security challenges faced by users of the grid are considerably different to those faced in previous environments. The adoption of pilot jobs systems by LHC experiments has mitigated many of the problems associated with the inhomogeneities found on the grid and has greatly improved job reliability; however, pilot jobs systems themselves must then address many security issues, including the execution of multiple users' code under a common 'grid' identity. In this paper we describe the improvements and evolution of the security model in the ATLAS PanDA (Production and Distributed Analysis) system. We describe the security in the PanDA server which is in place to ensure that only authorized members of the VO are allowed to submit work into the system and that jobs are properly audited and monitored. We discuss the security in place between the pilot code itself and the PanDA server, ensuring that only properly authenticated workload is delivered to the pilot for execution. When the code to be executed is from a 'normal' ATLAS user, as opposed to the production system or other privileged actor, then the pilot may use an EGEE developed identity switching tool called gLExec. This changes the grid proxy available to the job and also switches the UNIX user identity to protect the privileges of the pilot code proxy. We describe the problems in using this system and how they are overcome. Finally, we discuss security drills which have been run using PanDA and show how these improved our operational security procedures.

This paper reports on recent test beam and laboratory results performed on 3D strip and pixel detectors.The devices were produced using a special double-sided 3D technology aimed to simplify the fabrication process, where the columnar electrodes etched into the silicon do not pass through the full substrate thickness.Double-sided 3D n-in-p strip detectors show good electrical and charge collection characteristics after heavy irradiation up to 2 × 10 16 n eq /cm 2 .An effect of charge multiplication is observed at high bias voltages, both in laboratory tests with radioactive source and in beam tests with pions.This multiplication effect is beneficial for the signal-to-noise ratio for moderate voltages and values > 40 can be achieved.The detection efficiency and charge sharing properties of the 3D structure have been investigated in Medipix2 pixel detectors with micro-focus synchrotron X-rays and pion beams and compared to that of the standard planar technology.There is a drop in the detection efficiency over the pixel of the 3D sensors due to the central electrodes, however the corner electrodes do not represent a significant degradation of the efficiency compared to that of the planar devices.The 3D sensors show a considerably reduced charge sharing compared to planar detectors that makes them very interesting for imaging applications.

Family RelationsVolume 72, Issue 2 p. 383-387 ISSUE INFORMATIONFree Access Issue Information First published: 15 March 2023 https://doi.org/10.1111/fare.12708AboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Volume72, Issue2Special Issue: Doing Family OnlineApril 2023Pages 383-387 RelatedInformation

In November 2022, the HEP Software Foundation and the Institute for Research and Innovation for Software in High-Energy Physics organized a workshop on the topic of Software Citation and Recognition in HEP. The goal of the workshop was to bring together different types of stakeholders whose roles relate to software citation, and the associated credit it provides, in order to engage the community in a discussion on: the ways HEP experiments handle citation of software, recognition for software efforts that enable physics results disseminated to the public, and how the scholarly publishing ecosystem supports these activities. Reports were given from the publication board leadership of the ATLAS, CMS, and LHCb experiments and HEP open source software community organizations (ROOT, Scikit-HEP, MCnet), and perspectives were given from publishers (Elsevier, JOSS) and related tool providers (INSPIRE, Zenodo). This paper summarizes key findings and recommendations from the workshop as presented at the 26th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2023).

The evaluation of new computing languages for a large community, like HEP, involves comparison of many aspects of the languages' behaviour, ecosystem and interactions with other languages. In this paper we compare a number of languages using a common, yet non-trivial, HEP algorithm: the \akt\ clustering algorithm used for jet finding. We compare specifically the algorithm implemented in Python (pure Python and accelerated with numpy and numba), and Julia, with respect to the reference implementation in C++, from Fastjet. As well as the speed of the implementation we describe the ergonomics of the language for the coder, as well as the efforts required to achieve the best performance, which can directly impact on code readability and sustainability.

In this paper we document the current analysis software training and onboarding activities in several High Energy Physics (HEP) experiments: ATLAS, CMS, LHCb, Belle II and DUNE. Fast and efficient onboarding of new collaboration members is increasingly important for HEP experiments as analyses and the related software become ever more complex with growing datasets. A meeting series was held by the HEP Software Foundation (HSF) in 2022 for experiments to showcase their initiatives. Here we document and analyse these in an attempt to determine a set of key considerations for future experiments.

The Key4hep project aims to provide a turnkey software solution for the full experiment lifecycle, based on established community tools. Several future collider communities (CEPC, CLIC, EIC, FCC, and ILC) have joined to develop and adapt their workflows to use the common data model EDM4hep and common framework. Besides sharing of existing experiment workflows, one focus of the Key4hep project is the development and integration of new experiment independent software libraries. Ongoing collaborations with projects such as ACTS, CLUE, PandoraPFA and the OpenDataDector show the potential of Key4hep as an experiment-independent testbed and development platform. In this talk, we present the challenges of an experiment-independent framework along with the lessons learned from discussions of interested communities (such as LUXE) and recent adopters of Key4hep in order to discuss how Key4hep could be of interest to the wider HEP community while staying true to its goal of supporting future collider designs studies.

Detector studies for future experiments rely on advanced software tools to estimate performance and optimize their design and technology choices. The Key4hep project provides a flexible turnkey solution for the full experiment life-cycle based on established community tools such as ROOT, Geant4, DD4hep, Gaudi, podio and spack. Members of the CEPC, CLIC, EIC, FCC, and ILC communities have joined to develop this framework and have merged, or are in the progress of merging, their respective software environments into the Key4hep stack. These proceedings will give an overview over the recent progress in the Key4hep project: covering the developments towards adaptation of state-of-the-art tools for simulation (DD4hep, Gaussino), track and calorimeter reconstruction (ACTS, CLUE), particle flow (PandoraPFA), analysis via RDataFrame, and visualization with Phoenix, as well as tools for testing and validation.

The podio event data model (EDM) toolkit provides an easy way to generate a performant implementation of an EDM from a high level description in yaml format. We present the most recent developments in podio, most importantly the inclusion of a schema evolution mechanism for generated EDMs as well as the "Frame", a thread safe, generalized event data container. For the former we discuss some of the technical aspects in relation with supporting different I/O backends and leveraging potentially existing schema evolution mechanisms provided by them. Regarding the Frame we introduce the basic concept and highlight some of the functionality as well as important aspects of its implementation. The usage of podio for generating different EDMs for future collider projects (most importantly EDM4hep, the common EDM for the Key4hep project) has inspired new features. We present some of those smaller new features and end with a brief overview on current developments towards a first stable version as well as an outlook on future developments beyond that.

A performant and easy-to-use event data model (EDM) is a key component of any HEP software stack. The podio EDM toolkit provides a user friendly way of generating such a performant implementation in C++ from a high level description in yaml format. Finalizing a few important developments, we are in the final stretches for release v1.0 of podio, a stable release with backward compatibility for datafiles written with podio from then on. We present an overview of the podio basics, and go into slighty more technical detail on the most important topics and developments. These include: schema evolution for generated EDMs, multithreading with podio generated EDMs, the implementation of them as well as the basics of I/O. Using EDM4hep, the common and shared EDM of the Key4hep project, we highlight a few of the smaller features in action as well as some lessons learned during the development of EDM4hep and podio. Finally, we show how podio has been integrated into the Gaudi based event processing framework that is used by Key4hep, before we conclude with a brief outlook on potential developments after v1.0.

The automation of operations is essential to reduce manpower costs and improve the reliability of the system. The Site Status Board (SSB) is a framework which allows Virtual Organizations to monitor their computing activities at distributed sites and to evaluate site performance. The ATLAS experiment intensively uses the SSB for the distributed computing shifts, for estimating data processing and data transfer efficiencies at a particular site, and for implementing automatic exclusion of sites from computing activities, in case of potential problems. The ATLAS SSB provides a real-time aggregated monitoring view and keeps the history of the monitoring metrics. Based on this history, usability of a site from the perspective of ATLAS is calculated. The paper will describe how the SSB is integrated in the ATLAS operations and computing infrastructure and will cover implementation details of the ATLAS SSB sensors and alarm system, based on the information in the SSB. It will demonstrate the positive impact of the use of the SSB on the overall performance of ATLAS computing activities and will overview future plans.

The rapid evolution of technology and the parallel increasing complexity of algorithmic analysis in HEP requires developers to acquire a much larger portfolio of programming skills. Young researchers graduating from universities worldwide currently do not receive adequate preparation in the very diverse fields of modern computing to respond to growing needs of the most advanced experimental challenges. There is a growing consensus in the HEP community on the need for training programmes to bring researchers up to date with new software technologies, in particular in the domains of concurrent programming and artificial intelligence. We review some of the initiatives under way for introducing new training programmes and highlight some of the issues that need to be taken into account for these to be successful.

The increase in the scale of LHC computing expected for Run 3 and even more so for Run 4 (HL-LHC) over the next ten years will certainly require radical changes to the computing models and the data processing of the LHC experiments. Translating the requirements of the physics programmes into computing resource needs is a complicated process and subject to significant uncertainties. For this reason, WLCG has established a working group to develop methodologies and tools intended tocharacterise the LHC workloads, better understand their interaction with the computing infrastructure, calculate their cost in terms of resources and expenditure and assist experiments, sites and the WLCG project in the evaluation of their future choices. This working group started in November 2017 and has about 30 active participants representing experiments and sites. In this contribution we expose the activities, the results achieved and the future directions.

New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will be easier to develop and maintain, that it remains available in the future on new platforms, that it meets new needs, and that it is as reusable as possible. This report discusses a virtual half-day workshop on "Software Sustainability and High Energy Physics" that aimed 1) to bring together experts from HEP as well as those from outside to share their experiences and practices, and 2) to articulate a vision that helps the Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP) to create a work plan to implement elements of software sustainability. Software sustainability practices could lead to new collaborations, including elements of HEP software being directly used outside the field, and, as has happened more frequently in recent years, to HEP developers contributing to software developed outside the field rather than reinventing it. A focus on and skills related to sustainable software will give HEP software developers an important skill that is essential to careers in the realm of software, inside or outside HEP. The report closes with recommendations to improve software sustainability in HEP, aimed at the HEP community via IRIS-HEP and the HEP Software Foundation (HSF).

Abstract The long-term sustainability of the high-energy physics (HEP) research software ecosystem is essential to the field. With new facilities and upgrades coming online throughout the 2020s, this will only become increasingly important. Meeting the sustainability challenge requires a workforce with a combination of HEP domain knowledge and advanced software skills. The required software skills fall into three broad groups. The first is fundamental and generic software engineering (e.g., Unix, version control, C++, and continuous integration). The second is knowledge of domain-specific HEP packages and practices (e.g., the ROOT data format and analysis framework). The third is more advanced knowledge involving specialized techniques, including parallel programming, machine learning and data science tools, and techniques to maintain software projects at all scales. This paper discusses the collective software training program in HEP led by the HEP Software Foundation (HSF) and the Institute for Research and Innovation in Software in HEP (IRIS-HEP). The program equips participants with an array of software skills that serve as ingredients for the solution of HEP computing challenges. Beyond serving the community by ensuring that members are able to pursue research goals, the program serves individuals by providing intellectual capital and transferable skills important to careers in the realm of software and computing, inside or outside HEP.

The Grid and Web service community are defining a range of standards for a complete solution for security. The National e-Science Centre (NeSC) at the University of Glasgow is investigating how the various pre-integration components work together in a variety of e-Science projects. The EPSRC-funded nanoCMOS project aims to allow electronics designers and manufacturers to use e-Science technologies and expertise to solve problems of device variability and its impact on system design. To support the security requirements of nanoCMOS, two NeSC projects (VPMan and OMII-SP) are providing tools to allow easy configuration of security infrastructures, exploiting previous successful projects using Shibboleth and PERMIS. This paper presents the model in which these tools interoperate to provide secure and simple access to Grid resources for non-technical users.

Rucio is the next-generation data management system of the ATLAS experiment. The software engineering process to develop Rucio is fundamentally different to existing software development approaches in the ATLAS distributed computing community. Based on a conceptual design document, development takes place using peer-reviewed code in a test-driven environment. The main objectives are to ensure that every engineer understands the details of the full project, even components usually not touched by them, that the design and architecture are coherent, that temporary contributors can be productive without delay, that programming mistakes are prevented before being committed to the source code, and that the source is always in a fully functioning state. This contribution will illustrate the workflows and products used, and demonstrate the typical development cycle of a component from inception to deployment within this software engineering process. Next to the technological advantages, this contribution will also highlight the social aspects of an environment where every action is subject to detailed scrutiny.

Rucio is the successor of the current Don Quijote 2 (DQ2) system for the distributed data management (DDM) system of the ATLAS experiment. The reasons for replacing DQ2 are manifold, but besides high maintenance costs and architectural limitations, scalability concerns are on top of the list. Current expectations are that the amount of data will be three to four times as it is today by the end of 2014. Further is the availability of more powerful computing resources pushing additional pressure on the DDM system as it increases the demands on data provisioning. Although DQ2 is capable of handling the current workload, it is already at its limits. To ensure that Rucio will be up to the expected workload, a way to emulate it is needed. To do so, first the current workload, observed in DQ2, must be understood in order to scale it up to future expectations. The paper discusses how selected core concepts are applied to the workload of the experiment and how knowledge about the current workload is derived from various sources (e.g. analysing the central file catalogue logs). Finally a description of the implemented emulation framework, used for stress-testing Rucio, is given.

To prepare the migration to the new ATLAS Data Management system called Rucio, a renaming campaign of all the physical files produced by ATLAS is needed. It represents around 300 million files split between ~120 sites with 6 different storage technologies. It must be done in a transparent way in order not to disrupt the ongoing computing activities. An infrastructure to perform this renaming has been developed and is presented in this paper as well as its performance.

After the current maintenance period, the LHC will provide higher energy collisions with increased luminosity. In order to keep up with these higher rates, ATLAS software needs to speed up substantially. However, ATLAS code is composed of approximately 6M lines, written by many different programmers with different backgrounds, which makes code optimisation a challenge. To help with this effort different profiling tools and techniques are being used. These include well known tools, such as the Valgrind suite and Intel Amplifier; less common tools like Pin, PAPI, and GOoDA; as well as techniques such as library interposing. In this paper we will mainly focus on Pin tools and GOoDA. Pin is a dynamic binary instrumentation tool which can obtain statistics such as call counts, instruction counts and interrogate functions' arguments. It has been used to obtain CLHEP Matrix profiles, operations and vector sizes for linear algebra calculations which has provided the insight necessary to achieve significant performance improvements. Complimenting this, GOoDA, an in-house performance tool built in collaboration with Google, which is based on hardware performance monitoring unit events, is used to identify hot-spots in the code for different types of hardware limitations, such as CPU resources, caches, or memory bandwidth. GOoDA has been used in improvement of the performance of new magnetic field code and identification of potential vectorization targets in several places, such as Runge-Kutta propagation code.

The ATLAS experiment has successfully used its Gaudi/Athena software framework for data taking and analysis during the first LHC run, with billions of events successfully processed. However, the design of Gaudi/Athena dates from early 2000 and the software and the physics code has been written using a single threaded, serial design. This programming model has increasing difficulty in exploiting the potential of current CPUs, which offer their best performance only through taking full advantage of multiple cores and wide vector registers. Future CPU evolution will intensify this trend, with core counts increasing and memory per core falling. Maximising performance per watt will be a key metric, so all of these cores must be used as efficiently as possible. In order to address the deficiencies of the current framework, ATLAS has embarked upon two projects: first, a practical demonstration of the use of multi-threading in our reconstruction software, using the GaudiHive framework; second, an exercise to gather requirements for an updated framework, going back to the first principles of how event processing occurs. In this paper we report on both these aspects of our work. For the hive based demonstrators, we discuss what changes were necessary in order to allow the serially designed ATLAS code to run, both to the framework and to the tools and algorithms used. We report on what general lessons were learned about the code patterns that had been employed in the software and which patterns were identified as particularly problematic for multi-threading. These lessons were fed into our considerations of a new framework and we present preliminary conclusions on this work. In particular we identify areas where the framework can be simplified in order to aid the implementation of a concurrent event processing scheme. Finally, we discuss the practical difficulties involved in migrating a large established code base to a multi-threaded framework and how this can be achieved for LHC Run 3.

This is a pre-print of a paper from Proceedings of the Conference on Parallel and Distributed Computing and Networks 2010 published by ACTA Press. http://www.iasted.org/conferences/pastinfo-676.html

The DDM Tracer monitoring framework is aimed to trace and monitor the ATLAS file operations on the Worldwide LHC Computing Grid. The volume of traces has increased significantly since the framework was put in production in 2009. Now there are about 5 million trace messages every day and peaks can be near 250Hz, with peak rates continuing to climb, which gives the current structure a big challenge. Analysis of large datasets based on on-demand queries to the relational database management system (RDBMS), i.e. Oracle, can be problematic, and have a significant effect on the database's performance. Consequently, We have investigated some new high availability technologies like messaging infrastructure, specifically ActiveMQ, and key-value stores. The advantages of key value store technology are that they are distributed and have high scalability; also their write performances are usually much better than RDBMS, all of which are very useful for the Tracer monitoring framework. Indexes and distributed counters have been also tested to improve query performance and provided almost real time results. In this paper, the design principles, architecture and main characteristics of Tracer monitoring framework will be described and examples of its usage will be presented.

The ATLAS Production and Distributed Analysis System (PanDA) is a key component of the ATLAS distributed computing infrastructure. All ATLAS production jobs, and a substantial amount of user and group analysis jobs, pass through the PanDA system, which manages their execution on the grid. PanDA also plays a key role in production task definition and the data set replication request system. PanDA has recently been migrated from Brookhaven National Laboratory (BNL) to the European Organization for Nuclear Research (CERN), a process we describe here.

The ATLAS experiment at the LHC collider recorded more than 5 fb−1 data of pp collisions at a centre-of-mass energy of 7 TeV during 2011. The recorded data are promptly reconstructed in two steps at a large computing farm at CERN to provide fast access to high quality data for physics analysis. In the first step, a subset of the data, corresponding to the express stream and having 10Hz of events, is processed in parallel with data taking. Data quality, detector calibration constants, and the beam spot position are determined using the reconstructed data within 48 hours. In the second step all recorded data are processed with the updated parameters. The LHC significantly increased the instantaneous luminosity and the number of interactions per bunch crossing in 2011; the data recording rate by ATLAS exceeds 400 Hz. To cope with these challenges the performance and reliability of the ATLAS reconstruction software have been improved. In this paper we describe how the prompt data reconstruction system quickly and stably provides high quality data to analysers.

This paper summarizes operational experience and improvements in ATLAS computing infrastructure in 2010 and 2011. ATLAS has had 2 periods of data taking, with many more events recorded in 2011 than in 2010. It ran 3 major reprocessing campaigns. The activity in 2011 was similar to 2010, but scalability issues had to be addressed due to the increase in luminosity and trigger rate. Based on improved monitoring of ATLAS Grid computing, the evolution of computing activities (data/group production, their distribution and grid analysis) over time is presented. The main changes in the implementation of the computing model that will be shown are: the optimization of data distribution over the Grid, according to effective transfer rate and site readiness for analysis; the progressive dismantling of the cloud model, for data distribution and data processing; software installation migration to cvmfs; changing database access to a Frontier/squid infrastructure.

In this paper we explain how the C++ code quality is managed in ATLAS using a range of tools from compile-time through to run time testing and reflect on the substantial progress made in the last two years largely through the use of static analysis tools such as Coverity®, an industry-standard tool which enables quality comparison with general open source C++ code. Other available code analysis tools are also discussed, as is the role of unit testing with an example of how the GoogleTest framework can be applied to our codebase.

Summary form only given. In this paper, we present groundbreaking results of the simulation of 140,000 well scaled 13nm nChannel bulk MOSFETs, each microscopically different in terms of discrete dopant distributions. These devices were simulated using the well established Glasgow 3D drift/diffusion atomistic simulator. In order to undertake simulations of such a magnitude, we have employed advanced grid technology being developed at Glasgow as part of the nanoCMOS project.

Realizing the physics programs of the planned and upgraded high-energy physics (HEP) experiments over the next 10 years will require the HEP community to address a number of challenges in the area of software and computing. For this reason, the HEP software community has engaged in a planning process over the past two years, with the objective of identifying and prioritizing the research and development required to enable the next generation of HEP detectors to fulfill their full physics potential. The aim is to produce a Community White Paper which will describe the community strategy and a roadmap for software and computing research and development in HEP for the 2020s. The topics of event reconstruction and software triggers were considered by a joint working group and are summarized together in this document.

Data processing frameworks are an essential part of HEP experiments' software stacks. Frameworks provide a means by which code developers can undertake the essential tasks of physics data processing, accessing relevant inputs and storing their outputs, in a coherent way without needing to know the details of other domains. Frameworks provide essential core services for developers and help deliver a configurable working application to the experiments' production systems. Modern HEP processing frameworks are in the process of adapting to a new computing landscape dominated by parallel processing and heterogeneity, which pose many questions regarding enhanced functionality and scaling that must be faced without compromising the maintainability of the code. In this paper we identify a program of work that can help further clarify the key concepts of frameworks for HEP and then spawn R&D activities that can focus the community's efforts in the most efficient manner to address the challenges of the upcoming experimental program.

The design of a platform based video streaming server using the Xilinx microblaze processor and a custom H.263 hardware compression core is presented. The design uses a novel data structure to store the input images in external memory, allowing the H.263 core and the associated camera interface to utilise the external memory bandwidth more efficiently. The finished system is capable of encoding and streaming, using the real-time transport protocol, Dl sized video, at 30 frames per second in a Spartan-3 1500 FPGA device.

High-Energy Physics has evolved a rich set of software packages that need to work harmoniously to carry out the key software tasks needed by experiments. The problem of consistently building and deploying these packages as a coherent software stack is one that is shared across the HEP community. To that end the HEP Software Foundation Packaging Working Group has worked to identify common solutions that can be used across experiments, with an emphasis on consistent, reproducible builds and easy deployment into CernVM-FS or containers via CI systems. We based our approach on well-identified use cases and requirements from many experiments. In this paper we summarise the work of the group in the last year and how we have explored various approaches based on package managers from industry and the scientific computing community. We give details about a solution based on the Spack package manager which has been used to build the software required by the SuperNEMO and FCC experiments and trialled for a multi-experiment software stack, Key4hep. We shall discuss changes that needed to be made to Spack to satisfy all our requirements. We show how support for a build environment for software developers is provided.

This contribution presents a study on the applicability and usefulness of dynamic data placement methods for data-intensive systems, such as ATLAS distributed data management (DDM). In this system the jobs are sent to the data, therefore having a good distribution of data is significant. Ways of forecasting workload patterns are examined which then are used to redistribute data to achieve a better overall utilisation of computing resources and to reduce waiting time for jobs before they can run on the grid. This method is based on a tracer infrastructure that is able to monitor and store historical data accesses and which is used to create popularity reports. These reports provide detailed summaries about data accesses in the past, including information about the accessed files, the involved users and the sites. From this past data it is possible to then make near-term forecasts for data popularity in the future. This study evaluates simple prediction methods as well as more complex methods like neural networks. Based on the outcome of the predictions a redistribution algorithm deletes unused replicas and adds new replicas for potentially popular datasets. Finally, a grid simulator is used to examine the effects of the redistribution. The simulator replays workload on different data distributions while measuring the job waiting time and site usage. The study examines how the average waiting time is affected by the amount of data that is moved, how it differs for the various forecasting methods and how that compares to the optimal data distribution.

We compare a new CMOS Active Pixel Sensor (APS) to a Princeton Instruments PIXIS-XO: 2048B Charge Coupled Device (CCD) with soft X-rays tested in a synchrotron beam line at the Diamond Light Source (DLS). Despite CCDs being established in the field of scientific imaging, APS are an innovative technology that offers advantages over CCDs. These include faster readout, higher operational temperature, in-pixel electronics for advanced image processing and reduced manufacturing cost.

The volume and diversity of metadata in an experiment of the size and scope of ATLAS are considerable. Even the definition of metadata may seem context-dependent: data that are primary for one purpose may be metadata for another. ATLAS metadata services must integrate and federate information from inhomogeneous sources and repositories, map metadata about logical or physics constructs to deployment and production constructs, provide a means to associate metadata at one level of granularity with processing or decision-making at another, offer a coherent and integrated view to physicists, and support both human use and programmatic access. In this paper we consider ATLAS metadata, metadata services, and metadata flow principally from the illustrative perspective of how disparate metadata are made available to executing jobs and, conversely, how metadata generated by such jobs are returned. We describe how metadata are read, how metadata are cached, and how metadata generated by jobs and the tasks of which they are a part are communicated, associated with data products, and preserved. We also discuss the principles that guide decision-making about metadata storage, replication, and access.

Since their inception, Grids for high energy physics have found management of data to be the most challenging aspect of operations. This problem has generally been tackled by the experiment's data management framework controlling in fine detail the distribution of data around the grid and the careful brokering of jobs to sites with co-located data. This approach, however, presents experiments with a difficult and complex system to manage as well as introducing a rigidity into the framework which is very far from the original conception of the grid.

3D detectors are proposed as an alternative to planar silicon technology to withstand the high radiation environments in planned future high energy physics experiments. Here we review the characterization of double-sided 3D detectors designed and built at CNM and the University of Glasgow. A non-irradiated sensor is characterized in a pion test-beamutilizing the Timepix telescope. The charge collection and detection efficiency across the unit pixel are shown. Area of inefficiency can be found at the columnar electrodes at perpendicular angles of beam incidence while the pixels are shown to be fully efficient at angles greater than ten degrees. A reduction in charge sharing compared to the planar technology is also demonstrated. Charge collection studies on irradiated devices with a Sr-90 source show higher charge collection efficiency for 3D over planar sensors at significantly lower applied bias. The sub-pixel response is probed by a micro-focused laser beam demonstrating areas of charge multiplication at high bias voltages.

ATLAS has recorded almost 5PB of RAW data since the LHC started running at the end of 2009. Many more derived data products and complimentary simulation data have also been produced by the collaboration and, in total, 70PB is currently stored in the Worldwide LHC Computing Grid by ATLAS. All of this data is managed by the ATLAS Distributed Data Management system, called Don Quixote 2 (DQ2). DQ2 has evolved rapidly to help ATLAS Computing operations manage these large quantities of data across the many grid sites at which ATLAS runs and to help ATLAS physicists get access to this data. In this paper we describe new and improved DQ2 services: popularity; space monitoring and accounting; exclusion service; cleaning agents; deletion agents. We describe the experience of data management operation in ATLAS computing, showing how these services enable management of petabyte scale computing operations. We illustrate the coupling of data management services to other parts of the ATLAS computing infrastructure, in particular showing how feedback from the distributed analysis system in ATLAS has enabled dynamic placement of the most popular data, helping users and groups to analyse the increasing data volumes on the grid.

In this paper we summarise ATLAS operations from the STEP09 campaign in June 2009 through to ATLAS taking data in the first 7 TeV collisions at the LHC in 2010.We describe the lessons which were learned from the STEP09 challenge, both in proving which parts of the system were in good shape, but also in highlighting those areas which required improvement.We then describe the experience of ATLAS computing operations during the first LHC data taking era.The ATLAS experiment has successfully recorded, reconstructed, distributed and analysed millions of collision events delivered by the LHC at an unprecedented centre-of-mass energy of 7 TeV.The involved large-scale data processing operations, both the prompt reconstruction at Tier0 and the subsequent reprocessing campaigns in the Tier1 sites in the Grid, worked remarkably well.