Funnel: towards comfortable event processing

The funnel software package has solved for the ZEUS collaboration the problem of Monte Carlo event production; a problem faced by many HEP experiments. Thanks to extensive automation, a few man-hours per day are sufficient to resolve problems and to manage the entire ZEUS Monte Carlo production. Other than specifying the events to be produced, ZEUS physicists are thus freed from the chore of Monte Carlo production. As an additional benefit, the computing cycles required for production are nearly cost free since they replace otherwise idle cycles on hundreds of unix workstation and server computers, with minimal interference for their regular users. The computers are spread across a dozen sites around the world and continually deliver the effective equivalent of approximately one hundred dedicated computers. Funnel successfully demonstrates that generic independent tools can provide comfortable event processing. With an emphasis on automation and fault-tolerance, the tools manage all aspects of event processing including the job queues, the execution and failures of the processing program, parallel processing, as well as data buffering, archiving and remote transfer. The L3, HERMES and HI collaborations are presently creating Monte Carlo production systems, using the funnel experience and, to different extents, parts of the funnel software package. The experience gained with funnel encourages the construction of EVPRO, a general purpose software package for event processing. EVPRO would build on top of existing software; for example CPS or PVM for parallel processing. Whether on a dedicated farm of computers or using idle cycles, an application of any size could then easily enjoy the comfort of automated, fault-tolerant event processing. EVPRO aims to minimize application-specific event processing software, whose high development costs can only be justified for the largest of applications. A casual user may provide EVPRO with only the processing program and the data to be processed. A more complex or real-time application would tune EVPRO to its needs; for example, integrating custom hardware for the flow of event data. Making optimal use of the available computing resources, EVPRO would manage all aspects of the event processing. Monte Carlo production, event reconstruction and software triggers could use EVPRO, as could any computing application, inside or outside of HEP, which can be expressed in terms of events. In principle, event processing is a solved or even a trivial problem. Given an implementor, EVPRO could provide the trivial solution in practice