This paper discusses the issues of leak detection performance mapping, frequency and size distribution of false positives (i.e. false or non-leak alarms), and related issues in an attempt to elucidate the issues that must be considered when evaluating, testing, and reporting leak detection system performance. API 1130 recommends periodic testing of computational pipeline monitoring (CPM) systems to properly determine leak detection performance. Such testing can use commodity discharge, which is cumbersome, expensive, and limited in detail, but high quality in terms of the confidence in its results. Or it can be software based, implying potential low cost, speed and greater level of detail, with the admitted issue that software approaches always require real world validation. There is also a frequent desire and tendency to state the performance of a leak detection system (LDS) as a single number: detectable leak size. Generally, however, a more detailed view of leak detection system performance is provided by a set of performance maps that relate leak size to probability of detection as a function of elapsed time since the leak occurred for various leak rates. Frequency of false alarms is also a significant performance factor. Firstly, to lay the groundwork for this discussion, this paper presents a simplified analytic model which qualitatively predicts the probabilistic performance of leak detection systems and provides a basis for discussion of such probabilistic performance in volume balance systems. This simplified model predicts that leak alarms associated with small leak sizes will exhibit a probabilistic nature in terms of both sensitivity to real leaks, as well as creation of leak alarms or false positives when no leak is present. Secondly, this paper presents a methodology to use software-based test methods coupled with recorded real world pipeline data to provide complete probabilistic performance mapping and evaluation of false alarm rates for pipeline leak detection systems. This method is applied offline, and is relatively inexpensive and cost effective to implement if it is built into the CPM system as a system feature. Thirdly, this software testing approach is then applied to an existing leak detection system using actual pipeline data for a period of several months. The results are shown to be consistent with the qualitative results developed using the simplified theoretical analytic model presented earlier in the paper. For this real system, an attempt is made to present and evaluate true measures of the leak detection system performance, illustrating the abilities of the software based test methodology to present a quantitative analysis of leak detection sensitivity, the relationship between sensitivity and false-positives, and the development of measures that provide an objective measure of system efficacy. Finally, drawing on the authors' many years of experiences with installed pipeline leak detection systems, an attempt is made to combine both the theoretical and actual performance measures with subjective and behavioral responses to leak alarms and guidelines are presented for appropriate tradeoffs between leak detection sensitivity and the frequency and distribution (in leak size) of false positives.