Network Working Group Scott Bradner Internet-Draft Harvard University Allison Mankin USC/ISI Vern Paxson ACIRI February 2000 Advancement of metrics specifications on the IETF Standards Track 1. Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html A revised version of this draft document will be submitted to the RFC editor as a BCP (Best Current Practice) documenting an IESG procedure for the Internet Community. Discussion and suggestions for improvement are requested. This document will expire before July, 2001. Distribution of this draft is unlimited. 2. Abstract The Internet Standards Process [RFC2026] requires that all IETF Standards Track specifications must have "multiple, independent, and interoperable implementations" before they can be advanced beyond Proposed Standard status. This document specifies the test which the Bradner, Mankin & Paxson [Page 1] Internet-Draft Metrics Specification Advancement February 2000 IESG will use to determine if a metrics specification document meets these requirements. It also discusses the rationale for this process. 3. The Nature of the Problem The Internet Standards Process [RFC2026] requires that for a IETF specification to advance beyond the Proposed Standard level, at least two genetically unrelated implementations must be shown to interoperate correctly with all features and options. There are two distinct reasons for this requirement. The first reason is to verify that the text of the specification is adequately clear and accurate. This is demonstrated by showing that multiple implementation efforts have used the specification to achieved interoperable implementations. The second reason is to discourage excessive options and "feature creep". This is accomplished by requiring interoperable implementation of all features, including options. If an option is not included in at least two different interoperable implementations, it is safe to assume that it has not been deemed useful and must be removed before the specification can advance. In the case of a protocol specification which specifies the "bits on the wire" exchanged by executing state machines, the notion of "interoperability" is reasonably intuitive - the implementations must successfully "talk to each other", exchanging "bits on the wire", while exercising all features and options. In the case of a specification for a performance metric, network latency for example, exactly what constitutes "interoperation" is less obvious. This document specifies how the IESG has decided to judge "metric specification interoperability" in the context of the IETF Standards Process. The aim is to ensure that the dual goals of specification clarity and feature evaluation have been met using an interpretation of the concept of metric specification interoperability that strikes a balance between testing complexity and practicality. 4. On The Nature of metric specifications Compared to "state machine" protocols which focus on procedural specifications, a metric specification describes a method of testing and a way to report the results of this testing. One example of such a metric would be a way to test and report the latency that data packets would incur while being sent from one network location to Bradner, Mankin & Paxson [Page 2] Internet-Draft Metrics Specification Advancement February 2000 another. Since implementations of testing metrics are by their nature stand- alone and do not interact with each other, the level of the interoperability called for in the IETF standards process cannot be simply determined by seeing that the implementations interact properly. 5. Discussion and Recommended Process In order to meet their obligations under the IETF Standards Process the IESG must be convinced that each metric specification advanced to Draft Standard or Internet Standard status is clearly written, that there are the required multiple interoperable implementations, and that all options have been implemented. There may be multiple ways to achieve this goal; this memo documents the way that the IESG will use to determine if the requirements have been met. In the context of this memo, metrics are designed to measure some characteristic of a data network. An aim of any metric definition should be that it should be specified in a way that can reliably measure the specific characteristic in a repeatable way. Thus running the test a number of times on a stable network should produce essentially the same results. In the same way, sequentially running different implementations of software that perform the tests described in the metric document on a stable network, or simultaneously on a network that may or may not be stable should produce essentially the same results. Following these assumptions any recommendation for the advancement of a metric specification must be accompanied by an implementation report, as is the case with all requests for the advancement of IETF specifications. The implementation report must include reports of tests performed between sets of points on a network with two or more implementations of the software. The tests MAY be sequential on the same or different hardware, with each implementation being run after the previous one finishes, or they MAY be run in parallel with multiple implementations each on different hardware. It is RECOMMENDED that the tests be run not in deterministic order, but instead by executing a randomizing algorithm on the schedule, and interspersing tests from the several implementations at randomly selected times until all tests of all implementations are complete. This is a way of avoiding a biased result in relation to the network conditions and other factors while making the comparative tests. All of the tests for each set should be run in the same direction between the same two points on the same network. The tests should be Bradner, Mankin & Paxson [Page 3] Internet-Draft Metrics Specification Advancement February 2000 run simultaneously unless the network is stable enough to ensure that the path the data takes through the network will not change between tests. The tests should be run multiple times and the results compared and the mean and variance determined. The results of the tests using different implementations of the testing software must be within the same variance as the results obtained from multiple executions of the same software. In particular, if the variance using Method A is sigma^2, then the value yielded by Method B should fall within 2*sigma of the mean value reported by Method A at least ### % of the time (The percentage value will be filled in during a discussion of statistics and metrics in the upcoming IPPM meeting). Note that if Method A and Method B are identical, and the value they are estimating is distributed according to a Gaussian distribution, then we would expect that Method A's estimates would fall within 2*sigma of its mean value ### % of the time. We allow more deviation in recognition of the many pragmatic (and sometimes systemic) difficulties in realizing consistent network measurement, and the fact that many quantities related to networking have distributions quite different from Gaussian. In addition, we explicitly recognize the IESG's prerogative to relax the restriction of ### % within 2*sigma in light of the particular measurement factors and difficulties, providing these are expressed (in a communication with the relevant working group or document author) by the IESG. If the metric has options, all of the options must be tested in the same way. For example, if the metric includes a list of packet sizes that can be used, all of the listed sizes should be tested. If some of the options are not supported or tested they must be removed from the specification before the specification can be advanced on the standards track. As stated above, the measurements are made over a set of points in a network. The Area Director(s), in consultation with the working group chairs (if applicable), will recommend to the IESG their judgment as to the adequacy of the set of measurement points in covering the range of relevant network conditions. An implementation report is required for both the advancement from Proposed Standard to Draft Standard and from Draft Standard to Internet Standard. The implementation report for advancement from Draft Standard to Internet Standard can be an updated version of the one used for the advancement from Proposed Standard to Draft Standard. The prime concern of the IESG will be that the underlying reasons for the interoperable implementations are met, i.e. that the text of the Bradner, Mankin & Paxson [Page 4] Internet-Draft Metrics Specification Advancement February 2000 specification is clear and unambiguous, and that features of the specification which have not garnered support have been removed. The implementation report will be placed on the IETF web page along with the other pre-advancement implementation reports and will be specifically referred to in the IETF Last-Call. As with all such implementation reports, the determination of adequacy is made by the IESG upon recommendation by the Area Director(s) of the relevant IETF Area. This determination of adequacy can be challenged during the Last-Call period. 6. Security Considerations Some may view this policy as possibly leading to a reduction in the level of confidence people can have in metrics specifications, but the IESG feels that it will adequately ensure a reasonable evaluation of the level of clarity and ensure that unused options can be identified and removed before the advancement of a specification. Good, clearly written metric specifications can be of great assistance in the measurement and management of the Internet and thus assist in the reduction of some types of security threats. 8. Acknowledgements The basic format and some of the text for this memo came from [RFC2438], "Advancement of MIB specifications on the IETF Standards Track", which provides similar guidance for the advancement of MIBs. 8. References [RFC2026] "The Internet Standards Process -- Revision 3", Bradner, October 1996 [RFC2438] "Advancement of MIB specifications on the IETF Standards Track", O'Dell, Alvestrand, Wijnen, & Bradner, October 1998 9. Author's Addresses Scott Bradner Harvard University 29 Oxford St. Cambridge MA 02138 Email: sob@harvard.edu Phone: +1-617-495-3864 Bradner, Mankin & Paxson [Page 5] Internet-Draft Metrics Specification Advancement February 2000 Allison Mankin USC/ISI 4350 N. Fairfax Drive, Suite 620 Arlington VA 22203 Email: mankin@isi.edu Phone: +1-703-812-3706 Vern Paxson ACIRI 1947 Center St., Suite 600, Berkeley, CA 94704-1198 USA Email: vern@aciri.org Bradner, Mankin & Paxson [Page 6]