InterNIC Midterm Evaluation and Recommendations A Panel Report to the National Science Foundation December 1994 Prue Adler Assistant Executive Director Federal Relations and Information Policy Association of Research Libraries Kilnam Chon Professor Korea Advanced Institute of Science and Technology Daniel P. Dern Internet Analyst Alan Emtage V.P. for Research and Development Bunyip Information Systems, Inc. Peter S. Ford Member of the Technical Staff Los Alamos National Laboratory Joan C. Gargano Director, Distributed Computing Analysis and Support Information Technology University of California, Davis Martyne M. Hallgren Executive Director GateDaemon Consortium, CU-SeeMe Consortium Cornell University Anthony C. Hearn Resident Scholar for Information Sciences The RAND Corporation Ellen S. Hoffman Development Director Merit Network, Inc./University of Michigan Karl Jacob Chief Technical Officer On Ramp, Inc. Michael Mealling Network Information Coordinator Office of Information Technology Georgia Institute of Technology Barbara Morgan Director, Strategic Technology Planning University of California at Berkeley Mark Resmer Associate Vice President Information Technology Sonoma State University Anthony M. Rutkowski Executive Director Internet Society Gleason Sackman Coordinator, SENDIT North Dakota's K-12 Telecom Network Michael F. Schwartz (Panel Chair and Report Editor) Associate Professor Computer Science Department University of Colorado at Boulder Contents 1 Executive Summary : : : : : : : : : : : : : : : : : : : : : : : : : 2 2 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5 2.1 Background : : : : : : : : : : : : : : : : : : : : : : : : : 5 2.2 Mechanics of the Review Process : : : : : : : : : : : : : : : 5 2.3 Capsule Summary of Awardees' Agreed-Upon Services : : : : : : 6 2.4 Official Status of this Report : : : : : : : : : : : : : : : 8 3 Changes in the Internet Since the Solicitation : : : : : : : : : : 9 4 InterNIC Directory and Database Services - AT&T : : : : : : : : : : : 12 4.1 Directory of Directories (Yellow Pages) : : : : : : : : : : : 12 4.2 White Pages : : : : : : : : : : : : : : : : : : : : : : : : : 13 4.3 Database Services : : : : : : : : : : : : : : : : : : : : : : 14 4.4 Administration of Services : : : : : : : : : : : : : : : : : 14 5 InterNIC Information Services - General Atomics : : : : : : : : : : 15 5.1 Overview : : : : : : : : : : : : : : : : : : : : : : : : : : : 15 5.2 Overall Management : : : : : : : : : : : : : : : : : : : : : 15 5.3 Service Evaluation : : : : : : : : : : : : : : : : : : : : : : 17 5.4 IS Self Evaluation : : : : : : : : : : : : : : : : : : : : : 22 5.5 Evaluation Summary and Recommendations For Actions to the NSF : : : : : : : : : : : : : : : : : : : : : : : : : 22 6 InterNIC Registration Services - Network Solutions, Inc. : : : : : : 28 6.1 Service Overview : : : : : : : : : : : : : : : : : : : : : : 28 6.2 Evaluation and Recommendations : : : : : : : : : : : : : : : 28 7 NSF Meta-Questions : : : : : : : : : : : : : : : : : : : : : : : : : 31 7.1 Overview : : : : : : : : : : : : : : : : : : : : : : : : : : 31 7.2 NSF Management of the Cooperative Agreements : : : : : : : : 32 7.3 Requirements for Ongoing ``Customer'' Feedback : : : : : : : 33 7.4 Appropriateness of Informal Relationship between the InterNIC and CNIDR : : : : : : : : : : : : : : : : : 34 7.5 The Viability of the NIS Manager Concept : : : : : : : : : : 34 A InterNIC DS's Responses to Panel Questions : : : : : : : : : : : : 36 B InterNIC IS's Responses to Panel Questions : : : : : : : : : : : : : 41 C InterNIC RS's Responses to Panel Questions : : : : : : : : : : : : 47 D List of Acronyms : : : : : : : : : : : : : : : : : : : : : : : : : : 50 1 1 Executive Summary In 1992, the National Science Foundation (NSF) awarded three five year cooperative agreements, to American Telephone and Telegraph (AT&T) Company, General Atomics (GA), and Network Solutions, Inc. (NSI), to provide a Network Information Services Manager supporting Directory and Database Services, Information Services, and non-military Registration Services (respectively) for the NSFNET. The awardees adopted the name InterNIC for their joint role. The solicitation included provisions for an external review after 24 months of operation, which resulted in the current report. This report provides an evaluation to the NSF about the performance of the awardees, and a set of recommendations to the awardees about how they might better meet the needs of the U.S. Research and Education community during the remainder of the award period. The review was based on how well the awardees have provided their agreed-upon services, rather than on some other (possibly updated) characterization of the appropriate role of a Network Information Services Manager. The report presents recommendations from a review panel convened by the NSF to represent a broad range of expertise and interests. In the evaluation of the Database and Directory Services InterNIC component, the panel found that AT&T has provided all of the services specified in their cooperative agreement, and has performed admirably given the rapidly changing nature of information tools that have been introduced since the start of the InterNIC. The panel's critiques of AT&T's InterNIC component concerned insufficient community outreach to populate their directory-of-directories, and the use of software that is difficult to use during searches and registrations of both white and yellow pages services. The panel's recommendations for AT&T concern the longer-term appropriateness of a centralized directory-of-directories model, and the need to focus their white pages service efforts on fostering widespread adoption of distributed user white pages technology. In its evaluation of the Information Services InterNIC component, the panel found that GA has lacked leadership and direction, and has not satisfactorily provided most of the services specified in their cooperative agreement. The panel notes that GA has provided a small number of very good services, primarily as the result of two specific individuals (Susan Calcari and Kent England), who have accomplished their work apparently despite the management of the project rather than because of it. The 2 project's alarming rate of turnover, recent reorganization (placing into power what the panel perceived as a manager with inappropriate vision and insufficient community awareness), and inappropriate prioritization of its various efforts left the panel feeling that the current arrangement should not be continued. The panel had a split of opinions on what to recommend to the NSF. The majority (13 of 16 panelists) felt that the cooperative agreement with GA for the provision of IS should be terminated as soon as possible, and that the NSF should find a way to continue to support specific projects that the panel deemed as important and viable. Three minority opinions arose, two of which suggested continuing funding GA for portions of the project (with disagreement about which portions to continue), and one suggesting that the NSF continue funding a somewhat reduced set of functions over a probationary period, after which time another review be held to assess the situation. While the panel disagreed on these recommendations, there was general agreement that the NSF should continue to support the InfoScout, in the person of Susan Calcari. In its evaluation of the Registration Services InterNIC component, the panel found that NSI has provided an excellent service in the face of exponential growth in demand, and that they have adapted to this demand appropriately by delegating and distributing the registration activities in most cases. The panel also recognized NSI's success in fostering internationalization of RS functions, as well as the Information Sciences Institute's success in administering .US domain registrations as a subaward under NSI. The panel's recommendations included continuing to move the WHOIS legacy database function to more appropriate distributed technology, and moving to automate registration processing as much as possible. The panel also recommended that NSI begin charging for .COM domain name registrations (which pose the most significant growth problems) as soon as possible, and eventually charge for all IP registration services. The panel further recommended that the NSF be prepared to allocate more resources to NSI to help them meet their increasing work load. Finally, the panel encouraged the NSI and the NSF to develop suitable legal protection of the registration function, in response to NSI's experiences with companies' asserting trade and service mark claims in .COM domain name registrations. As part of the review process, the NSF asked the panel to answer a set of ``meta-questions'' about the viability of the Network Information Services Manager concept. The panel felt that 3 successful InterNIC services are important and should be continued, but that the current efforts are insufficient and in fact that no single program could meet the full spectrum of needs. The panel suggested in particular that the NSF consider ways of building Internet information services from both the top down and the bottom up, similar to how it so successfully helped to develop the physical network infrastructure. The panel also suggested that the NSF consider more focused awards of a shorter duration, and that NSF require specific methods for user feedback rather that the unsolicited individual user electronic mail offered by the InterNIC. The panel was also asked to consider the appropriateness of the informal relationship between the InterNIC and the Clearinghouse for Networked Information Discovery and Retrieval (CNIDR). The panel felt that the liaison has been beneficial (particularly for InterNIC Directory and Database Services) and should be continued, but that it is not advisable to complicate organizational relationships with formal inclusion of CNIDR in the InterNIC. Given GA's problems, the panel felt it important to consider the NSF's oversight role in the project. 1 The panel felt that the NSF has been more successful in its work to develop the physical network infrastructure than it has in developing Network Information Services. The InterNIC awards set the precedent of requiring significant self-coordination among a team of awardees, and requiring outreach to other Network Information Centers. The panel suggests that the NSF critically consider whether it is viable to expect significant self-coordination among a team of awardees in future awards. The panel also notes that the NSF's program management was not able to correct GA's problems early on despite excellent efforts by the NSF staff, primarily because the NSF staff were overextended by monitoring at least two major projects (the InterNIC and the NSFNET backbone) at once. The panel recommends that for future large scale efforts in the rapidly changing Internet environment, the NSF should form an ongoing advisory panel of outside experts or employ some external consultants to help manage such cooperative agreements, rather than waiting two years to call for a review. ____________________________ 1 Note: Peter Ford was asked to leave the room during this part of the panel discussions, because of his part-time consulting relationship with the NSF. 4 2 Introduction 2.1 Background In response to the rapid growth of the NSFNET and the expiration of the funding arrangement for the NSFNET National Service Center (NNSC), in the Spring of 1992 the U.S. National Science Foundation (NSF) issued Project Solicitation NSF92-24, calling for one or more Network Information Services Managers (NIS Manager(s)) to provide and/or coordinate (i) non-military Registration Services (RS), (ii) Directory and Database Services (DS), and (iii) Information Services (IS) for the NSFNET. The proposals were reviewed by an external panel in June 1992, after which time the NSF selected a set of awardees and negotiated five year cooperative agreements totaling over $12 million. The awards began January 1, 1993, and were required to reach full operation by April 1, 1993. The solicitation included provisions for an external review after 24 months of operation. The current report represents the evaluation and recommendations produced by that review. The purpose of the current review is twofold. First, the review should provide an evaluation to the NSF about how well the awardees have met their goals, and a set of recommendations about what should be done for the remaining 3 years of their cooperative agreements. Second, the review should provide recommendations to the awardees about changes that would better allow them to serve the U.S. Research and Education (R&E) community. 2.2 Mechanics of the Review Process The current review panel was selected by the NSF to represent a broad range of expertise and interests of the R&E community, but also included representation from the international community and the commercial sector. The membership of the current panel (hereafter referred to simply as ``the panel'') is mostly independent of that of the original proposal review panel, although for the sake of programmatic continuity there was some overlap between the panels. The panel met November 16-17, 1994, after having received and reviewed a set of materials from each of the awardees. During these two days the panel heard presentations from each of the awardees. After each presentation, the panel held a closed session, discussed the presentations, and constructed a set of 5 questions for the awardees to answer. These questions and answers are contained in appendices of this report. The current report resulted from detailed discussions among the panelists both during and after the meeting. Because the awardees were selected by a prior review process that considered how well the proposal fit the solicited functions, the panel decided that the evaluation should consider only how well the awardees provided the services specified in their cooperative agreements (outlined in Section 2.3 below), without further analyzing how well it met the goals of the original solicitation. A complicating factor in the review process is the fact that the Internet community, hardware, and software technology have evolved considerably since the time of the initial award. The panel decided that the evaluation should only consider these changes when the awardees needed to adapt in order to provide their agreed-upon services. In addition, the panel considered Internet changes in its recommendations to the awardees about changes that would better allow them to serve the R&E community. 2.3 Capsule Summary of Awardees' Agreed-Upon Services In brief, the awardees under review agreed to provide the following services in their negotiated cooperative agreements: o American Telephone and Telegraph (AT&T) Company provides the DS portion of the award. o General Atomics (GA) provides the IS portion of the award. o Network Solutions, Inc. (NSI) provides the RS portion of the award. Together, these three awardees adopted the name ``InterNIC''. Below we provide excerpts from a press release issued jointly by the InterNIC awardees in January 1993, primarily to provide the reader a brief sense of the InterNIC's goals. A complete list of goals is listed in the original proposal, available from the NSF. In the remainder of this report we assume the reader has read the proposal and hence is familiar with the services the InterNIC was responsible for offering, and the motivations for these services. AT&T agreed to: 6 o ``... develop and maintain a Directory of Directories, including lists of FTP (File Transfer Protocol) sites, lists of various types of servers available on the Internet, lists of white and yellow page directories, library catalogs and data archives.'' o ``... provide white and yellow pages type Directory Services. Access to these services will initially be provided through several currently popular in-use interface methods while migrating to the use of X.500 technology...'' o ``... [provide] database services, [including] the establishment of Database Services to extend and supplement the resources of the NSFNET, such as databases of contributed materials of common interest to the user community.'' o ``... offer database design, management, and maintenance to institutions and groups for inclusion in the Internet.'' GA agreed to: o ``...provide Information Services acting as the Network Information Center (NIC) of first and last resort and the NIC of NICs.'' o ``... include a full-service Reference Desk, a database of comprehensive networking materials called the Info Source, training classes and documentation, and coordination services among all appropriate groups in the community.'' o ``... [implement] several new approaches to distributing services [including] NICLink, a user-friendly hypermedia interface offering access to the Info Source and all the information it contains... and the Info Scout, an individual who will scout out new resources and innovative uses of the network for inclusion in the Info Source.'' In addition, although it was not included in the press release, GA had agreed to coordinate activities among the InterNIC partners. NSI agreed to: o ``... provide registration services as the IP registrar.'' 7 o ``issue IP numbers worldwide using delegated registries under the guidance of the Internet Assigned Numbers Authority.'' o ``register domain names, and track points of contact.'' o ``... [provide the information from these assignments] to the directory and database services provider to be made available to the entire Internet community.'' o ``periodically release the top level zone files to be used by all root Domain Name servers.'' 2.4 Official Status of this Report This report presents a set of panel recommendations. The National Science Foundation may choose to implement the recommendations operationally as it deems appropriate, and in the best interests of the program. 8 3 Changes in the Internet Since the Solicitation The Internet community, hardware, and software technology have evolved considerably since the time of the initial NIS Manager award. In this section we summarize some of the more salient changes, as background for recommendations to the awardees about changes that might better allow them to serve the R&E community in the future. The most significant change in the physical architecture of the Internet is the move from an NSF-funded backbone network providing connectivity between regional and national Internet service providers, to NSF-funded interconnection points for these same providers. The current NSF strategy is driven by the rapid growth in numbers of commercial Internet service providers in response to the rapid growth of the Internet market. The good news for the R&E portion of the Internet is that this growth has resulted in growth in the quantity and quality of available NIS's. There are for-profit and non-profit organizations offering IP connectivity services, and most have NIC or marketing staffs. There are more specialized NICs than ever before. Both the U.S. National Aeronautics and Space Administration (NASA) and the U.S. Department of Energy (DOE) continue to expand their own NICs. Others are just starting up. The number of commercial organizations providing training, documentation, consulting, or delivering some product via network information tools is rapidly increasing, fueling a highly competitive market place. Universities are examining new support models for help desks and consulting groups as applications focus on distributed computing and departments hire internal support people. Funding agencies such as the NSF and the Department of Education require ``support'' as part of approved proposals. Groups who do not offer connectivity services have also entered the NIS world. These groups, to name just a few, include public libraries and community groups such as SIG-WEB. SIG-WEB is an independent special interest group based in the San Francisco area. It draws representatives from a diverse group of organizations including the telecommunications industry, education, government, universities, and public institutions. New tools such as Gopher, the World Wide Web (WWW), and Mosaic, give users control over access to their own information as well as to other resources on the network. These tools are evolving rapidly. At the time the InterNIC proposal was written, the WWW 9 was just becoming known in the technical community involved with developing Networked Information Discovery and Retrieval tools. Today the WWW is one of the most visible aspects of the Internet, and it would not make sense to run a NIC without offering a WWW interface. The number of forums and meetings focusing on NIS's have increased. The Internet Engineering Task Force (IETF) has an energetic User Services Area. FARNET has an annual meeting focused on NIS topics. SIGUCCS has a Networking Task Force, in addition to two annual services meetings. Regional networks have their own user conferences (for example, NYSERNet, NorthwestNet, and MORENet). The InterNIC offers training as well as a NICFest. There is the annual European Network Services Conference plus many other conferences in the U.S. Finally, over the past few years a large number of books, magazines, commercial courses, CD-ROMs, online materials, videos, and consultants focusing on the Internet have become available. For example, there are currently at least 150 Internet-related books available from commercial publishers, with anywhere from another 50 to 100 known to be upcoming. These changes are significant to the goals of the InterNIC in a variety of ways. The growing number of NICs and commercial training/consulting organizations implies an increasing need for InterNIC outreach activities. Rapid change in networked information tools requires the InterNIC to upgrade their tool base regularly, both to provide more capable service and so that the InterNIC will not be viewed as ``outdated''. The increasing number of NIS-related forums implies that the InterNIC needs to target their outreach efforts carefully to reach those audiences that are most appropriate and receptive. Finally, the increasing number of Internet-oriented books and related materials implies that the InterNIC should not be in the business of creating such content. As will be discussed in this report, the InterNIC has adapted to these changes to varying degrees of satisfaction. More generally, the continuing commercialization of the Internet has major repercussions for NIC services. On the one hand, there is more information available in many forums -- online, in print, and from individuals (both professional and volunteer). This information is not of uniformly high quality, and it is increasingly difficult for many users to assess the quality of the information they find. At the same time, the overall effect of the changing Internet is a greater reluctance to share information because of its perceived commercial value. This also 10 means that there is a greater need for the InterNIC to help the flow of accurate information, and in particular to assist the R&E community, which cannot always rely on commercial interests to meet their specific requirements or to be objective in their assessments. This is a marked change from the period in which the solicitation was issued, when a smaller Internet community had a greater degree of shared value for such cooperative efforts and a greater focus on R&E requirements. The rapid growth of the Internet beyond the R&E community means that the InterNIC needs to be constantly reassessing its primary mission and the best ways to meet the challenges raised by the rapidly changing environment. 11 4 InterNIC Directory and Database Services - AT&T The panel congratulates AT&T on a job well done. AT&T has provided each of the services defined in their cooperative agreement, contributed some innovative technical solutions in the areas of yellow pages and white pages, and made good use of Internet information tools at a time of rapid change in these technologies. We feel the Directory and Database Services portion of the InterNIC provides a useful service to the R&E community. We provide critiques and suggestions about each of the DS components below. 4.1 Directory of Directories (Yellow Pages) The panel feels that AT&T has built the current incarnation of this service using appropriate technology (WWW and WAIS). We offer critiques in two areas: the shorter-term problem of the size and usefulness of the database, and a longer-term issue of the appropriateness of the current model. At present, the service receives 4,000 queries per day. While significant, this query rate is an order of magnitude lower than the typical query rates of other well-known, widely popular networked information services. We believe there are two main causes. First, although the underlying technology is reasonably functional and stable, the interface is difficult to use. The user is given limited instructions on how to build a search, of the form ``You may use compound searches, such as ...'' The panel recommends enumerating the possible types of compound statements, and providing some hints on how to improve searches when a search fails. It would also be useful to provide limits on the number of hits returned, and to allow users to narrow searches interactively. Second, because the service contains only 700 entries, it does not offer enough useful data to attract widespread use. We believe this paucity of entries can be traced to the difficulty of entering a directory into the service. At present users must read a lengthy explanation and then send an electronic mail message to enter data. The panel recommends that AT&T reduce the difficulty of entering information into the database, by providing a WWW form as an option for entering the data, and by reducing the amount of information that users must enter. 12 The longer-term issue we wish to raise is the appropriateness of a centralized directory-of-directories model. While a centralized database provides a convenient locus for registrations and queries, as the R&E community deploys increasing numbers of databases, this approach will have scaling problems. Moreover, even with the official status afforded by being the InterNIC DS, there is no reason why a particular directory-of-directories should be expected to become widely accepted as the authoritative source for such information. Indeed, there are many other projects providing such functionality, including a variety of distributed indexing services and many WWW pages that provide comprehensive lists of pointers to databases around the Internet. The indications are that such directories will become increasingly numerous over time, particularly given the increasingly commercial nature of the Internet, and the value to be provided by offering such a service. The panel recommends that AT&T again consider a more distributed approach to providing the directory of directories function, by continuing to work with R&D efforts under way in the IETF and elsewhere. 4.2 White Pages To date AT&T has provided all of the white pages services specified in its cooperative agreement on machines located within company premises. While this approach has worked satisfactorily to date, over time scaling problems will demand a much more distributed solution. We recommend that during the remainder of the cooperative agreement AT&T focus its white pages service efforts primarily on the increasingly critical task of building a distributed white pages directory of users, with the goal of making every U.S. R&E user accessible by name, organization and discipline. Such a service should deal gracefully with the situation where particular queries may result in thousands of hits, allowing users to narrow the search scope iteratively. Along with this technology, AT&T should establish a policy regarding organizations that do not want to provide their information or that only wish to provide partial information. We recommend that AT&T work with other groups in establishing this policy, including the North American Directory Forum (NADF) and the IETF. In order to help foster the widespread adoption of appropriate distributed white pages technology, we recommend that AT&T 13 identify sites that have successfully implemented campus-wide white pages services (such as the University of Michigan's X.500 LDAP deployment and the University of Illinois' CCSO system) and then work with the R&D community (and possibly with commercial service providers) to develop software distributions that will permit sites to install user white pages services with modest effort, and that allow interoperation among several popular white pages systems. The panel feels that AT&T should exploit its position as InterNIC DS provider to foster widespread adoption of such important technologies in the U.S. R&E community, and that if implemented well these services could spread to commercial services, such as AOL and CompuServe, and gain momentum similar to the way Gopher and WWW have. Part of the effort needed to make this happen is to develop tools that allow sites to populate a site's white pages database from information available in other common database formats. Doing this will require some background research to determine, for example, what database systems are in common use for university student registration systems, and then developing and packaging the needed conversion software. The other critical component is to focus on outreach, rather than on providing white pages services on centralized machines located within AT&T premises. 4.3 Database Services AT&T has provided the database services specified in their cooperative agreement, with inputs from the NSF program management. At the review meeting, AT&T agreed that this element of the offered services was of a lower priority than the Directory Services element. The panel sees a continued need for this service in order to support databases important to parts of the R&E community that lack resources to host such databases. The panel wishes to reinforce the intention that these databases be given permanent homes elsewhere as soon as it becomes feasible. 4.4 Administration of Services The panel commends AT&T for the excellent work in the administration of their services. The availability of each of the servers is exceptional in the light of dealing with technologies that are not completely stable. The rapidly changing landscape of Internet information tools makes AT&T's achievements even more noteworthy. 14 5 InterNIC Information Services - General Atomics 5.1 Overview ``The successful provision of information services during the next five years of explosive growth will depend on several important factors. These include the ability to stay in touch with the community and recognize important service needs and future trends; the ability to remain flexible and adapt quickly to changes in the network offerings and in the user base; the successful implementation of distributed services to the midlevel and campus NICs and to end users; and the establishment of a strong leadership role in the community.'' Network Information Services Manager for NSFNET and NREN Information Services, California Education and Research Federation Network May 1992 While a small number of the services developed by GA for the InterNIC have contributed to the usability of the Internet, they have not adequately provided the majority of services specified in their cooperative agreement. It appears that many of the failures can be traced to problems associated with the management of this part of the InterNIC by GA. Conversely, the successes appear to be the result of specific individuals on the staff, notably Susan Calcari and Kent England, who have accomplished their work apparently despite the management of the project rather than because of it. 5.2 Overall Management The failure of the Network Information Services group to meet the requirements of the cooperative agreement can be directly related to the project management provided by GA. The loss of one of the original Principal Investigators created a major disruption in the initial progress of meeting the cooperative agreement requirements. It created an ongoing lack of continuity in direction that was not resolved by the remaining co-principal investigator. It took eight months to install a new PI (Kent England) to provide the needed leadership. He has made good 15 progress in identifying problem areas and refocusing the group on activities in which they have a competence and strength. Even so, England's time and attention were only partially focused on the InterNIC project, as he was also responsible for the management of CERFNet. Unfortunately, several of the areas are still not being addressed sufficiently, and GA has decided to reorganize the IS group and again install new management. In attempting to assess the prospects for ongoing success under the new management of Karsten Blue, the blandness and uninformed quality of the strategic mission for the project at the senior management level is quite discouraging. To provide information about the Internet to the American public as the entire strategic mission statement for the project is hardly sufficient, and speaks to the limited vision of the new management. This comes at a time when the group has barely begun to reestablish itself in areas of core competencies, and has not even had the chance to make progress in new areas. The new management has no experience in providing network information services, and through the panel review process, the new management did not demonstrate a comprehension of the current Internet environment or a vision as to how the IS should evolve and be provided during this evolution. Considering the rate of change in Internet growth and service requirements, this lack of experience or leadership capabilities in this area severely compromise the ability of GA to perform at an acceptable level, let alone improve in areas that need attention. The continuing, alarmingly high rate of personnel turnover throughout the project team suggests that whatever factors are behind these problems remain unresolved. It is not clear how GA intends to address the causes of staff attrition, and thus whether the company is in a position to provide a stable environment within which the project could flourish. Associated with this issue is a concern that Kent England's departure leaves the quality of technical and strategic leadership of the project unknown. GA has positioned the IS team in the role of direct provider of all services, rather than a promoter and facilitator of designated services by others. Not only has this caused the InterNIC to compete with other service providers in such areas as the delivery of end-user workshops, but it has also obstructed one of the primary functions envisaged in the original proposal, namely the coordination of services. Support to other NICs has been relatively scant, the InterNIC Briefcase being one of the few tangible products in this area, and one that was only 16 produced very recently. Finally, the general approach taken in the management of the project appears to have been excessively reactive, and evidenced poor planning at various times. While some aspects of the evolution of the Internet have been extremely unpredictable, this is not true of relatively generic business activities where the project has experienced problems. In particular, it should have been possible to predict and take steps to avoid problems with such areas as book and CD-ROM distribution, and overloading of the 800 public-access number. The panel's Overall Quality evaluations are provided in Table 1. ____________________________________________________________________ | Cooperative Agreement Requirement | Implemented | Quality | -------------------------------------------------------------------- | Recording and tracking | yes | poor | | Assigning Internal Quality scores | no | unacceptable | | InterNIC Liaison Council | no | unacceptable | -------------------------------------------------------------------- Table 1: Overall IS Quality Evaluation 5.3 Service Evaluation Community Coordination GA has made some effort to act in a coordinating role for the other NICs, but there has been too much of an emphasis on providing NIC services directly to end users. Moreover, much of the interaction with other NICs has become information sharing rather than coordination. Under the leadership of Kent England, in the last six months GA has recognized this problem and started to take action to reorient their services. The creation of the InterNIC Briefcase was the first step in this direction. But the InterNIC Briefcase is only one step in the development of documentation, which should include much needed policies and procedures relating to network use. The value of such documentation and policies is obvious and remains an important function for someone to provide. NICs and all users would benefit from a forum within which such policies could be discussed, developed, and disseminated to the Internet community. Unfortunately, however, given the highly competitive nature of information services, the opportunity to create a web of support organizations and shared resources has been missed. The current GA organization is seen as a competitor by many service groups, 17 and these groups see no advantage to cooperate with GA. The opportunity to work with University NICs or specific disciplines may still open if the benefits can be clearly outlined; however the balance between working to strengthen the overall information infrastructure and delivering commercial services is delicate. Uninformed leadership at this stage could easily stifle this coordination opportunity. Not creating the NIC Liaison Council at the outset was a serious departure from the cooperative agreement, and significantly hampered the effectiveness of the InterNIC in its early stages. Kent England's decision not to convene it during his tenure was not made lightly but clearly speaks of GA's inability to understand the role of this council, either as a coordination/outreach tool or as a mechanism for leadership and evaluation of the project and the services it offered. The panel's Community Coordination evaluations are provided in Table 2. The panel's Prepared Material evaluations are provided in Table 3. ____________________________________________________________________ | Cooperative Agreement Requirement | Implemented | Quality | -------------------------------------------------------------------- | Community Coordination Services | yes | poor | | NIS Fest | yes | unacceptable | | Liaison Council | no | unacceptable | | International Cooperation | unclear | unreported | | Administrative Representation | yes | acceptable | | Educational Services | yes | poor | | Documentation Services | yes | poor | -------------------------------------------------------------------- Table 2: Community Coordination Evaluations (outreach to other NICS and organizations) InterNIC Partner Coordination One of the primary functions of GA within the InterNIC project was to serve as the public communications and coordinating arm of the enterprise. In this capacity, GA was ultimately responsible for the visibility of the InterNIC as a whole. Unfortunately, GA failed to provide visibility for its services or for its partners' services within the R&E community, and at the same time created inappropriate visibility within constituencies outside the primary target audience. GA is responsible for creating a seamless interface between the three InterNIC partners. An important component of InterNIC 18 coordination is a single point of contact for InterNIC services, using any of the forms of electronic communication. The ``800'' number concept failed due to the huge demand by unaffiliated users (i.e., the general public). GA has eliminated the 800 number and is instituting an automated telephone attendant that provides information on reaching the other InterNIC partners. The automated attendant service does not route calls to the other partners, which leaves open to debate the question about whether this service provides an adequate single point of contact. A common directory and database service is also critical to a unified InterNIC appearance. The current configuration of the database and directory service makes all of the data available through multiple access mechanisms, but the data are neither well organized nor easily searchable, and there is considerable duplication of information on the servers operated by the various partners. A great deal of work is needed to evaluate the usefulness of this system and to reorganize it to provide a high quality service. As an example of the problems, the three awardee sites have provided dissimilar electronic mail server interfaces -- requiring user commands in message bodies for IS and RS, but in message subjects for DS. Outreach and training is the area that seems to have suffered the most neglect, and is impacting all of the service providers. Outreach coupled with a communication plan might also be called market research. IS should contact customers to discover how they use services, what their current and future needs are, and what needs improvement. Simply relying on unsolicited comments is not sufficient. The outreach activities should include interviews, focus groups and measured uses of tools and services. Without these activities, the InterNIC can not hope to be responsive to the community it is charged to serve. The panel's Partner Coordination evaluations are provided in Table 4. Full Service Reference Desk The full service reference desk is a good idea for clearly delineated communities. GA made a good faith effort to provide these services to the R&E community. However, they have recognized that too much time was spent focused on supporting the end users. Eliminating the 800 number was an appropriate way to ensure that serving as NIC of first resort does not completely swamp available resources. The reference desk does not appear to have served the needs of 19 other NICs. While a mid-course correction to focus more on the provision of services to NICs has resulted in one tangible product (the InterNIC Briefcase), the reference desk itself has not been modified. In addition, the changes that have occurred in the Internet over the last three years call into question the desirability or feasibility of the model of a hierarchy of NICs. The InfoGuide provides very good online resources for the beginner, and is starting to build a good set of resources for the experienced user. The InfoScout, in the person of Susan Calcari, is the reason the InfoGuide and the Scout Report are successful. However, the creation of the InfoGuide for new users creates a problem of delivering that information to new users. Moving to entirely electronic networked information delivery creates a hurdle for new users who are not on the network, and may not address the needs of some users who need paper documents. While it may be appropriate for the InterNIC to create such materials, it is not appropriate for the InterNIC to distribute them directly to end-users. And given the recent explosion in publishing on the Internet, even the creation of such documents is again in question. The InfoScout relies on the net-happenings mailing list for information about current events on the Internet. While GA has provided listserve resources, net-happenings was started on the personal initiative of Gleason Sackman and is not a GA project 2. There would be severe damage to the InfoScout's ability to continue to do its good job if they did not have this valuable resource from which to build. The NICLink and X window whiteboard service are also interesting ideas that have not survived implementation. Even so, the panel recognizes that risks must be taken to create new services. The InfoScout service could have been overshadowed by an insurmountable workload; luckily, it has not. Good management is the key to learning from mistakes and ensuring the balance of the outcome leads to progress. The panel's Full Service Reference Desk evaluations are provided in Table 5. Education Training is another area where lack of focus or understanding the needs of the target audience clearly shows. Many of the ____________________________ 2 Sackman has no association (financial or otherwise) with the InterNIC, other than that the InterNIC provides the listserv software, disk space, and technical support for the net-happenings list. 20 educational seminars offered by GA have been directed at end users rather than at the staff of other NICs. They have entered the commercial marketplace by offering a seminar targeted at commercial firms. Education is one of principal areas in which GA has come to be perceived by many organizations as a competitor. The goal of providing high quality training services must start with the identification of the target audience and their needs, while recognizing that the commercial delivery of educational materials has burgeoned over the last three years. In addition, GA has missed an important and valuable opportunity to work with its partners to deliver training based on the Directory and Database Services of AT&T and the Registration Services of NSI. Both groups have clearly identified problems with their users that could have been addressed if GA had adequately fulfilled its coordination role. Outreach Services GA's outreach services suffer from the same lack of focus as previously described in various parts of this report. While they continue to participate in various network and educational forums and to produce and distribute the NSFNET Newsletter, the value of these efforts is hard to judge, given their ``shotgun'' approach to targeting services to audiences. While their presentations provide brief information on all three partners, they are barely achieving an adequate standard in presenting information on the InterNIC partnership. While NSI has received more visibility than AT&T through the popular press, the work of AT&T with directories and databases as well as their own development work has never been presented by GA. The R&E community largely has no idea of what AT&T is doing and thus perceives little value from them. This may be an added reason why AT&T is having difficulty getting entries in their directories. The NSFNET Newsletter continues to showcase information about NSFNET and associated projects, though it has not been widely enough disseminated to have served in any meaningful way as a communication tool beyond an already knowledgeable group. Net-happenings has been consistently excellent. Sackman's work in moderating this list has been widely recognized as one of the most useful digests of information on the Internet. 3 It should ____________________________ 3 Note: although Sackman is a member of the panel, he did not write the current section. 21 be noted that net-happenings is not a GA project, and is not a funded activity. The Internet Monthly report is subawarded to the University of Southern California's Information Sciences Institute (ISI) and is consistently delivered to the community via mailing lists and various Web servers. The panel's Community Outreach evaluations are provided in Table 6. 5.4 IS Self Evaluation There is little evidence to indicate that, other than monitoring such relatively minor items as reference desk contacts and server accesses, GA has made significant efforts to evaluate customer satisfaction with its services, as originally outlined in section J of the proposal. Informal communication with a number of NICs suggests that they typically either ignore the InterNIC Information Services entirely, or use them but have not had the opportunity for significant input into their design or delivery. It also appears that GA has not facilitated the Self-evaluation process for other NICs in the ways that were described in section I.3 of the proposal. 5.5 Evaluation Summary and Recommendations For Actions to the NSF While the panel was in general agreement that GA has not provided its agreed-upon services, the panel did not reach consensus on recommendations to the NSF. Below we provide the majority opinion (held by 13 of the 16 panelists) and three minority opinions (held by one panelist each). Majority Opinion Given the lack of leadership and direction shown in GA's overall management of the project (as seen in the alarming turnover of personnel and IS's recent reorganization), and given GA's inability to provide the services specified in their cooperative agreement, it is the majority opinion of the panel that the cooperative agreement with GA for the provision of IS should be terminated as soon as possible. Below we identify specific projects that this part of the panel feels the NSF should continue to support, as they have proven to 22 be valuable to the R&E community. We believe that the NSF should not, however, continue to fund GA in the provision of these services, as they are largely due to the efforts of two individuals despite the GA management rather than because of it. The following projects should be continued: o The InfoScout in the person of Susan Calcari. In particular, the Scout Report, based on information provided by net-happenings, is extremely valuable. Net-happenings is currently not funded by the NSF in any way. This service should be maintained, and expanded if necessary, if the volume of information becomes excessive for Gleason Sackman to handle it alone. o The NSFNET Newsletter has provided a record of the development of the NSFNET project since its beginning. Currently, the paper format is expensive and therefore its content and distribution should be targeted. The NSF should determine if this purpose continues to be valuable as they step out of the business of providing network services. While creating an electronic form of the magazine would allow it to be more widely distributed, there are several other magazines that could fill the information gap should this newsletter be discontinued. o The Internet Monthly Report is consistently delivered to the community electronically. The current subaward (at the University of Southern California's Information Sciences Institute) should be retained. o The NIC for NICs role should be re-evaluated. The InterNIC Briefcase, the project initiated by Kent England, should be completed and supported. Other projects of this nature should be more clearly defined in terms of target audience, delivery, and cost, and should be evaluated relative to whether the commercial marketplace can deliver the same services. The NSF should clearly identify the set of outcomes it wishes to facilitate through the development of NIC services within various distinct R&E community segments. o The information services available through InfoGuide are designed to appeal to new users, and serve as a significant source of information about the network. These services are valuable to portions of the R&E community, and should be maintained by AT&T. The InfoScout should continue to support 23 and maintain the information it provides. The InfoGuide may also serve as a distribution mechanism for the NSF. o Whatever projects the NSF continues to support, clear processes for evaluation should be implemented to assure that they continue to be valuable to a targeted R&E community. Minority Opinion #1 This panelist's overall recommendation is to continue funding GA for those portions of the project that are currently yielding results: InfoGuide, InfoScout, and further development on the InterNIC Briefcase. The panelist believes this would require 2-3 FTEs. The panelist thinks the remaining projects are not going anywhere and have already been given sufficient time to prove themselves. The notion of the IS portion of the award appears to be defunct. Minority Opinion #2 This panelist feels that the GA staff provides a high level of service in the research and development of online resources, but continues to struggle with all other aspects of the program requirements. The GA cooperative agreement should be reduced in scope to include only the following: o Common electronic identity. Maintain the single point of contact electronic mailing address for the InterNIC, reading mail and forwarding as required. o Common telephone identity. Maintain the single point of contact telephone number with an automated attendant and staff backup to forward calls as necessary. o InfoScout, InfoGuide and the InterNIC Mailing List. This is a very successful service. It should continue and build upon the current work, including the maintenance of current archives: ``How-to'' Templates, Resource Documents, Historical Archive, Network Security, and publication of the NSF Network News on the WWW and in limited hardcopy for distribution through outreach activities sponsored by R&E organizations. o InterNIC Newsletter and the Internet Monthly Report. 24 During the next year, the NSF should organize and sponsor focus groups of various constituencies (including commercial Internet service providers) to determine the need for NIC coordination activities, education and outreach to serve the R&E community. If this research determines that these services are still required, a clear strategy should be articulated for providing these services is a distributed fashion with explicit requirements for the measurement of service quality. Minority Opinion #3 This panelist feels that, given GA's mixed record of accomplishments as compared to their original plans; staffing turnover throughout the project; and uncertainties related to experience and goals of the new management, the NSF should not commit to full funding for the remaining years of the cooperative agreement at this time. Assuming a qualified co-principal investigator can be hired as InterNIC Director before the end of 1994, GA should receive some months of additional funding (perhaps through June 1995). A site visit and program review should be done in the Spring of 1995. Possibly some members of the 1994 InterNIC Review Panel could assist the NSF in that review. The NSF support should be reduced immediately to the support of a more modest Information and Education program, so that activities that are going well can continue. (These include InfoGuide, InfoScout, NSF Network News, and document templates, and may also include the Reference Desk and templates for courses.) The balance of services provided to the R&E community and to the general public needs to be clarified, relative to the NSF funding. The NSF should find some other appropriate group to tackle the InterNIC coordination and ``marketing'' roles, if those still seem desirable. 25 ____________________________________________________________________ | Cooperative Agreement Requirement | Implemented | Quality | -------------------------------------------------------------------- | Customizable Materials | one | poor | | (InterNIC Briefcase (new '94)) | | | | "How-to" Templates | yes | acceptable | | Resource Documents | yes | good | | Historical Archive | yes | acceptable | | Self Evaluation Guidelines | no | unacceptable | | Procedures and Policy Manual | no | acceptable | | (being done by other groups) | | | | Other documents | no | acceptable | | (being done by other groups) | | | -------------------------------------------------------------------- Table 3: Prepared Material Evaluations ____________________________________________________________________ | Cooperative Agreement Requirement | Implemented | Quality | -------------------------------------------------------------------- | InterNIC Coordination | partial | unacceptable | | InterNIC Transition | yes | acceptable | | Development of the unified | yes | poor | | InterNIC interface | | | | Common electronic identity | yes | acceptable | | Common telephone identity | yes | acceptable | | Common dir/dbase identity | yes | poor | | Outreach coordination | yes | poor | | Training coordination | partial | unacceptable | | Common Trouble Ticket System | no | unacceptable | | Communication Plan | no | unacceptable | | NIC Locator (new '94) | yes | poor | -------------------------------------------------------------------- Table 4: InterNIC Partner Coordination Evaluations 26 _____________________________________________________________________ | Cooperative Agreement Requirement | Implemented | Quality | ---------------------------------------------------------------------- | Full Service Reference Desk | yes | poor | | 800# (ended '94) | | unacceptable | | regular # + automatic referral | | acceptable | | InfoSource/InfoGuide | yes | good | | InfoScout | yes | excellent | | Scout Report (new '94) | yes | excellent | | NICLink | yes | poor execution | | Discipline specific info packets | no | unacceptable | | Internet User Guide | unclear | unacceptable | | InterNIC Mailing List | yes | poor | | NetHappenings | non-GA | excellent | | Smart Card | no | unacceptable | | X Windows whiteboard service | no | unacceptable | | Tracking queries | yes| | poor | | Common trouble ticket System | no | unacceptable | | Quarterly Reports | yes | poor | ---------------------------------------------------------------------- Table 5: Full Service Reference Desk Evaluations ______________________________________________________________________ | Cooperative Agreement Requirement | Implemented | Quality | ---------------------------------------------------------------------- | Outreach Services | yes | acceptable | | Training Seminars | yes | poor | | NSFNET Newsletter | yes | good | | Internet Monthly Report | yes | good | | Network Security | no | unacceptable | | Increasing Network Involvement (K-12) | unclear | unreported | | Technology in the Classroom | unclear | unreported | ---------------------------------------------------------------------- Table 6: Community Outreach Evaluations 27 6 InterNIC Registration Services - Network Solutions, Inc. 6.1 Service Overview The functions included under Registration Services include: o Overall administration of the Internet Protocol version 4 (IPv4) address space. NSI also coordinates with Reseaux IP Europeenne (RIPE) and the Asia Pacific Network Information Center (APNIC) in this task. NSI is responsible for delegating the address space to all first level NICs. NSI also operates and manages root domain name servers. o Registration and administration of U.S. IPv4 addresses. NSI is also the NIC of last resort for registration and administration of addresses and names for those parts of the Internet that do not have their own registration authority. o Administration and assignment of the .GOV, .NET, .COM, and .EDU Internet domain names. This task includes the operation of primary domain name servers for these name spaces. o Registration and continued administration of the legacy WHOIS database. o Through a subaward to ISI (Jon Postel as principal), the RS administers the .US Internet domain name space. o Development of tools for performing InterNIC RS functions, such as the Referral WHOIS (RWHOIS) effort. 6.2 Evaluation and Recommendations The panel congratulates Network Solutions Inc. (NSI) for their excellent work on providing registration services to the Internet community. The RS portion of the InterNIC project has responded well to the demands of the rapid growth of the Internet. NSI has directly facilitated Internet growth by effective allocation and administration of Internet names and addresses. NSI has responded to the exponential growth of the Internet by applying new approaches in the performance of their RS duties. In their role of overall administration of the IP name and address spaces, NSI has been responsible for the delegation of 28 management of the IPv4 address space and country-based name spaces (e.g., .FR, .UK, .JP, etc.). During NSI's tenure in operating RS functions, the Internet has shifted from centralized management of these functions to a distributed model of management and administration. This transition has gone very well, thanks in part to NSI and in part to the organizations that have stepped up to build this Internet infrastructure. The WHOIS database of network administrators has gotten to the point that a single centralized flat database is inappropriate. NSI has developed RWHOIS as a simple protocol to facilitate building a distributed database in place of the original WHOIS. It appears that the network operations community is picking up RWHOIS until a more general solution to Internet white pages becomes available. The panel observes that NSI will not be able to keep up with the overall load generated by the continued exponential growth of the Internet. NSI is encouraged to look for additional avenues for applying technology to solving some of their growth problems. Many of the tasks NSI performs currently have work steps that require human processing. The panel believes that NSI should continue to press to make these steps more dependent on computer processes if at all possible. The panel also recommends that the NSF support those efforts, and also be prepared to allocate more resources to NSI to help them meet their increasing work load. It is clear from the materials presented by NSI that a primary culprit in the RS work load is the .COM domain. The current management of .COM is not scalable since .COM is a flat domain name space, and thus the load of administering .COM falls solely on NSI. At present, the management of .COM is paid for by the NSF, and hence increasing demand for .COM registrations will require increasing support from the NSF. The panel recommends that NSI begin charging for .COM domain name registrations, and later charge for name registrations in all domains. Over the long run, the panel recommends that NSI charge for all IP registration services. During panel discussions a consensus emerged on a possible charging model that requests an initial fee for registering a name, and a recurring annual fee for subsequent administration of the name space, to cover costs due to updating entries, ensuring uniqueness of names in the name space, operation of root name servers, etc. It was noted that the charge for initial registration and recurring fees did not logically have to be the same amount. However, charging the same amount would allow NSI to state that everyone, including those who already have Internet 29 domain names, will have to pay the same amount within the initial 12 months and this might prevent a last minute, ``get them while they are free'' rush on domain names. NSI should consult with the NSF in the development of such a policy. The NSF needs to plan mechanisms for defraying the costs for institutions, such as U.S. R&E sites that would fall under the .EDU domain, that may not be able to bear the new charges directly. In the ideal scenario, any new plans should be in the direction of a fee to the end user of a name, and thus would facilitate the NSF's getting out of the name registration business. During the review, NSI discussed their encounters with the legal issues that have arisen in their management of the .COM domain. It appears that companies are considering pressing their trade and service mark claims into registration of similar names in the .COM domain. The panel was disturbed to hear that the NSF and/or the U.S. government does not provide a legal umbrella under which NSI can manage the Internet domain name space. The panel encourages the NSF to support NSI on legal and policy-related issues that stem from management of the domain name space, and also recommends that NSI and the NSF work to place the operation of the domain name system into a framework that provides suitable legal protection of the registration function. NSI should work in a leadership role in the development of such a policy. The panel recognizes NSI's success in fostering the internationalization of the RS functions to date. Many of the policy directions the panel is recommending, such as fee for service for name registration and an operational legal framework, will require an international framework if these policies are to be successful. NSI is encouraged to work within the Internet community to foster the development of the necessary international frameworks. The panel would also like to congratulate Jon Postel and ISI on their work in developing the administration of the .US domain as a subaward under the RS InterNIC award. In particular, ISI has developed informational materials on the .US domain that will help the future deployment of zones within the .US domain. The panel encourages ISI to continue to work on scaling the .US domain so that the U.S. Internet community does not have to revisit the problems exemplified by the current morass in the .COM domain. 30 7 NSF Meta-Questions As part of the review process, the NSF provided the panel a set of ``meta-questions'' intended to solicit recommendations about the viability of aspects of the Network Information Services Manager concept. Below we provide an overview, and responses to each of the meta-questions. 7.1 Overview The NSF has been incredibly successful in its work to develop the physical network infrastructure by providing funding and guidance at multiple levels, from the end user R&E institutions (connection grants) to mid-level and national backbone services. The exponential growth of the Internet can be directly attributed to the assistance given in its development through these NSF programs. A combination of small grants and larger cooperative agreements has allowed network growth in response to rapid changes in technologies and the user base. Because of the degree of community input during the solicitation process and the NSF's previous success with the physical network, there were high expectations throughout the R&E community that the InterNIC would serve as a similar NSF-funded focal point to jump-start the support and information services required by the Internet's education and research users. However, the expectation that the InterNIC would provide a focus for a similar high level of development and coordination for Information Services has not materialized. In its review of this project, the panel recognized that a number of issues remain before a cohesive network services program will develop. The InterNIC awards were the first Collaborative Cooperative Agreements requiring significant self-coordination among a team of awardees and the presentation of a ``seamless interface'' to the user community. In addition, there was the expectation that other NICs, particularly interdisciplinary NICs, would be developed that would draw on the InterNIC as a resource. As has been indicated in the review of the GA component (see Section 5), the failure by the organization with the lead responsibilities for coordination and outreach has had a negative impact on the better developed programs by the other partners. Further, it negatively impacted any potential for fostering development of other NIC programs. Instead of a cooperative effort that could draw on and focus the many projects already underway in the community, GA chose a predominantly solitary 31 course that did not create the vital interorganizational efforts that might have been possible. At the same time, there was little acknowledgment of the massive changes that were occurring in the Internet and the need for rapid redirection to meet these new challenges. 7.2 NSF Management of the Cooperative Agreements Given the problems that developed with this program, some consideration needs to be given to the role of the National Science Foundation in its work with the project. The advantage of a cooperative agreement is that it provides program flexibility in the face of changing circumstances. Through such agreements, the NSF can have continued input into the development of the program structure, and provide the participants of any single program with the knowledge that the NSF's staff has through work with multiple network projects. It can also ease the problems of rapid program change in the face of often slow government solicitation and award processes. The panel offers three suggestions in this context. First, the NSF should critically consider the viability of future cooperative agreements (and future proposal review panels' recommendations about these agreements) in light of the InterNIC experience. Perhaps it is simply not viable to expect significant self-coordination among a team of awardees. Second, the NSF should carefully review its capabilities for managing major cooperative agreements. Management theory is clear about the commitment required for maintaining multi-organiza-tional relationships. Because of the multiple agreements involved, the NSF took on a greater responsibility for that management and does not seem to have had adequate resources to fully assist in problem resolution, despite what seem to have been excellent efforts by NSF staff. While this does not necessarily indicate that such multiple awards are prone to fail, it suggests that NSF/DNCRI must carefully consider what its focus and management resources are, especially in the face of the pressures to continue in its physical infrastructure support projects. As major initiatives are started in new areas such as Information Services, careful consideration should be given to the needs for staff monitoring of such major projects. Third, the panel recommends that the NSF make more extensive use of outside experts throughout the life of each cooperative agreement. One of the outstanding achievements for the NSFNET 32 transition has been the external expertise brought about through hiring ``consultants'' such as Peter Ford to assist the NSF staff by bringing direct experience from the field. 4 A similar consulting group for Information Services would have provided valuable insights, and assured that critical issues were raised as the InterNIC took shape. Because the solicitation process moved slowly, such consultants would have provided additional insight into changing conditions as the awards were made. If the changing environment had been addressed early in the process, some problems (such as failure to focus on NICs rather than end-users) might have been flagged earlier and possibly resolved. The consultant model may be a direction for the NSF to pursue as it further studies what should be done for creating a cohesive Network Information Services framework. 7.3 Requirements for Ongoing "Customer" Feedback The InterNIC project failed to develop specific methods for user feedback other than unsolicited individual user electronic mail, which the panel feels is not an adequate means of evaluation. Any major project such as the InterNIC that is required to serve the community should have some formal process for community input. The NSF may wish to consider such as a requirement in future solicitations. For example, an external advisory board that had a solid sampling of ``customers'' could have assured issues were raised more speedily about changing environments without necessarily waiting two years for a review panel to be summoned. When millions of dollars are involved, the NSF needs ongoing methods to provide evaluation and ensure that the community that is supposed to be served does indeed receive the expected services. Given the flexibility possible in a cooperative agreement, such external advice to the NSF and project management can be of great value in providing ongoing corrections as well as ensuring that projects retain direct contact with ``customers.'' Such an external advisory board could also benefit the remaining years of the InterNIC project and should be considered. ____________________________ 4 Note: although Ford is a member of the panel, he did not write the current section. 33 7.4 Appropriateness of Informal Relationship between the InterNIC and CNIDR The Clearinghouse for Networked Information Discovery and Retrieval (CNIDR) has been funded by the NSF under a separate cooperative agreement from that of the InterNIC, in response to an unsolicited proposal. Because the NSF felt CNIDR's activities were of complementary value to the InterNIC, they initiated an informal relationship between CNIDR and the InterNIC. While a formal review of CNIDR would not be appropriate for this report, the NSF requested that the panel examine the relationship between CNIDR and the InterNIC. In general, the panel agreed that the liaison between the organizations has been beneficial, particularly with the Directory and Database Services (AT&T) where information tools are a key to service success. Based on the information available at the review, the panel recommends continuation of the informal collaborative relationship, which seems to be working well. Given the problems identified with coordination among the InterNIC organizations primarily attributable to GA's poor performance, it does not seem advisable to complicate organizational relationships with formal inclusion of CNIDR in the InterNIC at this time. 7.5 The Viability of the NIS Manager Concept The solicitation that resulted in the InterNIC awards was prepared nearly three years ago. Since that time, many changes have occurred (see Section 3). While the panel focused on evaluating performance based on submitted proposals, the issue of the InterNIC's overall effectiveness in establishing a network information services framework also arose. It was agreed that in general, this question could not be fully addressed because the primary problem of GA's failure to perform raised unanswerable questions about what might have been. There was little question from the panel's discussions, which represented voices from a broad sampling of the user community, that the needs identified in the solicitation still exist. While no consensus was reached on the issue of what changes should be implemented, there was agreement that no single program could meet the full spectrum of needs for Network Information Services. While the panel is in full agreement that the successful InterNIC services were important and should be continued, there was a sense that the current efforts are 34 insufficient. In general, the panel sensed that the InterNIC program may need to be supplemented with parallel projects if the goals are to be achieved, both to fill gaps not clearly ``owned'' by any of the participants, and to implement projects that have failed to materialize as expected from GA. While the discussion on this was brief, the panel recommends that the NSF reexamine the Information Services program to make it more comprehensive. An example was given by one panelist, who noted that physical infrastructure success has been the result of seeding at both the top (backbone) and bottom (stub network) levels, encouraging a community commitment to coordinated development. The panelist indicated that there was no similar multi-pronged assault on Information Services. A panel subgroup came up with a similar suggestion aimed at encouraging a complimentary program to assist other NICs in developing projects that would be widely disseminated to spread the base for NIC programs. In such a program, the NSF could encourage projects that were cooperative and broadly beneficial -- a contrast to the conflict between commercialization/self-support and community service that continues to be a difficult issue for the InterNIC participants to resolve. Through such an enhanced program, the remaining InterNIC services could be better leveraged and built within a community framework that would enhance the services that are already developing. The capability to encourage collaboration and cooperation also builds on strengths within the NSF that can be helpful in providing focus for the larger R&E community. The panel also suggests that the NSF consider how to reinvigorate this area so that ``the next time'' it will work. A more targeted approach may be appropriate, focusing either on communities of interest (e.g., K-12, or K-12 in Illinois, etc.) or on professional communities (professional societies, chemists, historical societies, etc.). The panel suggests care in looking at what InterNIC activities are heavily used today, and suggests trying to learn methods and overall strategies from them before issuing any new solicitations. The panel recommends planning grants to stimulate workshops and program/partnership development, and also suggests looking for more industrial partnerships (e.g., with Dow Jones, AOL education, Newspapers, CNN, Knight Ridder labs, etc.) Finally, some members of the panel felt that 5 year awards are a mistake, and that the NSF should consider shorter time horizons, such as 2 year awards. 35 A InterNIC DS's Responses to Panel Questions The questions below were answered by Erik Grimmelmann, PI of the DS portion of the InterNIC. Question The sense of panel is that the directory of directories is an extremely important service but is not used at present because it is not sufficiently populated. What are your numeric targets for populating this directory, and how do you expect to reach those targets? Response We agree with the committee that the directory of directories is extremely important and needs significant expansion. The Internet community is extremely large, and a popular resource can easily be swamped. Many resource owners cannot support large scale demand for their resources, so we believe we should list resources only when we have the ``owners'' permission. On the assumption that this policy remains in place, our targets are 1,000 entries by the end of contract year 2, 3,000 by the end of year 3, 6,000 by the end of year 4, and 10,000 by the end of year 5. We expect to reach these numbers by devoting a person to this process full time. This person will prepare descriptions of resources and submit them to the owner for approval. We will increase outreach activities, and ask third parties to submit information on resources they feel we should list (via electronic mail and via an HTTP form interface). Resource owners could also use the form interface to submit entries to the directory of directories. Question Given the work you do with directory services, do you believe it's appropriate that you provide database services to the community? Does it have an impact on your ability to deliver directory services? How do you prioritize/balance these demands? Response We believe that our support of directory services is significantly more important than our support of database services, and our resource allocations reflect this (only about 5% of our resources are dedicated to database support). 36 However, a number of the databases we hold (RFCs, IETF documents, etc.) are quite useful to the R&E community, especially since we WAIS index these documents. There is also synergy between the tools we need to support databases and the tools we need to support the directory of directories (WAIS, Gopher, WWW, mail server, and support utilities). We expect the current split in resource allocation will continue. Question If database services are deemed appropriate, what steps will you take to determine what the R&E community needs in terms of databases? Response Databases we handle for a fee help cover costs of running the InterNIC. In these cases the content owner determines the target audience. Databases that we support without a fee will be focused on communities such as K-12, libraries, and other affinity groups that primarily serve the R&E community. We will continue to seek guidance from the NSF on case by case basis for additional no-fee databases. Question How can you make your services more appealing to the community? Packaging, organization, outreach, ... Response While we will continue to refine our packaging and organization, we feel the greatest benefit will come from increased outreach. We will increase attendance at conferences in target communities and aggressively solicit and follow up leads. Question After your 2 years of experience, do you believe you are being required to deploy services that depend on solving some as yet unsolved research problems? For example, technical problems of providing a directory of directories, or privacy problems of white pages services? Response Yes. There is a demand for a large scale white pages service that cannot be fully met with current technology. RFC 1588 recognizes that the Internet needs systems that efficiently tie together a variety of directories using different tools. This is not yet a solved problem. The privacy dimension also presents issues in terms of the willingness of many organizations, especially corporations, 37 to make their directory information available. Directory systems must be made more secure. Accuracy will also remain a problem as long as directories are not tied to systems that are updated as part of normal business. Corporate directories are typically tied to payroll/personnel databases; published telephone directories are tied to telephone customers who pay a bill every month and thus validate their address. We need to find ways to tie our directory services into systems that already have accurate copies of directory data. Question The panel felt you provided a lot of data, but did not analyze/digest these data sufficiently to permit independent evaluation. What will you do to provide meaningful customer feedback analysis? Response We agree that some additional analysis could be useful, but much of the data is anecdotal and the commenting population is self-selected. We will explore inexpensive mechanisms to analyze the data we have. We have an REU student who has started to analyze the data, but it is quite diffuse and does not yet provide useful information for planning. We will also explore mechanisms to inexpensively survey the target community and help direct our outreach efforts. Question What do you see as your strategic directions through the end of the award? Response The Internet is changing so rapidly that a tactical focus is more important than a strategic focus. However, we do expect to continue our focus on service, and on providing high quality, reliable, accessible, and accurate services. For the directory of directories, we intend to follow the plan described in our answer to question 1 to increase the number of entries and thus the overall utility of the collection. As noted in our answer to question 6, we will also explore mechanisms to survey the target community and discover what subject areas of the directory of directories should be the focus of our efforts to increase the number of listings. For directory services, we will continue to work towards the Internet directory envisioned by RFC 1588. We will provide a testbed for real-world implementations of tools that provide gateways between different sorts of directories, and 38 will continue to work with the IETF and the research community on the design and architecture of such tools. As noted in our answer to question 2, we believe that databases are less important strategically than directory services, but we will continue to support databases that are of interest to the R&E community or which provide added value for the InterNIC. Question What are you doing to coordinate with your international counterparts? Response Our white pages effort includes participation in the world-wide PARADISE project. In general, we have interpreted our charter to target the U.S. R&E community. We have certainly not excluded overseas resources from the directory of directories, and we have begun to make copies of RARE RFCs available on our server under an agreement with RARE. Question What is the service you're providing that the U.S. R&E community can't do without? Response Our bringing together disparate services to provide the directory of directories and our directory services. We also provide a focus and a real world testbed for emerging technologies in information access and directory integration. In addition helped to revitalize the X.500 pilot in the U.S.. Question What are your opinions on the questions you asked the panel? What did you learn over the last 2 years? Suggestions? Response Issue 1: Resource owner must approve listing and description for inclusion in Directory of Directories. As noted in our answer to question 1, we believe we should continue the policy of requiring owner approval to list a resource. Issue 2: How to focus our services on the Internet R&E community As we focus our outreach activities on the R&E community, the center of gravity of our services will naturally tend to follow the needs of the community. In addition, some of the market research described in our answer to question 6 could 39 help us identify segments of the R&E community that we are not targeting adequately. Issue 3: Growth in volume: DS servers are potentially default targets of commercial services. Tony Hearn's analysis which he mentioned in the meeting indicates that AOL is a good target for K-12 teachers who are an important part of the R&E community. This is consistent with with anecdotal evidence we had heard previously. Given this input, no action seems to be necessary at present. Issue 4: Resource allocation: Do we aim for high availability for all services, or split servers so high usage applications do not hurt performance for all users? We believe that we should continue our emphasis on high availability. This is consistent with our intent to provide high quality service. We will investigate reconfiguring our systems so that different services target different servers as primary, but all servers will continue to offer the complete suite of applications. Issue 5: Should we put more emphasis on ``how to'' kits and on-line help and tutorials for directory applications? Yes. Where this involves significant effort, we will seek guidance from the NSF regarding the priority of such efforts and the likely impact on the community. 40 B InterNIC IS's Responses to Panel Questions The first question below was directed to Karsten Blue, the president of the newly reorganized CERFNet. All other questions were answered by Kent England, departing manager of the IS portion of the InterNIC. Question Karsten Blue: what is your strategic intent for the InterNIC? Response Our strategic intent for the InterNIC is to serve the Internet community by making it easier to use the Internet. Question The panel feels your activities lack focus, specifically for supporting the R&E community, and that you haven't achieved sufficient visibility in that community. Please tell us how you will change to solve the problems of the R&E community, and to achieve visibility in that community. Response It is true that the changing constituency of the Internet has caused us problems in reaching our target U.S. R&E audience. We have worked hard this year to re-orient our approach to finding the right targets in the ever widening Internet audience, by restricting uncontrolled access to our human resources by the general public through the open access Reference Desk (but not restricted from online resources, like InfoGuide). Our U.S. R&E audience consists of both end users and other NICs. Our objective is to support the needs of the R&E end user community directly primarily with online information resources and by supporting other NICs, which support R&E end users directly, through support for the basic functions of a NIC. The most effective way we can support R&E end users directly is through our online information resources, such as InfoGuide, Scout Report, NSF Network News, and net-happenings. The other effective way to support R&E users is through support to other NICs. I chose to tackle the issue of online information resources during the first quarter of this year by introducing the InfoGuide HTML resource, the NSF Network News bi-monthly (hardcopy and online), and the weekly Scout Report. I chose to tackle the issue of NIC support in the second and third quarters of 41 this year through the introduction of USV-Web, NIC Locator, and the InterNIC Briefcase and our interactions with FARNET, SIGUCCS, and Educom. I feel that these efforts are effectively focussed on the R&E community that is supported by our NSF funds. I feel that the combination of online resources and support to other NICs is our most effective means to serve the needs of diverse users within the U.S. R&E community. With respect to visibility, if I can assume that InterNIC Information Services needs to become more visible as CNIDR has achieved visibility, then I would say that the InterNIC Manager needs to focus more attention on coordination within the community, by attending more conferences and workshops and putting forward the InterNIC case and our service profile. I believe that the InterNIC ``Handout'', included in the black folder, is the story we have to tell about our services today. Another way we can achieve visibility is through more intensive support for specific pilot or demo programs. For example, we must work with others to create more discipline-specific NICs, such as the Agriculture Network Information Center (AGNIC). Another hypothetical example would be training and Help Desk support for a specific K-12 pilot program with the objective of demonstrating how a school system would develop user support for Internet access. The online R&E community can be reached by the InterNIC staff by more active participation in newsgroups and mailing lists and by offering pointers to the InfoGuide and other InterNIC services, when appropriate. For example, many questions asked time and again by beginners can be answered by browsing the InfoGuide or doing a search on the contents of the InfoGuide and the net-happenings archive. A directed effort to have a constant presence on the network would do much to spread the word on InterNIC services. Question The solicitation and your proposal response call for a NIC of NICs focus. The panel feels you do not meet this requirement. Is this a mistaken perception, and if so how are you meeting this need? In particular, what will you do to establish a NIC liasion council? Response The issue of NIC support has only recently been addressed, after the introduction of our online information resources. 42 The Briefcase and the Whole Internet seminar are the first two concrete InterNIC services for other NICs that we have recently introduced. Our current suite of services is outlined in our InterNIC ``Handout''. I feel that the NIC liaison council needs a focus and that the initial focus is provided by these two new services. If the NIC liaison council charter is to advise InterNIC on NIC support services, then we are ready now to solicit participation in an annual or semi-annual convocation of the Liaison Council, timed to coincide with an appropriate NIC conference or workshop. I would very much appreciate feedback from the Review Panel about which NIC conference or workshop we should coordinate with. I would suggest SIGUCCS or a CNI tie-in. I would also appreciate feedback on the makeup and size of the Council. Question It appears you have tried to take on many services that other providers can and do handle. How will you delegate or exit from efforts, and as a gov funded activity how will you achieve fairness to other providers/enterprises (commercial and otherwise) - authors, seminar presenters, service providers? How will you balance this w/ need to do cost recovery? Response We originally intended to sell books by mail, as well as CD-ROM subscriptions. These efforts are more effectively done by others and we have suspended these efforts. Those efforts that are wholly or substantially funded by NSF or other government funds should not be competitive with other commercially available services. These subsidized efforts should be undertaken to provide services that are not commercially available until such time as commercial alternatives are available. Those efforts that are wholly unsubsidized may be competitive with other commercially available services and the competition will not suffer through inappropriate cross-subsidy. The InterNIC seminars are an example of this. There is no reason why InterNIC can't offer seminars in competition with others when there are no subsidies involved. It is in the area of cost recovery from partially subsidized activities, such as the Help Desk, where the most difficulty 43 is present. I suggest that we attempt to maintain a separation between the subsidized efforts, such as the Scout Report, and the cost-recovered efforts such as the seminars, and avoid mixing of subsidized and unsubsidized activities. Question When you changed managers before, you lost a lot of momentum. How will you maintain momentum this time? Response General Atomics must not delay in recruiting a new Principal Investigator willing to accept responsibility for the InterNIC. GA must coordinate with NSF and the Review Panel to find a PI that is acceptable to all parties. I feel that GA has a limited time to accomplish this transition or else the momentum will again be lost. Question How will you provide a seamless service among the 3 InterNIC entities? Response Seamless service is achieved first by close cooperation among the partners and second by effective presentation to the clients. The four partners of InterNIC (General Atomics, AT&T, NSI and MCNC) are becoming an increasingly more coordinated team as we learn to work together. This has led to closer cooperation on a number of issues, particularly our support for RS. The seamless interface to our constituents is not simply a matter of a single phone number or a single point of contact. It is through the coordinated outreach, for which GA is primarily responsible, that we maintain a coherent front. As we achieve more visibility we will be recognized as the four constituents of one InterNIC service. Question What will you do to get real customer feedback, statistics, etc. that allow independent evaluation? Response Hard data comes from our online servers with statistics about information served via WWW, Gopher, FTP, electronic mail, etc. We have published statistics in our monthly reports. Please comment specifically on what you would like to see in our regular statistical reports that we do not currently provide. We also retain customer feedback in the form of kudos and comments. These comments have been reported to NSF and our partners in our quarterly review slide presentations which I 44 decided, for reasons of space, not to include in your binder. This has resulted in the oversight of not providing you with specific instances of this kind of qualitative feedback which we can rectify by sending you some of our presentation data. A way to get more objective evaluation is to conduct market research by asking a statistical sampling of our clients how we are doing. I think this would be premature at this point and expensive at any rate. Frankly, when InterNIC IS is sufficiently ``visible'' to the community, it will be sufficiently visible to you and the question of evaluation will be answered by you and by others in the community commenting in public media, such as on mailing lists. Question What is the service you're providing that the U.S. R&E community can't do without? Response All of the services that InterNIC IS provides are competitive in the sense that others can, will and do provide comparable services. We are not a monopoly franchise, like InterNIC RS. The most important thing that InterNIC IS can provide to the U.S. R&E community are certain value-added services that help organize the Information Reservoirs and Information Streams that we are in the process of building right now. NSF has been remarkably effective supporting network connectivity infrastructure. This is no longer necessary as the Internet has evolved to the present. We are in the early stages of building a real information infrastructure and we don't yet know how to provide the day-in day-out production services to manage this information infrastructure. You should not let the current start-up, bootstrapping, and management woes of InterNIC IS distract you from the real need of the U.S. R&E community for support of the information infrastructure. Please consider the evolving roles I outlined at the end of my presentation and then advise on how NSF may play a vital role in the evolution of the information infrastructure comparable to the role DNCRI played in the formation of the Internet connectivity infrastructure in the period from 1987 to 1994. 45 I firmly believe that NSF has a role in supporting this information infrastructure by providing structure for the reservoirs of information (such as InfoGuide) and editorial content on the streams of information (such as the Scout Report). The information architects and system mechanics are very important parts of the production services that InterNIC must invent in the future. 46 C InterNIC RS's Responses to Panel Questions The questions below were answered by Scott Williamson, PI of the RS portion of the InterNIC. Question What solutions does NSI see for their legal and scaling problems? Response NSI sees the need for an International non-profit organization to accept the responsibility for the liability associated with registrations. We see the need for strong policy/procedures for registration. The organization that has the liability must adopt the policy/ procedures for Internet registrations. Further, we feel that the Internet Society's treatment of the standards process should serve as a model for the registration process. Response NSI sees the need for our scaling problems as follows: Distribute both the data through RWHOIS and the registration allocation process through regional registries. Also, potentially establish third (3) level regional registries. Question How soon could you implement a charging scheme? Do you have a plan in place to handle billing? Response NSI could implement a charging scheme in two weeks starting with the acceptance of checks. Followed by a credit card charging scheme. To implement a credit card charging scheme NSI will exercise the first option of upgrading an existing account which would take from one (1) week to one (1) month to execute. Otherwise, if a charging scheme must be initiated from scratch, research tells us it will take from one (1) to four (4) months to implement. This scheme will include how to handle the billing. The charging scheme would apply to .COM, .ORG, and .NET. For existing records write software that will filter through the database and send a notice to the established POC via e-mail within their activation month. NSI recommends that this would begin three (3) months after activation of the charging system. 47 Question What is your plan for spinning up the 10,000+ DNS servers we will need in the U.S. 3 years from now, and how will this be supported in the future? Will the current volunteer scheme be able to meet the exponential growth? Response We expect the name servers to be volunteered by participating communities. They will initiate their incorporation into the domain system. Response NSI believes the volunteer scheme will be able to meet the exponential growth through additional levels of delegation. Question Upon which of the services offered by the other InterNIC partners are you dependent? What increases/decreases/changes would impact your activities? Response NSI is dependent on the HELP DESK services provided by Information Services (IS) to handle the basic Internet questions from the user community. If this service did not exists NSI would obviously experience a considerable increase in what is already a high volume of ``What is the Internet'' type questions. Response Better education of the Network Service Providers (NSPs) by IS would result in a decrease of inquiries via phone calls to NSI. Question How will NSI handle explosive growth in non-U.S. sites? What will you do to further the globalization of management of NSI? (Note: This question originally asked about both NSI and the Internet Assigned Numbers Authority (IANA), but it was pointed out that the IANA role is not funded under the InterNIC awards and hence that this question should be limited in scope to the NSI.) Response Domain Names in other countries are primarily under country codes and those are delegated to administrators in each country. Currently the number of non-U.S. sites registering in .COM is very small. Response Address Numbers in non-U.S. sites are primarily handled by regional registries. Additional registries may be established to handle the growth. Continue to strengthen the ties between registration and the Internet Society, which is an international organization. 48 Discuss registration policies in International technical committees such as in the IEPG (Internet Engineering Planning Group). Question The panel perceives a problem with international data being applied to the global information database in a timely manner. What will you do to remedy this problem? Response International updates go into the same queue as all other requests. We are working out automatic updates from regional registries, first with SWIP (Shared WhoIs Project), and later with complete delegation via RWhois. Question Do you foresee any situations that might arise in the future that could cause NSI to be unable to continue functioning in registration services? Response Litigation against NSI would effect NSI's ability to do this job. Any other changes to NSI should not effect NSI's commitment to continue this project. Question Beyond the obvious, which of the services you offer do you feel the U.S. R&E community cannot do without? Response It is clear that the R&E funding should not support the growth of the .COM domain, yet there are important elements of the .COM which are an active part of the R&E community. 49 D List of Acronyms AGNIC Agriculture Network Information Center (AGNIC) AOL America On Line (information service provider) APNIC Asia Pacific Network Information Center AT&T American Telephone and Telegraph Corporation CD-ROM Compact Disc Read-Only Memory CERFNET California Education and Research Federation Network (Owned and operated by General Atomics) CERN Conseil Europeen pour la Recherche Nucleaire (European Center for Nuclear Research) CNI Coalition for Networked Information CNIDR Clearinghouse for Networked Information Discovery and Retrieval CNN Cable News Network .COM Commercial institution (Domain naming branch) CU-SeeMe Video Conferencing Tool developed at Cornell University DNCRI Division of Networking and Communications Research and Infrastructure (NSF) DNS Domain Naming System DOE U.S. Department of Energy DS Directory Services (AT&T portion of InterNIC) .EDU Educational institution (Domain naming branch) EUNET European UNIX Network FARNET Federation of American Research Networks .FR France (Domain naming branch) FTE Full-Time Equivalent FTP File Transfer Protocol GA General Atomics Corporation .GOV Governmental institution (Domain naming branch) HTML Hypertext Mark-up Language (WWW) HTTP Hypertext Transfer protocol (WWW) IANA Internet Assigned Numbers Authority (currently operated by ISI) IEPG Internet Engineering Planning Group IETF Internet Engineering Task Force IP Internet Protocol IPv4 Internet Protocol, version 4 IS Information Services (GA portion of InterNIC) ISI University of Southern California's Information Sciences Institute InterNIC Name adopted by NIS Manager awardees for their unified service .JP Japan (Domain naming branch) K-12 Kindergarten through Twelfth Grade networking MCNC Microelectronics Center of North Carolina MORENET Missouri Research and Education Network NADF North American Directory Forum NASA National Aeronautics and Space Administration .NET Domain naming branch for network administrations 50 NIC Network Information Center NIS Network Information Service NNSC NSFNet Network Service Center (NIC award prior to the InterNIC) NREN National Research and Education Network (HPCC program component) NSF U.S. National Science Foundation NSFNET National Science Foundation Network NSI Network Solutions, Inc. NSP Network Service Provider NYSERNET New York State Educational and Research Network .ORG Domain naming branch for non-profit institutions PARADISE European X.500 deployment project PI Principal Investigator POC Point of Contact R&E Research and Education community (of the U.S. in this report) RARE Reseaux Associes pour la Recherche Europeenne (European Research Networks Association) REU Research Experiences for Undergraduates (NSF grant program) RFC Request For Comments (Internet documents, often used for protocol specifications) RIPE Reseaux IP Europeenne (European continental TCP/IP network operated by EUnet) RS Registration Services (NSI portion of InterNIC) RWHOIS Referral Whois protocol SENDIT North Dakota's K-12 Telecom Network SIG Special Interest Group SIG-WEB Special Interest Group on the WWW SIGUCCS ACM Special Interest Group on University & College Computing Services SWIP Shared Whois Project .UK United Kingdom (Domain naming branch) .US United States of America (Domain naming branch) U.S. United States of America WAIS Wide Area Information Servers system WHOIS Simple Internet white pages service (RFC 954) WWW World Wide Web (Internet hypertext-based multimedia information service) 51