Peer-to-peer simulation frameworks: a survey
DOCNAME

Abstract

Peer-to-peer (p2p) protocols, like all distributed protocols, are complex, and therefore harder to debug and study in the wild. This is more true of existing p2p protocols, where changing the behaviour of the protocol --- however minor the change may be --- may result in unknown manifestations on the dynamics of the swarm using that protocol. In lieu of the unintended consequences of perturbing a live swarm, researchers have resorted to simulation frameworks. However, simulation results obtained from one simulator are often hard to reproduce when using another simulation framework. This document surveys existing simulator frameworks prevalent in simulating p2p protocols today in order to quantify any assumptions and characteristics inherent in the simulator. This, we hope, will aid future researchers in choosing the right simulation framework for their abstraction.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on September 08, 2011.

Copyright Notice

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 2. Terminology
• 3. Criteria for evaluating simulation frameworks
• 4. List of simulation frameworks
• 5. Security Considerations
• 6. IANA Considerations
• 7. References
• 7.1. Normative References
• 7.2. Informative References
• Appendix A. Acknowledgments
• Author's Address

1. Introduction

Peer-to-peer (p2p) protocols, like all distributed protocols, are complex, and therefore harder to debug and study in the wild. This is more true of existing p2p protocols, where changing the behaviour of the protocol --- however minor the change may be --- may result in unknown manifestations on the dynamics of the swarm using that protocol.

Researchers contemplating on changing the behavior of an existing p2p protocol have to be careful still, least they inadvertently do more harm than good by introducing their changes. Furthermore, any changes to an existing p2p protocol or a newly developed p2p protocol must be tested and evaluated for validity and reproducibility by the research community. While analytical and mathematical modeling (fluid models, optimization and linear programming) is easily validated, it is harder to validate empirical experiments due to the dynamic nature of the networks, hosts, and interconnections between them. Simulation frameworks are attractive since they provide a controlled environment under which new behavior of p2p protocols can be studied and quantified.

The good news is that there is a plethora of simulation frameworks for p2p protocols available today, some of them are surveyed in Naicken et al. [naicken]. However, that survey is dated and does not include simulation frameworks like ns-3 [ns-3] and ProtoPeer [protopeer] that have become available since the survey was published.

The aim of this document is to update the state-of-art with respect to p2p simulation frameworks available today. We will survey simulator frameworks prevalent --- and actively used --- in simulating p2p protocols today in order to quantify any assumptions and characteristics inherent in the simulator. This, we hope, will aid future researchers in choosing the right simulation framework for their abstraction.

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

3. Criteria for evaluating simulation frameworks

This is a non-exhaustive list of all criteria that we should evaluate when surveying a simulation framework.

• Type of simulator: flow-level, message-level, or packet- level. Advantages and disadvantages of each.
• Does the simulator specifically target p2p networks? Some like ns-3 are general purpose simulators, but p2p models can be constructed and evaluated over a general-purpose simulator.
• Level of documentation (APIs, wiki, etc.)
• Support for building models: script level, compiled language, through a visualization editor, etc.
• System limitations imposed by the simulator framework, if any.
• Learning curves associated with the simulator framework.
• Support for trace-driven simulation (i.e., using live traces to inject events in the simulator queue.
• Scalability of the simulator.
• Whether or not the simulator framework supports distributed simulations synchronized on a common time source or event queue.
• Support for transitioning from a simulation environment to actual system implementation (or, can the code developed for a simulator be used with minimal or no modifications in a real host)? See Galuba et al. [protopeer].
• Support for modeling link-level (delay, latency, loss, data rate) and host-level characteristics (i.e., simulate both low-level events and application PDUs).
• Support for interfacing real hosts that inject events into the simulator.
• Support for collecting statistics and measurements from the models.
• Visualization tools for creating topologies, viewing the simulation in action, etc.
• Support for importing existing topologies (GT-ITM) and others.
• Support for exporting topologies in a standard graph markup language.
• Should we focus on only academic and research simulators or commercial simulators as well?
• ...

4. List of simulation frameworks

A list of simulation frameworks that we can survey appears below (original list is in Naicken et al. [naicken], I have added a couple more simulators). This is a rather exhaustive list, however, going forward, we should focus on those frameworks that are: newer, actively in use today, and those frameworks that are actively used today and have been surveyed before, but could stand to be looked at again in light of hardware and software advances in the last few years (multi-cores, parallel programming, etc.):

• ns-3 [ns-3].
• ProtoPeer [protopeer].
• GPS.
• PeerSim.
• P2PSim.
• OverSim.
• DHTSim.
• PlanetSim.
• VPDNS.
• Narses.
• Neurogrid.
• GnutellaSim.
• myNS --- we could probably drop this in favor of ns-3.
• Overlay Weaver.
• Query-cycle Sim.
• GTNetS [gtnets] --- seems to be abandoned.
• ...

5. Security Considerations

This document does not introduce any new security considerations in p2p protocols.

6. IANA Considerations

This document does not require any IANA considerations.

7. References

7.1. Normative References

7.2. Informative References

Appendix A. Acknowledgments

Author's Address