This page contains various ideas or questions that occured when designing and implementing Solipsis. Recall that this is a wiki. Feel free to comment, argue or answer on these topics. These are free discussions on networking issues that are certainly shared by other designers of peer-to-peer systems....

Reliable UDP

UDP is preferred in user-to-user applications because it allows transparent NatTraversal (which otherwise is impossible if both ends of the connection are behind NATs). However, there are situations where many of TCP's features would be welcome too:

• unbounded message size (without fragmentation at the IP level)
• automatic packet re-ordering
• automatic and efficient packet retransmission
• ...

Emulating some of these features upon the UDP protocol is commonly called reliable UDP. As strange as it may seem, many P2P applications rewrite their own reliable UDP protocol so as to enable NatTraversal.

NAT Traversal

The problem is when two nodes in the Solipsis network are each behind a NAT device (firewall, gateway, etc.) and want to connect to each other. There are in fact two problems :

• getting each other's addresses/ports as seen from the Internet (i.e. as shown to the outside by the NAT)
• connecting to each other by opening a hole through the NAT

As with most end-user applications, we don't want to rely on the system administrator to open dedicated inbound ports for us to listen to. Thus we need a way to play with the NAT's features/limitations so as to pass through.

STUN

Simple Traversal of UDP through NATs (http://www.faqs.org/rfcs/rfc3489.html) is an IETF protocol. It allows a NATted client to a connect to an arbitrary STUN server on the Internet (outside the NAT). This public STUN server will send back some information :

• the client's IP address and port as seen from the Internet
• what features and limitations the STUN server has been able to detect on the NAT device

With this information, the NATted client can know : 1) if the NAT can be traversed back by UDP packets depending on what UDP packets the client sends outside 2) what address and port number the client must give to its peers so that they can call it back.

STUN is not an elegant and complete solution. It is rather a kludge to alleviate the problem that each NAT has its own functioning. As per the RFC :

"This protocol is not a cure-all for the problems associated with NAT. It does not enable incoming TCP connections through NAT. It allows incoming UDP packets through NAT, but only through a subset of existing NAT types. In particular, STUN does not enable incoming UDP packets through symmetric NATs (defined below), which are common in large enterprises. STUN's discovery procedures are based on assumptions on NAT treatment of UDP; such assumptions may prove invalid down the road as new NAT devices are deployed. STUN does not work when it is used to obtain an address to communicate with a peer which happens to be behind the same NAT. STUN does not work when the STUN server is not in a common shared address realm."

uPNP NAT Traversal

Universal Plug'n'Play is a forum led by Microsoft and others aimed at making computer hardware discover each other seamlessly. Amongst others, it has a specification for applications to talk to NATs and tell them what ports they want to use/open : the Internet Gateway Device (IGD) Standardized Device Control Protocol V 1.0 (http://www.upnp.org/standardizeddcps/igd.asp).

There are several problems :

• it is said to be quite a complex protocol, involving a complex XML grammar
• it is based on the willingness of system administrators to allow arbitrary applications to tell the NAT to open ports on the outside ; it is unlikely many NAT administrators are willing to take that risk (any trojan / virus infecting a PC behind the NAT would then be able to open the private network to the outside)

Discussion of different methods

An interesting document in French can be found here (http://www.iro.umontreal.ca/~jaumard/Research_Projects/Mitacs_Project2/Publications/Report1.pdf).

Some SIP applications use STUN for traversing NAT. Although not quite elegant, this solution seems mature.

A recent discussion (http://www1.ietf.org/mail-archive/web/sip/current/msg10657.html) on the IETF SIP mailing-list.

Chownat (http://chownat.lucidx.com/) is an cool app allowing two parties, each behind NATs to communicate without a middle man or other 3rd party.

Implementations

Shtoom (http://www.divmod.org/Home/Projects/Shtoom/index.html) is a very interesting project (http://www.python.org/pycon/dc2004/papers/6/) that builds a complete phone-over-IP application in Python, including the SIP stack itself. It has built-in STUN support using Twisted (see here (http://www.divmod.org/cvs/trunk/shtoom/shtoom/stun.py?rev=699&root=Shtoom&view=log) and here (http://www.divmod.org/cvs/trunk/shtoom/shtoom/rtp.py?rev=576&root=Shtoom&view=log)).

UDP Fragmentation

Main discussion

The Solipsis network is probably suited to UDP best : one-way short notification messages, no critical content (at least not in the layer that handles topology discovery and maintenance). Also, Solipsis is meant to be usable on ad-hoc mobile networks, where TCP may not be a very good choice (?).

Obviously, UDP has the drawback of doing very little stuff compared to TCP. One of the issues is fragmentation ; as UDP is a datagram protocole, fragmentation has to be dealt with at whatever level.

UDP fragmentation is commonly handled by the IP layer ; that is, UDP datagrams are fragmented into several IP packets and then reassembled at the end point of the connection. This means fragmentation is theoretically handled by the OS and transparent to the application. There are several drawbacks :

• All implementations have limit to UDP packet size anyway. It seems 8192 is a common limit.
• Fragment reassembly is an all-or-nothing process ; this means if one fragment is lost (or arrives too late - it seems a common value for this timeout is around 20-30s.), the whole UDP datagram is discarded.
• Certain routers/firewalls block path MTU discovery, so that fragmentation cannot properly be done on the emitting side ; I don't know if it affects UDP as well as TCP (could UDP datagrams be fragmented later in the path ?).

Thus UDP fragmentation leads to more unreliable communications, and should probably be avoided. This means using, if possible, small enough UDP datagrams that fragmentation will not occur on common networks. Less than 1500 bytes is ok for Ethernet-based networks, and 500 bytes is said to be small enough for every network out there.

Other stuff about UDP

Simplistic file transfer over UDP : the [TFTP (http://www.faqs.org/rfcs/rfc1350.html)] protocol.

The SIP protocol can use UDP : see the [SIP RFC (http://www.faqs.org/rfcs/rfc3261.html)].

Misc

Standard Port Numbers

A very thorough list of port assignments (http://www.iana.org/assignments/port-numbers) is given by the IANA. see also [here (http://www.bekkoame.ne.jp/~s_ita/port/port5000-5399.html) for a comprehensive break-up)

The problem is twofold:

• mainly, choose a standard (UDP) port number for standalone nodes; for now we have temporarily settled on port 6010, but we will have to choose more carefully
• optionally, choose a range of port numbers for mass node hosting (especially world creation)

Security and authentication

There is a detailed discussion in the SIP protocol RFC (http://www.faqs.org/rfcs/rfc3261.html) (see chapter 26 and especially 26.4). In short:

• existing security solutions are not perfect (of course)
• they can re-use existing infrastructure: HTTP Digest Auth, S/MIME, TLS...
• none of them is really meant to work over UDP, because of session initialization overhead and packet size overhead

Authentication is probably more important than confidentiality in Solipsis. Of course, an ad-hoc Web-of-trust with public keys would be the best solution. It is not obvious how we can shoehorn this into a datagram-based connectionless protocol without ruining its efficiency.