| The GnutellaNet protocol |
Last update: 2001 Feb 5
Original version was by gene@wego.com.
This verson is updated to correct the endian-ness errors, and clarify and update the situation with the network size and TTL values.
Notes
Everything is in network byte order unless otherwise noted. Byte order of the GUID is not important.
Apparently, there is some confusion as to what "\r" and "\n" are. Well, \r is carriage return, or 0x0d, and \n is newline, or 0x0a. This is standard ASCII, but there it is, from "man ascii".
Keep in mind that every message you send can be replied by multiple hosts. Hence, PING is used to discover hosts, as the PONG (Ping reply) contains host information.
Throughout this document, the term server and client is interchangeable. Gnutella clients are Gnutella servers.
Thanks to capnbry for his efforts in decoding the protocol and posting it.
How GnutellaNet works
General description
GnutellaNet works by "viral propagation". I send a message to you, and you send it to all clients connected to you. That way, I only need to know about you to know about the entire rest of the network.
A simple glance at this message delivery mechanism will tell you that it generates inordinate amounts of traffic. Take for example the defaults for Gnutella 0.54. It defaults to maintaining 25 active connections with a TTL (TTL means Time To Live, or the number of times a message can be passed on before it "dies"). In the worst of worlds, this means 25×24^6, or 4,777,574,400 messages resulting from just one message!
Well, okay. In truth it isn't that bad. In reality, there are usually only a few thousand Gnutella clients on the GnutellaNet at any one time (and there have never been more than about 30,000). That means that long before the TTL expires on our hypothetical message, every client on the GnutellaNet will have seen our message.
During 2000, many Gnutella clients used smaller defaults for the TTL and the number of active connections. Some went so far as to lower both to 4. However, this is much too low. Even if the network were "connected" in the most perfect manner, 4 links per node and a TTL of 4 is only enough to connect 96 clients. Another popular combination is 4 links per node and a TTL of 7, which can connect 1155 clients, but again only if the network is "wired" perfectly. If all the nodes hd 4 links per node and the network connections changed purely at random, the TTL would have to be about 12 to 15 in order for most messages to be able to reach all nodes.
However, some network structure has evolved. Smarter clients and higher-bandwidth clients have formed a "backbone", with older clients and low-bandwidth users pushed off to the edges. If the backbone clients maintain a higher number of connections, the GnutellaNet can work even with a TTL of 7 to 10 even when most of the clients only maintain 4 connections.
GUIDs
Obviously, once a client sees a message, it's unnecessary for it to process the message again. The original Gnutella designers, in recognition of this, engineered each message to contain a GUID (Globally Unique Identifier) which allows Gnutella clients to uniquely identify each message on the network.
So how do Gnutella clients take advantage of the GUID? Each Gnutella client maintains a short memory of the GUIDs it has seen. For example, I will remember each message I have received. I forward each message I receive as appropriate, unless I have already seen the message. If I have seen the message, that means I have already forwarded it, so everyone I forwarded it to has already seen it, and so on. So I just forget about the duplicate and save everyone the trouble.
Topology
The GnutellaNet has no hierarchy. Every server is equal. Every server is also a client.
Each Gnutella server only knows about the servers that it is directly connected to. All other servers are invisible, unless they announce themselves by answering to a PING or by replying to a QUERY. This provides amazing anonymity.
Unfortunately, the combination of having no hierarchy and the lack of a definitive source for a server list means that the network is not easily described. It is not a tree (since there is no hierarchy) and it is cyclic. Being cyclic means that every message a client sends out will arrive back multiple times unless all (or at least most) of the clients are careful to use the GUIDs to drop the duplicates.
Connecting to a server
After making the initial connection to the server, you must handshake. Currently, the handshake is very simple. The connecting client says:
GNUTELLA CONNECT/0.4\n\n
The accepting server responds:
GNUTELLA OK\n\n
After that, it's all data.
Header
| bytes | summary | description |
| 0-15 | Message identifier | 16 bytes that will identify this message and distinguish it from all others sent on the network. Windows clients use a Windows GUID (which is 16 bytes). Other clients should generate 16 bytes based on something that will make it unique (like your local IP address, the current time, and some random numbers) |
| 16 | Payload descriptor (function identifier) | Value Function |
| 0x00 PING | ||
| 0x01 PONG (Ping reply) | ||
| 0x40 PUSH request | ||
| 0x80 QUERY | ||
| 0x81 HITS (Query reply) | ||
| 17 | TTL | Time to live. Each time a message is forwarded its TTL is decremented by one. If a message is received with TTL less than one (1), it should not be forwarded. |
| 18 | Hops | Number of times this message has been forwarded. |
| 19-22 | Payload length | The length of the ensuing payload. |
Payload: PING (function 0x00)
No payload
Routing instructions for PING
Forward PING packets to all connected clients. Most other documents state that you should not forward packets to their originators. I think that's a good optimization, but not a real requirement. A server should be smart enough to know not to forward a packet that it originated.
A cursory analysis of GnutellaNet traffic shows that PING comprises roughly 50% of the network traffic. Clearly, this needs to be optimized. One of the problems with clients today is that they seem to PING the network periodically. That is indeed necessary, but the frequency of these "update" PINGs can be drastically reduced. Simply watching the PONG messages that your client routes is enough to capture lots of hosts.
One possible way to really reduce the number of PINGs is to alter the protocol to support PING messages which includes PONG data. That way you need only wait for hosts to announce themselves, rather than discovering them yourself.
Payload: PONG (query reply) (function 0x01)
| bytes | summary | description |
| 0-1 | Port | IPv4 port number, using little-endian byte order: The low byte comes first. For example, if the port number is 6346, the first byte is 202 and the second byte is 24 (because 24×256+202=6346) |
| 2 | IP address | IPv4 address, first byte |
| 3 | IP address | IPv4 address, second byte |
| 4 | IP address | IPv4 address, third byte |
| 5 | IP address | IPv4 address, last byte. Please note this byte ordering not little-endian. For example, if the IP address is 10.23.45.67, byte 2 will be 10, byte 3 will be 23, etc. |
| 6-9 | Number of files | Number of files the host is sharing. |
| 10-13 | Number of kilobytes | Number of kilobytes the host is sharing. |
Routing instructions for PONG
Like all replies, PONG packets are "routed". In other words, you need to forward this packet only back down the path its PING came from. If you didn't see its PING, then you have an interesting situation that should never arise. Why? If you didn't see the PING that corresponds with this PONG, then the server sending this PONG routed it incorrectly.
Payload: QUERY (function 0x80)
| bytes | summary | description |
| 0-1 | Minimum speed | The minimum speed, in kilobytes/sec, of hosts which should reply to this request. |
| 2+ | Search criteria | Search keywords or other criteria. NULL terminated. |
Routing instructions for QUERY
Forward QUERY messages to all connected servers.
Payload: HITS (query reply) (function 0x81)
| bytes | summary | description |
| 0 | Number of hits (N) | The number of hits in this set. See "Result set" below. |
| 1-2 | Port | IPv4 port number. |
| 3-6 | IP address | IPv4 address. Same byte ordering as in PONG payload description above |
| 7-10 | Speed | Speed, in kilobits/sec, of the responding host. |
| 11+ | Result set | There are N of these (see "Number of hits" above). |
| bytes summary, description | ||
| 0-3 Index: Index number of file. | ||
| 4-7 Size: Size of file in bytes. | ||
| 8+ File name: Name of file. Terminated by double-NULL. | ||
|
Last 16 bytes | Client identifier | GUID of the responding host. Used in PUSH. |
Routing instructions for HITS
HITS are routed, the same way PONGs are -- send these messages back on their inbound path. That means, send them only to the connection from which you recieved the corresponding QUERY. The corresponding QUERY is the one with the same message identifier (GUID) in its header as this reply.
Payload: PUSH request (function 0x40)
| bytes | summary | description |
| 0-15 | Client identifier | GUID of the host which should push. |
| 16-19 | Index | Index number of file (given in query hit). |
| 20-23 | IP address | IPv4 address to push to. |
| 24-25 | Port | IPv4 port number to push to. |
Routing instructions for PUSH
Forward PUSH messages only along the path on which the query hit was delivered. If you missed the query hit then drop the packet, since you are not instrumental in the delivery of the PUSH request.
Downloading from a server
Downloading files from a server is extremely easy. It's HTTP. The downloading client makes a new connection (a TCP connection) directly to the IP address of the server with the file to be downloaded. It then requests the file by sending an HTTP header:
GET /get/1234/strawberry-rhubarb-pies.rcp HTTP/1.0\r\n Connection: Keep-Alive\r\n Range: bytes=0-\r\n \r\n
As you can see, Gnutella supports the range parameter for resuming partial downloads. The 1234 is the file index (from HITS packet described above), and "strawberry-rhubarb-pies.rcp" is the filename.
The server will respond with normal HTTP headers. For example:
HTTP 200 OK\r\n Server: Gnutella\r\n Content-type:application/binary\r\n Content-length: 948\r\n \r\n ds*GKh:RkFk@)gjGLgK\Gh@+$L__^KVU-`D@:`/:#%KfTYJ^Y(BWDFL$#:rltyh...
The important bit is the "Content-Length" header. That tells you how much data to expect. After you get your fill, close the socket. Also note the double \r\n before the beginning of the data. No special encoding is used -- if the file is bonary, the data will be binary.