Talk:Enhanced messaging protocol
From AzureusWiki
Contents |
older proposal
protocol format:
A: <protocol ident><supported messages><stream properties>|wait for B|<stream of messages> B: |wait for A|<protocol ident><supported messages><stream properties><stream of messages>
<protocol ident> = 1 byte length + "OpenBT-Packets" + 1 byte min version + 1 byte max version
<message> = 2 byte ID, 2 byte length of payload, payload
<supported messages> = special message; ID 0xFF00; <stream properties> = special message; ID 0xFF01; payload array of [length(1), key, length(1), value] sets that indicate stream and client properties, e.g. which ports are used by the client, the clients name, a hash of the associated content etc.
Messages
Each message consists of a 2 byte message ID + 2 byte length and its payload.
Bundles
Each bundle represents an atomic feature of a protocol. Each bundle can reserve a part of the ID space and map those dynamic IDs to relative, internal IDs that facilitate simple message processing.
Each bundle should be managed by an owner or a closed development group to ensure concurrent development of features doesn't lead to collisions in the versioning of the protocol or message IDs. Read as: If you're not maintaining a message bundle don't add extensions to it, either create your own bundle or try to contact the maintainers of that bundle.
BT_Main
bundle name: BT_Main
Version 0:
Represents the classic bittorrent protocol messages
- 0 - choke
- 1 - unchoke
- 2 - interested
- 3 - not interested
- 4 - have
- 5 - bitfield
- 6 - request
- 7 - piece
- 8 - cancel
The bittorrent handshake values are mapped to the following FF01 message keys:
- BT_infohash
- BT_clientID
- the reserved bitfield won't be mapped
Version 1:
Implements version 0, plus:
- 9 - stacked have 16 bit (array of 16 bit shorts representing indivdual piece numbers), allows bulk-sends of have messages and thus reduces the overhead compared to classic BT, esp. for torrents with many small pieces
- 10 - stacked have 32 bit (array of 32 bit integers representing individual piece numbers), shouldn't be used unless the torrent has more than 65536 pieces
redefined messages:
- 5 - bitfield, if the length of this message is 0 it is a seed
additional mappings are:
- BT_port - the listening port
- BT_DHTport - the UDP listening port for the distributed tracker
- BT_CName - the full name of the client
Reserved bundle
The reserved bundle has the base ID 0xFF00 and spans to 0xFFFF
0xFF00
This is a special message used to negoatiate the additionally supported bundles and reserve a part of the message ID space to them.
It must be sent after the initial protocol identifier. If one 0xFF00 message shouldn't be enough (since the message length is restricted to only 64KiB-1B) multiple messages may be send and must be interpreted as additional supported messages.
the payload is an array of the following data structures:
- bundle ID offset
- length: 16 bit
- This defines the mapping offset for each message in the bundle. A bundle reserves a part of the ID space to its own messages. It must as many message IDs as the most greedy supported version of this bundle needs, i.e. no other bundle may have an offset that falls within the offset of the specified bundle + the number of needed messages.
- min version
- length: 8 bit
- specifies the minimal supported version of this message bundle.
- max version
- length: 8 bit
- specifies the maximal supported version of this message bundle. If the peer does advertise support for this bundle but the min-max regions don't overlap the bundle has to be considered as unsupported by both sides. Otherwise the highest mutually supported version will be assumed for all messages recieved within this ID space
- N
- length: 8 bit
- specifies the length of the following field in byte
- bundle name
- length: N byte
- a unique identifier of the bundle [preferrably a human-readable string which is not null-terminated]
0xFF01
This is a special message used to advertise additional metadata associated to the the stream or client.
It can be send at any time in the stream to advertise changed conditions, e.g. port changes. Interpreting this message is up to the upper protocol layer.
The payload consists of arrays of the following data structure:
- N
- length 8 bit
- defines the length of the key
- key
- length N byte
- a unique identifier of the value [preferrably a human-readable string which is not null-terminated]
- M
- length 8 bit
- defines the length of the value
- value
- length M byte
- defines the value associated to the key
0xFF02
This is a 0-sized special message that indicates that the handshaking phase is done and the client doesn't have to wait for additional 0xFF00 messages anymore.
modular layered protocol
layer 0: protocol negotiation/handshake
handshake format for initiatior and responder:
<1 Byte; pattern size N> <N Bytes; ''bytepattern'' 1> <4 Bytes; ''MTU''> -- point 1 M times <1 Byte; pattern size O> <O Bytes; ''bytepattern'' 2 to M> -- point 2
- each bytepattern identifies a supported protocol, where the first byte pattern is the preferred one and the following ones are in descending preference order
- the responder may respond with any offered bytepattern and can append additionally supported patterns, this provides backwards compatibility and early protocol handshaking for future enhancements
- the MTU unit can be used to signal a maximum block size for the next layer, this can be useful for non-streamed transports (e.g. UDP)
- M can be 0 on both sides
- the derived MTU should be the maximal mutually supported MTU, i.e. negotiated MTU = min(MTU1,MTU2)
- The responder may immediately send its reply as soon as it has recieved up to point 1, but has wait up to point 2 (i.e. until the initiator has finished sending his patterns) if it does see a supported bytepattern. The initiator has to wait at least until it has recieved up to the responder's point 1 before it continues with the next protocol layer
layer 1: multi stream/message muxing
bytepattern for this muxing protocol (placeholder for the moment): 0x00EC4BA7 (length 4 bytes)
Note: This layer assumes that the lower layers constitute a reliable packet or stream transport
unfragmented message: <1 byte; status flags set to 0x00 (unfragmented)> <2 bytes; length N> <N bytes; payload for the next layer> fragmented message/stream announcement: <1 byte; status flags set to 0x01 (fragmented)> <2 bytes; multi fragment XOR stream identifier> <4 bytes; overall size> fragmented message chunk: <1 byte; status flags set to 0x03 (fragmented & payload)> <2 bytes; multi fragment identifier> <1 byte; sequence number> <2 bytes; chunk size N> <N bytes; chunk> stream terminator: <1 byte; status flags set to 0x05 (fragmented & term)> <2 bytes; stream identifier>
- streams are created by setting the overall size to 0
- Fragmented messages and Streams share the same payload and initiator message but do NOT share the same namespace
- each time a new fragmented message is announced the identifier should be incremented to avoid collisions (if the 2 byte limite is reached it should wrap around)
- each new stream must have an unused stream identifer, a simple wraparound of the ID is not valid since older streams still might be in use
- the hard limit for each chunk is 65535 bytes of payload, although an MTU of the lower layer may restrict that. But to ensure fast message interleaving any chunk shouldn't be bigger than 16KiB (+- some overhead). An implementation may and should choose to dynamically adjust the message slicing to the connection speed to ensure that small control messages can be interleaced between larger data (multi-chunk) messages.
- each stream and fragmented message must have its own counter for the sequence number which has to be increased with each chunk. It's purpose is to allow reordering on a small scale, once it reaches the 1 byte limit it has to wrap around
Purpose of this layer:
- This layer should provide a transparent messaging/streaming link between two endpoints that allows arbitrary sized messages (<4GB) or streams to be interleaved and thus give smaller, high priority messages a chance to arrive in time even when a larger transfer is in progress
layer 2: application level messages and handshaking
This layer makes use of layer 1 to send with different purposes to the other endpoint
each message conists of
<2 bytes: message ID> <N bytes: payload>
where N is derived from the upper layer's message size by substracting the 2 bytes for the ID
predefined message 0x0000 format:
<1 byte; size N of content descriptor> <N bytes; ''content descriptor''> M times <1 byte; package name size O> <O bytes; package name> <1 byte; reserved IDs for this package P> <1 byte; min version> <1 byte; max version>
- the content descriptor should provide information about the purpose of the messaging connection, i.e. a network identifier if this protocol is used for a large interconnected network or a file hash if it is used for content specific swarming.
- only message 0x0000 is predefined
- the used IDs start with 1 and are incremented by P for each package, the meaning of each submessage (and hence ID) is up to the message maintainer
- versioning is done by using the highest mutually supported version, if some versions use less IDs than other version the highest required number of IDs must be allocated
- if some circumstances require a major version break a new package should be used
layer 3: BT2 messages
bah... make something up... multihaves, push messages... whatever
Example for a BT2 connection
- placeholder
