[netfinity]

@tHeWiZaRdOfDoS
Well, since my SCT is only in alfa state I didn't bother adding any new tags but made the feature rely on the modname in the ET_MODVERSION tag. That means my mod cannot send do SCT to other mods than itself.

@SlugFiller
Well the devs seems to have a different view about that. They are not very keen on people adding their own tags, and especially not string tags.

This post has been edited by netfinity: 20 May 2006 - 07:01 PM

[tHeWiZaRdOfDoS]

Well I was about to use this method (string-type tag) but I also don't like it (even if it's just "rec" for "recommendations")... but I'll give it a shot... as there's unfortunately no "howto" available for official side... and I'm pretty sure other mods would want to adapt it (without sending "Tombstone" as the modstring )

[DavidXanatos]

OT @ Wizard
I can not recognise that other people (excepted netfinity) who add tags asked the devs first, so why should I?
You can not descripe tags in the hello packets as spam, since thay have a use and are required by other mods.
The neo does not send any not standard opcodes when the remote cleint does not support them.
So it works as in the describtion of FS, just there are not string tags but uint8 tags.

About the ICS v2, I think as SF made coments v2 he also does not ask the moders for their permission. So why should I?

I used the upper bytes of the ICS tag for SCT to save bandwidth, and since no one used them since many yeras i dont expect anyone in the next 3 years to have a real need for them, besides there are still 2 bytes unused in the ICS tag, and SCT, is in some kind similar to ICS booth descripes unfinished parts.
OT end

Back on Topic
As a sollution for the problem I would suggest the following:
Send an TAGTYPE_BLOB tag, always with the same tag id, that contains binary data, of a variable length > 4, the first 4 bytes would identyfy the feature itself, 4E9 combinations are enough and 4 chars are long enough to contain a short name like SCT_, ICS_, _REQ, or what ever. the bytes > 4 would be obtional and the moder could use them for any kind of informations he wants.
This ads more overhead that TAGTYPE_UINT32 tags but much less than TAGTYPE_STRING tags. And it prevents conflicts.

Using a bit field like netfinity suggest, may lead to to much conflicts, and remember not all features can be supported or not, there may be different versions and flags so for a feature we should reserve at least 4 bits than we can increase the version up to 15 or use only the 3 first than to 7 and the last one as a flag. This lead us to maximal 8 to 11 features per one tag.
The neo have already 10 protocol extensions Including the webcache tags (and the encryption needs more informations then just the version)

David

[Kry]

DavidXanatos, on May 21 2006, 02:46 PM, said:

So it works as in the describtion of FS, just there are not string tags but uint8 tags.

And that's the problem. I don't know if I'm going to be sad or amused when those opcodes start being used by the official for official protocol extensions that you won't be able to use unless you break your own backwards compatibility.

Anyway, sending lots of tags in the hello packet IS spam. You send that to EVERY client, increasing overhead.

[tHeWiZaRdOfDoS]

Well Kry, you understand me
That's exactly what I'm talking about... compatibility!
Hmmmm... how 'bout that:

After receiving Hello/Info packets and making sure your "opponent" is an eMule client and that it's a mod, then send a new OP_EMULEPROT package with a new OP_MODFEATURES including all the precious information?

That's a feature request to all modders now! What do you think?
I propose to put that into ConnectionEstablished() with a check for SO_EMULE && !m_strModVersion.IsEmpty() and let's say the next official release or 0.48a.

The modstring has still to be sent (or at least something on the CT_MODVERSION tag) but all other tags can be removed and put into the new packet...
We could also open up a new thread for all the features... backwards compatibility should be ensured as we can still send those tags in the OP_HELLOANSWER if it's an old client... this would take away lots of OH as not all clients will get spammed with unneeded tags...


EDIT: typo...

This post has been edited by tHeWiZaRdOfDoS: 21 May 2006 - 05:44 PM

[Kry]

While it's a good start, I see it far from perfect... what if some other non-emule client wants to follow a similar approach?

I guess they can send something in the mod tag for that... and the check for SO_EMULE can be removed. Then again, it seems sub-optimal. A new tag for "non-pure-protocol" could fit better for that task, as it won't force non-mule clients to send modtags.

As for the rest of the tags, yes, they can be moved to a new packet that is sent only for non-pure clients just as you say. The problem is ofcourse the backwards compatibility, so I think the test would be more like:

client send the non-pure tag
-> send new opcode
else
if SO_EMULE && client is <=0.47a
---> send old tags on helloanswer
else
---> do nothing

This means the new tag/opcode must be agreed/coded/etc before the next official, and so it should be used as soon as mod merge the next official.

This is good as new offcial / non-mule / etc won't get any more spam than this new tag. Problem is old clients keep getting it, but hey, at least only eMule ones (so this solve my spam problems )

Point out my mistakes, please.

[DavidXanatos]

hmm....
This don't grant apriori a backwards compatybility becouse at the point we sendt the first hello we dont know should we include the old tags for compatybility or can we use the new packet. So we are where we started again, at least for the backwards compatybility. We can use the old way and send the tags only in the first hello when the clinet have no modstring.

Howeever, an extra packet is indeed a good idea,
its exactly what the emule devs did long time ago for emule features with the OP_EMULEINFO packet, to dont spam donkys and co, befoure in the first 0.4x versions they merged all features to bitfilds inside of the hello packet.
We could create a new tag type, with 32 bit long ID's, so the moders would have an near to infinite pool to implement everythink thay can immagine. In standard packets like hello we can not include nonstandard tags, in a new packet we can.
And for all new features it means a much better handling of the compatybility and overhead.

Since currently only a few mods have more than ICS implemented, it may be worth to drop the backwards compatybility for the few features with older clinets and use such "modfeature" packet, exclusivly.

btw: I would also like to see an new protocol opcode OP_MODPROT for all mod packets, webcache already do this, we could just rename it and use for other features to.

I'll consider the implementation of such sollution in my next release...

David

[tHeWiZaRdOfDoS]

DavidXanatos, on May 21 2006, 07:15 PM, said:

This don't grant apriori a backwards compatybility becouse at the point we sendt the first hello we dont know should we include the old tags for compatybility or can we use the new packet. So we are where we started again, at least for the backwards compatybility. We can use the old way and send the tags only in the first hello when the clinet have no modstring.

Hmmmm... have to recheck but usually we will send an OP_HELLOANSWER in any case, no?

Quote

btw: I would also like to see an new protocol opcode OP_MODPROT for all mod packets, webcache already do this, we could just rename it and use for other features to.

I agree but...

Quote

I'll consider the implementation of such sollution in my next release...

please don't do that!
Even if it's only for once please let's find a COMMON solution! This is something that has to be accepted and adapted by the whole modding community so I totally dislike the idea that a single person should decided about it!

[Kry]

tHeWiZaRdOfDoS, on May 21 2006, 08:20 PM, said:

DavidXanatos, on May 21 2006, 07:15 PM, said:

This don't grant apriori a backwards compatybility becouse at the point we sendt the first hello we dont know should we include the old tags for compatybility or can we use the new packet. So we are where we started again, at least for the backwards compatybility. We can use the old way and send the tags only in the first hello when the clinet have no modstring.

Hmmmm... have to recheck but usually we will send an OP_HELLOANSWER in any case, no?

Either you send the hello without that capability and the other guy replies he has it, so you use it, or you get the hello from the other guy and you reply you have it, so the other guy can use it. I see no problem. as long as ONE of you knows the other guy is capable, it just has to send the new opcode and so you both know you both have it.

Quote

Quote

I'll consider the implementation of such sollution in my next release...

please don't do that!
Even if it's only for once please let's find a COMMON solution! This is something that has to be accepted and adapted by the whole modding community so I totally dislike the idea that a single person should decided about it!

My point exactly.

[Avi-3k]

to save at much as possible, i thing the official devs
should decide on a marking in the EMULEPROT packet
to let other clients know we support an extended mods features
(one bit is enough, it can be exchanged in the misc tag)

i agree with the rest of what Kry and Wiz said...

@David Xanatos
like others said, please wait before there's an agreement
so everyone will stay compatible with each other...

Regards,
Avi3k

[DavidXanatos]

Hi,
I wroted a first draft Suggestion of the "Mod protocol enhancement specyfication".
what do you think about it?

Suggestion for "Mod protocol enhancement specyfication" revision 0.1
A summary of ideas from David, netfinity and tHeWiZaRdOfDoS

Quote

Support/Version flags for mod protocol extensions must be placed as tags in a special mod packet with the opcode OP_MODFEATURES.
The packet  reading function must support the new tyg type CModTag (for tag format specyfications see *).
This packet is to be sendt on UpDownClient::ConnectionEstablished() to clients that identyfy them selvs as mods that explicitly support that feature and the OP_MODPROT protocoll type. (for identification note see **)
There are 4 basic types of Mod Protocoll Extensions:
0. Mod data inserted in regulary packets, not recomended.
1. Own mod packets, all such packets are to be sendt as OP_MODPROT (***), don't use the ed2k/emule packets to prevent conflicts with future official client versions.
2. Multi packet extensions (that are datas added to the eMule MultiPacket packet), OP_MODEXTENSION must be always used as container for thet extensions, it contains a sub list of opcodes and content like the multi packet it self, in this way conflicts with official features are avoided.
3. UDP packet (OP_REASKFILEPING/OP_REASKACK) extension, mod extensions are to be placed on the very end of the packet, and formated  in the same way like the content of the multi packet.
All above extensions can be used only when the remote clinet informs that he support them.
Recived extended data even if they have a known Opcode are to be droped when the remote cleint does not said that he support it.

* The CModTag have to haldle 16, 32, 64 bit long ID's to grand enough space for mod features with less overhead than string named tags, and to be usable with "switch".
** A single hello tag can be used for this purpose, sending the packets apriori to all mods is a waist of overhead, idealy would be to het a bit from the MISC tag from the devs.
*** The values from the webcache prot can be used here.

Comments, improvement ideas?

David

[Devil Doll]

Quote

Quote

Support/Version flags for mod protocol extensions must be placed as tags in a special mod packet with the opcode OP_MODFEATURES.

Comments, improvement ideas?

When (during the handshake) and to whom will this opcode be sent? You're talking about an ed2k protocol addon, thus donkeys, amules etc. are allowed to participate in this game if they want to - right?

[DavidXanatos]

Devil Doll, on May 22 2006, 05:50 PM, said:

Quote

Quote

Support/Version flags for mod protocol extensions must be placed as tags in a special mod packet with the opcode OP_MODFEATURES.

Comments, improvement ideas?

When (during the handshake) and to whom will this opcode be sent? You're talking about an ed2k protocol addon, thus donkeys, amules etc. are allowed to participate in this game if they want to - right?

The mod feature packet will be sendt directly after the handshake was finished to everyone who says us during the handshake that he supports mod extensions.
donkys and co are free to use this technick.

David

[Xman1]

one sidenote:
sending an extra packet means an extra socket->Send() which means a Overhead of 40 Bytes (TCP/IP Header). You should consider this.

Because Xtreme doesn't use any additional opcodes (except modstring) I can't give you any suggestion.

[CiccioBastardo]

Extra packet is uselss when the same thing can be put into the hello packet under a unique tag id.
You have to consider 40 bytes for the overhead and the delay of an additional packet that requires an answer (I think the other part would like to know what the other client supports as well).
You can compress it to few bytes to send at least once (on presentation, if you don't know the other client is going to support mod extensions) and it can be removed altogether when the other mod doesn't show a mod string.

I think bandwidth can be sapred this way, but I may be wrong.

[Devil Doll]

That's why I was asking when and to whom the new opcode will be sent: The later the time and the fewer the clients, the less relevant will be those 40 bytes overhead, compared to sending the features opcode to every client.

[tHeWiZaRdOfDoS]

Hi there... I was sick so I couldn't answer... I dunno wether we really need a CModTag type (imho 255 opcodes should be enough) but well, it won't hurt...
Using the WebCache Prot seems to be a good idea as many clients already implemented this system...

Considering the TCP OH... might there be a way to "combine" multiple packets to a single Send()? Though I guess in the long run we would save a lot more overhead than we would waste....

[DavidXanatos]

Hmm...
indeed the IP overhead is also there, I would sugest the following solution:

We will bind the modinfopacket to the end of the hallo answer and hello when we know that the client supports it.
When we don't sent it and gets the hello answer that says he supports it we will send the info packet separatly so the ip overgead will be a one time and only between mods that supports it...

David

[Kry]

That's basically what I said above.

So, all that is needed is a bit on the hello packet to mark you have mod extensions, then you can do whatever you agree to do with the new protocol/tags/etc, as long as only is sent to those having that bit set? That solves the hello/helloanswer thing, if 1 bit can be erserved by the official developers for this feature, so other people won't get such packets.

Are you all ok with having that bit? That means not sending off-the-protocol packets or tags to clients that do NOT have that bit set.

[Xman1]

it's so easy:
we already send an identification that we are using a mod: the modstring. This modstring should stay like it is to be compatible with all the other clients (also official which supports the mod string).

If you receive a modstring you can tell the other clients which features you support.