[DavidXanatos]

I have prepared today a reference implementation of a UTP based NAT-Traversal feature for eMule.
You can find it on: https://gitorious.or...eomule_reloaded

The implementation is designed to be strait forwards and simple, for data streaming over UDP an existing state of the art library is used. The Micro transport Protocol (uTP) https://github.com/bittorrent/libutp released by the uTorrent developers. It provides a generic Streaming system over arbitrary UDP sockets optimized for High Speed low latency P2P transfers.

The reference implementation Mod contains Nat-Traversal v3 and Extended Source Exchange, all switchable via preprocessor. The code is untested, will do that when I find some more hours of spare time.
Whats missing ts the last eserver protocol extension from lugdunum for NAT-T I will add that later.
Also I will add a reference implementation for IPv6 Support, and possibly some variation of SCT.

David X.

[petermrg]

Great work!

These are two major features emule is missing.
I will try to get some time to test it!

[]Axl[]

Really a great work!

Now I wait a mod to test the new code

[tHeWiZaRdOfDoS]

Great work, David
I hope I'll find some time for it, soon.


Found some time under my bed, so...
There are some misses in the #ifdef parts (try to compile without them should indicate them) - nothing problematic, though.
Also, in PartFile.cpp and KnownFile.cpp:

Quote

if(forClient->SupportsSourceExchangeExt())
{
const uint8 uDirectUDPCallback = cur_src->SupportsDirectUDPCallback() ? 1 : 0;
const uint8 uSupportsNatTraversal = cur_src->SupportsNatTraversal() ? 1 : 0;
byCryptOptions |= (uSupportsNatTraversal << 7) | (uDirectUDPCallback << 3);
}

Shouldn't that be

Quote

if(forClient->SupportsNatTraversal())
{
const uint8 uDirectUDPCallback = cur_src->SupportsDirectUDPCallback() ? 1 : 0;
const uint8 uSupportsNatTraversal = cur_src->SupportsNatTraversal() ? 1 : 0;
byCryptOptions |= (uSupportsNatTraversal << 7) | (uDirectUDPCallback << 3);
}

or rather write that info regardless of the remote clients' state? Doesn't hurt in any way...

Similar part in otherfunctions.cpp @ GetMyConnectOptions - we could and should indicate our support for NAT-T not only in callback case - what do you think?

This post has been edited by tHeWiZaRdOfDoS: 05 August 2013 - 10:45 AM

[DavidXanatos]

Will look over the preprocessors in the evening.

about changing if(forClient->SupportsSourceExchangeExt()) to if(forClient->SupportsNatTraversal()) the callback stuff is useless without knowing the UDP port that normal SX does not transmit, one could do a test for booth conditions though.

Or we could employ a trick from bittorent, use the same port number for TCP and UDP, so if our router does not change this (like when its only a firewall and not a nat) we can try the TCP port we know for UDP and if we are lucky get a connection, or a callback, though any half decent firewall wil block this and reqhire a bidirectional hole punch, those a buddy to synchronize this attempt.

That is why I said in the past that NAT-T needs an extended SX to work really well.

about otherfunctions.cpp @ GetMyConnectOptions to my understanding the callback argument is only set to false when writing con options for firewall checking, buddy request and in a direct callback request,
for normal file publishment in kad for example the NAT-T status will be always sent.

Cheers
David X.

This post has been edited by DavidXanatos: 05 August 2013 - 12:04 PM

[tHeWiZaRdOfDoS]

Well, the point is: we send that UINT anyways, so there's no harm informing the remote user at any time possible.
Also, the if(forClient->SupportsSourceExchangeExt()) doesn't make sense at that point because you checked it before for NeoXS, thus that "if" will never be true.
AFAIR eMule doesn't allow the TCP and UDP port to be the same but that would be a nice trick, indeed.

[DavidXanatos]

I dont quite follow, why should the check fail in this part of the code we always enter no mather if nei sx or normal sx.

abot the uint sue we always sent it but no point in settin glags oothers can not understand

[tHeWiZaRdOfDoS]

Ah, I see, I got confused with the brackets. Anyways, that function is not present for USE_NAT_T except USE_NEO_SX is used.

Though, while using USE_NEO_SX along with USE_NAT_T really makes sense, I'd suggest implementing both for an "official" NAT-T implementation, i.e. simply indicating NAT-T support also implies that the new SX system is supported - what do you think?

[DavidXanatos]

I'm not sure its wise to use one flag for booth protocols.
A client that supports NAT-T could opt for a different extended XS implementation, one that if for example much less flexible but in turn much more compact. This would not affect his NAT-T support directly.

So I think its better to keep the two flags separated.

David X.

[tHeWiZaRdOfDoS]

Yes, they could - but IMHO they should AT LEAST support the current implementation.

EDiT: you see, as we are just negotiating about an official implementation here, we have the opportunity to make NAT-T really helpful by including extended XS into NAT-T specifications

This post has been edited by tHeWiZaRdOfDoS: 06 August 2013 - 05:12 PM

[DavidXanatos]

hmm... I'm not sure its a clean design to force multiple features together that in principle to some extend could work on their own.

For example some one could opt for supporting Extended SX only, to be able to share LowID sources with kad callback, but not add NatT.

And in case there will be a more streamlined Ext SX version in a few years one may want to drop support for old Ext SX.

I don't think it is reasonable to maintain full compatibility over decades. For example NeoLoader does not support Mule SX v1, only the current SX v2 protocol and the new Extended SX.

Maintaining all old features is ok for old clients, but for example if someone would want to port Mule on lets say java to run it as an native android app (not a Qt one that only runs on selected CPU's) the easiest for him is if he can assemble the most up to date feature set and can safely drop reasonably outdated extensions.

Also a 32 bit long tag has plenty of place for many many various extensions

David X.

[tHeWiZaRdOfDoS]

That is - of course - correct. Still, we are talking about the "official" NAT-T code that will be used in a lot of mods & clients. Thus, we should use that opportunity to "enforce" some absolutely useful extensions.
The big advantage would also be that we would NOT rely on an additional inofficial hello tag - I'm pretty sure that 99% of all adapting mods & clients would NOT support any of the additional "Neo" features and thus it'd be a (near) complete waste to include that tag.

It'd be a different thing if we could agree on using a new & clean tag in the mod prot for "modding community" features, i.e. features that are not only used by NeoMule and NeoLoader but a large part of the modding community.

Again, I think that those feature definately are nice. I just think we should implement it as powerful and clean as possible.

[DavidXanatos]

How else than through a hello tag we would advertise the support?
To my understanding the question is only if we use one or two bits in the new tag.

Also from the top of my head there are 2 more features that would go into the tag, IPv6 support and what ever version of SCT.

Extended SX is also needed for IPv6.

A community extension tag would definitely be a cool thing.

I currently use the NeoTag like this:

		UINT
		ExtendedSourceEx	: 1, // Extended Source Exchange with variable source info
		SupportsNatTraversal: 1, // NAT-T Simple Traversal UDP through NATs 
		SupportsIPv6		: 1, // IPv6 Support
		//SupportsRangeMap : 1, // SCT
		// We write generic eMule protocol extensions from the top
		Reserved			: 27,
		// And NeoLoader specific protocol extensions from the bottom
		NeoKadVersion		: 1, // Neo kademlia support
		HostCache			: 1; // Hoster Cache

I think it is a reasonable compromise between saving overhead and having plenty of space for new extensions.
I don't really see adding more than 8 Neo Only extensions, I mean all this features have real value only if supported by sufficiently many clients, may be with a few minor exceptions.
So for the most of them one would need to agree on a community implementation anyways.

Actually I think that even NeoKad once hardened and open sourced (not GPL though but something like http://piratepad.net/x43RU73nQN ) could be used by mods as there will be some kad for IPv6 needed as eMules kad is made for IPv4 only.

Though if you don't like it we could also select a dedicated tag fully for the community, I'm fine with booth options.

Cheers
David X.

This post has been edited by DavidXanatos: 06 August 2013 - 07:40 PM

[DavidXanatos]

So what do we do?

New Community Tag or do we use the Neo Tag ?

[tHeWiZaRdOfDoS]

Well, I vote for a comm tag
It'd be nice to hear other (active) modders' opinions, though...

[Tuxman]

I as an active modder don't care how you name it.

[tHeWiZaRdOfDoS]

It's not about the name but I'd suggest using a clean opcode and not one that's already used for some Neo-only features.

[Neoo26]

The features are "Neo-Only" because only Xanatos implemented it, actually.

Quote

I as an active modder don't care how you name it.

[tHeWiZaRdOfDoS]

... and will.
SCT will receive a separate tag/structure, IPv6 and NeoKad (as SS explained somewhere else) do not work as expected and/or won't be used due to other problems (license, etc.), hoster support will/should never be added to eMule which leaves the remaining NAT-T and XS tags which make sense and should be adapted into a clean comm tag.

[Neoo26]

In less word, you saying that a developer will not implement it because of the name ?

"Neo" mean "new" not "wrote by david x".