Official eMule-Board: Ipv6 - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Ipv6 Rate Topic: -----

#1 User is offline   Andu 

  • Morph Team
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 13015
  • Joined: 04-December 02

Posted 08 February 2011 - 01:18 AM

I know this is an old topic and it's in the FAF as a feature that will be introduced in the future. Considering that IANA has now assigned the final 5 IP blocks that were still available to the RIRs this time seems to have finally come. If estimations are right the Internet will run out of IPs by the end of the year. So the pressure to deploy IPv6 is growing on all the major stakeholders of the internet. Sometime in June there will be an IPv6 day which is supposed to act as a test run for how prepared the internet is to switch to IPv6.

Either way it seems like we will have to face the IPv6 change sooner rather than later. So I think it would be good idea if the devs would take the time to dig into this since we still have at least months till this will become an issue.
Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.


Dark Lord of the Forum


Morph your Mule

Need a little help with your MorphXT? Click here

0

#2 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 08 February 2011 - 04:33 PM

Most stuff is already written in the old thread (problems, general plan to work on it, etc), so just one comment: It will no doubt be years or decades untill this becomes a real issue meaning that there will be a consumer line which doesn't supports IPv4 anymore.

Quote

Sometime in June there will be an IPv6 day

AFAIK that only means that some big web companies put their IPv6 stuff into the AAA records of their domains (how it was always intended, but not done yet because of some buggy implementations here and there), and not that anything is switched from IPv4 to IPv6.

#3 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 19 April 2011 - 11:50 AM

View PostSome Support, on 08 February 2011 - 05:33 PM, said:

Most stuff is already written in the old thread (problems, general plan to work on it, etc), so just one comment: It will no doubt be years or decades untill this becomes a real issue meaning that there will be a consumer line which doesn't supports IPv4 anymore.

Quote

Sometime in June there will be an IPv6 day

AFAIK that only means that some big web companies put their IPv6 stuff into the AAA records of their domains (how it was always intended, but not done yet because of some buggy implementations here and there), and not that anything is switched from IPv4 to IPv6.
IPv6 day is when the large vendors really will turn on IPv6 as a trial. People like Akamai, Google, Facebook are all taking part. It is a major event that may have a serious impact on web traffic as for example Windows users suddenly start resolving IPv6 AAAA records for these services.

I would just like to point out to you if you don't already know that IPv6 and the depletion of IPv4 addresses is going to have a major (and potentially catastrophic) impact on eMule very soon now. Likely before the end of this year, and certainly not years away.

I've looked through the code and note the following:

1) there is NO IPv6 support at all. Nada. Meanwhile Transmission and other torrent p2p networks already have IPv6 built in.

2) eMule code consistently refers to uIP (an IP address) as a 32 bit integer. That assumes the only address family in use on the Internet is IPv4. IPv6 is already here. It also has no concept of a session, and equates a remote IP address to a single remote client in several places.

3) The eMule NAT traversal and UDP code also consistently assumes that a received IP address is equivalent to an end node, and that the IP address is largely invariant over time.

Maybe it is today. But you need to read up about NAT444 and Carrier Grade NAT.

IPv4 addresses have already run out in Asia Pacific. Pretty soon now, whole clouds of consumers are going to disappear behind Carrier Grade NAT.

CGN will perform a 2nd layer of NAT at multiple points WITHIN the Internet cloud, which means that the association between a remote node and it's "public" IP address may be broken. It could also mean that the received IP address is path dependent, and may for example be different between node X and node Y and between node X and node Z. Your current peer discovery mechanism may fail, as they assume a single NAT that applies to all traffic from Node X.

eMule's KAD, as implemented today, is potentially going to have serious problems with CGN.

Although this statement depends a lot on the precise implementation of the CGN and the persistence of associations across sequential NAT requests, you have to ask yourself two questions:

If I send two sessions from node X to node Y and node Z destination nodes shortly one after the other, do they both arrive at the remote eMule client with the same NATted source address?

and also over time:

if I send a packet to node X today, has Node X got the same NATted IPv4 address as yesterday if node X has been idle in the meantime, so I can establish a bootstrap to the peer to peer network?

The answer is "probably not" in both cases.

Here are a couple of examples:

1. The function CPacketTracking::InTrackListIsAllowedPacket tracks outstanding opcode request status keyed on remote IP address and opcode only. If many nodes now appear behind the same address (due to CGN and NAPT port overloading) you could very well start discarding valid requests, because they will all appear to come from the same remote IP address, even though they are different clients.

2. eMule learns its own public IPv4 address through the responses from communication with a number of other KAD peers via the receipt of the KADEMLIA_FIREWALLED_RES opcode in CKademliaUDPListener::Process_KADEMLIA_FIREWALLED_RES.

this calls

CKademlia::GetPrefs()->SetIPAddress(uFirewalledIP);

Which checks whether the IPv4 address reply has changed from previous responses, and that the last response matches this response (built in hysteresis of two identical addresses being returned sequentially to prevent spoof replies triggering a change of address)

The number of client tests is set via #define UDP_FIREWALLTEST_CLIENTSTOASK 2.

Therefore the current eMule KAD code assumes a single (public) globally unique IPv4 address, that rarely changes and that can be easily detected (2 tests).

If you think about CGN, you now have a large NAT pool of outgoing public IPv4 addresses, rather than the single largely invariant IPv4 address of CPE NAT. So the chance of two sessions having the same source address may be greatly diminished (depends on the exact NAPT allocation scheme and how sticky a client is to a NATted association). Secondly the time a particular association is persistent is expected to be a lot lower (I have seen 90 seconds in Cisco CGN docs compared to days of CPE NAT) Thirdly, there may be different NATs in the path between node x and Y and X and Z.

Wake up and smell the coffee, or pretty soon you're going to start to lose serious amounts of p2p connectivity.

The hard way to fix this is to spend many man hours trying to figure out NAT traversal over DS Lite and NAT444/ CGN. It's very tough. Look at the Teredo protocol to see how hard it is.

The easy way is to add on IPv6 support. Good luck!
0

#4 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 19 April 2011 - 03:05 PM

Quote

1) there is NO IPv6 support at all.

I do suspect that most devs are aware of this.

Quote

2) eMule code consistently refers to uIP (an IP address) as a 32 bit integer.

Which is a direct result of 1.). No point in replacing this as long as we don't support Ipv6 anyway.

Please read the prior thread about Ipv6. The main problem is that eMule (Kad) needs to have a rare ressource like IP to work properly the way it is build currently. I'm not sure if the system can be ported to IPv6 at all without major changes (not talking about codechanges to accept IPv6 addresses).

Pointing out the carrier NATs might grow is a valid problem, but fixing this is not as easy it sounds. KAD cannot accept multiple Nodes from one IP, as again having one rare ressource (IP) is currently the keystone of network security and there is no easy replacement in a decentralized system.

#5 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 19 April 2011 - 03:56 PM

View PostSome Support, on 19 April 2011 - 04:05 PM, said:

Quote

1) there is NO IPv6 support at all.

I do suspect that most devs are aware of this.

Quote

2) eMule code consistently refers to uIP (an IP address) as a 32 bit integer.

Which is a direct result of 1.). No point in replacing this as long as we don't support Ipv6 anyway.

Please read the prior thread about Ipv6. The main problem is that eMule (Kad) needs to have a rare ressource like IP to work properly the way it is build currently. I'm not sure if the system can be ported to IPv6 at all without major changes (not talking about codechanges to accept IPv6 addresses).

Pointing out the carrier NATs might grow is a valid problem, but fixing this is not as easy it sounds. KAD cannot accept multiple Nodes from one IP, as again having one rare ressource (IP) is currently the keystone of network security and there is no easy replacement in a decentralized system.

Would you be so kind as to post a link to the original thread? I searched on IPv6 and this was the only (obvious) thread that popped up.

At first glance I'm amazed that porting to IPv6 would be so hard. TCP and UDP are the same. IPv6 will almost certainly be non-natted end to end. Nodes will generally pick up a node identifier based on their unique 48 bit MAC address and this should be largely invariant [SLAAC process] unless someones really turns on privacy (although the original SLAAC address should also still respond to allow the node to run services).

I guess I really need to read that thread of why my assumptions are incorrect, rather than repeat the discussion.

This post has been edited by cgn: 19 April 2011 - 04:00 PM

0

#6 User is offline   Andu 

  • Morph Team
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 13015
  • Joined: 04-December 02

Posted 19 April 2011 - 04:14 PM

My point was that as we aren't exactly well stocked with devs it might be prudent to start working on IPv6 and all the reworking that it entails now since it's going to take quite a while anyway. Why delay the work for a year or two if then you will need a year or two on top of that to actually get it done?

If the work is too extensive it might be worth considering building a new network/client from the ground up that addresses all the issues that everybody has been annoyed about anyway.
Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.


Dark Lord of the Forum


Morph your Mule

Need a little help with your MorphXT? Click here

0

#7 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 19 April 2011 - 05:51 PM

View Postcgn, on 19 April 2011 - 07:56 PM, said:

Would you be so kind as to post a link to the original thread? I searched on IPv6 and this was the only (obvious) thread that popped up.

Search function is no the top right of this page. Entering ipv6 in Find words and selecting proper subforums (General discussion, Feature requests... add anyting you like) brings a number of topics. Definetely more than only one.

This post has been edited by fox88: 19 April 2011 - 05:51 PM

0

#8 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 19 April 2011 - 07:55 PM

Relevant (English) topics:
->this
- Ipv6! - Internet Protocol version 6
- P2p Over Ipv6 - Making emule work over IPv6
- Is Kad Ready For Ipv6 ?
- Ipv6

This post has been edited by xilolee: 19 April 2011 - 07:57 PM

INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non è serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dà subito l'allegrIa! Immagine Postata
0

#9 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 19 April 2011 - 08:28 PM

View Postfox88, on 19 April 2011 - 06:51 PM, said:

View Postcgn, on 19 April 2011 - 07:56 PM, said:

Would you be so kind as to post a link to the original thread? I searched on IPv6 and this was the only (obvious) thread that popped up.

Search function is no the top right of this page. Entering ipv6 in Find words and selecting proper subforums (General discussion, Feature requests... add anyting you like) brings a number of topics. Definetely more than only one.

I've read before whilst lurking on this forum that you've got an attitude problem, and you've just proven it.

Just to be clear: I don't use eMule at all. I have no benefit whatsoever that it continues working.

In fact if it failed gracefully and disappeared off the Internet altogether I wouldn't be at all upset.

And yet because I care about the Internet I've spent a couple of days of my time going through the code in detail to check whether the introduction of CGN will cause problems for eMule (and it probably will) and reported them to you, or vice versa whether the current version of eMule (which has been downloaded 555 million times) will cause problems for the Internet when CGN is introduced (and it might because the public address will be the address of the CGN box, and so ALL peer to peer eMule traffic within an ISP might end up traversing the limited resources of the single central CGN box twice, rather than just traversing and being switched on local high-bandwidth links)

So I'm gonna ignore your posts completely from now on.

View Postxilolee, on 19 April 2011 - 08:55 PM, said:


Thank you kindly xilolee for your positive and helpful input. I shall read and comment those topics.

->this
Info on the likely negative affect of CGN here, but certainly incomplete.

- Ipv6! - Internet Protocol version 6
Lot's of noise about NAT. No serious information.

- P2p Over Ipv6 - Making emule work over IPv6
leuk_he understood the problem 5 years ago and correctly diagnosed the benefits of IPv6 (no NAT, globally unique public addresses, possible use of multicast technology at least within a LAN). eMule developers understood that the KAD protocol was hard-coded to 32 bit IPv4 address rather than socket structure. So problem was understood, but there was apparently no follow up

- Is Kad Ready For Ipv6 ?
Lot's of talk of why or why not to implement, but only two technical posts of the difficulties (which were largely incorrect IMHO).

- Ipv6
More denial.

So to summarise the three real technical difficulties I came across were:

1) It hasn't been done before, and the protocol is highly linked to IPv4 32 bit addresses.

True.

This is an argument for doing nothing. Which was apparently used successfully in the past.
But my humble opinion is that you cannot ignore IPv4 address depletion any more.
It has already happened in Asia Pacific. http://www.apnic.net...austion-details

If you want new eMule users, especially in China or India, you have no choice.

If you want a workaround of CGN (which is coming to an ISP network near you soon) you can try to work out how to get around multiple NATs in series, or you can watch your user community at best stay stable, or at worst dwindle to nothing, or you can implement IPv6.

2) Should there be a separate KAD network?

There could possibly be two independent KAD networks.

But I think you guys are lacking some imagination here. You have already implemented a buddy system for nodes stuck behind firewalls. What's stopping you tracking which nodes have IPv4 only connectivity, which have IPv6 only connectivity to the (single) KAD network, and which have both IPv4 and IPv6 connectivity and could act as proxy / buddy servers?

So the dual connected nodes in the network can act as buddies for the single connected nodes.

Maybe there won't be enough dual connected nodes and the network will split, you'll have to see. But as I said before 2 CGN's head to head may well split your IPv4 only KAD network anyway as there will no longer be persistent IPv4 addresses to contact, and you have not implemented Teredo style bubble packets to overcome any restrictive NAT or dual NAT situations.

3) According to Some Support.

All distributed systems are prone to attacks and so is Kad. One of the main security "feature" on Kad are therefore scare ressources, which is basically the IP address. If you have millions of good ip addresses in the network, we can make sure (or at least we can try) that 1000 bad ones don't cause too much harm. If a client behaves bad, spams us, tries stupid things or is otherwise detected to be evil we can ignore or even ban him by its IP. However with IPv6 that will be gone, because every single guy can get millions of IPs, so one single spammer with a 100 MBit line might own 50% of the network's nodes (which would make it unusable). Thats a real problem and it makes no sense to ignore it because it would haunt us back at some point. Ideas, corrections and research papers on this topic welcome ;)

This one deserves further discussion.....

You should look at the structure of an IPv6 address and the features that are mandatory.

Firstly: there is no NAT. So the IPv6 address you see on the end of a session is generally the address you see.

If you use a three way session initiation handshake (like TCP) you can be fairly sure that the remote guy isn't just getting a free ride by blasting spoofed packets at you. To "increase" security you could exchange MD5 HMAC hashes that get flushed through the KAD network and rotate fairly regularly. At least someone attacking you would probably have to modify the client code then. See e.g. OSPF for how that is done.

Secondly: an IPv6 address is split into 2 components. The network dependent portion, and the host portion.
Use of SLAAC states that this will be split on a /64 boundary. The network dependent portion is still governed by an ISP, and is therefore still a relatively scarce resource. The host portion is free for the local admin to assign, and you are correct in assuming this is no longer scare (2^64 hosts per LAN)

So as a first cut to avoid the Sybil problem you could check for uniqueness of the network portion (the upper 64 bits of the 128 bit address). That limits the number of spoofing nodes an individual site could host to the length of the subnet assigned to an end user. This could be as high as 2^16 still (/48) but will most likely be 1 (/64 assigned per household)

A second cut could be use of ICMP to discover the path the the remote host. IPv6 will generally allow ICMP. So traceroute should work (we'll see in practice whether this is really tue but so far so good) So you could trace the route to the remote peer and check for a unique path. If people decide to hide their last hop you could exclude them from the KAD network, well they would no longer have a unique path.

You could also check whois information to see what allocation each user has (last ditch effort) or maybe more smartly, insist that a remote host contacts the all_routers address and communicates the address of its next hop. OK that can also be spoofed but it's more work for your attacker.

In any case, if you see suspicious activity, you could first ban a /64 rather than a /128. If that fails to stop them then ban a /63, then a /62 etc. until the bad guys are locked out.

So in any case, there's a whole load of ideas to experiment with. None are proven or full proof, but I don't see any other workable suggestions anywhere else either.

Most importantly from my point of view, I personally find your current NAT traversal code pretty naive. I humbly request that you try to make eMule fail gracefully in the presence of CGN, and that it does not become a menace to the Internet. If 555 million of your nodes suddenly start up a UDP broadcast storm, that could become very serious very quickly.

Good luck and thanks for your attention.

This post has been edited by cgn: 19 April 2011 - 09:46 PM

0

#10 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 19 April 2011 - 09:42 PM

Quote

My point was that as we aren't exactly well stocked with devs it might be prudent to start working on IPv6 and all the reworking that it entails now since it's going to take quite a while anyway. Why delay the work for a year or two if then you will need a year or two on top of that to actually get it done?

Because at this point its not even sure if the current Kad protocol will work out at all for IPv6. There isn't much point investing countless hours of work only to figure out that it was in vain in the end. I mean sure you could say "hey lets bother about those other issues lateron", but thats probably just like the worst protocols (in terms of security) of the internet were created which hunt us till today (*cough* SMTP, DNS, etc *cough*).
As far as i can see, in the end there are like 3 possible solutions:
- Inventing a new scare ressource. Problem: At this point this only leaves processing power, scales terrible for normal users vs. attackers
- Centralizing it. Wouldn't be pure DHL then anymore and it central server apperently don't work out for eMule that good anymore
- Inventing some new mechanism, like some form of chain of trust. (Problem: We haven't come up with anything yet ;) )

Quote

If the work is too extensive it might be worth considering building a new network/client from the ground up that addresses all the issues that everybody has been annoyed about anyway.

Well thats always an option and given that time goes on and eMules codebase gets older, this could be considered (but not by us). However the problem stays the same for now. I have not looked into BitTorrent and if they switched their DHL copy to IPv6, but if they did i'm willing to bet i could kill the whole network within some hours with a 100MBit line (or worse, kill anything else like the root dns server with the help of the network, but that might be harder and depends on the size of the network as well as other basic security implementations).

#11 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 19 April 2011 - 09:58 PM

Quote

If you use a three way session initiation handshake (like TCP) you can be fairly sure that the remote guy isn't just getting a free ride by blasting spoofed packets at you. To "increase" security you could exchange MD5 HMAC hashes that get flushed through the KAD network and rotate fairly regularly. At least someone attacking you would probably have to modify the client code then. See e.g. OSPF for how that is done.

eMule has already spoof protections and having to modify the client is not a valid protection against anything.

Quote

Secondly: an IPv6 address is split into 2 components. The network dependent portion, and the host portion.
Use of SLAAC states that this will be split on a /64 boundary. The network dependent portion is still governed by an ISP, and is therefore still a relatively scarce resource. The host portion is free for the local admin to assign, and you are correct in assuming this is no longer scare (2^64 hosts per LAN)

It remains to be seen how they are assigned. If $GlobalISP1 gets one /64 address for all its customers and $InterestedPrivateUser1 can order one too, this cannot be used for filtering anything.

Quote

A second cut could be use of ICMP to discover the path the the remote host. IPv6 will generally allow ICMP. So traceroute should work (we'll see in practice whether this is really tue but so far so good) So you could trace the route to the remote peer and check for a unique path. If people decide to hide their last hop you could exclude them from the KAD network, well they would no longer have a unique path.

Besides the fact that the last hops can easily be faked (once it is in your network, you can add whatever hops you want), this is not practical. You can't do ICMP lookups for every contact.

Quote

You could also check whois information to see what allocation each user has (last ditch effort) or maybe more smartly, insist that a remote host contacts the all_routers address and communicates the address of its next hop. OK that can also be spoofed but it's more work for your attacker.


Same as above.

Quote

Most importantly from my point of view, I personally find your current NAT traversal code pretty naive.

What code are you actually talking about in the official eMule? And how do you think it would misbehave (meaning not that it would fail to work but disrupt/bother something) in any situation (based on the sourcecode)?

#12 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 19 April 2011 - 11:31 PM

Quote

It remains to be seen how they are assigned. If $GlobalISP1 gets one /64 address for all its customers and $InterestedPrivateUser1 can order one too, this cannot be used for filtering anything.


You are way behind the curve. This has already rolled out on major ISP's in many countries. Global providers are getting /32 (that is bigger than the current Internet many times over). Home users are going to get /64 most likely as an absolute minimum. Some will get /48. You need to re-read my post.

Translate your thinking from today's IPv4/32 being linked to a single KAD node/host, to a whole IPv6/64 user LAN as the minimum identifier size, and they are pretty much equally rare. No-one is going to share a /64 in effect (same restriction as today's code that allows one KAD node to one IPv4 address. Instead now it's one KAD node to one ipv6/64 range of addresses, but that still equates to one real person acting as the LAN administrator no matter how you look at it)

Then instead of blocking a single IPv4/32 address on detecting improper action you'd block a whole range of IPv6/64 or 18446744073709551616 addresses in one go. That'll cope with most end user LANs on the first infraction, expect for the odd user who'll get /48. Then you could use a strategy of ever increasing block sizes (that double in size every time a spoof action is detected to cover one additional bit of mask.) Worst case would be then that the bad guy with a /48 could spam or attack you 16 times (corresponding to the 16 bits difference of a /48 and a /64).

How many addresses is that in a /48? 79228162514264337593543950336.

I think that's a pretty good ratio to cover that in just detecting 16 infractions.

And there's still a factor of 2^16 difference between a global ISP with a /32 and the worst case bad guy who has a /48, or 32 bits between a global ISP with a /32 and a best case bad guy with a /64.

That's the same ratios as an entire Class B IPv4 network today to an individual IPv4 address, or the entire IPv4 Internet to a single IPv4 address respectively.

The ratios between the bad guy's address space to the address space in use on the Internet as a whole remain roughly the same (approximately the same rarity value thus). It's just the IPv6 addresses are longer, and the relevant piece of information has been arithmetically shifted left along the address.

Quote

Quote

Most importantly from my point of view, I personally find your current NAT traversal code pretty naive.

What code are you actually talking about in the official eMule? And how do you think it would misbehave (meaning not that it would fail to work but disrupt/bother something) in any situation (based on the sourcecode)?

I have looked mainly at the KademliaUDPListener.cpp, PacketTracking.cpp, UDPfirewallTester.cpp ListenSocket.cpp, sockets.h, otherfunctions.h, listensocket.cpp, AsyncsocketEx.cpp, ServerSocket.cpp, Kademlia.cpp, ClientUDPSocket.cpp

It's mainly the UDP / NAT traversal code that concerns me most in KAD.

There's two things: a storm, and unwanted interaction with an ISP's normal traffic.

To be honest, it's difficult to say offhand exactly what would cause a storm. I'd have to do a lot more analysis on this, and even then it is almost impossible to say for certain because the behaviour of CGN is not 100% defined in terms of persistent handling of related sessions. Some manufacturers ensure that requests from address 1 will always translate to address 2. Some will use port overloading to save IPv4 space. Others will use pure round robin etc. Others will have different timeouts of NAT associations, so there could be race conditions. So the protocol definitely needs to be conservative. TCP has many hundreds of man years development behind it, and not without reason.

Needless to say I have seen very real UDP storms in the past. This isn't just seeding panic. e.g. Appletalk version 2 ZIP and NBP protocol, which implemented a distributed directory not dissimilar to KAD in that there was no central repository of information, and when a user opened up a finder window it started trying to find "local" servers and the services they offered.

See e.g. http://www.cisco.com...080093cca.shtml

Novell IPX also used to have this issue sometimes, and this was abused to create a DOS attack.

http://securityvulns...ocument290.html

You also have "ping" and user triggered searches in your code.

The danger of the combination of UDP and NAT is the lack of predictability of the relationship of the source and destination addressing, together with the need to DIY so many functions that are typically forgotten about during normal application development.

And the current code makes all sorts of assumptions about NAT and the underlying IP network.

If I make a list of what I suspect for hidden dependencies I get the following so far:

1) where the NAT is located (next to the KAD node and applicable to all traffic from the node outside the local subnet = path invariance),
2) how many NAT devices a conversation will traverse (0 or 1),
3) the type of NAT device is not tracked or tested at all (cone, restrictive, symmetric)
4) how many firewalls are in the path (0 or 1) and that they are invariant (stateful firewalls may also have timeouts on UDP sessions),
5) that NAT and firewall functions are combined when they are in reality separate and may be implemented in completely different locations
6) that all traffic is treated the same way (no consideration of temporary or permanent reroutes or load balancing),
7) no secure handling of packet loss or duplication (no transaction id per request)
8) no rate throttling of outbound traffic (tracks inbound counters for discards but not outbound for when last opcode was sent)
9) single node is located behind a single IP address
10) Own IP address rarely changes
11) Information obtained by Node X from Node Y about node Z is also directly applicable to a session set up from node X to Node Z (see path invariance)
12) remote source IP address used as a key for a large number of data stores (rather than a session)
13) retransmission mechanisms are not explicitly defined
14) session status tracking is limited to whether an individual opcode is expected, rather than a defined finite state machine
15) apparently no formal definition of acceptable transactions and combinations of transactions.
16) open source code, so relatively easy for (inexperienced/malicious) developers to insert/exploit transactions into the KAD network as a whole
17) No clear congestion detection and back off mechanisms on the control channel as opposed to the file transfer channel (aka TCP window size) which are triggered by e.g. DSCP/WRED in Internet routers/ traffic shaping/ policing.
18) Ignores standard OS calls and IPv6 transition mechanisms. (not sure if this could be a real problem, but the Windows stack is becoming increasingly complex and likely to hide certain functions from the app, like creating automatic tunnels)
19) there only seem to be two global timers defined, rather than timers per event per session.
20) Ignores official naming systems like DNS and detects peers and its own IP address dynamically.
21) Peer address vary slowly over time (temporal invariance)
22) AFAIK there are no official specifications of the KAD protocol, no "fix-ups" for new NAT functions, nor formal testing for interactions before implementation of protocol/network changes

Do I need to go on? Or are you just going to deny it all and say the code is great?

As I said, one of the main defences (opcode counting within a time period in PacketTracking) is based on the above assumptions like IP source address invariance, which could well soon be broken. That could trigger a denial of service in one place that causes other nodes to retransmit requests elsewhere.....

In one respect your code is proven in production. In another, I doubt whether the protocol has ever been fully defined and analyzed.

It's not going to be one obvious thing that causes a storm. It's generally a combination of things, plus some timing/ congestion issues that trigger retransmission of a request.

The other real major concern is the detection of "public address" and unintended interaction with an ISP's internal routing and traffic processing.

If CGN is implemented, the eMule firewall detection code and detection of a client's own IP address looks really unreliable. It only asks two other nodes and then learns it's own IP address from the reply from peers (after checking that it is invariant in prefs->SetIPAddress. That learned address is then sent via SendMyDetails via the KAD channel if I recall correctly, for use by other nodes wanting to transfer files to this node. eMule knows nothing about what an ISP is doing on the lower layers, nor DNS, so second guessing this is dangerous. If that public address is ever incorrect, the traffic could easily be mis-routed, potentially triggering other devices, such as a network intrusion detection system.

If the firewall detection code works properly though, it is going to detect the outside NAT address of the ISP's central CGN box, and not the outside address of the node's CPE NAT router.

That CGN box is probably going to be a fairly limited resource (it will be super expensive and can probably still only handle 10Gbps total traffic and a million sessions) 10Gbps and a million sessions is probably peanuts compared to an ISP's internal link bandwidths. So worst case scenario could be that instead of the peer to peer transfer traffic between nodes located within an ISP being switched within the MPLS backbone (as of today) it is all directed at the CGN box, and has to to traverse it twice. That could be a serious bottleneck for ALL traffic from an ISP to the rest of the Internet. Major DOS.

Please bear in mind that's the result of 2 days work trying to get my head around code that deliberately obscures certain elements of its operation.

But basically if you compare the UDP exchange processes in the current source code to a standard TCP implementation, there's a whole lot of functions missing related to reliability that you'd get free of charge with a standard socket call to create a TCP session. I'd be a lot less concerned with native IPv6/TCP transport sessions between nodes. And since there's no NAT in IPv6, there'd also be no need for NAT traversal code.

This post has been edited by cgn: 20 April 2011 - 09:22 AM

0

#13 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 20 April 2011 - 07:56 AM

It seems to me looking at your code in UploadBandwidthThrottler that it only works on policing the outbound queue by delaying sending on a per slot basis to achieve a (maximum) outbound transmission rate. The rate is user defined.

This strategy would seem equivalent of stuffing as much into a pipe as possible, no matter whether it gets delivered or not. Again, pretty naive, or perhaps even irresponsible IMHO.

I don't see any congestion detection, congestion avoidance, or back off mechanism in your protocols to prevent overloads at network bottlenecks (like a CGN) adversely affecting other network traffic traversing that section of the network that is experiencing (temporary) congestion.

I searched on "congestion" and "retransmit" and got 0 hits.

Use of a TCP session would achieve that effect automatically through the interaction of e.g. WRED in the ISP network routers triggering appropriate packet discard, and this interacting with the normal TCP retransmit/ window management mechanism running on the end host, thus ensuring that the application acted as a good network citizen and backed off gracefully.

This TCP based congestion avoidance mechanism has been extensively studied, tested, and deployed on the Internet.

If you use UDP, it's up to you to back off properly in your application, otherwise your application could be responsible for denying other network users fair access to the service.

FYI.

http://en.wikipedia....work_congestion
http://cciesp.net/20...tion-avoidance/
http://www.cisco.com...de_Chapter.html
www.cisco.com/en/US/technologies/tk543/tk759/technologies_white_paper0900aecd80295abb.pdf
http://www.cisco.com...de_Chapter.html
http://www.cisco.com...e/qcfpolsh.html

I think I've done my bit in properly informing you of my views in a straight forward and professional manner, and it's up to you how you act/react to this information.

Good luck in making the correct decisions, and thanks once again for reading.

This post has been edited by cgn: 20 April 2011 - 09:31 AM

0

#14 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 20 April 2011 - 08:29 AM

Quote

eMule knows nothing about what an ISP is doing on the lower layers, nor DNS, so second guessing this is dangerous.

DNS? What has DNS to do with anything here?

Quote

If that public address is ever incorrect, the traffic could easily be mis-routed, potentially triggering other devices, such as a network intrusion detection system.

The address won't be incorrect, just possibly out of date (which might happen faster than expected for certain NATs). In this case some packets are misrouted, but not enough to be a problem. If some NIS is triggering, thats the problem of the guy who set it up this way.

Quote

it is all directed at the CGN box, and has to to traverse it twice

That is again an ISP problem. The ISP deceides how to route traffic (as well as how to set up its CGN boses). It is transparent to clients, so they don't have to nor should care. Setting up undersized boxes which can't actually handle the IPv4 traffic it is supposed to is not something we have to bother about.

Quote

trying to get my head around code that deliberately obscures certain elements of its operation.

Right.... sneaky little devs.

Quote

But basically if you compare the UDP exchange processes in the current source code to a standard TCP implementation, there's a whole lot of functions missing related to reliability that you'd get free of charge with a standard socket call to create a TCP session.

Functions which are missing are those we don't need, like ensuring that packets are not lost, are received in the proper order, uncorrupted, etc. Security related functions (anit-spoof, anti-flood, etc) are implemented.

Quote

...

You post all sort of general assumptions and papers, some completely unrelated to eMule or protocol. Posting flaws of other protocols, searching the sourcecode for keywords and making wild guesses is just not helpful. Sorry but i don't have the time to explain you the protocol in-depth here. If you have a valid concern that a part of eMule might misbehave under certain conditions, i'm very willing to look into this. However the whole protocol was worked on with security in mind and while for example some routing taks for Kad sure won't work under certain conditions (which is equal to having the firewalled status right now), i'm pretty sure it cannot be abused for any attack nor will it accidentally flood/disrupt something else.

If you want to have a look into the protocol there are quite a lot papers available and esp. Kad has been analized by several researchers.

#15 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 20 April 2011 - 09:54 AM

Sorry, but IMHO you and other posters on this forum have demonstrated a complete lack of deep technical knowledge of modern networks and the developments that have taken place in the last few years. If you want to correctly code protocols that run over UDP, you need that knowledge.

You seem to have zero knowledge of the upcoming (drastic) changes required for implementing IPv6 and ensuring continued service for IPv4 for the majority of Internet users in the face of huge challenges like IPv4 address space depletion. You couldn't even correctly identify IPv6 address allocation lengths.

Claiming that is all the "ISP's problem" is naive and downright irresponsible.

The ISP is NOT responsible for testing all applications deployed on the Internet.
You are responsible for how your code behaves.

There are a lot of very worried ISP's and equipment manufacturers right now, I can tell you quite frankly, wondering if their business is going to survive this transition.

For example, implementing a CGN box will cost at least half a million dollars per box, and probably several million per ISP (see http://www.zdnet.com...n-router/136278 http://www.networkwo...k-will-increase http://www.fcc.gov/D...OC-303870A1.pdf ).

At the end of the day, it is Internet users, like eMule users among others, who have to finance these purchases.

Even then, these massive and expensive boxes are physically incapable of switching all the traffic in an ISP, and so need to be handled with some respect.

We are already at, or sometimes beyond, the limits of what is financially and technologically sustainable on the Internet IMHO.

Your code (which has been downloaded 555 million times) could well be responsible for denying other network users access to the Internet on a massive scale. It certainly isn't helping any transition. I wouldnt bother posting otherwise. I don't get paid for this. It isn't up to me to re-engineer your code in detail.

You have now been properly warned about the risks, and it is up to you how you act on that information.

Switching to TCP and standard IPv6 transports would remove a lot of risk. [the BitTorrent clients I examined already used IPv6/TCP]

I shall not bother you further. As I said. Good luck.

This post has been edited by cgn: 20 April 2011 - 10:13 AM

0

#16 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 20 April 2011 - 10:20 AM

View Postcgn, on 20 April 2011 - 12:28 AM, said:

This is an argument for doing nothing.
...
But I think you guys are lacking some imagination here.

It appears you came here to tell devs they are lazy and unimaginative. Sure you have no attitude problems, have you?
What practical steps you suggested having superior imagination? 'Try to do something' was your best advice so far, which is not even funny.

View Postcgn, on 20 April 2011 - 12:28 AM, said:

I've spent a couple of days of my time going through the code in detail

Waste of time it was, because you did not do your homework properly.

This post has been edited by fox88: 20 April 2011 - 10:31 AM

0

#17 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 20 April 2011 - 10:27 AM

Quote

a complete lack of deep technical knowledge

Quote

You seem to have zero knowledge

Quote

You couldn't even correctly identify IPv6 address allocation lengths.

Quote

naive and downright irresponsible.


Quote

I don't get paid for this.

Too bad, i mean paid trolling would be the job of my dreams :)

#18 User is offline   ducho 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 98
  • Joined: 22-October 06

Posted 20 April 2011 - 10:57 AM

View Postcgn, on 20 April 2011 - 07:54 AM, said:

I shall not bother you further. As I said. Good luck.


Could this mean that he/she/it is gone for good?

Oh please my Lord :devil:

:clap:
0

#19 User is offline   cgn 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 7
  • Joined: 19-April 11

Posted 20 April 2011 - 03:02 PM

Thank you for posting the link to those research papers on Kadmelia. Interesting.

I would note that all of the papers look from the perspective of looking at the efficiency and security and performance from the perspective of the Kamdelia network.

None of them that I scanned look at the efficiency, security, and potential negative effect on the underlying Internet infrastructure caused by running the Kadmelia network over the Internet (with perhaps the exception of preventing someone exploiting Kadmelia to perform a DDOS attack).

You might want to prompt some of your research buddies to look at this.

View Postducho, on 20 April 2011 - 11:57 AM, said:

View Postcgn, on 20 April 2011 - 07:54 AM, said:

I shall not bother you further. As I said. Good luck.


Could this mean that he/she/it is gone for good?

Oh please my Lord :devil:

:clap:


Yes the poster "cgn" is gone after this last post.

You can carry on in whatever way you see fit: it's your code and your responsibility.

But CGN is here to stay. Don't forget the name. It's coming to more than one ISP near you. Soon. Real soon.

www.nanog.org/meetings/nanog46/presentations/Wednesday/Ward_AFT_panel_N46.pdf

Moving eMule to TCP/IPv6 would make you part of the solution, rather than being potentially a major part of the problem.

To quote Forrest Gump: "And that's all I have to say about that"
0

#20 User is offline   Stulle 

  • [Enter Mod] Dev
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5804
  • Joined: 07-April 04

Posted 20 April 2011 - 05:15 PM

What I wonder most about... what makes you so superior to all of the devs around here?

Besides, you make it sound as if eMule's KAD network has the potential to destroy the internet (just like Jane did in episode 3x04). Well, I think there are two major problems with that assumption. Firstly, KAD is not responsible for the majority of the stress put on ISPs... It's the file transfer. Secondly, it's up to the ISP to provide the server, hence the name Internet Service Provider. Although resources should obviously not be wasted, it's not the coders responsibility to make sure the ISPs service is not disrupted because they did not set their network up in a way it will handle technology which has been around for a reasonable number of years.

So much rather than flaming at the folks here for not making a transition they don't have to do you should come up with something that helps making a transition or just support another project that goes better ways to prepare for the future. Worst case scenario for the people here is that some potential new users won't join the network or that eventually people will wander off to another piece of software. Either way, the network infrastructure will most likely remain utterly unbothered.

Lastly, before pointing fingers ask yourself if eMule and KAD are actually the real culprits. For all that is known and scientifically proven, this software does a reliable job. There is so much crap-ware in the vast internet that has the sole purpose of pwning machines, DDOSing, spamming and generally doing bad stuff, they are a lot more dangerous than an aged piece of reliable software.
I am an emule-web.de member and fan!

[Imagine there was a sarcasm meter right here!]

No, there will not be a new version of my mods. No, I do not want your PM. No, I am certain, use the board and quit sending PMs. No, I am not kidding, there will not be a new version of my mods just because of YOU asking for it!
0

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users