Official eMule-Board: Improvement Of Emule-Kad Lookup Performance - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Improvement Of Emule-Kad Lookup Performance packet loss; inconsistency of round-trip UDP ports

#21 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 05 September 2011 - 06:38 PM

Do senior members here have any comment on this patch?
0

#22 User is offline   liubingshuang 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 11
  • Joined: 31-August 11

Posted 06 September 2011 - 09:15 AM

No,until now. Perhaps senior members are too busy? :flowers:

View PostEnig123, on 05 September 2011 - 06:38 PM, said:

Do senior members here have any comment on this patch?

0

#23 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 06 September 2011 - 11:06 PM

By just looking at the patch in the thread, it seems to me the main component of it is to resend packets which were not answered. I don't think this is a good idea. While it is true that UDP packets can get lost and resending might help, most often lost packets mean that either our or the receiver line is congested and in this case it is better not to bother this node anymore but use another one - after all certain "hot keyword" nodes will be flooded with request so its intended that they cannot answer them all. If all clients start to reask x times, that makes the situation worse and won't increase answer rates. The resend time is way too short btw. Also this patch will increase overhead of course.

If i missed something, feel free to point out.

#24 User is offline   Nissenice 

  • clippetty-clopping...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4231
  • Joined: 05-January 06

Posted 07 September 2011 - 01:45 AM

Hi liubingshuang

About this part in CSearch::ProcessResponse
-			CUInt128 uDistance(((CContact*)*itContactList)->GetClientID().Xor(m_uTarget));
+			//CUInt128 uDistance(((CContact*)*itContactList)->GetClientID().Xor(m_uTarget));//there modfy (*itContactList)'s clientID by mistake
+			CUInt128 uDistance(CUInt128(((CContact*)*itContactList)->GetClientID()).Xor(m_uTarget));

Have you checked this that the (*itcontactList)'s clientID actually is modified by mistake? From what I can see by looking at the original code it seem to be correct. That's why I'm wondering.

This post has been edited by Nissenice: 07 September 2011 - 02:03 AM

0

#25 User is offline   liubingshuang 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 11
  • Joined: 31-August 11

Posted 07 September 2011 - 01:28 PM

Yes, sometimes the resending policy could have the problem that you warried. But in our experiment, there are more than 10% of responses got by resending request packets. So pakcet loss is the main problem. What's more, we indeed raise lookup success ratio up about 10%.

View PostSome Support, on 06 September 2011 - 11:06 PM, said:

By just looking at the patch in the thread, it seems to me the main component of it is to resend packets which were not answered. I don't think this is a good idea. While it is true that UDP packets can get lost and resending might help, most often lost packets mean that either our or the receiver line is congested and in this case it is better not to bother this node anymore but use another one - after all certain "hot keyword" nodes will be flooded with request so its intended that they cannot answer them all. If all clients start to reask x times, that makes the situation worse and won't increase answer rates. The resend time is way too short btw. Also this patch will increase overhead of course.

If i missed something, feel free to point out.

0

#26 User is offline   liubingshuang 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 11
  • Joined: 31-August 11

Posted 07 September 2011 - 01:31 PM

Ok, I am so cautious that I make a mistake here. Sorry, please ignore it.

View PostNissenice, on 07 September 2011 - 01:45 AM, said:

Hi liubingshuang

About this part in CSearch::ProcessResponse
-			CUInt128 uDistance(((CContact*)*itContactList)->GetClientID().Xor(m_uTarget));
+			//CUInt128 uDistance(((CContact*)*itContactList)->GetClientID().Xor(m_uTarget));//there modfy (*itContactList)'s clientID by mistake
+			CUInt128 uDistance(CUInt128(((CContact*)*itContactList)->GetClientID()).Xor(m_uTarget));

Have you checked this that the (*itcontactList)'s clientID actually is modified by mistake? From what I can see by looking at the original code it seem to be correct. That's why I'm wondering.

0

#27 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 07 September 2011 - 02:36 PM

Quote

Yes, sometimes the resending policy could have the problem that you warried. But in our experiment, there are more than 10% of responses got by resending request packets. So pakcet loss is the main problem.

That isn't a proper conclusion. Lets say one node is congested, so 50% of all packets send to (or from) it are dropped. Now if you create a single client which resends all requests 3 times, there is a good chance that at least one package/answer gets through and you get your reply - because you ask more often you had a better chance. But consider now that all clients do the same: Not only is your chance to receive an answer as low as before (or lower of the nodes down channel is congested) but you are also causing 3 times as many overhead for this node, as well as for all IPs which were part of the network before but aren't anymore (for example because the dynamic IP was reassigned to another user). These are three huge disadvantages in turn for the (imho) very rare case that a packets get really lost without congestion problems in the target or source node.

#28 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 07 September 2011 - 05:14 PM

Some Support is right about the resends are a bad idea. It might be useful if the problem with packet loss is on your end, but if it is that high that the 10 node redundancy isn't enought then you might have a lot bigger problems than innacurate Kad lookups.

From my experience when experimenting with speeding up Kad searches by reducing the node lookup timeout to only cover 95% of the nodes, I noticed that it often used less number of lookups while the results where often of better quality. I haven't done any scientific measurements but the tendance seem to point into the direction that congested (slow or non responding) nodes are of limited use.

I've noticed that you try to map responses from a node, that doesn't match the ip/port number in the tried list, you attempt finding one where only the ip matches. First I have hard to see when this could happen, and secondly if a node responds from a different port it must be something wrong with it as it's not according to the protocol and is probably not very useful either.

As I see it you have tried to increase the success rate from individual nodes when the protocol is designed with a built in redundancy where the information is stored on more than one node and finding just one of them is enought. It seems overkill to me and adds overhead without any real benefit from what I can see. Maybe you have made some observations in addition to the ones you already mentioned that justifies these changes. It would be nice to hear your reasoning!
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
2

#29 User is offline   liubingshuang 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 11
  • Joined: 31-August 11

Posted 08 September 2011 - 10:46 AM

Thx for your helpful suggestions. In our measurement, we found that 14% of responses got by resending request packets. So the congestion of network link does not matter , probably. There is a technical report about the patch: http://www.rapidshar.../file-4767.html

View Postnetfinity, on 07 September 2011 - 05:14 PM, said:

Some Support is right about the resends are a bad idea. It might be useful if the problem with packet loss is on your end, but if it is that high that the 10 node redundancy isn't enought then you might have a lot bigger problems than innacurate Kad lookups.

From my experience when experimenting with speeding up Kad searches by reducing the node lookup timeout to only cover 95% of the nodes, I noticed that it often used less number of lookups while the results where often of better quality. I haven't done any scientific measurements but the tendance seem to point into the direction that congested (slow or non responding) nodes are of limited use.

I've noticed that you try to map responses from a node, that doesn't match the ip/port number in the tried list, you attempt finding one where only the ip matches. First I have hard to see when this could happen, and secondly if a node responds from a different port it must be something wrong with it as it's not according to the protocol and is probably not very useful either.

As I see it you have tried to increase the success rate from individual nodes when the protocol is designed with a built in redundancy where the information is stored on more than one node and finding just one of them is enought. It seems overkill to me and adds overhead without any real benefit from what I can see. Maybe you have made some observations in addition to the ones you already mentioned that justifies these changes. It would be nice to hear your reasoning!

1

#30 User is offline   Nissenice 

  • clippetty-clopping...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4231
  • Joined: 05-January 06

Posted 08 September 2011 - 12:02 PM

Thanks!

Just click on the same icon as before as in the image below. (No need for everyone to click everywhere... :P)

http://img221.images...387/patchit.png


I guess this link will do too: http://www.rapidshar...7-LP0VAUdd.html

This post has been edited by Nissenice: 08 September 2011 - 12:06 PM

0

#31 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 08 September 2011 - 03:21 PM

Quote

In our measurement, we found that 14% of responses got by resending request packets. So the congestion of network link does not matter

Repeating this doesn't makes it true.

Also on a side note, one of Kads key security measures is to try to never send more packets to an unknown target than we have received to lead to such a request. This means if we receive a malicous routing response with fake IPs we will only send one packet to such an IP - the same amount as was needed by the malicous node to sent the routing answer. This is important to make sure that the Kad network cannot be used to amplify an attackers available bandwidth for an DDoS attack. If you resend all requests x times, this increases the potential as the attacker has to sent one packet to you and in return you send x packets to the victim.
Now in this case it is still not a major issue because the attack vectors are weak for routing answers, but still not a good idea security wise (besides the other problem i have pointed out before).

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

3 User(s) are reading this topic
0 members, 3 guests, 0 anonymous users