Official eMule-Board: Intelligent Chunk Selection - Diverse Parts For Rare Files - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Intelligent Chunk Selection - Diverse Parts For Rare Files Rate Topic: -----

#1 User is offline   mik1 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 1
  • Joined: 13-April 07

Posted 13 April 2007 - 05:02 AM

Not sure if it's a suggestion or a bug report.
Guess it would improve downloading of rare files, if they are requested by more than one user.
User A has a complete copy of the file.
The file is requested by users B and C.
Please make e-Mule force user B download chunks of the file different from user C, and vice versa.
It would allow users B and C using each outher as an additional seed for missing chunks.
(read about Intelligent Chunk Selection, but does not work as described here for my eMule 0.47c)
1

#2 User is offline   PacoBell 

  • Professional Lurker ¬_¬ (so kyoot!)
  • PipPipPipPipPipPipPip
  • Group: Moderator
  • Posts: 7296
  • Joined: 04-February 03

Posted 13 April 2007 - 05:40 AM

View Postmik1, on Apr 12 2007, 10:02 PM, said:

Please make e-Mule force user B download chunks of the file different from user C, and vice versa.
What, you mean like chunk hiding? It's been discussed and implemented in several mods already with varying degrees of success. Overall, such techniques tend to confuse the client's chunk selection algorithm.

Quote

(read about Intelligent Chunk Selection, but does not work as described here for my eMule 0.47c)
Could you describe "does not work"? I'm not a mind-reader ;)
Sed quis custodiet ipsos custodes
Math is delicious!
MmMm! Mauna Loa Milk Chocolate Toffee Macadamias are little drops of Heaven ^_^
Si vis pacem, para bellum DIE SPAMMERS DIE!

#3 User is offline   raron 

  • Member
  • PipPip
  • Group: Members
  • Posts: 47
  • Joined: 07-September 04

Posted 14 April 2007 - 09:18 AM

View Postmik1, on Apr 13 2007, 07:02 AM, said:

Not sure if it's a suggestion or a bug report.
Guess it would improve downloading of rare files, if they are requested by more than one user.
User A has a complete copy of the file.
The file is requested by users B and C.
Please make e-Mule force user B download chunks of the file different from user C, and vice versa.
It would allow users B and C using each outher as an additional seed for missing chunks.
(read about Intelligent Chunk Selection, but does not work as described here for my eMule 0.47c)


I agree, but I think eMule already does it like this?

At least it's how its explained in the "User Made Guide and FAQs" - "Why are my downloads so slow", under the heading "A basic overview of the ED2K network" http://www.emule-pro...p;rm=show_topic
(*sigh* It's almost longer to explain where it is than to quoute it)

I'm no expert though, it could be its just a random (but needed) chunk that gets uploaded. Which would also work, but somewhat less efficient I guess.
0

#4 User is offline   nobody1000 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 12
  • Joined: 24-March 06

Posted 14 April 2007 - 04:57 PM

I think Mik1 is right. I've noticed that for new shared files everybody takes the same chunk. For multimedia files this is normal that first and last chunk for preview are downloaded), but it happens also to every chunk. I think some good randomization would be required. Random number genrator could be initialized with user hash or timer etc.
0

#5 User is offline   cafebean 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 490
  • Joined: 30-September 06

Posted 14 April 2007 - 05:12 PM

View Postnobody1000, on Apr 14 2007, 09:57 AM, said:

I think Mik1 is right. I've noticed that for new shared files everybody takes the same chunk...
I don't think that is really the case. More likely, that part was uploaded to someone with a high upload, who quickly shared it amount many peers. Then many people only have that part to upload because the source of the file is slow.
0

#6 User is offline   PacoBell 

  • Professional Lurker ¬_¬ (so kyoot!)
  • PipPipPipPipPipPipPip
  • Group: Moderator
  • Posts: 7296
  • Joined: 04-February 03

Posted 16 April 2007 - 05:41 AM

View Postcafebean, on Apr 14 2007, 10:12 AM, said:

I don't think that is really the case. More likely, that part was uploaded to someone with a high upload, who quickly shared it amount many peers. Then many people only have that part to upload because the source of the file is slow.
Indeed, and keep in mind that eMule will generally try to complete a chunk it's already started, so if a fast client one degree away from the releaser begins to upload a new chunk to you before the releaser does, regardless of its rarity relative to your current list of sources, eMule will attempt to download that chunk from as many sources as it can find. HTH.
Sed quis custodiet ipsos custodes
Math is delicious!
MmMm! Mauna Loa Milk Chocolate Toffee Macadamias are little drops of Heaven ^_^
Si vis pacem, para bellum DIE SPAMMERS DIE!

#7 User is offline   zz 

  • -
  • PipPipPipPipPipPipPip
  • Group: Debugger
  • Posts: 2014
  • Joined: 30-November 02

Posted 16 April 2007 - 03:07 PM

PacoBell: That's not entirely true any longer. If it has the chance to get a very rare chunk, it will rather take that than complete a common chunk. Check out the chunk choosing changes in one of the recent releases (version/history file + source) for details.

/zz B)

This post has been edited by zz: 16 April 2007 - 03:07 PM

ZZUL - get control of your uploads: ZZUL Forum
0

#8 User is offline   nobody1000 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 12
  • Joined: 24-March 06

Posted 16 April 2007 - 08:03 PM

View Postcafebean, on Apr 14 2007, 10:12 AM, said:

I don't think that is really the case. More likely, that part was uploaded to someone with a high upload, who quickly shared it amount many peers. Then many people only have that part to upload because the source of the file is slow.

Unfortualelly you didn't understand me. I was depicting situation when 10 clients were downloading the same file simultaniously and asking for exactly the same chunk from the middle of shared file even if all parts were available for them download and they had no parts at all by themselves.
0

#9 User is offline   cafebean 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 490
  • Joined: 30-September 06

Posted 16 April 2007 - 09:24 PM

View Postnobody1000, on Apr 16 2007, 01:03 PM, said:

Unfortualelly you didn't understand me. I was depicting situation when 10 clients were downloading the same file simultaniously and asking for exactly the same chunk from the middle of shared file even if all parts were available for them download and they had no parts at all by themselves.
So you were the uploader? Otherwise there is no way for you to know that. Then that would only happen if that part was the rarest part, therefor everyone are trying to download it. Can you conform that other parts of the file are not more available than the chunk that they are all downloading?
0

#10 User is offline   nobody1000 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 12
  • Joined: 24-March 06

Posted 17 April 2007 - 01:33 PM

View Postcafebean, on Apr 16 2007, 11:24 PM, said:

So you were the uploader?
Can you conform that other parts of the file are not more available than the chunk that they are all downloading?

I was a releaser and thus uploader (the only one). I will try to catch some screens when it happens again.
0

#11 User is offline   cafebean 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 490
  • Joined: 30-September 06

Posted 17 April 2007 - 04:52 PM

View Postnobody1000, on Apr 17 2007, 06:33 AM, said:

I was a releaser and thus uploader (the only one). I will try to catch some screens when it happens again.
I wasn't looking for prove (I trust that you want to make emule better). Just that you know what you're talking about. Like "the only [releaser]", is that file not available anywhere else on the internet so that file couldn't be shared by someone else, or have you tried to download that file with highid and no ipfilter to see no other sources?
0

#12 User is offline   nobody1000 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 12
  • Joined: 24-March 06

Posted 26 April 2007 - 02:03 PM

View Postcafebean, on Apr 17 2007, 06:52 PM, said:

"the only [releaser]", is that file not available anywhere else on the internet so that file couldn't be shared by someone else

that's it
0

#13 User is offline   omeringen 

  • löl
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 983
  • Joined: 01-January 06

Posted 12 June 2008 - 01:35 PM

Hi,
First of all i want to say i have no knowledge about how code works. So, just asking a small question ;

What is the chunk choosing differences between official eMule and some mods with ICS(intelligent chunk selection) feature ?

Regards.

This post has been edited by omeringen: 08 May 2010 - 07:16 PM

0

#14 User is offline   omeringen 

  • löl
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 983
  • Joined: 01-January 06

Posted 08 May 2010 - 07:28 PM

After two years. . .
There is another two topics about it;
More Intelligent Chunk Selection
[patch] Intelligent Chunk Selection As in enkeyDEV.5 mod
The "patch" topic is very old, maybe there is some changes, i am not sure. . .

My question still stands. Sometimes eMule is becoming real dummy about chunk selection. And you know some of the mods have Sotn feature to get over it while releasing. And most of them have ICS feature. . . This subject has been talked about for a long time at mods section, so let's continue here. :flowers:

Why there is no ICS at official client ?

This post has been edited by omeringen: 08 May 2010 - 07:37 PM

3

#15 User is offline   taz-me 

  • I'm taz (a modder)
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 587
  • Joined: 07-December 06

Posted 11 May 2010 - 07:23 AM

View Postomeringen, on 08 May 2010 - 10:28 PM, said:

The "patch" topic is very old, maybe there is some changes, i am not sure. . .


enkeyDEV code contains a fault (preview chunks priority logic is reversed), as well as questionable code for chunks selection mod from current client (we're downloading from) based on all chunks not just those it holds, resulted in a bit complex ICS implementations map :

1. MorphXT and derived - if download preview chunks first is selected ICS is bypassed (official logic is used) until those chunks are downloaded.
2. eMuleFuture (< 1/0) and derived (perhaps even additional mods) - if download preview chunks first is selected they are likely to be downloaded very late.
3. AcKroNiC, SharkX & zBOOM (frozen project) - preview chunks priority logic as well as chunks selection mod are fixed.

As far as I know there are complementary code implementations (named differently and perhaps with small changes) which try to reveal information about hidden chunks (by SOTN / HideOS) via history of chunks requests by clients - if they were not sources (1'st full sources of files, or in other words this info is gathered if these clients requested chunks from us). This info is used by ICS to determine selection mod (according to chunks "rareness"). This info can be used (and is) also by anti leecher mechanism for detecting suspicious clients (clients asking for chunks they seem to poses).

Quote

Why there is no ICS at official client ?


It might have to do with SOTN (& HideOS) hiding chunks messing ICS functioning (due to improper figuring of chunks "rareness"). Perhaps revised SOTN (that won't hide chunks from ICS identified clients - which can be detected via handshake protocol) will increase official devs motivation to consider replacing official chunk selection to ICS (or an alike).

This post has been edited by taz-me: 11 May 2010 - 07:27 AM

P2P is about sharing, ed2k is my choice !
0

#16 User is offline   omeringen 

  • löl
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 983
  • Joined: 01-January 06

Posted 11 May 2010 - 02:57 PM

But Sotn and hideOS has been implemented because there is no ICS at official, am i wrong ? Otherwise no-one would need these features. . . If majority of network (means official client) have had ICS, there would be no part hiding features on mods. Personally i(and most of the releasers) like Sotn, it is doing it's job nicely but i think official client should be more clever about part selecting. The all a releaser want is to upload parts efficiently.

This post has been edited by omeringen: 11 May 2010 - 03:03 PM

0

#17 User is offline   taz-me 

  • I'm taz (a modder)
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 587
  • Joined: 07-December 06

Posted 11 May 2010 - 03:24 PM

Problem is 2 sided :

1. For old (<= 0.50a) official clients : SOTN might be "doing good".
2. For ICS (or for that matter : all "smart" / "advanced" chunk selection) clients - current SOTN is messing up accurate chunk status, thus damaging.

To reduce further damage to ICS based clients, newer versions of mods with SOTN must make sure not to hide chunks from clients with ICS (can be identified via protocol handshake). Usage of current implementation of SOTN in new versions of mods reduces the chances of official client migrating to ICS ...
P2P is about sharing, ed2k is my choice !
0

#18 User is offline   omeringen 

  • löl
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 983
  • Joined: 01-January 06

Posted 12 May 2010 - 05:59 AM

@taz-me,
Part hiding features are not the point here. . . I am sure that all the coders would remove Sotn/hideOS features if official client have had ICS, because there will be no need to use them.

This post has been edited by omeringen: 12 May 2010 - 06:00 AM

0

#19 User is offline   taz-me 

  • I'm taz (a modder)
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 587
  • Joined: 07-December 06

Posted 12 May 2010 - 11:11 AM

@omeringen: Don't get me wrong here : it seems that at least on this, we're seeing things eye to eye (i.e.: it's a great idea for official client to migrate to ICS).

BTW: it's taz (always was, is, will be), since it's taken here (likely by myself however at the time my email was at a former employer and password probably gone ...).

This post has been edited by taz-me: 13 May 2010 - 05:46 AM

P2P is about sharing, ed2k is my choice !
0

#20 User is offline   James R. Bath 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 790
  • Joined: 02-August 04

Posted 13 May 2010 - 12:44 AM

Adding ICS seems a much better use of time than adding VMCS (Virtual Mule Chunk Selection). As much as I'd like to see which chunk an actual mule would pick, I'd prefer ICS.
Currently recommending and using: eMule beba 2.63
For slot control only, currently recommending: Tombstone Xtended 1.0 (or higher) if you absolutely must have slot control


Posted Image

Quote

Where there is a mule there is fuel. Where there is a stool sits a fool. - Winston Churchill

-1

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users