[eklmn]

DavidXanatos, on Dec 23 2006, 11:53 AM, said:

...
When we have a part complete that the remote cleint wants on his reask we only need to send him part status no crumb status as at this point for him it _only_ mathers do we have somethink vor him or not, it is irrelevant is it a part or a crumb.
...
I hope this is clear enough explained

now it's better because you finaly explained why you proposed such design and i have a different opinion in thak case. IMO the crumbs status information is not only matter of two clients you and him, but whole network. Look how part status information is used. By request of the next part all selection algorithms are using availability information or in other words part status from another clients. So IMO the crumbs status information should be used same way, i.e. this info supposed to help us to choose the part properly.

DavidXanatos, on Dec 23 2006, 11:53 AM, said:

If we dont have any part he may want we must send him our crumb status so that he know we have somethink for him he wants.
Thats to the reask part.
In case he gets an upload slot we have to send to him our complete crumb map in order to allow him to choice the crumbs/parts he needs the most.

As you wrote above the crums status should be sended in 2 cases:
1) on normal reask if we detected that remote client does not need any part from us,
2) on gettig an upload slot.
IMO the sending of the crumb status is redundand because:
1) Since we don't know what crumbs the remote client already have we need to send him an info about all crumbs. So the client will have already an info about whole status
2) If client got an upload invitation from NNS sources (no parts & no crumbs to request base on previus reask) he must to request the current status (not another way around, i.e. spaming with crumbs info for whole on upload invitation)

BTW, David, Merry Christmas!

[DavidXanatos]

Hi,
I understand your point, ofcause it is an advantage to can obtimise also crumb selection,
My neo actualy smartly selects only parts the crumbs inside a choisen part are downloaded lineary from the begin of the part.
I think that as long as the crumb system is only an addon to the regular part system my solution is sufficient,
but when we use the crumb system as our primary system we indeed need to update the crumb status regulary.

@netfinity
About the incremental status update and the a4af stuff,
I think we can make the permanent storring of status informations for all files permanently like in NetF and Neo a requirement for the imminet updates when we gets an part complete,
the status update on a regulary reask may for the time being remain as it is: a full update I think.

PS: Merry Christmas!

David

[netfinity]

I have been modifying my code to have a "Part Status Abstraction Layer" now, so that most parts of eMule doesn't have to bother at all with what size chunks and how many they are. This makes changing the size of the crumbs very easy and also makes it possible to mix protocols (eg. BT).

The first prototype will use eDonkey crumbs and will send complete crumb sets. However, hashing will not be part of the initial release! Currently crumbs are chosen lineary from the beginning of the part, but maybe I will fix that before releasing.

When it comes to sending the part status maps, it is probably best to always send them with the complete crumb set. The reason is that we can then use it to obsolete the ICS system which isn't widely used anyway. By knowing how many crumbs that are complete within a part we can get a much better guess about the availability of the part on the net in whole.

Quote

I think we can make the permanent storring of status informations for all files permanently like in NetF and Neo a requirement for the imminet updates when we gets an part complete,

Yes, my thought. This will not be required for basic operation, but if you want to have the most recent status information this would be a requirement. Doing so would also allow a client to receive status updates for a file even thought it is currently in A4AF.

/netfinity

This post has been edited by netfinity: 29 December 2006 - 06:58 PM

[netfinity]

Reviving this old thread, then I finally got the feature in place. A number of things has changed since that time. For example, using up 60 MB of RAM for storing hash sets are not considered an issue anymore as most clients have gigabytes of RAM.

So, I have taken the old hybrid crumbs protocol and extended it to better fit eMule. I intend to publish the updated and backwards compatible protocol here.

The reason to stay backward compatible is mainly for allowing clients to implement the feature in steps. This allow developers that don't have time to implement sub-chunk verification or handling chunks of varying size to simply just implement the ability to download crumbs. This is easy and still benefits the network even if they don't re-share non-complete parts. Reason why it is beneficial for the network is that the precious resources of a complete source of a rare file wouldn't be bothered with clients asking for the same chunks as already downloaded by another client.

Stay tuned!!!

[tHeWiZaRdOfDoS]

IMHO this is a *very* useful addition. I'm positive that it'll help spreading rare files and improve overall *mule performance.
SCT v2 support will be added to the next kMule build - I'm just still fiddling with some codeparts that have to be "converted" from threaded hashing code

[Neoo26]

Nice that it will be added to kMule

[netfinity]

tHeWiZaRdOfDoS, on 24 August 2013 - 08:15 AM, said:

SCT v2 support will be added to the next kMule build - I'm just still fiddling with some codeparts that have to be "converted" from threaded hashing code

Yes, I know the threaded stuff messes up feature separation. Have thought that if I get enough time and there is enough interest, I might do an SCT only mod. However, that would most likely only happen if the official devs showed interest in merging this feature into the official client.

I'm looking also into making a patch for MLDonkey to partially implement SCT v1. v2 would take some additional workings, and I don't feel comfortable in ocaml. Reason why I'm especially interested in porting to this particular client is that it is popular in embedded systems like NAS, which tend to be on 24/7. But, anyone creaming loudly "me too, please!" might get my attention.




One thing that has to be investigated is how this feature will affect overhead, and how to best mitigate it. The retrieving of the recovery hash data causes insignificant overhead, but the part status vectors are another story if running on a narrow internet connection. One thing I have been thinking of is to require the upload to be set to 20kB upload or more to enable it, but I'm not sure if that is a good idea.