[Jaff - MIDI maniac]

Store on HDD last 8, 10 or 16 served clients (they need to have at least one chunk complete) to give them as possible sources for those who download from us. Here's a possible format, something like fakes.dat:

eD2k_hash_1,file_size ip:port, ip:port, ip:port, ...
eD2k_hash_2,file_size ip:port, ip:port, ip:port, ...
eD2k_hash_3,file_size ip:port, ip:port, ip:port, ...

or you can store this data in pure binary data.
This will be good for rare files but it will fail for dinamic ip's. Also it will be great to keep the first 3 (at least) possible sources the clients with file complete (ex. we've downloaded the file from them or we've uploaded the last chunk).

[Stulle]

no use. if we are still downloading the file we will store them anyway and if not we wouldn't be asked anyway.

[leuk_he]

downlaoding sources can be stored (for a limited time). Uploading is not a option because the rpotocol does not suggest to annoucne a file (unless you want to request a file you do not download...... .)

[James R. Bath]

Jaff - MIDI maniac, on Feb 19 2008, 10:29 PM, said:

Store on HDD last 8, 10 or 16 served clients (they need to have at least one chunk complete) to give them as possible sources

I like the idea in general. Utorrent, and maybe other torrent clients, keeps a session file that makes resuming where one left off lightning quick compared to eMule. Anyway, this really is a feature request since whatever eMule does now, it's basically like someone who's been knocked unconscious after closing it.

[Stulle]

dude, src are being stored. what's the point?! we are going to ask sources and get someone to ask us for upload in the same instant.

[Jaff - MIDI maniac]

Stulle, on Feb 20 2008, 09:04 AM, said:

no use. if we are still downloading the file we will store them anyway and if not we wouldn't be asked anyway.

Stored sources are for those who are asking (download) this file. What sources have you stored for downloaded files? None! I want to give more sources to the others. This will help rare files to spread more faster. Refresh of sources is done by replacing the older source with other new (last) served client. leuk_he said (for a limited time) 3-7 days would be great. You can't give sources that dosen't exist anymore.

@James R. Bath: You're in the wrong forum. Like Stulle already said, sources are being stored (but sadly only until file is completed. Are not keeped to serve others clients).

[Stulle]

first of all, the time span you propose is way too long.

further, you propose this for rare files but unfortunately will sources for rarefiles most likely know each other already and furthermore they would find themselves pretty fast because both of them are searching for the other.

this is quite different for complete sources. this is a crucial point for rare files. those complete sources are likely to have the parts the downloaders want but none of the downloaders will have them. furthermore will they only be found but not search for new sources (like downloaders). unfortunately can we not find them as another complete source because we don't need them. the only thing we can probably do is advise ANYONE to use KAD and it'll be fine. if all downloaders search for like one new complete sources they will find that one source sooner than if only one downloader would search that new complete source. once the new source is found it will be distributed via source exchange and other downloaders will get aware.

there is no need to change anything at all. it works just fine and as a complete source we can not change nothing.

[the Doctor]

Stulle, on Mar 3 2008, 09:53 AM, said:

further, you propose this for rare files but unfortunately will sources for rarefiles most likely know each other already and furthermore they would find themselves pretty fast because both of them are searching for the other.

Automatic bread to sense converter activated.

Search: rare
Replace: incomplete

stulle to sense, on Mar 3 2008, 10:18 AM, said:

Further, you propose this for incomplete files but unfortunately will sources for incomplete files most likely know each other already and furthermore they would find themselves pretty fast because both of them are searching for the other.

One counter question:

Sources are being stored, but are hashes of those sources stored?


Question 2:

If you have a file complete and request the file via some source modification from a downloader you saved when you shut down morph, would not the peer see this as a a passive request? Is this possible? Or would this ed2k protocol abuse explode in subspace?

[Stulle]

the Doctor, on Mar 3 2008, 10:27 AM, said:

Stulle, on Mar 3 2008, 09:53 AM, said:

further, you propose this for rare files but unfortunately will sources for rarefiles most likely know each other already and furthermore they would find themselves pretty fast because both of them are searching for the other.

Automatic bread to sense converter activated.

Search: rare
Replace: incomplete

stulle to sense, on Mar 3 2008, 10:18 AM, said:

Further, you propose this for incomplete files but unfortunately will sources for incomplete files most likely know each other already and furthermore they would find themselves pretty fast because both of them are searching for the other.

One counter question:

Sources are being stored, but are hashes of those sources stored?


Question 2:

If you have a file complete and request the file via some source modification from a downloader you saved when you shut down morph, would not the peer see this as a a passive request? Is this possible? Or would this ed2k protocol abuse explode in subspace?

unfortunately is the doctor wrong.

Quote

This will help rare files to spread more faster.

so he is proposing to save parts of our queue when we got the file completly to improve the sharing of rare files.

as to your first question, when we save sources they will be stored according to the naming of the part file. so we will load one source list for each partfile (if present of course).

for your second question, i think it might eventually be possible to make such a hack for it is no less than a hack. if we have the file complete it is redundant to ask another client for data he can give us. this leads us to the next problem, how do we know the other client really still wants us? he might have stopped or finished the file while we were offline.

while storing sources works just fine storing the queue (partially or fully) is more likely to fail. we have basically no idea if the other client wants us and we should not spam him for any reason, especially not with some dirty hack.

This post has been edited by Stulle: 03 March 2008 - 10:05 AM

[Jaff - MIDI maniac]

1) time span is not too long. If served client with source is not able to connect to recommended source (it might be offline / changed his IP or hasn't the file anymore) it will ignore the source and find his own sources in traditional way but at least it might work. (Don't forget the fact that you still have the file in share) Why I'm proposing 7 days? If you don't run eMule 24/24, 7/7 you'll see that you ain't served anyone certain file, so... no sources updates, no more served clients. Stored sources will expire.

2) I've proposed this for folowing reasons:
a ) spreading faster the rare files (think as a releaser) It's not important the upload ratio but the faster spreading of file (and sources of course)
b ) our/they incomplete files - give the known sources to others downloaders (if we download a file and we've finished before them)
NOTE: before fisnishing a file you're more able so store sources with complete files than after completing your own file because you'll store only served clients (by you) with last (sub)chunk missing
c ) more sources for clients who are not connected on KAD
d ) compensate in a way the actual source exchange. Assume you've just started your eMule. Everyone who download from you will ask: more sources, more sources... What sources can you give them? You have 2 choices 1) some from queue (if you have clients on it asking the same file, having at least a part from file) 2) usualy NONE or 3) from stored sources
e ) some files shared are ignored on some servers. We can recommend sources connected to them who due to server limitations "don't share a certain file"

I've forgot the time counter there for every (X) known source. It might look like this:

eD2k_hash_1,file_size ip:port (time), ip:port (time), ip:port (time) ...
eD2k_hash_2,file_size ip:port (time), ip:port (time), ip:port (time) ...
eD2k_hash_3,file_size ip:port (time), ip:port (time), ip:port (time) ...

@the Doctor:
1) There's no need to check your stored sources if they still have the file. Clock is ticking, sources are replaced / deleted when time's up for them.
2) There's no need to store clients hashes unless you have a method to search a client on server(s) or KAD by his hash.

This post has been edited by Jaff - MIDI maniac: 05 March 2008 - 01:25 AM

[Stulle]

we shall not hammer innocent ips and we cannot garantuie we get the same client after 24h. even below that timespan it is somewhat unlikely. that is exactly the point why save sources was somewhat changed in the past due to official rules.

here an small exert from the source code: "source is not complete and file is very rare"
and only if this applies source exchange will be granted. there is no use in discussing this any further under the point of view from distributing sources.

further, i told you why it is bad to use either some dirty hack or save the queue in parts. we cannot know what the other side is really like and if we assume that it will be way too random.

i think we can quit this discussion now, everything has been said and despite the positive facts you may see is noone willing to code hacks or completly new systems of file distribution.

reread the part about the source finding in my above post. even if a client has not enabled KAD (which is very rare these days) will he get such sources via KAD. we are also searching sources globally via the ed2k servers so we might very well get the sources from the server a client is connected to.

file closed.

[Jaff - MIDI maniac]

Stulle, on Mar 5 2008, 10:00 AM, said:

we shall not hammer innocent ips and we cannot garantuie we get the same client after 24h. even below that timespan it is somewhat unlikely. that is exactly the point why save sources was somewhat changed in the past due to official rules.

Hammering what? Saved sources are given when somebody asks for them and time for them is not infinite. Probably you've also skipped reading the refresh of them. Btw, not everyone has dinamic IP and if you don't want to keep sources for that time nobody stops you. Until eM session is closed is enough to me. I'll keep it running for weeks.

Stulle, on Mar 5 2008, 10:00 AM, said:

further, i told you why it is bad to use either some dirty hack or save the queue in parts. we cannot know what the other side is really like and if we assume that it will be way too random.

what's random when a client completes a file from you or storing complete sources before finishing downloading a file? I'm sorry but I can't see things from your point of view. "the other side" is very clear:
1) has the complete file or full chunks from it (sources stored before we complete the same file from download/upload queue)
2) has finished file / a full chunk from us (sources stored from served clients)

Stulle, on Mar 5 2008, 10:00 AM, said:

i think we can quit this discussion now, everything has been said and despite the positive facts you may see is noone willing to code hacks or completly new systems of file distribution.

you think we can quit the discution. I'm not hurry up and still wait others opinion.
"Patience is the companion of wisdom" - Saint Augustine

Stulle, on Mar 5 2008, 10:00 AM, said:

even if a client has not enabled KAD (which is very rare these days) will he get such sources via KAD.

What you're saying is like "You'll sill recive a mail from me even if you don't have a mail adress"
Take a look at eM in statistics and you'll see yourself percent of connected clients on KAD/serves

Stulle, on Mar 3 2008, 10:53 AM, said:

the only thing we can probably do is advise ANYONE to use KAD and it'll be fine.

why not reduce KAD trafic? Why not serve clients with sources before they re-re-research on KAD / global servers?

Stulle, on Mar 3 2008, 10:53 AM, said:

there is no need to change anything at all. it works just fine and as a complete source we can not change nothing.

[Stulle]

Jaff - MIDI maniac, on Mar 5 2008, 09:15 PM, said:

Stulle, on Mar 5 2008, 10:00 AM, said:

we shall not hammer innocent ips and we cannot garantuie we get the same client after 24h. even below that timespan it is somewhat unlikely. that is exactly the point why save sources was somewhat changed in the past due to official rules.

Hammering what? Saved sources are given when somebody asks for them and time for them is not infinite. Probably you've also skipped reading the refresh of them. Btw, not everyone has dinamic IP and if you don't want to keep sources for that time nobody stops you. Until eM session is closed is enough to me. I'll keep it running for weeks.

it might not always be the truth that the other client has a dynamic ip but nevertheless do we have to assume this. this is exactly the reason why the official prohibited to save sources for too long. that's a fact and needs no further discussion here for neither we nor you make the rules.

Jaff - MIDI maniac, on Mar 5 2008, 09:15 PM, said:

Stulle, on Mar 5 2008, 10:00 AM, said:

further, i told you why it is bad to use either some dirty hack or save the queue in parts. we cannot know what the other side is really like and if we assume that it will be way too random.

what's random when a client completes a file from you or storing complete sources before finishing downloading a file? I'm sorry but I can't see things from your point of view. "the other side" is very clear:
1) has the complete file or full chunks from it (sources stored before we complete the same file from download/upload queue)
2) has finished file / a full chunk from us (sources stored from served clients)

again, sources are saved. so if we do not have the file completly we are saving sources for that very file.
if we completed the files we are not even being asked for sources because we finished the file.
if we have someone in queue we can only guess if he still wants something from us. nevertheless, this network is not build up on the rather wild guess if somebody wants something but soley on the request for something. whenever somebody wants something we have he will ask for it and there is no point in changing a thing unless you want to create a ridiculous network where we ask random clients if they want something we have. no bloody point!

Jaff - MIDI maniac, on Mar 5 2008, 09:15 PM, said:

Stulle, on Mar 5 2008, 10:00 AM, said:

i think we can quit this discussion now, everything has been said and despite the positive facts you may see is noone willing to code hacks or completly new systems of file distribution.

you think we can quit the discution. I'm not hurry up and still wait others opinion.
"Patience is the companion of wisdom" - Saint Augustine

yeah and the wheel needs reinvention because it could run even smoother... we are getting no where and if i am not totally mistaken you are the person that already started a ridicolous and ages long discussion about a matter that has no chance of being implemented some time ago.
what is your problem? don't you know what the meaning of "no" is? go and fetch a dictionairy. unless you do it on your own you will have to accept other peoples decisions. i am really sure any coder in his right mind will agree with my reasoning and if your sole purpose is to discuss you are more than wrong in this board.
oh and just for the record, i don't give a damn about quotes of wise people because they don't make the person quoting them look clever but rather make them look stupid for they were not able to phrase it themselves in a proper manner.

Jaff - MIDI maniac, on Mar 5 2008, 09:15 PM, said:

Stulle, on Mar 5 2008, 10:00 AM, said:

even if a client has not enabled KAD (which is very rare these days) will he get such sources via KAD.

What you're saying is like "You'll sill recive a mail from me even if you don't have a mail adress"
Take a look at eM in statistics and you'll see yourself percent of connected clients on KAD/serves

small mistake there when i was in a rush this morning. let me correct. "even if a client has not enabled KAD (which is very rare these days) will he get such sources via Source Exchange."
this is a simple fact and if you think a bit you will (eventually and hopefully) come to the same conclusion.

Jaff - MIDI maniac, on Mar 5 2008, 09:15 PM, said:

Stulle, on Mar 3 2008, 10:53 AM, said:

the only thing we can probably do is advise ANYONE to use KAD and it'll be fine.

why not reduce KAD trafic? Why not serve clients with sources before they re-re-research on KAD / global servers?

we are not reducing a thing for kad. we are doing the researches ever and ever again while a certain limit of sources is not reached. this limit however is proportional to the hardlimit of a file.
you are talking about rare files we need to support. this softlimit will never be reached and thus will we keep on searching. we change not a thing if we eventually ask a client that really wants something from us.

Jaff - MIDI maniac, on Mar 5 2008, 09:15 PM, said:

Stulle, on Mar 3 2008, 10:53 AM, said:

there is no need to change anything at all. it works just fine and as a complete source we can not change nothing.

what is that supposed to mean? am i right with the assumption that you are only discussing for the sole purpose of discussion? you are getting pretty low, this board is not a discuss-for-fun-board but rather a board to discuss technical issues of emule and in this particular section of emule mods.

still you ignore the fact that any change we would have to make in order to make clients aware of us having a file completly would be just a very dirty hack. the problem of dirty hacks is that they have very bad downsides. any hack is exactly what it sounds like, it is something bad and you cannot and even less should build up a network upon hacks. the only way around a hack would be to make this part of the protocol. this however is nothing a mod could do and i assure you that the officials would not even consider it just because of the ridiciolous of your request.

@ leuk_he: i humbly ask you to close this thread, this is leading no where.

This post has been edited by Stulle: 05 March 2008 - 08:54 PM

[leuk_he]

Stulle, on Mar 5 2008, 09:50 PM, said:

@ leuk_he: i humbly ask you to close this thread, this is leading no where.

If one party want to stop discussing this that is a good option...

Anyway, there should be very very good reasons for hacks in the protocol.


Thus: