Official eMule-Board: Compare To Local File - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Compare To Local File Force Check a current transfer aganst a Complete file Rate Topic: -----

#1 User is offline   larek 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 31-January 13

Posted 31 January 2013 - 09:33 PM

The now 3 year old "force recheck" thread is still not implemented, and I doubt it ever will (as discussed below) as Edk links are not built like .torrent files.

However I think/proposing, that eMule add the ability to:
1. Force Check against a local file, when requested (much like "Force Recheck) (prompt for file to compare to.)
2. Force Check against current shares, to see if its currently sharing the file that it's trying to download from the network.
BUT:
A: The File Must be Complete.


My common use for the "force recheck" command on a torrent is when I get the file some other way for a torrent that's either slow, or not fully seeded. I Stop the Torrent, Replace the unfinished file, with my other_sourced file. Then do a "force recheck" to see if the other_source file is the correct one. (I can then also seed the complete file).

There is no Good equivalent in eMule. If you share the other_source file, you can then manual check the Hashes but that's very tedious. Also eMule will continue to download the file, not noticing that it is sharing the complete file, until you Download the entire file from the network which is wasteful. The only workable, solution I have is to use a Second PC, with the same edk links and have my normal eMule upload to it and download from it. But then I have all these copies, and It's not that simple to manage.


The File you are comparing against would have to be a complete file, because of the nature of edk links. The links only Store the complete file hash (most of the time) so there is not a way to see if you have good sub parts. This is why we don't see a torrent like "force recheck" command (As I understand it) (torrents do have hashes for each part). While it is possible to get sub-hashes, you must find at least one source on the network, and this was requested three years ago and has still not been implemented. This is a simpler will_always_work version, that I hopeful will get implemented in a much short time frame.


I would like to additional see an option to compete the download from the file, so that I get a file named the way the edk stipulates, and it go to the normal output directory. Just as if I had full downloaded the file off the network.

Checking against current shared files is another way to go. It at least saves from downloading a file you have. But as the collision is currently coded you lose the edk file name and its record in the transfers list, and the edk file name. I would rather have the check against file, or at least the current transfer record to not disappear.


Update:
For #1, yes you can share each file, read the hash, and then manually compare it to the the file your trying to get, but it very tedious to do for 50 files at a time
For #2, If your getting a file, and then you share out a complete version, eMule doesn't notice, you will still download the file, and Share it. Only when you complete the Download, does eMule notice.
::Neither of theses methods let you 'complete' the download. I.E. you don't get a file named as the edk link stipulates, in the directory you had setup for the download

This post has been edited by larek: 05 February 2013 - 08:28 PM

0

#2 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 01 February 2013 - 07:37 AM

View Postlarek, on 01 February 2013 - 12:33 AM, said:

1. Force Check against a local file, when requested (much like "Force Recheck) (prompt for file to compare to.)

Standard MS binary comparison command line utilities could do that. There are third party utilities as well, some of them with GUI.

View Postlarek, on 01 February 2013 - 12:33 AM, said:

2. Force Check against current shares, to see if its currently sharing the file that it's trying to download from the network.

You cannot add a file to downloads if you are already downloading it or have completed file in shares.

View Postlarek, on 01 February 2013 - 12:33 AM, said:

I would like to additional see an option to compete the download from the file, so that I get a file named the way the edk stipulates, and it go to the normal output directory. Just as if I had full downloaded the file off the network.

It adds unnecessary complexity, and I'm not sure we need two differently named files in two directories.


That last case is the reverse of point 2: add a complete file to shares while downloading the same file.
Unfortunately, this event is totally ignored by eMule. I understand that a warning in dialog window should be avoided here, but there is no record in the Log and even debug verbose log has nothing. Total absense of any signs of "collision" does not look right.
0

#3 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 02 February 2013 - 08:44 PM

View Postfox88, on 01 February 2013 - 08:37 AM, said:

View Postlarek, on 01 February 2013 - 12:33 AM, said:

1. Force Check against a local file, when requested (much like "Force Recheck) (prompt for file to compare to.)

Standard MS binary comparison command line utilities could do that. There are third party utilities as well, some of them with GUI.

Not as long as the eMule download is incomplete, however...

View Postlarek, on 31 January 2013 - 10:33 PM, said:

My common use for the "force recheck" command on a torrent is when I get the file some other way for a torrent that's either slow, or not fully seeded. I Stop the Torrent, Replace the unfinished file, with my other_sourced file. Then do a "force recheck" to see if the other_source file is the correct one. (I can then also seed the complete file).

... means he can simply add the file to eMule's share and compare the hashes. It's not that complicated, I've done that many times.

Complicated it gets at the moment when you find out, that the file from other source differs in for example one or two chunks (out of many), because eMule has no possibility to import single chunks from other files or downloads as I requested here. So right now situations like that involve using tools like MetFileRegenarator (which is not working really clean with the met files, much information missing after the usage) and/or hexeditor if you want to copy some chunks between the files.



View Postfox88, on 01 February 2013 - 08:37 AM, said:

View Postlarek, on 01 February 2013 - 12:33 AM, said:

2. Force Check against current shares, to see if its currently sharing the file that it's trying to download from the network.

You cannot add a file to downloads if you are already downloading it or have completed file in shares.

No, but you can do it the other way around, which is an issue than and I have requested that to be fixed here. That would make the verification very easy, if eMule aborts the download, that you got the correct file, if not you either got a different file or it gets complicated as described above.



View Postlarek, on 31 January 2013 - 10:33 PM, said:

The File you are comparing against would have to be a complete file, because of the nature of edk links. The links only Store the complete file hash (most of the time) so there is not a way to see if you have good sub parts. This is why we don't see a torrent like "force recheck" command (As I understand it) (torrents do have hashes for each part).

That's not entirely true. ed2k-links are indeed posted mostly like that on webpages, but once you've started the download you get the hashset from the sources, so unless you have no sources at all, you should have a hash for every 9500KB of the file, that's what my feature request about handling of almost identical files is based on. You can see if the hashset is already available in the properties of each download, you can also view it in the ed2k-link tab (you have to check "show hashset" or whatever this option is called in english).
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#4 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 03 February 2013 - 01:24 AM

View PostLink64, on 02 February 2013 - 11:44 PM, said:

Not as long as the eMule download is incomplete, however...

If you compare with local file, calculate it's hash and there is hash value in ed2k link of the download.

View PostLink64, on 02 February 2013 - 11:44 PM, said:

No, but you can do it the other way around, which is an issue than and I have requested that to be fixed

As said in my previous message, at least there should be a warning in the Log when adding a file to shares and eMule is already downloading the same file.
0

#5 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 03 February 2013 - 08:51 AM

View Postfox88, on 03 February 2013 - 02:24 AM, said:

View PostLink64, on 02 February 2013 - 11:44 PM, said:

Not as long as the eMule download is incomplete, however...

If you compare with local file, calculate it's hash and there is hash value in ed2k link of the download.

And which "standard MS binary comparison command line utility" is able to do that?



View Postfox88, on 03 February 2013 - 02:24 AM, said:

View PostLink64, on 02 February 2013 - 11:44 PM, said:

No, but you can do it the other way around, which is an issue than and I have requested that to be fixed

As said in my previous message, at least there should be a warning in the Log when adding a file to shares and eMule is already downloading the same file.

And who is going to see that message there? Popup window with this information is IMO the only way to make sure the user gets this info while in the background eMule should automatically stop downloading the corresponding download, delete it and share the complete file. No point in wasting even a single byte on something the user already has.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#6 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 03 February 2013 - 09:59 AM

View PostLink64, on 03 February 2013 - 11:51 AM, said:

And which "standard MS binary comparison command line utility" is able to do that?

Why it's so difficult to get?
We have two cases: two complete files, or download in progress and complete file.
In the first case the names of utilities are comp and fc. In the second you only compare hashes and file length.

View PostLink64, on 03 February 2013 - 11:51 AM, said:

And who is going to see that message there? Popup window with this information is IMO the only way to make sure the user gets this info while in the background eMule should automatically stop downloading the corresponding download, delete it and share the complete file. No point in wasting even a single byte on something the user already has.

Again you missed the point.
Right now there is no message anywhere at all, the collision is ignored.
While the situation deserves to be be mentioned in the log at least; those bytes would not be wasted.

On the subject of popup dialog and automatic actions.
There could be a popup window in case of user action (button click), but not in automatic mode (for example, when shared directories are rescanned at startup). Automatic stopping & deleting should not be implemented.
0

#7 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 03 February 2013 - 03:32 PM

View Postfox88, on 03 February 2013 - 10:59 AM, said:

View PostLink64, on 03 February 2013 - 11:51 AM, said:

And which "standard MS binary comparison command line utility" is able to do that?

Why it's so difficult to get?
We have two cases: two complete files, or download in progress and complete file.
In the first case the names of utilities are comp and fc. In the second you only compare hashes and file length.

It's difficult to get because you don't write even half of what you are trying to say. And as now you hopefully see, you can't get the hashes from MS command line utilities, so need another tool for that. Than better add it to eMule's share.



View Postfox88, on 03 February 2013 - 10:59 AM, said:

View PostLink64, on 03 February 2013 - 11:51 AM, said:

And who is going to see that message there? Popup window with this information is IMO the only way to make sure the user gets this info while in the background eMule should automatically stop downloading the corresponding download, delete it and share the complete file. No point in wasting even a single byte on something the user already has.

Again you missed the point.
Right now there is no message anywhere at all, the collision is ignored.
While the situation deserves to be be mentioned in the log at least; those bytes would not be wasted.

Right, now we have no message at all, but when implementing something, put it in the most suitable place and not somewhere, where it will be almost useless, because almost no one will see it. People have better things to do than watch the log, important/error messages should always come as a popup, where the user has to click on OK/close when he has read it (or not). In addition to the popup this information including deleting of the now unnecessary download should be of course written to the log. And BTW yes, downloading files, which the user already has, is waisting network bandwidth.



View Postfox88, on 03 February 2013 - 10:59 AM, said:

On the subject of popup dialog and automatic actions.
There could be a popup window in case of user action (button click), but not in automatic mode (for example, when shared directories are rescanned at startup). Automatic stopping & deleting should not be implemented.

Why not? Downloading those files is just waisting bandwidth, which could be used for something someone does not already have. If the user does not notice or ignores the message, eMule still has to take all necessary steps for to not waste upload bandwidth of other users for nothing. Also if the user has many (>100) downloads, it would be quite comfortable if eMule finds and delete the right download. If there are some reasons, which don't come to my mind right now for not deleting the files (you know any?), eMule should at least instantly stop the download.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#8 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 04 February 2013 - 12:01 AM

View PostLink64, on 03 February 2013 - 06:32 PM, said:

you can't get the hashes from MS command line utilities, so need another tool for that. Than better add it to eMule's share.

Obviously there is no MS utility, but you could get Link Creator.

View PostLink64, on 03 February 2013 - 06:32 PM, said:

the user has to click on OK/close when he has read it (or not).

When a file is moved into shared direcotry, it could remain unshared until eMule's restart. The user might be away at that moment.
Maybe the rightmost pane in eMule's status bar should be used more actively.

On second thought, popup in a separate thread might be used too.

View PostLink64, on 03 February 2013 - 06:32 PM, said:

Why not? Downloading those files is just waisting bandwidth

It would not make situation worse than it is now.
Just in case: file hash is used as a unique identifier, while it is not.

This post has been edited by fox88: 04 February 2013 - 12:07 AM

0

#9 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 04 February 2013 - 01:38 PM

View Postfox88, on 04 February 2013 - 01:01 AM, said:

View PostLink64, on 03 February 2013 - 06:32 PM, said:

the user has to click on OK/close when he has read it (or not).

When a file is moved into shared direcotry, it could remain unshared until eMule's restart. The user might be away at that moment.
Maybe the rightmost pane in eMule's status bar should be used more actively.

On second thought, popup in a separate thread might be used too.

It should be a popup window that remains there until the user clicks it away. The type of window you get for example from your web browser when you leave https site, not that what we see at the end of a download for few seconds. Of course eMule should continue it's normal operation in the background.



View Postfox88, on 04 February 2013 - 01:01 AM, said:

View PostLink64, on 03 February 2013 - 06:32 PM, said:

Why not? Downloading those files is just waisting bandwidth

It would not make situation worse than it is now.
Just in case: file hash is used as a unique identifier, while it is not.

Yes, it would not make it worse, but we can have it better. The current behavior is buggy and needs to be fixed, the issues with it can be found in my feature request. eMule also does not allow to start a download of a file which is already in the share, why shall it allow to continue a download of a file which got into the share after the download was started? That's not consistent.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#10 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 04 February 2013 - 04:58 PM

View PostLink64, on 04 February 2013 - 04:38 PM, said:

Of course eMule should continue it's normal operation in the background.

I think eMule already has that kind of popup windows.

View PostLink64, on 04 February 2013 - 04:38 PM, said:

The current behavior is buggy and needs to be fixed

Funny, I got a minus in that topic for saying that I remember a similar request. :)

View PostLink64, on 04 February 2013 - 04:38 PM, said:

That's not consistent.

It is not consistent, but it could be discussed which way is the right one.
Right now only the file ID is used for the check, not even file size.

This post has been edited by fox88: 04 February 2013 - 05:03 PM

0

#11 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 04 February 2013 - 08:14 PM

View Postfox88, on 04 February 2013 - 05:58 PM, said:

It is not consistent, but it could be discussed which way is the right one.
Right now only the file ID is used for the check, not even file size.

File size and if available the AICH hash should be used IMO, file hash alone is indeed not enough.

Really not the file size? That would be a pretty bad implementation since file hash and size are both available in the search results. That could be even considered as a bug.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#12 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 05 February 2013 - 04:12 PM

View PostLink64, on 04 February 2013 - 11:14 PM, said:

File size and if available the AICH hash should be used IMO, file hash alone is indeed not enough.

Generally speaking, all of these together are not enough.
That's why destructive automatic actions like deletion should not be used.

View PostLink64, on 04 February 2013 - 11:14 PM, said:

Really not the file size? That would be a pretty bad implementation since file hash and size are both available in the search results. That could be even considered as a bug.

No, not that bad.
In ED2K network file ID is postulated to be a unique identifier. File size in that sense is a reference information.
Therefore, not a bug. Not even a feature.
0

#13 User is offline   larek 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 31-January 13

Posted 05 February 2013 - 08:15 PM

I think I should Clarify a few things

View PostLink64, on 02 February 2013 - 03:44 PM, said:

View Postfox88, on 01 February 2013 - 08:37 AM, said:

View Postlarek, on 01 February 2013 - 12:33 AM, said:

1. Force Check against a local file, when requested (much like "Force Recheck) (prompt for file to compare to.)

Standard MS binary comparison command line utilities could do that. There are third party utilities as well, some of them with GUI.

Not as long as the eMule download is incomplete, however...

View Postlarek, on 31 January 2013 - 10:33 PM, said:

My common use for the "force recheck" command on a torrent is when I get the file some other way for a torrent that's either slow, or not fully seeded. I Stop the Torrent, Replace the unfinished file, with my other_sourced file. Then do a "force recheck" to see if the other_source file is the correct one. (I can then also seed the complete file).

... means he can simply add the file to eMule's share and compare the hashes. It's not that complicated, I've done that many times.


Yes I know you can manually compare hashes this way, Its not simple quick however, when your doing it to 50 files on a regular basis. Which is why I'm submitting this feature request.


View PostLink64, on 02 February 2013 - 03:44 PM, said:

View Postfox88, on 01 February 2013 - 08:37 AM, said:

View Postlarek, on 01 February 2013 - 12:33 AM, said:

2. Force Check against current shares, to see if its currently sharing the file that it's trying to download from the network.

You cannot add a file to downloads if you are already downloading it or have completed file in shares.

No, but you can do it the other way around, which is an issue than and I have requested that to be fixed here. That would make the verification very easy, if eMule aborts the download, that you got the correct file, if not you either got a different file or it gets complicated as described above.


Thats right! If your Emule file is not downloading well, so you give up and grab it in another way, then Share that file, you will continue to download it from the network, (and share it too).

Also I commonly want the File Name as described in the EDK link. So The I would like the Collision, to ask if I want to compete the download from the other file. So that I get a correctly named file.


View PostLink64, on 02 February 2013 - 03:44 PM, said:

View Postlarek, on 31 January 2013 - 10:33 PM, said:

The File you are comparing against would have to be a complete file, because of the nature of edk links. The links only Store the complete file hash (most of the time) so there is not a way to see if you have good sub parts. This is why we don't see a torrent like "force recheck" command (As I understand it) (torrents do have hashes for each part).

That's not entirely true. ed2k-links are indeed posted mostly like that on webpages, but once you've started the download you get the hashset from the sources, so unless you have no sources at all, you should have a hash for every 9500KB of the file, that's what my feature request about handling of almost identical files is based on. You can see if the hashset is already available in the properties of each download, you can also view it in the ed2k-link tab (you have to check "show hashset" or whatever this option is called in english).


Most of the files that I seek other sources for, have not found sources for a day or two on the network. And since we haven't gotten the sub hash version, in the past 3 years. I'm pushing for at least a full file check version.


I've updated the first post to address these concerns

This post has been edited by larek: 05 February 2013 - 08:30 PM

0

#14 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 05 February 2013 - 09:58 PM

View Postfox88, on 05 February 2013 - 05:12 PM, said:

View PostLink64, on 04 February 2013 - 11:14 PM, said:

File size and if available the AICH hash should be used IMO, file hash alone is indeed not enough.

Generally speaking, all of these together are not enough.
That's why destructive automatic actions like deletion should not be used.

View PostLink64, on 04 February 2013 - 11:14 PM, said:

Really not the file size? That would be a pretty bad implementation since file hash and size are both available in the search results. That could be even considered as a bug.

No, not that bad.
In ED2K network file ID is postulated to be a unique identifier. File size in that sense is a reference information.
Therefore, not a bug. Not even a feature.

Could you please decide wether you think that the ed2k hash is enough to be considered as a unique identifier or not? I mean, right now you say in one sentence, that we can't use it as unique identifier, few lines later it's not a bug that eMule uses it like that. Both can't be right. Either we can use it as unique identifier (eventually with file size and AICH hash) or eMule has a serious bug, which makes it think it has downloaded files even if it didn't.

This post has been edited by Link64: 05 February 2013 - 10:33 PM

So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#15 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 05 February 2013 - 10:16 PM

View Postlarek, on 05 February 2013 - 09:15 PM, said:

Yes I know you can manually compare hashes this way, Its not simple quick however, when your doing it to 50 files on a regular basis. Which is why I'm submitting this feature request.

That was exactly the reason for my feature request, the idea was that I can simply add them to the share and eMule does the rest including deleting unnecessary downloads. I have sometimes way over 1000 downloads, finding all downloads, that can be deleted is a pain in the butt.



View Postlarek, on 05 February 2013 - 09:15 PM, said:

Also I commonly want the File Name as described in the EDK link. So The I would like the Collision, to ask if I want to compete the download from the other file. So that I get a correctly named file.

That's something I haven't thought about. Maybe the confirmation window should have the possibility to choose the filename or even give the possibility to edit it if we are not happy with both of them.

Technically we would still not "finish" the download from the complete file, we would delete the temp file and use complete file in the share. No need to copy maybe even GBs of data only for to end up with two identical files.



View Postlarek, on 05 February 2013 - 09:15 PM, said:

Most of the files that I seek other sources for, have not found sources for a day or two on the network. And since we haven't gotten the sub hash version, in the past 3 years.

Well, file hash and eventually the size seem to be the standard way for eMule to distinguish files, so why not here. Small files under 9500KB have no hash set anyway.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#16 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 06 February 2013 - 12:42 AM

View PostLink64, on 06 February 2013 - 12:58 AM, said:

Could you please decide wether you think that the ed2k hash is enough to be considered as a unique identifier or not? I mean, right now you say in one sentence, that we can't use it as unique identifier, few lines later it's not a bug that eMule uses it like that. Both can't be right. Either we can use it as unique identifier (eventually with file size and AICH hash) or eMule has a serious bug, which makes it think it has downloaded files even if it didn't.

I thought it should be quite obvious, no?

If hash has less bits than a file, then there are more different files of that size than the total number of distinct hash values.

In ED2k hash is used as if it is a unique identifier.
Probably server would have troubles with two different files with the same ID, therefore it's pointless to share or download files with the same ID.
Therefore eMule behaves correctly by forbidding downloads and multiple shares of files with the same hash.

But I would not delete user's file with the same hash. Because file could be different, and most users would not like to have their files deleted just because a smart program thought it was the same. It should be user's decision.

Is that explanation clear enough?

This post has been edited by fox88: 07 February 2013 - 12:57 AM

0

#17 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 10 February 2013 - 11:20 AM

View Postfox88, on 06 February 2013 - 01:42 AM, said:

In ED2k hash is used as if it is a unique identifier.
Probably server would have troubles with two different files with the same ID, therefore it's pointless to share or download files with the same ID.

Wouldn't the server rather think it's the same file? I mean, you can maybe stop one eMule from sharing two files with the same hash, but not many different eMules. It would be nice at this point to hear from some dev, if file size is indeed not taken into account. I mean, the server has it, what would it do in case two different eMules sharing two files with the same hash but different size? Also considering that servers are more or less deprecated, we should more think about Kad than them and eventually find a better way of handling such cases in Kad.



View Postfox88, on 06 February 2013 - 01:42 AM, said:

But I would not delete user's file with the same hash. Because file could be different, and most users would not like to have their files deleted just because a smart program thought it was the same. It should be user's decision.

We still need to find a way out of this. I also never sugested using the file hash by itself. We also have the file size and eventually the AICH hash and the entire hashset (for the complete file we have tham for sure only not always for the download), they should definitely also be used.

One possibility would be to tell the user what eMule found (what is equal for both files, what not) and give him the option either to delete the download or continue it and make a binary comparison when ready (no, I don't expect the user to do it in 21st century). Also binary comparison of so far completed parts would be possible. However I'm not sure that all this efford is actually worth it, in ed2k we have certain way to distinguish between files which works so far good enough I think. Also you're not allowed to download a file which you have in your share, downloading it simply because you do it the other way around only breaks everything.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#18 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 11 February 2013 - 12:06 AM

View PostLink64, on 10 February 2013 - 02:20 PM, said:

if file size is indeed not taken into account.

What's the difference? Think of hash+size combination as one bigger hash. Add another hash or even hashes. Then apply the same ideas as above.
Thera are MD4/MD5 collision generators, by the way.
0

#19 User is offline   Link64 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2153
  • Joined: 25-January 04

Posted 11 February 2013 - 05:19 PM

View Postfox88, on 11 February 2013 - 01:06 AM, said:

What's the difference? Think of hash+size combination as one bigger hash. Add another hash or even hashes. Then apply the same ideas as above.

The difference is that in one case eMule and ed2k-server can distinguish between two files with the same file hash but different size, in the other case not. In one case we can download both files, in other not even if they are obvoiusly different.

The question here is not if the combination of file size and hashes really gives us a unique identifier (no), but wether it's good enough for the suggested feature or not. And I don't mean here some theoretical things, I mean "is that good enough?". Many other things are checked with the help of checksums and it seems to be good enough, so why not here, specially that we have more than one, that's better than in many other applications.
So poste ich richtig! (besonders Punkt 2 beachten)
Für alle, die was heruntergeladen haben und nicht wissen was sie damit anfangen sollen: endun.gen.

BOINC ...and you can always say you're working on a science project.
0

#20 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4974
  • Joined: 13-May 07

Posted 11 February 2013 - 07:58 PM

View PostLink64, on 11 February 2013 - 08:19 PM, said:

The difference is that in one case eMule and ed2k-server can distinguish between two files with the same file hash but different size, in the other case not. In one case we can download both files, in other not even if they are obvoiusly different.

There is no difference.
Once again: same size, same hash, but different contents - collisions of that kind must exist starting from moderately sized files. The only sure way to distinguish is binary comparison.

View PostLink64, on 11 February 2013 - 08:19 PM, said:

"is that good enough?"

So far it was; otherwise I bet the weakness would have been used against eMule.
The safety is in low probability of random collision and in the difficulty of creating collisions at will.
Moving to uncompromised algorithm would have been great, but difficult undertaking.

This post has been edited by fox88: 11 February 2013 - 08:07 PM

0

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users