Official eMule-Board: Sr13-importparts-003 (45b Based) - Official eMule-Board

Jump to content


  • (4 Pages)
  • +
  • 1
  • 2
  • 3
  • Last »

Sr13-importparts-003 (45b Based) Last Updated: 2005/04/07

#1 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 19 November 2004 - 10:27 PM

As promised in this thread, I'm releasing my mod, that allows you to import data from any file to eMule download.

SR13-ImportParts-003 (based on eMule 0.45b)

What this mod does:
This mod allows you to import data from file to a download in eMule. Any time you have file, which contet partially matches your download, be that interrupted FTP/other P2P download, corrupted file from old CD, corrupted version of same file you've downloaded from eMule, you can use this mod to verify this corrupt file and import verified parts of data from such file to download.

How it works:
Right-click on any download (if you just added it, wait until hashset is obtained first), select "Import parts", then select target file to add import to queue. Since it uses same queue as file hashing, import may not start immediately until all previosly started hashing is done. You can add as many imports to queue as want - just click different download and follow same steps.

This mod imports from external file a) all parts that match hashset and B) all parts that DO NOT match hashet, but still have no single byte of data downloaded, so A/ICH can try to recover something from corrupt data. If part doesn't match and there's at least something downloaded already for it - it won't be imported.

History:

* 003
x New threading code - everything integrated in eMule hashing queue.
x Allow very long file names for files to be imported.
x Flushing tweaked a bit - there should less writing conflicts.
* 002
x Uses threading code from SiRoB of MorphXT mod.
x Added simple locking to prevent starting two imports at once.
x Added flushing - now eMule is much more responsive when import is in progress and it no longer eats memory equivalent to size of file being imported as well.
* 001
x First release, no threading.

TODO:
* If some data already downloaded for corrupt part in import source file, try to combine it with already downloaded data and see if that will make valid part.
* Add additional mode to import data from .part/.part.met pairs that would import only actually downloaded data.
Pros: faster (not important for importing from local HDD, but will noticeable imporve importing from network drives). Cons: won't work or will work worser if .met file is corrupted or if import .part file once downloaded data that did not pass part verification. I'm still not sure what to do with partial data in source file that doesn't fill entire part. Implementing combination mentioned above will take care of some cases.

Links to sources and binaries are here.

This post has been edited by roversr13: 13 April 2005 - 11:33 AM

0

#2 User is offline   geearf 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 350
  • Joined: 21-December 03

Posted 19 November 2004 - 11:37 PM

Would it be possible to download two files which should be the same, but having different hashes, and merging them automaticallly ?
If no i guess you can still do it by hand,
well that is a good idea,

Thanks.
0

#3 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 20 November 2004 - 12:41 AM

geearf, on Nov 20 2004, 02:37 AM, said:

Would it be possible to download two files which should be the same, but having different hashes, and merging them automaticallly ?
View Post


If hashes are different then it is not same file. However, you will recover any matching 9,28Mb parts with this mod - that's exactly what it was made for.
0

#4 User is offline   jpkoester1 

  • Premium Member
  • PipPipPipPipPip
  • Group: Members
  • Posts: 303
  • Joined: 05-June 04

Posted 20 November 2004 - 01:37 AM

Just to expand the idea, I'm sure it would be a lot more coding work, but I have this idea that might utilize this import feature nicely.

The idea is to use categories to 'link' two downloads together. An example might help to visualize this idea.

1. I search for a file I would like to have
2. I find several versions of the same size that I assume are almost the same except maybe metadata.
3. I put both of these files in the same category and choose to 'link' downloads in this category.
4. After both hashsets are received we compare which chunks are the same in both files
5. The file with more sources becomes the main file
6. The file with less sources becomes the 'helper'-file
7. We try to download chunks from the helper file that match the main file and then import them to the main file when they are finished.

Some restrictions should apply to this so we don't waste other peoples upload bandwidth.
1. The helper file should only request chunks the main file doesn't have yet.
2. Chunks from the helper file that don't match the main file should never be requested

Hope the idea makes sense to anyone else than me.

Let me know what you think.

Cheers,
JP

PS.: or does your mod already do that as you might have implied in your last post?
"I cna tyep 300 wodrs per minuet :D"
0

#5 User is offline   overlook 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 11
  • Joined: 16-November 04

Posted 20 November 2004 - 01:48 AM

I understand what he means, it had crossed my mind before.

He is talking he has 2 of the same files each is corrupted in a seperate place.
But he will need the correct hash so they can be merged correctly. :o

So it won't be acceptable from 2 inputs, he needs 3 inputs for this 2 work.
Hashsets are different for the same file if one has a corrupted archive. :)
0

#6 User is offline   jpkoester1 

  • Premium Member
  • PipPipPipPipPip
  • Group: Members
  • Posts: 303
  • Joined: 05-June 04

Posted 20 November 2004 - 02:10 AM

overlook, on Nov 20 2004, 03:48 AM, said:

I understand what he means, it had crossed my mind before.

He is talking he has 2 of the same files each is corrupted in a seperate place.
But he will need the correct hash so they can be merged correctly. :o

So it won't be acceptable from 2 inputs, he needs 3 inputs for this 2 work.
Hashsets are different for the same file if one has a corrupted archive. :)
View Post


If that is a reply to my post, that is not what I mean at all. I am not talking about corrupted files at all.

Certain file types (mp3, avi, etc) have metadat (movie name/artist/....). If a downloader changes any of this data he changes the file, so although 99.99% of the file are the same the filehash changes. That is why you often see several files of exactly the same length, same filename, etc but different filehash when you search for a file. My idea is to put these similar - but not exactly the same files into one category and then compare the subhashes to see which chunks are the same in both versions.

The file with most sources would then become the main download. The helper file(s) would only be used to request those chunks which have the same hash as the one from the main download.

Hope that makes it a little clearer.

Cheers,
JP
"I cna tyep 300 wodrs per minuet :D"
0

#7 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 20 November 2004 - 09:16 AM

Nice idea, jpkoester1, but, alas, this would require lots more work, than this on-demand import mod. You still can try to achieve this effect by importing parts back and forth between downloads periodically.
0

#8 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 20 November 2004 - 09:23 AM

Just a bit more about how it works:
This mod imports from external file a) all parts that match hashset and B) all parts that DO NOT match hashet, but still have no single byte of data downloaded, so A/ICH can try to recover something from corrupt data. If part doesn't match and there's at least something downloaded already for it - it won't be imported.

When you need this mod:
Anytime you have file, which contet partially matches your download, be that interrupted FTP/other P2P download, corrupted file from old CD or even corrupted version of file you've just mistakenly downloaded from eMule.

This post has been edited by roversr13: 04 February 2005 - 01:31 PM

0

#9 User is offline   overlook 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 11
  • Joined: 16-November 04

Posted 20 November 2004 - 12:18 PM

Well that is simple.

roversr13: you have to make the program stip the id3 v1 and v2, and then do a hash of the file.

then we can solve the problem we are unable to find sources to download from. ;)

This post has been edited by overlook: 20 November 2004 - 12:29 PM

0

#10 User is offline   geearf 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 350
  • Joined: 21-December 03

Posted 20 November 2004 - 01:48 PM

In fact, my question was quite the same thing jpkoester1 said.

This would be a lot better, but might be a lot of work also.
0

#11 User is offline   MRIS 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 78
  • Joined: 07-January 04

Posted 21 November 2004 - 06:19 AM

I've been trying to download this binary using emule on both razorback and KAD for the last 12 hours but zero sources. Can someone share this one please??
0

#12 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 21 November 2004 - 07:16 PM

Keep trying. My connection was dead for past 24h, but I've just came back online. I'm mostly on Razorback 2.
0

#13 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 24 November 2004 - 11:29 AM

overlook, on Nov 20 2004, 03:18 PM, said:

roversr13: you have to make the program stip the id3 v1 and v2, and then do a hash of the file.
View Post


Sorry, but I really dislike idea of stripping metadata or otherwise altering (even virtually) files in any way, especially because I, for example, tend to share files with tags in correct unicode, while there are lots of broken local encoding tags. IMHO, files that are even one byte different must never be joined automatically.
0

#14 User is offline   Skrybe 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 107
  • Joined: 07-February 03

Posted 03 December 2004 - 08:33 AM

This sounds like something I've been looking for. But I've got a couple questions...

You said, "This mode imports from external file a) all parts that match hashset and B) all parts that DO NOT match hashet, but still have no single byte of data in download, so A/ICH can try to recover something from this. If part doesn't match and there's at least something downloaded already for it - it won't be imported."

So if I have a file with 100 chunks and one chunk is missing, but I find another file with the same filesize and name (but a different hash) I can download it and then merge it into my first file. Is that right?

I assume the second file doesn't have to be complete? eg: As soon as the 2nd file got downloaded the chunk that is missing from the first file I could merge them?

If that's all true then your mod is exactly what I've been looking for. BUT I have a question/suggestion. Is there any way to make the mod force the 2nd file to download specific chunks? Because in the example above I could have to wait for 900MB to be downloaded before I finally got the ONE chunk I needed to complete the first file. So it'd be nice to say "just download chunk 42" then do the merge. :)

edit: I think I may have just come up with an answer to my own request... If you start the second download and get the hash but no parts you could import the 99 parts from the original (incomplete) file into the new one right? And then it'd only try to download the one chunk it needs. Is that the way it works?

This post has been edited by Skrybe: 03 December 2004 - 08:36 AM

0

#15 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 03 December 2004 - 10:08 AM

Skrybe, on Dec 3 2004, 11:33 AM, said:

This sounds like something I've been looking for. But I've got a couple questions...
[skip]
I assume the second file doesn't have to be complete? eg: As soon as the 2nd file got downloaded the chunk that is missing from the first file I could merge them?


Yes, it does all of that.

Quote

So it'd be nice to say "just download chunk 42" then do the merge. :)


No it doesn't control download at all. It doesn't care, actually where did you get file you're importing - it can come from anywhere.

Quote

edit: I think I may have just come up with an answer to my own request... If you start the second download and get the hash but no parts you could import the 99 parts from the original (incomplete) file into the new one right? And then it'd only try to download the one chunk it needs. Is that the way it works?


Yes, that's excatly how I recover corrupt files that I get from other sources than ed2k.
0

#16 User is offline   Sir_Boagalott 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 470
  • Joined: 23-September 02

Posted 05 December 2004 - 09:03 AM

I :love: you!! :kiss:

:lol:

I've been able to acomplish this for a long time, but up untill now, its been a huge pain in the ass.

Thanks!! :respect:
0

#17 User is offline   Skrybe 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 107
  • Joined: 07-February 03

Posted 06 December 2004 - 03:44 AM

Excellent! Thanks a bunch roversr13. I'm going to give this a whirl tonight.
0

#18 User is offline   crf 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 417
  • Joined: 09-September 04

Posted 15 December 2004 - 06:59 PM

It would be nice if you could use the aich hashes to recover bits of incomplete large parts as well.
0

#19 User is offline   roversr13 

  • -=/ SR 13 /=-
  • PipPipPipPip
  • Group: Members
  • Posts: 217
  • Joined: 25-December 02

Posted 15 December 2004 - 07:13 PM

roversr13, on Nov 20 2004, 12:23 PM, said:

Just a bit about how it works:
This mod imports from external file [....] all parts that DO NOT match hashet, but still have no single byte of data downloaded, so A/ICH can try to recover something from corrupt data. [....]
View Post

This post has been edited by roversr13: 15 December 2004 - 07:16 PM

0

#20 User is offline   crf 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 417
  • Joined: 09-September 04

Posted 16 December 2004 - 12:52 AM

Ooops :), I forgot to read carefully.

(PS -- can a mod edit the ed2k link for the binary to remove the space between the period (.) and the extension).

This post has been edited by crf: 16 December 2004 - 02:07 AM

0

  • Member Options

  • (4 Pages)
  • +
  • 1
  • 2
  • 3
  • Last »

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users