[eStallion]

Following happened:

My data hard disk went full while downloading some Gigabyte size files. When eMule popped up "disk full" messages, one after the other of the files giving this message vanished from the eMule Transfer list.

After solving the free-space problem I restarted eMule. But the files didn't re-appear in the "Transfer" list. I was rather surprised...


So I examinated the Temp folder to see what happened: Those files that still where listed in the Transfer list consisted of three files:

• 002.part
  
• 002.part.met
  
• 002.part.met.bak

The others, which didn't re-appear, only consited of two files:

• 001.part
  
• 001.part.met

AFAIK, the *.part.met.bak files aren't required by eMule. So I investigated the contents of those .part.met files that didn't have an accompanying .part.met.bak file... The result: They were empty! No... they consisted of loads of Space characters!

I'm unable to continue these downloads now. The empty .part.met files are giving neither eMule nor me the chance to do some manipulation to use the existing .part files to resume these downloads... Very frustrating...


I guess, eMule updates the .part.met files occasionally by creating a backup file and doing some modifications. My suggestions:

• Either have eMule create .part.met.bak files right from the beginning or refrain from using them at all.
  
• Allocate enough empty space for the .part.met files right from the beginning so that appending to the file won't be required.

[torpon]

Have you tried How To Recover Corrupt Downloads ?
Cheers

This post has been edited by torpon: 19 September 2010 - 08:28 AM

[xilolee]

Hi

options extended:
- check diskspace: 4095
- safe met/dat file writing: always

If the files were empty, you don't need to recover them (only re-download them).

another post that explains how to recover files (click)

[eStallion]

@xilolee

Thanks for your help!

I've set safe met/dat file writing to always, checked create temp files non-sparse and set check disk space to 4095.


@torpon

Thanks for the hint. I would like to refrain from installing Java and the whole bunch just to rescue my downloads. I prefer an eMule internal solution, working from now on.

This post has been edited by eStallion: 29 September 2010 - 12:39 AM

[DatHebIkWeer]

I had the same problem. And now (after the fact of course) I neatly changed the settings as suggested and hope that will work. (Except the sparse setting that doesn't have anything to do with this problem.)
I couldn't recover the files anymore with metfilegenerator (as suggested in the link). It seems that program basically just takes the ###.part.met.bak file and puts it back as ###.part.met file. This works of course if the .bak file is unaffected. But in my case the .bak file was affected just as the .met file.
This of course should never happen. The destroyed .bak file is a serious bug in itself.

A strange detail is that at least 1 file that was destroyed was not changed during that session. I am quite positive about that because it had missing parts that had been missing from the beginning for about a month. The whole file had downloaded except the missing parts. I could recover the file by just renaming it to the proper extension, but it is still incomplete.
Of course this means another bug. It means that eMule does superfluous writes or at least superfluous file open/close operations on files that don't need to be changed in the first place. This is another serious bug that affects the reliability of the program needlessly (as proven by this disastrous occurrence).

Now the files were lost irrecoverably by the machine. I could just rescue a handfull by renaming them, and then only partially. Some files were still readable by vlc player so I could check them and try to do a new search for them, but most are lost without recovery. This is mainly due to the fact that the ed2k link is not stored safely. The ed2k link does not change during the download process, so it should not be stored in a way that is affected by it. It should of course allways be stored in a user readable read only file of some sort, so that the .part file can be rehashed at all time and we can try to reload the file if all else fails. If that information is needed eMule should access the file not with open exclusive write, but with open shared read. In that way eMule will never destroy it unwantedly.
So no safe storage of vital read only information is another serious bug I would say.

And then of course: eMule gave beeps when it encountered the write errors that caused this. So it knew something was wrong but I couldn’t do anything about it. But just sounding an alarm is not proper error handling. If a known and foreseeable error situation occurs there should be damage preventing handling of some sort. In this case just halting all file i/o and a dialog explaining what was wrong and asking what I want to do would have helped me a lot.

More details about the situation:
When I checked the disk had 0 bytes left on it. In the process of emptying space I closed the program without thinking. (I know that was stupid.) and in the closing process the beeps and errors occurred.
I use eMule 0.50a on a Windows 7 machine. My temp folder is on a dedicated external hdd. The disk is usually just half full. But for some reason at the time of the crash it just filled up with (500 GB of) cache of some sort. If the caching came from eMule, my virus scanners, Windows or the disk itself I don't know. I reinstalled eMule, reconfigured my virus scanners, made a new directory and moved my .part files there. Now everything seems to work better. So I cann't say if that is caused by a bug in eMule or not. I think not.


Of course the directory itself may be have been corrupted since it is just a database with limited capacity. Many files and extensive file changes and file fragmentation may cause it to just go bananas. I use the sparse file option on an NTFS disk to save space. This of course generates a tremendous fragmentation, specially with a lot of large files and when the disk gets fuller.

[Link64]

DatHebIkWeer, on 07 July 2012 - 03:11 AM, said:

This is mainly due to the fact that the ed2k link is not stored safely. The ed2k link does not change during the download process, so it should not be stored in a way that is affected by it. It should of course allways be stored in a user readable read only file of some sort, so that the .part file can be rehashed at all time and we can try to reload the file if all else fails.

Saving the ed2k link in a separate file was requested a while ago here. With an implementation like I have described in post #5 of that thread, all those recovery tools would be redundant.

DatHebIkWeer, on 07 July 2012 - 03:11 AM, said:

When I checked the disk had 0 bytes left on it. In the process of emptying space I closed the program without thinking. (I know that was stupid.) and in the closing process the beeps and errors occurred.

You should keep at least 10% of the disc space free, i.e. for a 500GB drive you should set in eMule's preferences to leave at least 50GB free.

[DatHebIkWeer]

Link64, on 07 July 2012 - 03:48 PM, said:

You should keep at least 10% of the disc space free, i.e. for a 500GB drive you should set in eMule's preferences to leave at least 50GB free.

I back your solution in that thread.
I know but I set up eMule wrong. Changed that now. My foolishness is not really the issue here I think. We should focus on getting eMule to be resistant to monkeys like me.

This post has been edited by DatHebIkWeer: 07 July 2012 - 03:46 PM

[Link64]

DatHebIkWeer, on 07 July 2012 - 05:43 PM, said:

I know but I set up eMule wrong. Changed that now. My foolishness is not really the issue here I think. We should focus on getting eMule to be resistant to monkeys like me.

I agree, hardcoded minimum of free disc space (5% or something like that) would be a good idea.

[DatHebIkWeer]

Link64, on 07 July 2012 - 08:00 PM, said:

I agree, hardcoded minimum of free disc space (5% or something like that) would be a good idea.

I think hardcoding any limit is never a good idea (or hardly ever). It is the quickest and surest way to make your program outdated even before you released it.
But the default setting of the disk space monitor enabled with a free space of for instance 500 MB would help I guess. I think the current way of setting the check diskspace option is misleading or prone to human error. You can fill in a disk space and leave the check diskspace option unchecked. In that case the system will not check diskspace although the user may assume it does. It would be more transparant if the checkbox was removed and you just fill in the number of MB's (or a percentage of diskspace) you want. If you set it to zero the program should understand that you mean no space check.


5% on a 2 TB disk is 100 GB. That may be a nice margin to prevent .part files from getting too fragmented, but eMule can safely keep filling the disk way past that limit (I hope).

[Link64]

DatHebIkWeer, on 09 July 2012 - 03:07 PM, said:

Link64, on 07 July 2012 - 08:00 PM, said:

I agree, hardcoded minimum of free disc space (5% or something like that) would be a good idea.

I think hardcoding any limit is never a good idea (or hardly ever). It is the quickest and surest way to make your program outdated even before you released it.

Than make it preferences.ini only and allow/accept settings over GUI only above that. OTOH hardcoding limits, which prevent issues like this one, is completely OK IMHO, better than let the users trash files and waste network resources.

DatHebIkWeer, on 09 July 2012 - 03:07 PM, said:

5% on a 2 TB disk is 100 GB. That may be a nice margin to prevent .part files from getting too fragmented, but eMule can safely keep filling the disk way past that limit (I hope).

In theory you can use the entire capacity of a disc, however that works only good, if that disc is used for long term storage, so once the files have been written to it, nothing changes, or at least very seldom. You would also not use NTFS (or similar file system mainly designed for daily use) on such disc. On a drive, where lots of files are permanently written to it or deleted and replaced with a new ones, some reasonable margin has to be kept for to prevent too high fragmentation. 5% is actually already pretty low, for ext2/3/4 about 10% are usually recommended. With current drive sizes and prices it should also not be a problem to use such a small fraction of a drive for to prevent performance (and other) issues.

[DatHebIkWeer]

Link64, on 09 July 2012 - 05:20 PM, said:

Than make it preferences.ini only and allow/accept settings over GUI only above that. OTOH hardcoding limits, which prevent issues like this one, is completely OK IMHO, better than let the users trash files and waste network resources.

I thought of that too. Preferences.ini gives it a kind of monkey proof threshold. But make it a reasonable value that people can understand. It has to be a safety precaution, not a dictate on how to use hardware properly. If it is a preferences.ini thing than it should be so that in normal circumstances a user will not have to change it.
If you set the value too high it will provoke a lot of frustration and anger. And remember: standards change continuously. I talk about a 2 TB disk because I bought 1 last week. On such a disk a space limit of 5% will be an unexplainable 100 GB. I could also buy a 4 TB disk but that was relatively expensive (and had no USB 3). In 2 years time disks will be 6 TB or 8 TB. Then 5% will be 300 or 400 GB. A hard setting that seems reasonable now may seem outrageous in the near future. (That is of course another reason why a new version of eMule is needed.)

I used to be a supervisor over a small team of programmers in the past. If one of them came to me with a hardcoded limit, he (or she) would hear it. You can make limits dependent on relevant criteria, but you always have to be careful with them. Because usually you have an inclination to look at the past, but your program will be used in the future. You can never know all the situations a user will bring the program into.
10 years ago we had 2 Mb ADSL internet and had to develop with many people having modem connections in mind. Now we can get 50 Mb internet in some places and even faster. Standards and limits based on modem speeds and 15 year old specifications are hopelessly outdated.

[Link64]

DatHebIkWeer, on 09 July 2012 - 09:54 PM, said:

It has to be a safety precaution, not a dictate on how to use hardware properly.

Sure, it can be designed as a pure safety precaution, or something that also ensures, that the users won't be pissed off too much because they used wrong settings (than come here, get to know what they did wrond and complain about why is eMule not done in a way, that prevents that from happening). Usually system requirements of a program should consider both, simply to keep support requests low and also to prevent people from thinking, that your program breaks their systems. Can be also done by a combination of hardcoded safety precaution and preferences.ini only setting, which can be lowered untill the safety precaution level by advanced users. So together with the GUI setting we end up with 3 limits. The safety precaution however would also have to take into account, that eMule can have it's folders on system drive, where leaving "just few MB" might still lead to serious issues.

DatHebIkWeer, on 09 July 2012 - 09:54 PM, said:

I talk about a 2 TB disk because I bought 1 last week. On such a disk a space limit of 5% will be an unexplainable 100 GB.

It's the same "unexplainable" as 15 years ago 100MB of a 2GB disc were (and for sure a lot cheaper than those 100MB). 5% is 5% regardless of the drive size. If the margin is not too small, it could also be designed more "soft", i.e. if we are above 5% and eMule wants to write a file, it could go also a little below that limit and suspend than all not started downloads.

Anyway, before talking about numbers and eventually implementing that in eMule one would simply have to test, how NTFS behaves on such large drives with eMule's activities on it, it's possible, that for large drives lower values might be OK.

DatHebIkWeer, on 09 July 2012 - 09:54 PM, said:

Standards and limits based on modem speeds and 15 year old specifications are hopelessly outdated.

A program, which has not been updated for 15 years is probably quite unusable anyway, if eMule was not updated since let's say 0.30, we couldn't use it (without some tricks) on Vista/7 and without Kad and considering current server situation pretty much nobody would be using it. There's no need to plan for the next decades, next 2-3 years are more than enough.

[coluche]

hej,

first, I agree that this should be seen as a bug, and a persitent one, for so long time now.

The bak files imo. are not only totally useless, but even worse, as they make think there was a working backup thingy, and thus keeping users from taking their own precautions.

I can't understand why a .met and the according met.bak can be corrupted at the same time.

-----------------------------

ps - a really tiny minimum of free diskspace is enough. With me having way too little harddisk space, I had emule working fine with disk filled up to 100% minus 20 MB.

yeah, fragmentation and all, I know, but that seems to cause hdd performance problems only, not data errors.

This post has been edited by coluche: 10 July 2012 - 12:43 PM

[Link64]

coluche, on 10 July 2012 - 02:31 PM, said:

ps - a really tiny minimum of free diskspace is enough. With me having way too little harddisk space, I had emule working fine with disk filled up to 100% minus 20 MB.

yeah, fragmentation and all, I know, but that seems to cause hdd performance problems only, not data errors.

Sure, to prevent errors it should* be enough (as long as not on system drive), hence I posted the idea of 3 different limits, one configurable over the GUI which is limited by the preferences.ini only limit which is again limited by the hardcoded one. In default configuration the preferences.ini only one should have a value, which do not lead to unnecessary high performance issues, so the avarage user can set the limit higher, if he wants to leave more space free than that. Advanced users, who think they know what they are doing, could edit the preferences.ini value and basically do what they want as long as they don't get to the hardcoded safety limit.

Alternatively we could have just the hardcoded and the GUI setting with a clear warning (confirmation popup or so) in case the user chooses something below x% of total disc space. Might be eventually a better solution than 3 limits now that I think about it.


*) can still lead to problems if eMule's config folder is on that drive and safe saving of met/dat files is activated, IIRC I have succeeded myself something similar, when I tried running eMule out of 100MB TC container (just for eMule and config, no downloads)

This post has been edited by Link64: 10 July 2012 - 02:01 PM

[Meuh6879]

use the 4Gb minimum disc space ... it's the limit of a FAT32.
many users use an external (NTFS or FAT32) hard drive with a "portable" version of emule (well, emule is portable at the begin, lol...)

[DatHebIkWeer]

I could live with a hardcoded 20 MB limit that users can increase but not decrease if that is what eMule needs.
Don't base standards on Fat32. It is outdated and silently being phased out by MS. Its maximum file size is only 4 GB and that is too small for today’s standards. If I buy an external HDD today it is default formatted NTFS with a 4 kB block size.
Actually I never really understood what MS wanted with Fat32 anyway. When it was introduced NTFS was already up and running and clearly superior to it in every way. Maybe it was a copyright matter.

You should put the temp directory on a system disk if you really like trouble.

[Link64]

DatHebIkWeer, on 11 July 2012 - 01:59 PM, said:

Actually I never really understood what MS wanted with Fat32 anyway. When it was introduced NTFS was already up and running and clearly superior to it in every way.

FAT32 does not need so much space for itself, so specially for small drives it's way better than NTFS. The features of NTFS are simply not needed everywhere or even might have some negative effects, hence we also have now exFAT, which from my experience is quite good for not too large external drives. On large drives (2TB or so) the clusters get so big, that NTFS is better again, even if at the beginning it seems to use about 2x more space for itself.