Official eMule-Board: [emule 0.50a] Rehash Of Files On Windows Version Switch - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

[emule 0.50a] Rehash Of Files On Windows Version Switch

#1 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 09 May 2023 - 11:34 PM

Is a normal behavior on eMule 0.50a to rehash files when switching the Windows OS version?

The eMule installation is the same. It has the same path and the same partition letter. Permission is granted for all users on eMule and all directories that uses.

When I switch from XP 32 bits Service Pack 2 to Windows 7 Service pack 1 64-bits and vice-versa, eMule rehashes the files and cast the following error on verbose for each shared file when rehashing:

[time and date] AICH Hashset to write should be already present in known2.met [file hash here]


Is this normal, can be avoided? (on an HDD is a killer when you have several (dozens, hundreds) huge files shared)

CPU: Core 2 Duo E8400
RAM: ~3.75GB on XP 32-bits // 10GB on Windows 7 64-bits

This post has been edited by squidlogic: 09 May 2023 - 11:36 PM

0

#2 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 12 May 2023 - 08:47 PM

When you are using windows xp, verify if you have the known2.met or the known2_64.met (or both) in emule config folder.
When you close emule, verify which one of the two has been updated.
Then do the same when using windows 7.
I suspect emule creates/rebuilds the known2.met when you are using windows xp; and it creates/rebuilds the known2_64.met when you are using windows 7.
Because most likely the known2_64 is not compatible with 32bit os versions...
INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non č serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dā subito l'allegrIa! Immagine Postata
0

#3 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 13 May 2023 - 01:00 AM

(Note: when I use the word intallation I mean the "portable" installation folder, not implicit assumption of using an installer as I always used the zip binaries)

I ever and only had known2_64.met on this eMule installation that has been carried from a Windows 2000 system.

So, I only have known2_64.met. Should both be there?

I meant known2 above. I have known.met, and known2_64.met, but as known2 root filename, only known2_64.met. Not any known2.met.

When I closed on Windows 7 the file known2_64.met was rebuilt.

When I went back to XP known2_64.met was rebuilt.

Could be that the log in verbose is legacy and actually means both known2.met and known2_64.met.

The oldest backup of the installation folder I have at hand on this machine dates back 2011 (I must have some backups on some old backup CD/DVD), and even then I had only known.met and known2_64.met.

Note that until recently I never fiddle with a 64-bit system, so couldn't be that back then was build that file if it is related to a 64-bit system.

I'm cheking with a disk catalog of old backup DVDs and I can find, dated 2007, an installation that only had known.met and known2_64.met as well.

No known2.met anywhere, I fear :S

So if known2_64.met might not be compatible with 32-bits systems... and I always had that file since Windows 2000 (and it was only 32-bit)... what is wrong here?


The only thing that I can't be sure, as I don't have a time machine, is that, maybe..., I could have carried from emule Plus their files (as about 20 years ago I used it), but, still, if that could be true, why official eMule still used, always, known2_64.met intead known2.met on a 32-bit system?

What I'm sure is that since at least 0.4x there were only known.met and known2_64.met but no known2.met on the installation folder.


If a fix could be delete the file... what would I lose? :S Would help rename to known2.met?

This post has been edited by squidlogic: 13 May 2023 - 01:05 AM

0

#4 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 13 May 2023 - 01:18 AM

I think all the above I just have wrote has no sense as per Help pages:

Quote

o Known2_64.met
Stores AICH hashes of all downloaded and / or shared files. If you delete this file, eMule will have to rehash all files on the next restart. If you don't want eMule to remember downloaded files, disable the "Remember downloaded files" option in the preferences.
o Known2.met
This file is not used by eMule anymore. If you do not plan to downgrade to an earlier version, you can delete this file.


So, something else might be involved.

Anyway, no explanation of why the rehash :(

This post has been edited by squidlogic: 13 May 2023 - 01:18 AM

0

#5 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 14 May 2023 - 08:24 AM

known2_64.met was introduced with emule 0.47a (26.01.2006):
Spoiler
Did you search/download/upload large files (over 4GB)?
If yes, then I would add to the list: if you downloaded/uploaded large files and later you use a 32bit windows version, the known*.met files will be rebuilded.

This post has been edited by xilolee: 14 May 2023 - 08:24 AM

INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non č serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dā subito l'allegrIa! Immagine Postata
0

#6 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 15 May 2023 - 04:51 PM

View Postxilolee, on 14 May 2023 - 10:24 AM, said:

Did you search/download/upload large files (over 4GB)?


Probably yes. I could think of a couple of files beyond that limit that I could have downloaded in the past (even a decade ago).


View Postxilolee, on 14 May 2023 - 10:24 AM, said:

If yes, then I would add to the list: if you downloaded/uploaded large files and later you use a 32bit windows version, the known*.met files will be rebuilded.


:confused:

But... this has no sense to me because, until I started using (actually fiddling with) a 64-bit Windows version, a few months ago (this year), I always have been on a 32-bits Windows version.

As so, if it depends on the above (actions on files over 4GB) then, there is the reason I have a known2_64.met. Ok, right.

But not because I have used in the past a 64-bit Windows version. I never used a 64-bit Windows version. Ever.

In fact, the first rebuild on known2_64.met didn't happen because I switch from a 64-bit Windows version to a 32-bit Windows version, but just the opposite, this is from XP 32-bit to 7 64-bits. When I ran it on 7 64-bit, the rehash/rebuild of the file happened.

Then when I switched back again from Windows 7 64-bit to XP 32-bit, it rebuilt again. Rehashed again.

And a side note. When I was running it on Windows 7 64-bit, I didn't download anything, I was just sharing files. Just testing that all was working... but it surprised me the results :S

And it concerns me because these switches, both, might be "killer" to the HDD. To date SSD is not an option and is not the topic here, anyway, but to understand the reason of the rehashes and, if possible, fix it.


But, from what I see, there is no fix at hand, right?. I should need some script, then, that when using some Windows version uses one known2_64.met or the other so it doesn't rebuild it, over and over and over, if I switch back and forth of Windows installations.

If it is just as you say I'm just confused :confused:

I'll accept it if there isn't other explanation, but I don't see where the problem lies, at least I don't see the correlation of that explanation with my usage of eMule till I recently used a 64-bit Windows version first time.


Anyway, thanks for your efforts in helping me, xilolee, I really appreciate it :)
0

#7 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 15 May 2023 - 05:00 PM

You could save the known*.met files in both pcs.
When you use the x86 os version, you'll replace the known*.met files in the config folder with the ones you saved.
And the same for the x64 os.
INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non č serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dā subito l'allegrIa! Immagine Postata
0

#8 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 16 May 2023 - 12:11 AM

@xilolee, in the end is what I already thought, using an script, some bat that checks the Windows version and swaps files, but maybe I was doing something wrong and I wanted to fix it, so the reason of this topic, with no better luck sadly :/

By the way, is the same PC, just two different Windows installations sharing volumes/partitions with full rights, one among them where eMule resides.

Thanks for all your help :)
0

#9 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 16 May 2023 - 05:26 AM

Let me (us) know if you resolve it like I wrote. ;-)
INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non č serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dā subito l'allegrIa! Immagine Postata
0

#10 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 20 May 2023 - 07:00 PM

After a couple of days on Windows 7 64-bits and create an script to replace/swap known2_64.met files, depending on the OS version, it works, yes.

Back on XP 32-bit, it didn't rehash everything. Only a couple of files I downloaded on Windows 7 that weren't recorded on the XP known2_64.met.

But, still, the point here is WHY? Why does this happen? Where is the incompatibility of both files?

As known2_64.met is a binary file, if there were a reliable known2_64.met viewer (search results point to a met viewer for known.met instead), I'd really like to have a look to both files to see where is the problem.

Because, to give some unrelated but similar scenario. While being on Win7 64-bit I was using the 64-bit version and the 32-bit version of an Internet browser (Mozilla based), some time one, and some time the other, but, both, pointing to the same user profile.

The result is the expected. It doesn't matter if you are on one or the other Windows version or a 32-bit or 64-bit software, the profile data is the same and didn't cause any bad behavior. Cookies were there, history was there, bookmarks/favorites were there, add-ons were there. All was working.

It is no sense why doesn't this happen with eMule.

As I said, I'd really like a reliable known2_64.met viewer to see what is the conflicting difference.
0

#11 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 20 May 2023 - 07:45 PM

But still you can not use firefox x64 in xp 32bit.
You can use firefox 32bit in x64 OSs because windows developers thought about it (i.e., 32bit programs), creating special folders, files, variables and so forth.
And firefox probably made its corrections too.

It is possible that emule developers thought (at that time) the users won't use emule x86/x64 config files on both systems and then didn't encounter this problem.
Hence they didn't find it nor correct it.
INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non č serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dā subito l'allegrIa! Immagine Postata
0

#12 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 20 May 2023 - 11:47 PM

View Postxilolee, on 20 May 2023 - 09:45 PM, said:

But still you can not use firefox x64 in xp 32bit.
You can use firefox 32bit in x64 OSs because windows developers thought about it (i.e., 32bit programs), creating special folders, files, variables and so forth.
And firefox probably made its corrections too.


There aren't special folders or files.

The profile files are a compound of files holding data. That same data is compatible with any OS (including *nix) and any architecture.

There aren't any special files for one or the other OS or architecture. The file format for bookmarks is the same; the file format for the sqlite holding the history, is the same; the sqlite for cookies, is the same (note that both history and cookies were earlier stored in simple plain text files, long ago and still compatible); and so on...

I can't agree with that argument that the developers thought about it.

It is like saying that a text file, that is just a compound of ones and zeros, that depending on the desired character encoding can be set in one or the other way, could be read, or not, depending on the OS or architecture.

That is why I say that, if the file wasn't in binary format, I'd like to see what are the differences when storing on Win7 64-bit and XP 32-bit (and I'd like to know what happens on Win7 32-bit to compare).

There is no much sense, to me, that depending on the OS and architecture the file is saved differently. What a headache or nightmare for the developers :o

If I knew coding I'd love to check the source code to see what is going on there :S

It is certainly weird.

Anyway, this is going around in circles, so let's leave it here :)

Swap the file was the obvious workaround, but I'd really love to know why this happens :)
0

#13 User is offline   xilolee 

  • eMule 0.50b BETA1 user
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 7983
  • Joined: 20-August 08

Posted 21 May 2023 - 07:36 AM

Developers or someone that know how this works could read this topic, hence maybe we will have an answer... Some day.
INCONCEIVABLE! - You keep using that word. I do not think it means what you think it means.
come ottenere aiuto italian guides - guide della sezione italiana
italian support - sezione italiana scaricare la lista server
ottenere id alto impostare le porte nel router
recuperare file corrotti i filtri ip
Sembra talco ma non č serve a darti l'allegrIa! Se lo lanci e poi lo respiri ti dā subito l'allegrIa! Immagine Postata
0

#14 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 24 May 2023 - 08:26 PM

I'm not sure if I can follow the whole discussion, but what I can say is that it doesn't make a difference for eMule if it runs on a 64bit or 32bit OS. Since it's compiled for 32bit it will always run in the 32bit architecture/compabiltiy layer. The rehashing is not related to that.
Most likely the rehashing is a timestamp problem. Files are considered as changed as soon as there timestamp doesn't match with the value in the known.met.

#15 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 25 May 2023 - 01:31 AM

@"Some Support"

I know that should run on 32-bit mode, but as the behavior was as of a change of Windows version and architecture... I couldn't find other reason. From W2000 to WXP, 32-bits both, I didn't have these issues.


What exactly do you mean by timestamp? Doesn't eMule store just a number like "epoch"?

Because, if otherwise, I have different date formats on both Windows as well as on one of them I have Daylight Saving Time correction disabled. Does that matter?

I have to admit that this brings more doubts :S

Is fixable on user side? What corrections could I do?

EDIT:

Anyway, re-thinking, what timestamp. I mean, what timestamp should store of files? Also, I'm talking about known.met, but known2_64.met.

This post has been edited by squidlogic: 25 May 2023 - 01:46 AM

0

#16 User is offline   megaT 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 62
  • Joined: 09-May 20

Posted 25 May 2023 - 01:41 PM

@SomeSupport

Hi, do you know where the part about checking the timestamp of the file is?
I checked a bit in the code, it seems that a thread is spawned in order to check the file (SharedFileList.cpp), then
there is a Run which call's method's from KnownFile.cpp - in fact it create's a new record.

Right now I don't see where those record's are actually checked and/if the file was being processed (whether comparing against a timestamp or something else?).
Im just seeing that CKnownFile class store's m_tUtcLastModified as uint32.
There is also GetUtcCFileDate which uses CTime (MFC, which stores __time64 internally) but it's not used in the CKnownFile (nor in the database?)

Edit:
Well ok think I've found it, it's CheckAndAddSingleFile (SharedFileList.cpp, line 1606...).
It check's the file's last write time and then offers this to the method FindKnownFile.
This method (KnownFileList.cpp) compares the last write time (UTC), the file size and the file name,
so the file will be tried and rehashed if it was copied to the new windows installation.

This post has been edited by megaT: 25 May 2023 - 02:15 PM

0

#17 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 25 May 2023 - 09:22 PM

View PostmegaT, on 25 May 2023 - 03:41 PM, said:

so the file will be tried and rehashed if it was copied to the new windows installation.



But nothing is copied anywhere...

Anyway, if it helps, imagine the setup this way:

Disk ---> nth number of partitions

x partition holds Windows XP 32-bit
y partition holds Windows 7 64-bit
z partition holds eMule folders (this includes the eMule exe, config, log, etc directories as well as incoming and shared folders)

z partition has a letter, and the letter is the same to both Windows installation.


But still, I don't understand why check the date.

I actually understand it as, instead to check the whole file hash each time eMule runs, it checks if it has been modified and then reshases it.

But, also, I don't find any circumstance under a shared/downloaded file is modified unless, maybe with a archive file (rar, zip, etc), or an office document (xls, doc, etc, that auto-rewrites on opening as backup)?

That aside, isn't the UTC time the same on both systems?. I'll check where is the difference, because DST shouldn't be the problem, I think..., and the timezone is also the same, but if we are talking about UTC, again, it doesn't matter... the offset.

I'm lost of why any of the above is a problem here XD

Did I just find a bug by mistake? :confused:

By the way, I used a virtual machine to run a Windows 7 32-bit and... the same eMule, the same letter the same config, and the same problem with known2_64.met. It brought the same error and started rehashing everything:

[time and date] AICH Hashset to write should be already present in known2.met [file hash here]



So, the architecture is not the problem :S

And as I understand by the help files on the page, the reference to known2.met in the error message is legacy, if now is being used known2_64.met.

This post has been edited by squidlogic: 25 May 2023 - 09:24 PM

0

#18 User is offline   megaT 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 62
  • Joined: 09-May 20

Posted 25 May 2023 - 10:38 PM

Well number of disk's isn't important here, this is all on the file system level.
The date is been checked in order to determine if the file was changed, if it was then it need's to be reprocessed.

Quote

I actually understand it as, instead to check the whole file hash each time eMule runs, it checks if it has been modified and then reshases it.

Which is exactly what happens, the date is checked in order to avoid rehashing... which however doesn't seem to work on your side - hence the file is being rehashed.

Quote

That aside, isn't the UTC time the same on both systems?

It should be, some systems however have an invalid date/time set.
But it shouldn't matter because the timestamp of the file is compared against what's in the index file,
I assume that something doesn't work correctly with this 32/64 index file transation/conversion.

Quote

As known2_64.met is a binary file, if there were a reliable known2_64.met viewer (search results point to a met viewer for known.met instead), I'd really like to have a look to both files to see where is the problem.

You can use any hex editor you like, the format is as specified in Indexed.cpp (I think, check the destructor).
It isn't super elegant, but quite an simple approach of serialization.

This post has been edited by megaT: 25 May 2023 - 10:41 PM

0

#19 User is offline   squidlogic 

  • Member
  • PipPip
  • Group: Members
  • Posts: 26
  • Joined: 14-August 10

Posted 26 May 2023 - 12:43 AM

View PostmegaT, on 26 May 2023 - 12:38 AM, said:

Well number of disk's isn't important here, this is all on the file system level.
The date is been checked in order to determine if the file was changed, if it was then it need's to be reprocessed.


It was just to give an idea of how is the setup :)

... to show that no file is copied or modified, either way.


View PostmegaT, on 26 May 2023 - 12:38 AM, said:

Quote

I actually understand it as, instead to check the whole file hash each time eMule runs, it checks if it has been modified and then reshases it.

Which is exactly what happens, the date is checked in order to avoid rehashing... which however doesn't seem to work on your side - hence the file is being rehashed.


Then, so, I thought... why not CRC32?

I barely remember reading that the CRC32 of a file is saved in the MFT (on NTFS, or maybe was on FAT32). Wouldn't be as quick as check the date and less prone to error? The file system has already a way to tell if a file has been modified without checking for its date.

After all it is also a hash of the file. There are more possible collisions than with other hashes, but could be checked with other data, just in case, such as the file size or whatever, and in doubt, rehash.

Told from the point of view of an user, not a programmer :)


View PostmegaT, on 26 May 2023 - 12:38 AM, said:

I assume that something doesn't work correctly with this 32/64 index file transation/conversion.


Note that I tested on a virtual machine with a 32-bit version Windows 7, and did the same as Windows 7 64-bits. It didn't accept whatever is on known2_64.met from the XP system :-?

Weird, weird, weird.

If it is my system, I'll double check the dates on one or the other system, or whatever. Anyway, it is weird.


View PostmegaT, on 26 May 2023 - 12:38 AM, said:

Quote

As known2_64.met is a binary file, if there were a reliable known2_64.met viewer (search results point to a met viewer for known.met instead), I'd really like to have a look to both files to see where is the problem.

You can use any hex editor you like, the format is as specified in Indexed.cpp (I think, check the destructor).
It isn't super elegant, but quite an simple approach of serialization.


:-S

I'll have a look, despite I don't know coding on C/C++.

I can check with an HEX editor the server.met, and similar files, but even then, with large files is crazy.

There should be some tool to convert to human readable :-S

If I just knew coding... :-S


Thanks for the hints, megaT :)

This post has been edited by squidlogic: 26 May 2023 - 12:44 AM

0

#20 User is offline   megaT 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 62
  • Joined: 09-May 20

Posted 26 May 2023 - 01:21 PM

Quote

I barely remember reading that the CRC32 of a file is saved in the MFT (on NTFS, or maybe was on FAT32). Wouldn't be as quick as check the date and less prone to error? The file system has already a way to tell if a file has been modified without checking for its date.

Well I don't think that NTFS does CRC32 on the fly, basically the timestamp's are the most common and default way to check if the file was modified.
Back then you'd had to do everything in order to spare yourself IO and disk access, nowadays with SSD it might be not so problematic anymore,
generating the CRC will also require to iterate the file completely.

Quote

After all it is also a hash of the file. There are more possible collisions than with other hashes, but could be checked with other data, just in case, such as the file size or whatever, and in doubt, rehash.

No, in fact CRC32 is a checksum - not a cryptographic hash, it cannot safeguard against file collisions, it's only for data integrity.
These day's I guess we'd rather use SHA anyways for this instead of CRC32... it depends what it cost's and if it needs to be fast on hardware level, CRC32 is still good to use depending on the use case.
On your every day PC nowadays of course hashing is fast enough.

Quote

There should be some tool to convert to human readable :-S

If I just knew coding... :-S

Well feel free to start coding... it's never too late. ;)
Im a bit more active here now because I have time again to tinker with eMule/ed2k.
0

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users