[fox88]

WentloogWhix, on 06 October 2009 - 09:20 PM, said:

an ipfilter.dat file that eliminates bogon IP addresses.

Only bogon?

[WentloogWhix]

fox88, on 06 October 2009 - 10:19 PM, said:

WentloogWhix, on 06 October 2009 - 09:20 PM, said:

an ipfilter.dat file that eliminates bogon IP addresses.

Only bogon?

You may recall that no ipfilter.dat file is supplied with eMule 0.49c, so any additional blocking is an improvement. If you supply me with your IP range I'll be happy to add it in <cheeky grin>

[Wulp]

WentloogWhix, on 06 October 2009 - 09:01 PM, said:

Now for the bad news: none of the corrupt sources have been blocked, and they continue to spew out their corrupt data, and eMule continues to accept it, no questions asked.

Just to be on the safe side: Your test machine which is downloading the file does really have the AICH hash (as can be seen in file details)?

There are also two things which I'm not sure about:
- As far as I know when a client was detected that sent corrupted data it is not banned immediately but is rather banned after several times of repeating of sending corrupted stuff.
- I think in order that AICH can work correctly, there must be at least one genuine source because one needs to receive the AICH hashset from someone (not to be confused with the normal hashset of chunk hashes). (But I'm really not an expert here...)

This post has been edited by Wulp: 07 October 2009 - 12:07 AM

[WentloogWhix]

Wulp, on 07 October 2009 - 01:05 AM, said:

Just to be on the safe side: Your test machine which is downloading the file does really have the AICH hash (as can be seen in file details)?
- I think in order that AICH can work correctly, there must be at least one genuine source because one needs to receive the AICH hashset from someone (not to be confused with the normal hashset of chunk hashes). (But I'm really not an expert here...)

I can confirm I gave it the COMPLETE hashset as generated by selecting "Add eMule AICH Hash for advance corruption handling" as well as "Add complete hashset" in the "eD2K Links" tab. I created a small text file using notepad, pasted the information in, copied the file to the test machine, and used this information in Tools --> Paste eD2K links to get the download started.

I also wish to confirm that I posted the complete hashset in my second post, which was subsequently edited. I'm not going to post it again.

I will leave the download running (I stopped after 2 chunks) but last time I did this eMule happily downloaded 380MB, which is greater than the size of this file. It seems an awful waste of bandwidth just to prove what I have already stated.

[fox88]

WentloogWhix, on 07 October 2009 - 01:54 AM, said:

You may recall that no ipfilter.dat file is supplied with eMule 0.49c, so any additional blocking is an improvement.

That's an excellent reason not to use a decent IP filter.

WentloogWhix, on 07 October 2009 - 01:54 AM, said:

If you supply me with your IP range I'll be happy to add it in <cheeky grin>

I already supplied you with an idea to read Support forum. Now I got an additional argument that this case does not require a FR.
Have fun.

[WentloogWhix]

fox88, on 07 October 2009 - 08:05 AM, said:

WentloogWhix, on 07 October 2009 - 01:54 AM, said:

You may recall that no ipfilter.dat file is supplied with eMule 0.49c, so any additional blocking is an improvement.

Now I got an additional argument that this case does not require a FR.

You want me to block an entire german ISPs subscriber range? Given that this forum is hosted in Germany, I think that's a tad ironic. Why not block all subscriber ranges in all countries?

Perhaps a second feature request should be to provide an ipfilter.dat file as part of the standard install, like MorphXT does. Not that it would make any difference, because the rogue software wasn't blocked by that ipfilter either.

I guess what this forum is saying is that the rogue software has broken the system and that until it starts targeting the files they care about nothing will be done. Hopefully by then it will not be too late. In the meantime work proceeds on version 0.50 and I should hope that fixes the problem.

eMule is awesome software, and I guess resistance to adding features is a reason why it isn't bloatware.

[Famerlor]

Try the following: Get a fake free server.met, i.e. from gruk.org (link is available in support-subforum). Use the ipfilter.dat from sourceforge.net/emulepawcio.

Additionally you could try a mod with variable cbb-ban-threshold, i.e. eMule Spike2. eMule defaults to ban a client after 33% of all data received from it was corrupt. With variable cbb-ban-threshold you can lower this limit manually.
Manual banning of clients will never be implemented in official eMule or any legit mod.


Greetz--


Spike2



P.S.: And about the ClientAnalyzer-thing Tuxman was referring to: Its new version is unreleased so far, but will be called "eMule Tombstone 2.0" when released.

[WentloogWhix]

Famerlor, on 07 October 2009 - 12:49 PM, said:

Try the following: Get a fake free server.met, i.e. from gruk.org (link is available in support-subforum). Use the ipfilter.dat from sourceforge.net/emulepawcio.

Ipfilter.dat from sourceforge.net/emulepawcio doesn't exclude the IP range of the rogue clients, and I didn't use a server (fake or otherwise) to start the download.

The server I connected to is one of the ones on the gruk.org list.

Please note: I am NOT talking about fake files, but non-fake files that are being actively poisoned by rogue software.

I will check out Spike 2. Thanks for the info.

[WentloogWhix]

I followed the steps in "How To Get A Reliable Server List & A Good Ipfilter" and guess what?
Absolutely no change. eMule continues to download junk from the rogue software. Why am I not surprised?

[WentloogWhix]

I tried the OfFixed mod, but little has changed, except the number of servers.

From the server log:

08/10/2009 02:02:26 AM: Downloaded part 16 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 02:36:55 AM: Downloaded part 19 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 02:54:13 AM: Downloaded part 24 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 03:19:31 AM: Downloaded part 10 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 03:58:57 AM: Downloaded part 15 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 04:44:07 AM: Downloaded part 7 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 05:11:58 AM: Downloaded part 0 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 05:40:29 AM: Downloaded part 15 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 06:13:03 AM: Downloaded part 16 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 06:39:41 AM: Downloaded part 5 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)
08/10/2009 07:07:51 AM: Downloaded part 14 is corrupt :( (2009l ([color="#ff0000"]****[/color]) Unabridged.zip)

From the statistics page for this session, bearing in mind that the total file size to be downloaded is 226.62MB

eMule v0.49c OfFixed v1.0 Statistics [http://emule-project.net]
Session
 Downloaded Data: 102.17 MB
 Completed Downloads: 0
 Active Downloads (chunks): 2
 Found Sources: 18
 Download Sessions: 601
 Gained Due To Compression: 0 Bytes (0.0%)
 Lost Due To Corruption: 96.74 MB (94.7%)
 Parts Saved Due To I.C.H: 0
 Total Overhead (Packets): 2.35 MB (42.45 k)

At what point do you guys think eMule would wake up to the fact that it is getting corrupt data from rogue clients, or at the very least, unreliable clients. It seems to me that ALL clients are trusted, unless you block them in ipfilter.dat

This post has been edited by torpon: 08 October 2009 - 05:50 AM
Reason for edit:: Once again. avoid references to copyrighted stuff

[Famerlor]

Well, I did not suggest OfFixed, but Spike2-Mod. In the "Security Options" page you could lower the CBB-ban-threshold down to i.e. 25% or even 15% and see if sth. changes. (Yes, I know, Spike2mod is still 0.48a but a short test shouldn't hurt...)

[WentloogWhix]

Famerlor, on 08 October 2009 - 10:59 AM, said:

Well, I did not suggest OfFixed, but Spike2-Mod.

Sorry, I thought they were equivalent. Busy downloading now.
http://hostex.de/1190236146
I hope that this is the right version
eMule-0.48a-Spike2-1.2-bin.rar

This post has been edited by WentloogWhix: 08 October 2009 - 11:45 AM

[fox88]

WentloogWhix, on 07 October 2009 - 11:32 AM, said:

Perhaps a second feature request should be to provide an ipfilter.dat file as part of the standard install, like MorphXT does.

Binary does not include ipfilter.dat at all.
Including .dat file could be considered useless if you take into account that installer is not updated for many months.

WentloogWhix, on 07 October 2009 - 11:32 AM, said:

Not that it would make any difference, because the rogue software wasn't blocked by that ipfilter either.

You do understand how ip filter is made, don't you? This is a wake up call for you.

This post has been edited by fox88: 08 October 2009 - 05:02 PM

[WentloogWhix]

fox88, on 08 October 2009 - 05:57 PM, said:

You do understand how ip filter is made, don't you?

Yes, I know how they are made,a nd how they are edited. See earlier post:

WentloogWhix, on 05 October 2009 - 10:32 PM, said:

Even after I banned the ISPs entire IP range by manually editing the ipfilter.dat file, it was still impossible to download the file under attack, and other eMule 0.49c clients appeared to be stalled or distributing corrupted chunks.

fox88, on 08 October 2009 - 05:57 PM, said:

This is a wake up call for you.

Actually I think this is ironic, because this post is intended to be a wake up call to the developers of eMule that the protocols are not designed to thwart rogue software that is trying to poison legitimate, non-fake files, and I think this should change.

The ipfilter.dat is not sufficient to identify individual rogue software that is useing IP addresses in the range provided to broadband and dialup customers in Germany (84.135.69.193, 85.181.38.247, 92.228.200.119).

 84.135.69.193 Whois Information
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net[Who Is Domain][trace][Reverse DNS Search]/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
% To receive output for a database update, use the "-B" flag.

% Information related to '84.128.0.0[Who Is IP][trace][Reverse IP Search] - 84.135.255.255[Who Is IP][trace][Reverse IP Search]'

inetnum: 84.128.0.0[Who Is IP][trace][Reverse IP Search] - 84.135.255.255[Who Is IP][trace][Reverse IP Search]
netname: DTAG-DIAL19
descr: Deutsche Telekom AG
country: DE

 85.181.38.247 Whois Information
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net[Who Is Domain][trace][Reverse DNS Search]/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
% To receive output for a database update, use the "-B" flag.

% Information related to '85.176.0.0[Who Is IP][trace][Reverse IP Search] - 85.182.127.255[Who Is IP][trace][Reverse IP Search]'

inetnum: 85.176.0.0[Who Is IP][trace][Reverse IP Search] - 85.182.127.255[Who Is IP][trace][Reverse IP Search]
netname: HANSENET-ADSL
descr: ALICE DSL
descr: HanseNet Telekommunikation GmbH
descr: ADSL Pool Customers
country: DE
admin-c: HNT-RIPE
tech-c: HANO-RIPE
status: ASSIGNED PA
mnt-by: HANSENET-MNT
mnt-lower: HANSENET-NOC
mnt-routes: HANSENET-MNT
source: RIPE # Filtered

 92.228.200.119 Whois Information
% Information related to '92.228.0.0[Who Is IP][trace][Reverse IP Search] - 92.231.255.255[Who Is IP][trace][Reverse IP Search]'

inetnum: 92.228.0.0[Who Is IP][trace][Reverse IP Search] - 92.231.255.255[Who Is IP][trace][Reverse IP Search]
netname: HANSENET-ADSL
descr: ALICE DSL
descr: HanseNet Telekommunikation GmbH
descr: ADSL Pool Customers
country: DE
admin-c: HNT-RIPE
tech-c: HANO-RIPE
status: ASSIGNED PA
mnt-by: HANSENET-MNT
mnt-lower: HANSENET-NOC
mnt-routes: HANSENET-MNT
source: RIPE # Filtered

Unless you think that all germans should be barred from using eMule. ???

My current tests are using http://www.emulefutu...re/ipfilter.txt
which has a date of 20090925, with 6512 filters loaded.

Last time I posted a screen shot, but that was edited out, so I have carefully edited the file name so you won't be able to find it. It shows my test machine is happily downloading a poisoned version of a legitimate file.

This post has been edited by WentloogWhix: 08 October 2009 - 08:20 PM

[fox88]

WentloogWhix, on 08 October 2009 - 11:41 PM, said:

which has a date of 20090925, with 6512 filters loaded.

This is from my log: 243622 IP filters loaded.
You missed your wake up call.

[WentloogWhix]

fox88, on 08 October 2009 - 09:13 PM, said:

This is from my log: 243622 IP filters loaded.

I'll show you mine if you show me yours.

Does yours exclude the IP addresses above? I doubt it. Is it relevant? No. Could we please get back to the issue of poisoned files?

[fox88]

WentloogWhix, on 08 October 2009 - 03:49 AM, said:

I followed the steps in "How To Get A Reliable Server List & A Good Ipfilter" and guess what?
Absolutely no change.

Less than 7000 IP filters? You did not follow it properly.

WentloogWhix, on 09 October 2009 - 12:23 AM, said:

Does yours exclude the IP addresses above? I doubt it. Is it relevant? No.

You doubt even before knowing. You claim irrelevance without even learning what is normal IP filter.

WentloogWhix, on 09 October 2009 - 12:23 AM, said:

Could we please get back to the issue of poisoned files?

With incorrect setup behaviour of mule could be unpredictable.
Therefore right now there is no clearly visible huge issue.

[WentloogWhix]

eMule v0.48a Spike2 v1.2 Statistics [Spike2-Mod-User]

Session
   Downloaded Data: 207.18 MB
   Completed Downloads: 0
   Active Downloads (chunks): 1
   Found Sources: 12
      On Queue: 11
      Queue Full: 0
      No needed parts: 0
      Asking: 0
      Receiving hashset: 0
      Connecting: 0
      Connecting via server: 0
      Too many connections: 0
      Cannot connect LowID to LowID: 0
      Problematic: 0
      Banned: 0
      Asked for another file: 0
      Unknown: 0
      via eD2K Server: 0
      via Kad: 4
      via Source Exchange: 8
      via Passive: 0
      eD2K: 12 (100.0%)
      Kad: 12 (100.0%)
      eD2K/Kad: 12 (100.0%)
      UDP File Reasks: 225, Failed: 123 (54.7%)
      Dead Sources: 83 (27 + 56)
   Download Sessions: 1231
      Successful Download Sessions: 1192 (96.8%)
      Failed Download Sessions: 39 (3.2%)
      Average Downloaded Per Session: 177.98 KB
      Average Download Time: 1:18 Minutes
   Gained Due To Compression: 0 Bytes (0.0%)
   Lost Due To Corruption: 170.82 MB (82.4%)
   Parts Saved Due To I.C.H: 0
   Total Overhead (Packets): 4.00 MB (77.62 k)

Server log:

08/10/2009 10:29:04 PM: Obfuscated connection established on: eMule Security (85.10.193.45:4007)
08/10/2009 10:29:04 PM: New client ID is 129929513
08/10/2009 10:43:06 PM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
08/10/2009 11:21:48 PM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
08/10/2009 11:42:11 PM: Downloaded part 8 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 12:10:12 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 12:27:50 AM: Downloaded part 14 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 12:49:01 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 01:25:09 AM: Downloaded part 9 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 01:37:08 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 02:05:02 AM: Downloaded part 13 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 02:28:01 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 03:04:24 AM: Downloaded part 4 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 03:15:44 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 03:50:02 AM: Downloaded part 21 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 04:01:29 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 04:28:46 AM: Downloaded part 12 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 04:49:37 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 05:16:06 AM: Downloaded part 16 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 05:33:04 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 06:04:58 AM: Downloaded part 1 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 06:17:02 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 06:40:06 AM: Downloaded part 6 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 07:03:59 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 07:32:09 AM: Downloaded part 10 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 07:52:28 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 08:16:36 AM: Downloaded part 11 is corrupt :(  (Yet Another Poisoned File.zip)
09/10/2009 08:31:05 AM: Downloaded part 24 is corrupt :(  (Yet Another Poisoned File.zip)

[WentloogWhix]

fox88, on 08 October 2009 - 09:51 PM, said:

Less than 7000 IP filters? You did not follow it properly.

With incorrect setup behaviour of mule could be unpredictable.
Therefore right now there is no clearly visible huge issue.

OK, so we understand one another perfectly. I'm saying that even with the most accurate and up-to-date ipfilter.dat out there, excluding 243622 IP ranges, I doubt if that filter is excluding the ADSL Pool Clusters of the 3 ISPs in Gernamny.

@fox88: Please confirm whether your current ipfilter excludes 84.135.69.193, 85.181.38.247, 92.228.200.119, or not.

You're saying that the "issue" is a non-issue because all you need to do is add those IP ranges to your IPFilter and the problem is solved.

My response is that this is too heavy-handed, because the legitimate users in the ADSL pool are punished (by being blocked) because of the rogue activities of 20 machines. Extending this to its logical extreme, we could simplify ipfilter.dat to read:

0.0.0.1         - 255.255.255.255   ,   0 , The internet is a bad place

Clearly this example is extreme, but I'm trying to illustrate a point.

So my feature request is to add a facility to temporarily block the IP and user ID of a user who is spitting out random corrupt junk for a temporary period, based on the assumption that either he is having technical issues which is corrupting the data, or he is trying to poison the data. I am NOT talking about fake files.

In order to implement this feature the model of trust would need to change. Currently it seems to assume that non-blocked IPs will provide legitimate data, except that sometimes it may get corrupted, and the corrupted stuff will be downloaded, detected and ignored; downloaded, detected and ignored; and downloaded, detected and ignored; ... until eventually the user gets bored or runs out of bandwidth.

I accept that setting up eMule may require a little more that running the install program, but even after I ran the installer on a test machine, and followed the instructions provided, my setup wasn't good enough. That means that at least 80% of eMule users need "a wake up call" because the default settings out of the box don't work.

You may recall that I initially provided the full hashset of the poisoned file so that members of this forum could try it on their own correctly setup systems, but the hashset was deleted and no one has asked for it privately, which means that you don't aren't willing to try to replicate the problem to see if it's an issue.

I guess that I have failed to illustrate the issue, in spite of screen shots, sample logs, hashsets, descriptions of what happened, and so on.

I have reinstalled eMule on the test machine. I await further instructions on which server to connect to and which ipfilter.dat and nodes.dat file to use, and what other settings are required to convince me that we are dealing with a non-issue.

Right now I feel like I've got flu and my doctor is telling me to take two Asprin and call him in the morning. Statistically it isn't likely to be H1N1, even if I have all the symptoms, right?

[Famerlor]

@Wentloog: Which setting did you enter for "CorruptionBlackBox-ban-Threshold" in the "Security Options"-page in Spike2mod ?

Btw: The ipfilter.dat from emulefuture.de is fine of course. We don't see the need to block such wide ranges but only the worst known. That's why this IPfilter just has 6512 filters.

This post has been edited by Famerlor: 09 October 2009 - 01:07 PM