[James R. Bath]

Update: Since there's a lot going on in this thread and the limit/ban is an ongoing work, the currently recommended limit/ban is this:
IF Count(Req blocks not yet completed) > 4 AND Max(Transferred) < 540, THEN ban
The more ideal fix if ever implemented. Be sure to see this post to understand the context.

There is a problem in eMule where it allows an infinite repetition of:

remote client decided to stop/complete the transfer (got OP_OutOfPartReqs)

After uploading about only 200KB. The file hashing is circumvented and so is any banning or anything like:

Increased set m_fFailedFileIdReqs to 1

These short uploads need to be tracked and limited and the clients banned if they exceed a certain point. In testing this, I had clients upload over 60MB of data that was later found corrupt. They never got banned. I'm thinking the equivalent of a chunk or a certain number of the "decided to stop"s with "Req blocks not yet completed" to institute first a temporary ban and then a session ban if it continues after the first ban.

It goes in cycles like this

date time: Download session started. User: ip 'user name' (eMule v0.49c,OnQueue/None/None) in SetDownloadState(). New State: 0
date time: Download session ended: The remote client decided to stop/complete the transfer (got OP_OutOfPartReqs). User: ip 'user name' (eMule v0.49c,Downloading/None/None) in SetDownloadState(). New State: 1, Length: 51 secs, Payload: 212.29 KB, Transferred: 212.29 KB, Req blocks not yet completed: 2.

The client does this roughly every 3 minutes and eMule just keeps on swallowing. The amount transferred increases, the completed portion of the file jumps around but never enough to be displayed under Shared Files (so at least something in the Shared Files program loop is identifying there's something wrong). Note the 61.66 MB below and I have a verbose log for anyone who would like to see much more of the same repeated:

date time: Download session ended: The remote client decided to stop/complete the transfer (got OP_OutOfPartReqs). User: ip 'user name' (eMule v0.49c,Downloading/None/None) in SetDownloadState(). New State: 1, Length: 30 secs, Payload: 61.66 MB, Transferred: 152.88 KB, Req blocks not yet completed: 2.

I've seen these stops work on both files that are entirely fake and good files where the amount transferred ends up being over 2 times and a few hundred MB more than the file size. Since testing had to involve copyrighted material, because someone apparently trying to discourage its sharing was generating the corrupt data, I can't post details here, but I'd be happy to send specific hashes and logs privately.

This post has been edited by James R. Bath: 20 October 2009 - 04:46 PM

[tHeWiZaRdOfDoS]

This might happen with official mods (e.g. ZZUL/Morph) a lot because they try to "minimize" the open slots and put you back on queue if your slot isn't actually needed... I don't think it's such a big issue, though annoying...

[Enig123]

Most of those situations actually, are caused by a bad mod known as Thunder or Xunlei. They can share yet not finished chunk and when there's nothing to transfer, it decided to stop.

Don't think zzul is the cause due to there's not too much of zzul mods around and in the meantime these kind of logs happen quite frequently. If Thunder (Xunlei) been banned, those logs barely appears anymore.

This post has been edited by Enig123: 12 October 2009 - 08:27 AM

[fox88]

Evidently this is a client which was designed to pollute eMule's network; it does not download at all but pretends having a full source.

tHeWiZaRdOfDoS, on 12 October 2009 - 11:15 AM, said:

This might happen with official mods

Even more probable with poorly configured mods. I remember one releaser who could've been banned for very short upload sessions.

[tHeWiZaRdOfDoS]

There is no ban for short upload sessions - if so, you are using a bad/erroneous client.
Again: you cannot force someone to upload a certain amount to you and you should be glad to get something at all
I know that some clients try to pollute files though I still don't get how they do it (AICH doesn't jump in), though those clients transfer some BYTES only, not kB!

[Andu]

The point is that AICH only jumps in once ICH does. And that means that you have to dl the chunk first then it gets to recover data. And if the malicious clients send bad data for every block in a chunk AICH won't be able to salvage jack.

[James R. Bath]

fox88, on 12 October 2009 - 07:51 PM, said:

Evidently this is a client which was designed to pollute eMule's network; it does not download at all but pretends having a full source.

That's more or less precisely what's happening. They may actually have a full source, but they never (or very rarely) upload anything valid, and they do it on files that are real/valid/not-fake too. The result is that you need to DL 2-3 times the file size to complete real files, and the fakes never complete.

tHeWiZaRdOfDoS, Enig123:
This has nothing to do with Xunlei (no Thunder in my logs), which is treated as a leecher and blocked by the mod I use (Morph 11.3). The problem here is not leeching. The clients identify as valid eMule v0.49c and are sending corrupt data on files they claim to have complete/100%.

[fox88]

tHeWiZaRdOfDoS, on 13 October 2009 - 10:01 AM, said:

There is no ban for short upload sessions

I know. That was a joke, if you did not get it.

[Enig123]

James R. Bath, on 14 October 2009 - 02:59 AM, said:

tHeWiZaRdOfDoS, Enig123:
This has nothing to do with Xunlei (no Thunder in my logs), which is treated as a leecher and blocked by the mod I use (Morph 11.3). The problem here is not leeching. The clients identify as valid eMule v0.49c and are sending corrupt data on files they claim to have complete/100%.

In that case, an ipfilter list may help a lot more than Limits/bans.

[James R. Bath]

Enig123, on 14 October 2009 - 03:40 AM, said:

In that case, an ipfilter list may help a lot more than Limits/bans.

A limit on the number of "Req blocks not yet completed" (3 or 4) a client can have during a session, followed by a session ban, would actually be a better solution if the specific hole that's being exploited can't be fixed. Here's why:

Ipfilter (preferably a firewall that incorporates it with other block lists) help over the very short term (as in days). The IPs aren't fixed. They originate out of ISPs and shift. The only close to foolproof way, for the moment, is to ban entire ISP ranges. There are probably some good sharers in those ranges that wouldn't appreciate that.

I'm adding the IPs over at bluetack and am about to update what I've already submitted. The list has nearly tripled since. Despite having up-to-date block lists (both before and during my test), including the badpeers list, I've had to manually add the IPs to my firewall.

Last, the stats show that while there isn't a widespread problem - yet - unless you do happen to want a file being corrupted:
Stats:
Prior to test:
Average "Req blocks not yet completed" per 24 hours: 48
During test of corrupt/fake files:
Average "Req blocks not yet completed" per 24 hours: 44,000

After IP blocks added: 300

My test is essentially complete at this point and the IPs will probably not be further updated, so a week++ from now as different files are corrupted and as the exploit potentially expands into leechers using it to accrue credit points and clients re-sharing corrupt data they've received, I'd want eMule putting a limit & ban on it.

[fox88]

James R. Bath, on 14 October 2009 - 10:18 PM, said:

A limit on the number of "Req blocks not yet completed" (3 or 4) a client can have during a session, followed by a session ban, would actually be a better solution if the specific hole that's being exploited can't be fixed.

There could be other reasons for termination of an upload session. Also see my post above about poorly configured client. It was a real life example - there were many upload sessions, but I got only a handful of megabytes from a releaser of 200+MB file.
I think more sound algorithm is needed for banning; especially because eMule is open source.

James R. Bath, on 14 October 2009 - 10:18 PM, said:

Ipfilter (preferably a firewall that incorporates it with

It's up to you, but I don't think you need to block IPs with filter in a firewall.
By the way, your link to bluetack requires registration to read the post.

[James R. Bath]

Registration at Bluetack is free and quick. All the post did was list current IPs I'm blocking that abuse this "feature" of eMule for poorly configured clients. I do see what you're talking about in my logs, but it's comparatively rare. A tweak in the banning algorithm to fix it would be to add a Transferred limit. If it transfers < 500 per connection and Req blocks not yet completed > 4, ban. I don't see any cases in my logs where setting a Req blocks not yet completed limit to 4 would hurt any good clients provided there's an exception for the You cancelled the download cases.

Limiting it to just the following cases would also stop most of corrupt transfers:

Download session ended: The remote client decided to stop/complete the transfer (got OP_OutOfPartReqs).

But there are also a few cases of:

Download session ended: Disconnected: CClientReqSocket::Disconnect(): Close User:

And I haven't seen any indication that eMule makes special exceptions for the type of disconnection, so it appears the corrupt senders could disconnect however they wish, such as via timemout, then starting to upload again after the timeout limit.

[fox88]

James R. Bath, on 15 October 2009 - 09:00 PM, said:

All the post did was list current IPs I'm blocking that abuse this "feature" of eMule for poorly configured clients.

The IPs in question have nothing to do with poorly configured clients.

James R. Bath, on 15 October 2009 - 09:00 PM, said:

A tweak in the banning algorithm to fix it would be to add a Transferred limit. If it transfers < 500 per connection and Req blocks not yet completed > 4, ban.

If you want to discuss algorithm: if I'm not mistaken eMule asks for 3x180KByte parts. Therefore condition "> 4" is useless for one upload session; and it would not take much effort to modify malicios clients so that upload would be slightly above 500KByte.

[jestheonlyone]

hi

James R. Bath, on 15 October 2009 - 07:00 PM, said:

Limiting it to just the following cases would also stop most of corrupt transfers:

Download session ended: The remote client decided to stop/complete the transfer (got OP_OutOfPartReqs).

Yes, obviously.
But this would also stop most transfers...

Even if "Req blocks not yet completed" is also taken into account, BTW.

Please look more carefully at your logs: Many (if not most of) DL sessions end with both "decided to stop" AND "Req blocks not yet completed > 0"...

This post has been edited by jestheonlyone: 16 October 2009 - 01:59 AM

[WentloogWhix]

Andu, on 13 October 2009 - 09:39 AM, said:

The point is that AICH only jumps in once ICH does. And that means that you have to dl the chunk first then it gets to recover data. And if the malicious clients send bad data for every block in a chunk AICH won't be able to salvage jack.

Why then does the eMule client continue to trust the sources of the polluted data? Surely at some point it should realise that everything being downloaded is junk? Ite seems to me that the error recovery model is incomplete.

[James R. Bath]

fox88, on 15 October 2009 - 06:45 PM, said:

If you want to discuss algorithm: if I'm not mistaken eMule asks for 3x180KByte parts. Therefore condition "> 4" is useless for one upload session; and it would not take much effort to modify malicios clients so that upload would be slightly above 500KByte.

I think you need to reread my opening message under this topic, and again statistics later on, which made it very clear I'm not talking about the incomplete "Req blocks" for a given download session. To make this very clear, I included a list and count in another topic that also, I thought, made it clear that incomplete "Req blocks" is not the issue, but the infinite repetition of them.

This post has been edited by James R. Bath: 17 October 2009 - 02:35 AM

[James R. Bath]

jestheonlyone, on 16 October 2009 - 02:58 AM, said:

James R. Bath, on 15 October 2009 - 07:00 PM, said:

Limiting it to just the following cases would also stop most of corrupt transfers:

Download session ended: The remote client decided to stop/complete the transfer (got OP_OutOfPartReqs).

Yes, obviously.
But this would also stop most transfers...

Even if "Req blocks not yet completed" is also taken into account, BTW.

No, they don't. Look at the statistics I posted. These also apply to the "client decided to stop/complete the transfer" or "OP_OutOfPartReqs". OP_OutOfPartReqs is even rarer than "Req blocks". To compare, you need to have a log file where you download nothing but valid files. For that matter, most fakes don't have this kind of problem, which is repeatedly uploading incomplete blocks.

[James R. Bath]

tHeWiZaRdOfDoS, on 13 October 2009 - 07:01 AM, said:

There is no ban for short upload sessions - if so, you are using a bad/erroneous client.

I think you might misunderstand what's going on here. Hopefully this count and excerpt from the verbose log makes it very clear.

tHeWiZaRdOfDoS, on 13 October 2009 - 07:01 AM, said:

I know that some clients try to pollute files though I still don't get how they do it (AICH doesn't jump in), though those clients transfer some BYTES only, not kB!

This convinces me you're thinking of something else. We are talking about KB. KB that never exceeds 500 and is mostly around 200, but KB just the same. And they add up to MB and GB in incomplete payload when clients decide to exploit it repeatedly. The user constantly watching and blocking IPs or recognizing the huge disproportion between transferred and completed are the only mechanisms of protection, and neither is realistic if not completely defeating of the idea of a user-friendly, content-driven p2p client.

[fox88]

James R. Bath, on 17 October 2009 - 06:34 AM, said:

I think you need to reread my opening message under this topic, and [url="http://forum.emule-project.net/index.php?showtopic=147161&view=findpost&p=1033531"]again statistics later on

I already questioned the need to split discussion of the same problem into multiple threads. Now what you have to do is to make cross references in both topics - very useful and helpful feature.

James R. Bath, on 17 October 2009 - 06:34 AM, said:

which made it very clear I'm not talking about the incomplete "Req blocks" for a given download session

James R. Bath, on 14 October 2009 - 10:18 PM, said:

A limit on the number of "Req blocks not yet completed" (3 or 4) a client can have during a session, followed by a session ban, would actually be a better solution if the specific hole that's being exploited can't be fixed.

I suspect you used the word 'session' more than in one meaning.
It might be advisable to use clear and consitent terms, especially when describing something called algorithm.

[WentloogWhix]

fox88, on 18 October 2009 - 06:05 PM, said:

It might be advisable to use clear and consitent terms, especially when describing something called algorithm.

It might be advisable to contribute to the discussion, rather than sitting on the sidelines and firing cheap shots.

James is trying to point out that the corruption handling system is broken, and you seem to be more concerned about how many threads we should have. I'm not trying to be personal, but you really need to contribute more than you criticise.

The corruption handling system is broken because it was never assumed that someone would deliberately mislead other clients, and that file corruption was only ever inadvertent. Today this is not the case. There are dozens of machines not excluded by the IPFilter that are deliberately and maliciously exploiting the weaknesses of the CH process, and the couch potatoes in this forum are denying that it is going on. Denialism never solved any technical problems.

This post has been edited by WentloogWhix: 18 October 2009 - 06:24 PM