Official eMule-Board: Throttling - Workaround Ideas - Official eMule-Board

Jump to content


Page 1 of 1

Throttling - Workaround Ideas

#1 User is offline   Reven 

  • Member
  • PipPip
  • Group: Members
  • Posts: 30
  • Joined: 02-December 02

Posted 19 December 2005 - 02:03 AM

My ISP (Shaw in Edmonton, Canada) has suddenly started severely restricting eMule transfers. My ports 4662 and 4672 are both being blocked for incoming connections. Outgoing connections seem to be severely throttled. Connections are restricted to about 1.5 kb/s. One big problem is that the throttling has a coarse time resolution. I can upload at 4+kb's for a second or two, then the throttling occcurs for 5-10 seconds before packets can flow again. eMule most often decides the connection has lagged, and closes it.

I don't know how widespread this file-sharing-antagonistic behavior is on the part of ISPs. From what I can tell it seems to be growing. My gut feeling is that it will get worse before it gets better.

I would like to suggest that an encryption mechanism be put in place to prevent ISPs from determining whether a connection is indeed eMule-related. I propose a two step approach.

Step One - Simple Symmetric Encryption
The first step is a very simple symmetric-only encryption. Once a connection is opened to a peer, all data is encrypted using the remote host's ID as a key. If the remote host doesn't understand the packet, then eMule can fall back to an unencrypted connection. This will defeat current throttling technology that depends on sniffing the connection for protocol data. This is not a long-term solution, however, as ISPs will be able to break this encryption fairly easily as most IDs are simply a manipulation of the end user's IP address. It will take some time before the ISPs would have the infrastructure to be able to accomplish this. Protocol identification is currently based on analyzing the first packet or two on a connection for certain byte sequences. For ISPs to adopt protocol analyzing that utilizes decryption is something that would take quite a while and could cost them a significant amount of money. Some ISPs could give up on throttling at this point as being too costly for the gain.
PROs:
  • Can be implemented almost immediately - simple changes
  • No changes to eMule servers are needed
CONs:
  • ISPs that are determined to "enforce" throttling restrictions can adapt

Step Two - Public Key Infrastructure
If ISPs do start adapting, then the second step is to use a simple public-key infrastructure. Each eMule client would generate a public/private key pair (or use their existing secure identification keys). Each eMule client then operates a ftp or http server on an open port. This server would exist only to serve up that client's public key. When one client is going to upload to another, it first downloads that client's public key. Then, a symmetric key is negotiated when the actual file transfer connection takes place. The negotiation has to be done with no fixed bytes in the packet - nothing that a protocol sniffer can identify. The first 128 bits of the first packet sent is the PK-encrypted symmetric key. The rest of that packet is encrypted with the symmetric key so that the entire contents of the packet are encrypted.
PROs:
  • Unfeasable for ISPs to adapt
CONs:
  • Requires larger changes to eMule
  • As presented above, cannot be used with firewalled clients

None of the ideas presented here will prevent an ISP from knowing you are running eMule. The only thing this will do is prevent an ISP from easily being able to determine what connections are associated with eMule. It is still possible that upon detecting eMule from your IP that an ISP could throttle your entire connection. My belief is that this is a step that ISPs wouldn't take - it's one thing to throttle one program's data, another thing to throttle the entire connection. But, it is still a possibility, and one that is not easily countered. I believe, though, that if the above is done, it will help send a message to ISPs in general, that users demand protocol/data neutral service.
0

#2 User is offline   niclights 

  • lost in space
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 10288
  • Joined: 01-November 04

Posted 19 December 2005 - 02:22 AM

Interesting and well thought through. The finer points are beyond my knowledge, but I understand the principal.
I do hope this envokes a serious discussion on the subject since this is clearly a real issue now and only a matter of time before it affects more of us, assuming ISP's decide they must restrict for 'quality of service' etc.
People keep going on about filehashes and whether they will be 'broken' etc. This is highly unlikely, but the throttling technology is already here and is working.

I agree with your final comments. The idea of throttling everything is not something an ISP would like to do, much as ultimately terminating a users contract. P2P is perfectly valid and does not have to be an illegal practice. I would rather ISP's stopped promoting higher and higher speeds and instead invested in the server architecture to back up the lower speeds first! What is the point in 2000001Mbps when you can only utilise it for a short time?
0

#3 User is offline   leuk_he 

  • MorphXT team.
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5975
  • Joined: 11-August 04

Posted 19 December 2005 - 02:53 PM

1. Read the sticky in feature requests. (in short: this will not happen) (/edit: and do a search in feature request, this has heen request numourous times)
2. The mule (or any p2p application) can still be detected by traffic analyzing. p2p application connect to lots of different ip's with a lot of traffic.

The best solution (if possible) is to pick an other ISP that allows you to do with your connection what you want. Hurt your stupid throttling isp in the wallet!


by the way: http encapsulation might be useful fo those behind a proxy. SO that might happen.
You can already run your mule on port 80. (I already do this to help webcache users behind a transparant proxy) It helps agains dump traffic shapers that only look to the port number.

This post has been edited by leuk_he: 19 December 2005 - 02:55 PM

Download the MorphXT emule mod here: eMule Morph mod

Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.

Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
0

#4 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 19 December 2005 - 06:29 PM

And I renew my request - mark emule traffic as "bulk" so ISPs can put it on low priority and don't have to resort to blocking it altogether (if they are in good faith).

I don't think playing cat and mouse with ISPs is a good idea, despite the fact that an encrypted protocol will be interesting to see. Misunderstanding ISPs problems, on the other hand, is a grave sin. There is a forum (always forget the name) where many ISPs' employees post regularly and discuss hands-on experience on filtering technology. One of the top concerns is the sheer number of connections P2P makes. Just imagine a Wi-Fi hotspot with several P2P-ers around - really gives you an idea why it was called a "hot" spot.
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

#5 User is offline   Reven 

  • Member
  • PipPip
  • Group: Members
  • Posts: 30
  • Joined: 02-December 02

Posted 19 December 2005 - 09:28 PM

leuk_he, on Dec 19 2005, 09:53 PM, said:

1. Read the sticky in feature requests. (in short: this will not happen) (/edit: and do a search in feature request, this has heen request numourous times)

I read that FAQ. The entry in encryption in that post speaks to encryption as a way of securing a connection with respect to content, not as a way to defeat traffic analysis. Most people crying for encryption want it to save them from prosecution for copyright violations. I proposed encryption, yes, but for a completely different reason, so I don't think that FAQ applies.

I agree that there is little that can be done to make any P2P program actually secure - who knows who's on the other side? There is, however, much that can be done to hide the purpose of any single connection with respect to throttling. The encryption I propose is easy to implement, requires no changes to ED2K servers, and very cheap in terms of CPU time.

leuk_he, on Dec 19 2005, 09:53 PM, said:

2. The mule (or any p2p application) can still be detected by traffic analyzing.  p2p application connect to lots of different ip's with a lot of traffic.

As I mentioned in my original post, my suggestions will not hide the fact that you are running eMule. This was not the intention in the first place. The intention is to make it unfeasable for an ISP to know what individual TCP connections from your machine to another machine are eMule transfer connections.

leuk_he, on Dec 19 2005, 09:53 PM, said:

The best solution (if possible) is to pick an other ISP that allows you to do with your connection what you want. Hurt your stupid throttling isp in the wallet!

I agree - however the problem is, where I live, there is very little choice of broadband providers. Canada is great for having been one of the first adopters of broadband. The down side here is that in any particular area there is generally only one or two providers.

ISPs adopt this technology because they can get away with it. We need to keep them from getting away with it.

leuk_he, on Dec 19 2005, 09:53 PM, said:

by the way: http encapsulation might be useful fo those behind a proxy. SO that might happen.

That is also a good idea, but even more easily defeatable than my step one and much more difficult to implement. Within the http transfer will be emule protocol bytes. All the traffic shaping has to do is look at a different offset inside the first packet or two to detect an eMule connection. This is easily done with existing connection shaping software/firmware in routers. Attempting to decrypt a packet using a mangled destination IP as the key so as to detect protocol-identifying bytes is not within the capability of existing connection shaping software/firmware.
0

#6 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 19 December 2005 - 09:56 PM

gcostanza, on Dec 19 2005, 08:29 PM, said:

And I renew my request - mark emule traffic as "bulk" so ISPs can put it on low priority and don't have to resort to blocking it altogether (if they are in good faith).

I don't think playing cat and mouse with ISPs is a good idea, despite the fact that an encrypted protocol will be interesting to see. Misunderstanding ISPs problems, on the other hand, is a grave sin. There is a forum (always forget the name) where many ISPs' employees post regularly and discuss hands-on experience on filtering technology. One of the top concerns is the sheer number of connections P2P makes. Just imagine a Wi-Fi hotspot with several P2P-ers around - really gives you an idea why it was called a "hot" spot.
View Post
Would be nice to hear the ISP engineerers view of the problem. Maybe some of them could be solved without breaking the fundamental idea of P2P networking. However the likely issue is the number of connections created that is the problem. Preferably we should only ask that many sources as we need download slots, but that would require some major redesign of how slots are given and how sources are choosen. The client would need to know on forehand which sources that will be the fastest. Very complicated.

The "bulk" traffic marking is something I have wanted to implement for a long time but I have only found it possible to do on Ethernet level. On the IP level, packets always seem to be tagged with the lowest priority by default which makes it quite useless. Besides M$s specification how to set these tags is incredible hard to understand, which I guess is the reason why I never seen any product use this feature. However according to some manufacturers of cable modems they do support QOS settings to improve quality for VoIP and similar services. Unfortunatly I never saw any clear specification, but it might just be a simple multi queue that is always enabled.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#7 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 20 December 2005 - 01:39 AM

You are right - there is no easy way to achive packet marking under NT. The easy way would have been setsockopt() with IP_TOS parameter but support for that is crappy) at best. Even if you edit the registry... Yet at least two major P2P clients use this approach - official bittorrent and Azureus. To my knowledge there is absolutely no evidence that this will work for tcp - Microsoft do not explicitely say so unlike for UDP and ICMP - even though, I figure, the other devs must have tested it or they would drop a note.

I guess using GQoS is the way and it is quite a jungle there between admin rights, need for overlapped sockets on 2000 but not on XP, rsvp signalling not supported on XP and above. I can't make heads or tails but I'd like to one day...
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

#8 User is offline   VcdGuy 

  • Member
  • PipPip
  • Group: Members
  • Posts: 48
  • Joined: 05-June 03

Posted 02 January 2006 - 12:42 PM

My ISP is now throttling Emule activity also.
I've been using Emule and this ISP for about three years, and have had no problem until now. Emule has worked perfectly all that time.

They started out by blocking the standard Emule ports. My Emule suddenly went to LowID. I changed to other ports, and HighID returned. A short time later, I noticed that Emule's speed would be slow most of the time, but return to its normal fast speed during the night. I performed a number of online speed tests and noticed that the throughput of my connection seemed to fluctuate a lot, and was often much lower than it should be. I concluded that the ISP was upgrading or configuring equipment, and waited.

Sometime later, I noticed that other types of internet access seemed OK and fast, even when Emule was running slowly. Online speed tests confirmed this. I checked the standard Emule ports again, and they were now unblocked again. I tried many different ports, but none of them change the speed of Emule. It runs perfectly fine at night, for about 8 hours, always at the same time of day. During the rest of the time, they've restricted the bandwidth, and Emule runs pathetically slowly.

I of course don't like this either, and would like to find a way to remove this restriction. But is it possible ? Earlier in this thread was talk about encrypting bytes so that the ISP wouldn't know it's Emule. In regard to this, I wish to ask a stupid question. Wouldn't the ISP then still easily be able to defeat this by throttling all encrypted data ? Do other apps encrypt data in this way that the ISP would NOT want to throttle ? If Emule is the only thing encrypted, then the disguise is useless.
0

#9 User is offline   leuk_he 

  • MorphXT team.
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5975
  • Joined: 11-August 04

Posted 02 January 2006 - 02:13 PM

VcdGuy, on Jan 2 2006, 01:42 PM, said:

Wouldn't the ISP then still easily be able to defeat this by throttling all encrypted data ?  Do other apps encrypt data in this way that the ISP would NOT want to throttle ?  If Emule is the only thing encrypted, then the disguise is useless.
View Post



No, since encrypting or hiding the data would them have to throtle all traffic which defeat trafic shaping. (Well, they just could buy less bandthwith then to get the same effect.

But they can still detect p2p application by behaviour: many connections to many sources.
Download the MorphXT emule mod here: eMule Morph mod

Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.

Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
0

#10 User is offline   VcdGuy 

  • Member
  • PipPip
  • Group: Members
  • Posts: 48
  • Joined: 05-June 03

Posted 02 January 2006 - 02:58 PM

leuk_he, on Jan 2 2006, 02:13 PM, said:

No, since encrypting or hiding the data would them have to throtle all traffic which defeat trafic shaping.
View Post


I don't think you understood what I meant.

If their packet sniffing software can inspect all packets, and separate them into groups. Those groups with known contents get throttled or not, according to the ISP's desires and needs. Those packets which are encrypted, and therefore impossible to determine what kind of program is sending it, ALL get throttled. In this way, all known kinds of data that does not "harm" their network throughput will be passed along with NO throttling, while Emule (even though encrypted) will still be throttled. All traffic will NOT be throttled.

My question is : Will this be enough to render ineffective any attempt to bypass the throttling in this way ? Second question : Will this also throttle other kinds of data that the ISP does not want to throttle (do other apps use encryption in this way) ?
0

#11 User is offline   niclights 

  • lost in space
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 10288
  • Joined: 01-November 04

Posted 02 January 2006 - 03:06 PM

Just an offtopic. @VcdGuy: What is name of ISP - useful for support :)

I don't have answers to questions, but I'm glad this has started some sort of discussion. This is something that cannot be ignored. I think it is inevitable it will affect all of us sooner or later. GCostanza's post regarding multiple connections was enlightening. I had never considered this.
0

#12 User is offline   viola_2003 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 10
  • Joined: 09-September 04

Posted 04 January 2006 - 10:40 PM

...we are unfortunately having the same problem here in Italy... I do not know about the whole lot of provider we have here... but Libero/Infostrada is just doing what you all were talking about in this post. <_<
I can get a decent download speed only 8 hours a day... mostly during the night... even if I am using Azures... I know they can track down the number of connections and limit them... but I do not think is quite fair when you are paying for a 24 hours service... :argue:
... and I cannot even argue that I cannot do something illegal (meaning p2p protocols)... :confused:

Maybe we can just exchange whole slots of files beetween selected users so to limit the number of connections...
The upload capacity is not affected anyway... so what it takes 8 hours to download... it just goes back during the rest of the time! :-k

Hope this may be really sorted out... :unsure:
0

  • Member Options

Page 1 of 1

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users