Upload Throttling Performance Improvements support for high speeds on single slot
#21
Posted 13 February 2006 - 09:50 AM
#22
Posted 13 February 2006 - 10:07 AM
#23
Posted 15 February 2006 - 12:51 PM
Any help on this point is apprieciated.
#24
Posted 15 February 2006 - 04:20 PM
- The splitting of the packages before uploading (they are split into 10240 bytes packages). If I disable splitting, I can transfer much faster. With default request size, that means I send 180KBytes packets.
- The disable of the sleep that is suggested by lupzz. There's also an if(timeSinceLastLoop == 0) that sets bytesToSpend = 0. It needs to be changed to set bytesToSpend = realBytesToSpend/1000.
- The buffering from disk. Current standard eMule makes sure there's at least 100 KBytes in the buffer after the disk loading is done (this means one req block at 180Kbytes is what's most often read, since it read entire blocks at the time). Making this buffer dynamic so it can grow for fast sockets improves the transfer speed in a LAN. Slugfiller also have an implementation that requests the buffer to be filled on demand, but I haven't tested that implementation and don't know if it helps with performance. Growing the buffer size theoretically seems good, since it can read more data at the same time, hopefully leveraging cache and read-ahead better.
- Size of req blocks from the remote clients. Normally 3 req blocks at 180 KBytes each is requested. Maximum allowed to be requested from a standard eMule is 3*180KBytes in one req block, otherwise the upload is aborted (IDS_ERR_LARGEREQBLOCK). For really high speeds it would be recommended to request that size. I also tried requested one chunk at a time, and that increased speed even more, but that can't be done from a normal eMule client.
- The socket buffers: These doesn't seem to affect much on a LAN if the above modifications are in. I think socket buffers mostly internet transfers.
My current record in speed is 8.5 MBytes/s on a single socket over my Gigabit LAN. As a comparison, the Windows file sharing client (mount a remote hd) transfers at about 16 MBytes/s over the same LAN.
Over internet the record for a single socket is 80KBytes/s (100% of eMule's allowed bandwidth at the time). I think the larger socket buffers helped with this internet transfer. This transfer was done to a friend in the same country, but not at the same ISP and not in the same town.
/zz
This post has been edited by zz: 15 February 2006 - 04:26 PM
#25
Posted 15 February 2006 - 04:26 PM
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
Dark Lord of the Forum
Morph your Mule
Need a little help with your MorphXT? Click here
#26
Posted 15 February 2006 - 04:27 PM
/zz
#27
Posted 17 February 2006 - 02:20 PM
zz, on Jan 23 2006, 07:49 PM, said:
Have you tested with and without that part, but the other changes in place? Any difference?
I'm experimenting with skipping the sleep call, I'm guessing that and the larger send buffers would have the most effect. But since I have a slower connection I can only regression test them, I can't test that they have any effect on a faster connection.
/zz
sorry for the late reply, i've eventually found an available good tester on a 6mbit dsl (that is quite good to test the upload anyway)
i uploaded to him with my sym 10mbit connection and the real limit is 630kb/s (tested with different programs)
well you were right about the other mods, i've left them there because i started with thos but doensn't help the upload speed, here are my results:
original patch: 610-625kb/s
only with the sleep mod: 630kb/s steady
zz, on Feb 15 2006, 05:20 PM, said:
so i tested also these ones and the results (with only the sleep mod)
zz, on Feb 15 2006, 05:20 PM, said:
580kb/s
zz, on Feb 15 2006, 05:20 PM, said:
correcting bytesToSpend
610kb/s
zz, on Feb 15 2006, 05:20 PM, said:
still 630kb/s steady but i think this one with multiple uploads can help..
zz, on Feb 15 2006, 05:20 PM, said:
don't tested because needed a little more work to test this
zz, on Feb 15 2006, 05:20 PM, said:
i haven't tested because i've already changed there and there is no improvement with bigger buffers when the mods are in (as you've noted too)
btw good work i think the important one is only the sleep as you pointed to me the first time, with that one the speed goes to the top
the disk buffer i think can help when sending to that speed but to test this i need another 10mbit where to reach the full speed is really more difficult than the 6mbit
as always will make you know, and thanks for your tests
#28
Posted 17 February 2006 - 03:19 PM
It's a little strange though that the speed decreased when you stopped setting bytesToSpend = 0. Maybe I have missed something when I reviewed that code. Or possibly it was just random fluctuations of the speed.
As you say, the sleep-change is the most effective part, the other one's are just extras that may not be needed for internet transfers. I'm not sure how often eMule is used at LAN partys, etc, so maybe it's not necessary to optimize for a LAN with low pings and high bw.
/zz
#29
Posted 17 February 2006 - 05:29 PM
zz I think that eMule is used on some university networks. I guess it's pretty common in those alongside with DC. BT doesn't make much sense in those networks since searches are impossible.
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
Dark Lord of the Forum
Morph your Mule
Need a little help with your MorphXT? Click here
#30
Posted 17 February 2006 - 07:48 PM
@zz
i will test more throughly all the mods with another 10mbit
if you can explain what you change better, i will try to that tests too
#31
Posted 18 February 2006 - 03:38 PM
Note 10mb bb should be 1MB as an min (as i seen some one stateing 630KB download on an 10mb line thats 5mb bb ish)
#32
Posted 18 February 2006 - 03:57 PM
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
Dark Lord of the Forum
Morph your Mule
Need a little help with your MorphXT? Click here
#33
Posted 18 February 2006 - 04:02 PM
you be the first emule to ever flat line my download (with emule any way)
This post has been edited by leexgx: 18 February 2006 - 04:04 PM
#34
Posted 02 April 2006 - 11:02 AM
1) the setsockopt's on windows are a must... last time i've tested only amule on linux which without the setsockopt goes better
on windows instead without the send buffer enlargement you get very low performance
this is due to the linux send buffer auto-tuning which on windows isn't done
2) actually on windows the linear scaling of fragment number helps too, on linux not... i think i need more tests of this one
so actually i get the best results on windows only with the very first version of the mod
p.s.: remember the rule-of-thumb to compute the optimal buffer size is: 2 * rtt * capacity
we've about 30ms rtt and 10mbit so: .03 * 10^7 / 8 * 2 / 2^10 = 73k buffer
but with the line congested by the mule uploads and some other os buffering you can get to 100ms so i've found out that the best window size for our lines is: .1 * 10^7 / 8 * 2 / 2^10 = 244k ~= 256k
remember that a bigger window is always a preferable choice because the cwnd (is really reno on windows ? ) already does the "dirty job" of limiting the packets you put on the network
obviously this tuning is a lot line-dependant
#36
Posted 02 April 2006 - 04:48 PM
when you increase the sendbuffer, you only think about the throughput to a single client. You are right, you can increase this throuput. But you have to consider the throuput of all sockets... and there is one danger with a big sendbuffer:
at clients, which are blocking very oftenly, the sendbuffer will be filled up. then it can take up to a few seconds and the whole sendbuffer will be sent. In practice this means, there are times, when only sendbuffers are filled and there are times when many sockets send it's buffer.
Increasing the buffer -> your upload becomes more "zigzag" and is more oftenly saturated or you send too less.