Official eMule-Board: Harddisk Bandwith Limiting - Official eMule-Board

Jump to content


  • (4 Pages)
  • +
  • 1
  • 2
  • 3
  • Last »

Harddisk Bandwith Limiting Rate Topic: -----

#1 User is offline   Tommy Cobalt 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 15-November 04

Posted 15 November 2004 - 10:36 PM

Every time I mount my partitions in other way (I'm doing it quite often) or every time I add few GBs of share to eMule it starts hashing. In some conditions, like slow HD or having one big partition for entire system (yes there are still few ppl doing this) it makes operating the PC a pain in the back. If this is possible, I'd suggest giving eMule option to limit speed of reading and writing to harddisk. Having PC disbled for 6 hours every time I start eMule because of shitty IDE controler on MOBO or/and drivers is not so fun.

1) This FR is not an urgent one
2) I personally see it quite useful.

What do you think about this?

This post has been edited by Tommy Cobalt: 16 November 2004 - 12:17 AM

0

#2 User is offline   MasterJunior 

  • no more then I have to, if that. EndGame
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1313
  • Joined: 01-April 04

Posted 16 November 2004 - 10:28 PM

the way Ares lets you do it huh, sounds good
I'm wasted away, i make a million mistakes, theres a storm in my head, it rains on my bed, when your not here, im not afraid of dyin, but i am afraid of loosing you, maybe im addicted, im outta control, but your the drug that keeps me from dyin, maybe im a liar, but all i really kno, is your the only reason im tryin, when your lying next to me, your love is flowing through to me oh its beautiful, everythings clear to me, untill i hit reality and then i loose it all...... I love you eMule!!!

Bored??? Release your inner mule and get involved with your favorite file sharing program, EMULE!!!!!! Check out my feature request.
Modifications!!!....................Speed Up eMule!!! Partly Done
On Queue 2........................Individual Max Sources/File Partly Done
Advanced Sorting.................Slot for Small Files
Easy Switching.Partly Done....Advanced Priority System
To accomplish great things, we must not only act, but also dream; not only plan, but also believe
0

#3 User is offline   Tommy Cobalt 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 15-November 04

Posted 23 November 2004 - 04:15 PM

MasterJunior, on Nov 17 2004, 01:28 AM, said:

the way Ares lets you do it huh, sounds good
View Post


What is Ares?
0

#4 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 23 November 2004 - 11:12 PM

Interesting programming problem. Pretty difficult one according to me - how do we measure disk access speed (especially in dynamic conditions as requested), how do we resolve conflicts with the caching system of windows and most importantly - how do we make the hashing slower. Anyone ran a profiler on the hashing sybsystem?
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

#5 User is offline   Tommy Cobalt 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 15-November 04

Posted 29 November 2004 - 09:28 PM

So file hashing and file copying (after download has been completed) is realised by Windows' system routines? I've never seen in any program using these routines any option to slow down disk access rate, so I dunno if it's even possible, despite fact that it would be darn usefull in some cases. For slowing hasing there is of course a way - limit cpu operations that can be made by hashing algorithm in a given time, this should make it.
0

#6 User is offline   slowsilver 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2118
  • Joined: 30-September 02

Posted 29 November 2004 - 09:46 PM

You appear to have a different kind of problem, if your files have to be rehashed for 6 hours every time you start eMule. Hashing files is a one-time procedure.
Some files are rare because nobody wants them.

* * *

eMule has enough anti-corruption measures.
-- SF, Oct 30 2005, 07:08 PM
0

#7 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 29 November 2004 - 10:01 PM

Not if he deletes/corrupts/moves its known.met file.

Slowing the hashing should not be a problem. What about a Sleep(xxx) in the thread doing the hashing? Just a first thought.

/edit:
Maybe an option to make the hashing algorithm work full power (foreground) or in a nice (and maybe even very-nice) setting (background).

So fg setting would be good when one want to publish/use its files in less time possible, while the bg setting would be for those that use eMule silently in the background and could care less if the hashing process for the lastest downloaded 600MB file takes 2 minutes or 15 minutes if it does not interfere with the other interactive/higher priority processes (burning, file copying, code compilation...).
New point to add to my mod to-do list, probably.

This post has been edited by CiccioBastardo: 29 November 2004 - 10:44 PM

The problem is not the client, it's the user
0

#8 User is offline   Dune 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 80
  • Joined: 01-August 03

Posted 29 November 2004 - 10:17 PM

This FR is close to me also, I'm doing lots of CD/DVD burning, and huge disk access times that eMule does from time to time (as hashing) really interfearing with burning. Also with today's technology disks are O.K. after multipile stops, this sometimes really unpleasant. Even slowing down 15% of the process will give noticeable results.

Sorry for my English,
and thank you.
0

#9 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 30 November 2004 - 12:09 AM

Would thread priorities really do it in all cases? I mean when I issue a read request for all I know I might be reading off the HDD, the HDD internal buffer, the kernel buffer, some driver's buffer, the windows cache or who knows where else. Not always there will be a spike in HDD activity. Chances are my thread gets another slice again too soon no matter it's user-mode priority. At this point won't it be better to do a controlled read (at some low MB/s) of a VERY large portion of the file and hash that in memory.
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

#10 User is offline   Tommy Cobalt 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 15-November 04

Posted 16 January 2005 - 11:51 PM

Just decreasing priority won't help. Disk accesses aren't held by applications.
As far as I know after reading some literature, I can say, that it would be the best done by disk buffer's stream careful controlling. But it arises security / stability questions. I fear if malicious code could be executed to access from remote disk's data when reading / writing, after implementing somthing that controls whole system's feature into emule.

I'm waiting to see ot in action. Maybe next version.
0

#11 User is offline   bugsan 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 92
  • Joined: 02-November 02

Posted 23 January 2005 - 01:50 AM

I have just the evidence that this option is very important

This is a screenshot of my emule :

Posted Image

You can see a severe decrease of the download rate. At this time, eMule was completing one or two files... and cpu was 100% .. and a lot of connections dropped.

After that, download rate starts to increase again :)
---------------
Just an idea :
=> we can control the speed with the eMule cpu usage, not the global cpu usage. The best should be the kernel cpu usage ...
1) First we need a system to control the transfer rate, very simple :)
2) if the process cpu usage is above XX %, decrease the transfer rate, else increase it :)
start at 1MB/s, every 2 seconds :
< 30% => increase by 2
else <50 % => increase by 1 MB/s
else > 50% => decrease by 1 MB/s
else > 80% => divide by 2

remember, it's just an option for peoples who want it.

This post has been edited by bugsan: 23 January 2005 - 02:09 AM

0

#12 User is offline   asturcon3 

  • Miembro con bola
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 7487
  • Joined: 26-July 04

Posted 23 January 2005 - 03:28 AM

Anyway, launching the hashing as a low priority thread should solve the connection dropping, and it's relativelly easy to do, compared with throttling the hdd access. And sure it's easier than a programmed control of the process CPU usage, which in fact can't be done in w9x/me.
Imagen enviada Imagen enviada Imagen enviada
0

#13 User is offline   fabtar 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 880
  • Joined: 14-March 04

Posted 23 January 2005 - 03:04 PM

asturcon3, on Jan 23 2005, 03:28 AM, said:

Anyway, launching the hashing as a low priority thread should solve the connection dropping, and it's relativelly easy to do, compared with throttling the hdd access. And sure it's easier than a programmed control of the process CPU usage, which in fact can't be done in w9x/me.
View Post


I'm agree
0

#14 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 23 January 2005 - 03:39 PM

I am pretty disappointed with this low-priority idea especially coming from CB who definitely knows better. It's already bad that emule plays with process priorities. This is totally arbitrary decision especially on NT systems and are we really sure the problems don't derive exactly from that? The fact is the whole IO subsystem in emule is quite inefficient. If only the devs could drop a line on why it works the way it does. The only sensible reason is Win9x compatibility...

How can thread priorities solve anything? They have nothing to do with IO requests. They are dynamic in the 0..15 range with a boost (apart from other reasons) exactly based on IO load. Changing the priority is an ad-hoc thing that may even worsen the situation. So we'll have x% of the time that works and the rest not.

The whole discussion starts to be very similar to some held on the Wine mailing lists not long ago (in wine-time :)). On the Linux Kernel I believe the problem (similar) is referred to "reads-starved-by-writes" and the IO scheduler in Linux takes specific measures to prevent that. There are some excellent analyses done by Andrea Arcangeli.


Meanwhile the solution is pretty obvious. AIO on Linux and Completion Ports on NT. If that is not possible and the Sync model needs to stay for some weird reason the control should be at the request level, not at the thread priority. If we need to solve anything let's do it the right way.
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

#15 User is offline   bugsan 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 92
  • Joined: 02-November 02

Posted 23 January 2005 - 04:18 PM

The thread priority is relative...
We could have 2 processus : eMule and eMule_hasher. eMule with a normal priority and eMule_hasher with an idle or lower priority.

eMule_hasher would be used to hash and copy files from Temp to Incoming.
I think in this case it could work.
0

#16 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 23 January 2005 - 04:27 PM

You will starve the IO Manager (the one called by Ntxxx functions - all Ioxxx calls) and in turn emule as well. User-mode solutions will do almost nothing. They might work but again they may not. The thread scheduler itself is unpredictable.
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

#17 User is offline   bugsan 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 92
  • Joined: 02-November 02

Posted 23 January 2005 - 04:48 PM

i understand...
if i'm running virtualdub with a normal priority, the hashing system will be completely stalled.
well, bad.

So... if we can set a limit to the transfer rate, not a dynamic limit, but an user static limit (2-3MB/s), it will solve 90% of the problem, no ?

And hey, 2MB/s it's 120MB/min, ~6min for a 700MB file. It's not so long.
36min for a 4.37GB file, lol but it's nothing compared to the 7 days of download :D
0

#18 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 23 January 2005 - 08:38 PM

Quote

am pretty disappointed with this low-priority idea especially coming from CB who definitely knows better.

:huh: You mean me? :confused:
I have never talked about process priorities. Infact I proposed a Sleep() to delay the hashing algorithm (and so the I/O accesses).
The problem is not the client, it's the user
0

#19 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 23 January 2005 - 08:45 PM

Quote

am pretty disappointed with this low-priority idea especially coming from CB who definitely knows better.

:huh: You mean me? :confused:
I have never talked about process priorities. Infact I proposed a Sleep() to delay the hashing algorithm (and so the I/O accesses).
The problem is not the client, it's the user
0

#20 User is offline   gcostanza 

  • Philosopher
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 805
  • Joined: 10-October 04

Posted 23 January 2005 - 09:32 PM

Sorry CB - your Sleep() suggestion is not even worth commenting (let's say you were brainstorming :-k ) :P
I was dissapointed with your foreground/background play especially after asturcon3 and fabtar picked up on it. Don't be a bad example :respect: :flowers:
"Computer Science is no more about computers than astronomy is about telescopes."
-- E. W. Dijkstra
"Computers are useless. They can only give you answers."
-- Pablo Picasso
0

  • Member Options

  • (4 Pages)
  • +
  • 1
  • 2
  • 3
  • Last »

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users