[Tommy Cobalt]

Every time I mount my partitions in other way (I'm doing it quite often) or every time I add few GBs of share to eMule it starts hashing. In some conditions, like slow HD or having one big partition for entire system (yes there are still few ppl doing this) it makes operating the PC a pain in the back. If this is possible, I'd suggest giving eMule option to limit speed of reading and writing to harddisk. Having PC disbled for 6 hours every time I start eMule because of shitty IDE controler on MOBO or/and drivers is not so fun.

1) This FR is not an urgent one
2) I personally see it quite useful.

What do you think about this?

This post has been edited by Tommy Cobalt: 16 November 2004 - 12:17 AM

[MasterJunior]

the way Ares lets you do it huh, sounds good

[Tommy Cobalt]

MasterJunior, on Nov 17 2004, 01:28 AM, said:

the way Ares lets you do it huh, sounds good

What is Ares?

[gcostanza]

Interesting programming problem. Pretty difficult one according to me - how do we measure disk access speed (especially in dynamic conditions as requested), how do we resolve conflicts with the caching system of windows and most importantly - how do we make the hashing slower. Anyone ran a profiler on the hashing sybsystem?

[Tommy Cobalt]

So file hashing and file copying (after download has been completed) is realised by Windows' system routines? I've never seen in any program using these routines any option to slow down disk access rate, so I dunno if it's even possible, despite fact that it would be darn usefull in some cases. For slowing hasing there is of course a way - limit cpu operations that can be made by hashing algorithm in a given time, this should make it.

[slowsilver]

You appear to have a different kind of problem, if your files have to be rehashed for 6 hours every time you start eMule. Hashing files is a one-time procedure.

[CiccioBastardo]

Not if he deletes/corrupts/moves its known.met file.

Slowing the hashing should not be a problem. What about a Sleep(xxx) in the thread doing the hashing? Just a first thought.

/edit:
Maybe an option to make the hashing algorithm work full power (foreground) or in a nice (and maybe even very-nice) setting (background).

So fg setting would be good when one want to publish/use its files in less time possible, while the bg setting would be for those that use eMule silently in the background and could care less if the hashing process for the lastest downloaded 600MB file takes 2 minutes or 15 minutes if it does not interfere with the other interactive/higher priority processes (burning, file copying, code compilation...).
New point to add to my mod to-do list, probably.

This post has been edited by CiccioBastardo: 29 November 2004 - 10:44 PM

[Dune]

This FR is close to me also, I'm doing lots of CD/DVD burning, and huge disk access times that eMule does from time to time (as hashing) really interfearing with burning. Also with today's technology disks are O.K. after multipile stops, this sometimes really unpleasant. Even slowing down 15% of the process will give noticeable results.

Sorry for my English,
and thank you.

[gcostanza]

Would thread priorities really do it in all cases? I mean when I issue a read request for all I know I might be reading off the HDD, the HDD internal buffer, the kernel buffer, some driver's buffer, the windows cache or who knows where else. Not always there will be a spike in HDD activity. Chances are my thread gets another slice again too soon no matter it's user-mode priority. At this point won't it be better to do a controlled read (at some low MB/s) of a VERY large portion of the file and hash that in memory.

[Tommy Cobalt]

Just decreasing priority won't help. Disk accesses aren't held by applications.
As far as I know after reading some literature, I can say, that it would be the best done by disk buffer's stream careful controlling. But it arises security / stability questions. I fear if malicious code could be executed to access from remote disk's data when reading / writing, after implementing somthing that controls whole system's feature into emule.

I'm waiting to see ot in action. Maybe next version.

[bugsan]

I have just the evidence that this option is very important

This is a screenshot of my emule :



You can see a severe decrease of the download rate. At this time, eMule was completing one or two files... and cpu was 100% .. and a lot of connections dropped.

After that, download rate starts to increase again
---------------
Just an idea :
=> we can control the speed with the eMule cpu usage, not the global cpu usage. The best should be the kernel cpu usage ...
1) First we need a system to control the transfer rate, very simple
2) if the process cpu usage is above XX %, decrease the transfer rate, else increase it
start at 1MB/s, every 2 seconds :
< 30% => increase by 2
else <50 % => increase by 1 MB/s
else > 50% => decrease by 1 MB/s
else > 80% => divide by 2

remember, it's just an option for peoples who want it.

This post has been edited by bugsan: 23 January 2005 - 02:09 AM

[asturcon3]

Anyway, launching the hashing as a low priority thread should solve the connection dropping, and it's relativelly easy to do, compared with throttling the hdd access. And sure it's easier than a programmed control of the process CPU usage, which in fact can't be done in w9x/me.

[fabtar]

asturcon3, on Jan 23 2005, 03:28 AM, said:

Anyway, launching the hashing as a low priority thread should solve the connection dropping, and it's relativelly easy to do, compared with throttling the hdd access. And sure it's easier than a programmed control of the process CPU usage, which in fact can't be done in w9x/me.

I'm agree

[gcostanza]

I am pretty disappointed with this low-priority idea especially coming from CB who definitely knows better. It's already bad that emule plays with process priorities. This is totally arbitrary decision especially on NT systems and are we really sure the problems don't derive exactly from that? The fact is the whole IO subsystem in emule is quite inefficient. If only the devs could drop a line on why it works the way it does. The only sensible reason is Win9x compatibility...

How can thread priorities solve anything? They have nothing to do with IO requests. They are dynamic in the 0..15 range with a boost (apart from other reasons) exactly based on IO load. Changing the priority is an ad-hoc thing that may even worsen the situation. So we'll have x% of the time that works and the rest not.

The whole discussion starts to be very similar to some held on the Wine mailing lists not long ago (in wine-time ). On the Linux Kernel I believe the problem (similar) is referred to "reads-starved-by-writes" and the IO scheduler in Linux takes specific measures to prevent that. There are some excellent analyses done by Andrea Arcangeli.


Meanwhile the solution is pretty obvious. AIO on Linux and Completion Ports on NT. If that is not possible and the Sync model needs to stay for some weird reason the control should be at the request level, not at the thread priority. If we need to solve anything let's do it the right way.

[bugsan]

The thread priority is relative...
We could have 2 processus : eMule and eMule_hasher. eMule with a normal priority and eMule_hasher with an idle or lower priority.

eMule_hasher would be used to hash and copy files from Temp to Incoming.
I think in this case it could work.

[gcostanza]

You will starve the IO Manager (the one called by Ntxxx functions - all Ioxxx calls) and in turn emule as well. User-mode solutions will do almost nothing. They might work but again they may not. The thread scheduler itself is unpredictable.

[bugsan]

i understand...
if i'm running virtualdub with a normal priority, the hashing system will be completely stalled.
well, bad.

So... if we can set a limit to the transfer rate, not a dynamic limit, but an user static limit (2-3MB/s), it will solve 90% of the problem, no ?

And hey, 2MB/s it's 120MB/min, ~6min for a 700MB file. It's not so long.
36min for a 4.37GB file, lol but it's nothing compared to the 7 days of download

[CiccioBastardo]

Quote

am pretty disappointed with this low-priority idea especially coming from CB who definitely knows better.

You mean me?
I have never talked about process priorities. Infact I proposed a Sleep() to delay the hashing algorithm (and so the I/O accesses).

[CiccioBastardo]

Quote

am pretty disappointed with this low-priority idea especially coming from CB who definitely knows better.

You mean me?
I have never talked about process priorities. Infact I proposed a Sleep() to delay the hashing algorithm (and so the I/O accesses).

[gcostanza]

Sorry CB - your Sleep() suggestion is not even worth commenting (let's say you were brainstorming )
I was dissapointed with your foreground/background play especially after asturcon3 and fabtar picked up on it. Don't be a bad example