Official eMule-Board: Cached Read And Writes - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Cached Read And Writes Rate Topic: -----

#1 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 06 March 2021 - 06:30 PM

Hi Fox88

I've got 32 GB of memory and a Raid 5 HDD disk array. Still eMule seems to be able to saturate the disks easily. The memory and memory cache seems to be barely used by eMule.
Is it possible eMule starts using cached read and writes using OS cache?
The eventual file check on completion of the download should be uncached. But nearly everything else could be cached in my opinion.
0

#2 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 06 March 2021 - 10:07 PM

Which eMule version are you using? Disk IO has been reworked after the latest official version (0.50a) which can be found either in the latest beta or more accessable in the community version.

But that aside, caching is done by the OS (or even the drive firmware, depending on which cache level) and transparent for user level applications. Meaning eMule has no influence on disk cache usage. What eMule could do is keep more data in the ram memory and writing it to the disk in certain intervals only (which of course it already does to a certain degree as not every received bit is written right away), but this has a lot more downsides than upsides and isn't good practice. Even more so with SSDs being pretty common those days, which don't have any problems with high random read/write load.

#3 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4973
  • Joined: 13-May 07

Posted 06 March 2021 - 10:36 PM

Are we talking about disk reads (uploading) or disk writes (downloading)?
0

#4 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 07 March 2021 - 01:53 AM

@Some Support I'm using 0.60b
I'm talking about the standby cache for reads and the modified cache for writes. A program can instruct the OS to either perform a normal read or write (which uses the OS cache) or to perform it uncached. eMule seems to largely perform uncached reads and writes.


@fox88 Both reading and writing

When i check the standby cache with RAMMap, There are a lot of files in the standby cache. But none of them are because of eMule.

This post has been edited by Soul--Reaver: 07 March 2021 - 02:41 AM

0

#5 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 07 March 2021 - 04:20 AM

Actually it's possible for optimization of eMule's writing buffer behaviors, as what I have done with my mod, the responsiveness of the disk has been greatly improved even during massive downloads.

To be more specific, files with buffered writing will firstly read from the disk (reading), then the read buffer be changed by writing to buffer (changing, no immediate disk activity) and the last step is flush (writing), my buffer system separated these 3 steps, to prevent reading/writing occurred at the same time.

Other important step is, don't flush buffer if it's not maturely received, just keep it in memory for further receiving, by doing that, can avoid multiple writing for small amount of data, which I called it 'fragmented buffer writing'.

Third part is use some randomized technique to initiate buffer flushing sequence in a relatively randomized manner, in that way, coupled flushing activities from different downloading files can be avoided statistically, consequently disk i/o concurrency can be reduced.

Edit: @SS,
In order to achieve data consistency, eMule actually proactively initiate flushing order to the OS, in that sense, eMule's buffer is not completely handed over to the OS.

This post has been edited by Enig123: 07 March 2021 - 04:38 AM

0

#6 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4973
  • Joined: 13-May 07

Posted 07 March 2021 - 08:23 AM

View PostSoul--Reaver, on 07 March 2021 - 04:53 AM, said:

Both reading and writing

These two things are handled very differently in eMule since 0.50b.
What exactly we are talking about is unclear so far.
For a start, what data rates this is all about?
I have 16 MB of RAM and a non-RAID hard drive. Copying a file on this HDD commonly goes at 60+ MB/s.
Hence simultaneous uploads and download at 1 MB/s is like nothing.

View PostSoul--Reaver, on 07 March 2021 - 04:53 AM, said:

When i check the standby cache with RAMMap, There are a lot of files in the standby cache. But none of them are because of eMule.

I have a bunch of files related to eMule in RAMMap - config and a few from the shared ones.
0

#7 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 08 March 2021 - 11:49 PM

Pure disk read performance varies between 272-424 MB per sec depending on the location of the disk.
Reading and writing at the same time on the same partition averages between 100-120 mb per sec.
When i download a file and eMule reaches around 15-20 MB download speed the load on the disk averages at 80% and makes eMule feel sluggish in its user interface. In my opinion eMule does not properly utilize the OS standby and write cache. But i've never looked at the eMule code so the reason behind this is unknown to me.
So my request in simplest form is: Is it possible to make eMule use the OS cache in a better way or implement its own cache? eMule has a file buffer but it's still limited to 1.5 MB. Maybe increasing that would help because that limit is pretty ancient when internet connections were a lot slower.
0

#8 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 09 March 2021 - 02:24 AM

View PostSoul--Reaver, on 08 March 2021 - 03:49 PM, said:

Is it possible to make eMule use the OS cache in a better way or implement its own cache? eMule has a file buffer but it's still limited to 1.5 MB. Maybe increasing that would help because that limit is pretty ancient when internet connections were a lot slower.


Will you please try my mod? I have sent you a pm with a download link.
0

#9 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 09 March 2021 - 10:09 PM

Alright i tested your version and these are my experiences.


0.50b Enig123 version
www . dropbox . com/s/0gjgntv5thrnxib/Untitled_0.50b_test%20version.png?dl=0
The first part of the "system information" window is file allocation so should be ignored. Did cause the rest of "system information" for disk and i/o to not be properly readable.
Actual download speed and disk usage fluctuated a lot more. eMule interface stayed decently responsive.


0.60b x64 Fox88 version
www . dropbox . com/s/tmwodtd6fmwx59q/Untitled_0.60b_latest%20version_v1.png?dl=0
When i continued the same downloads on the latest Fox88 version, eMule interface actually froze but everything else still worked

www . dropbox . com/s/0dtg21pf6uyqqze/Untitled_0.60b_latest%20version_v2.png?dl=0
Eventually the interface unfroze (took almost finishing one of the downloads). The dip and spike right in the middle of "system information" was when the interface unfroze.
You'll notice the disk usage is a lot more stable and even. Whether that is a good or bad thing... well thats a matter of point of view.
The interface was slow and sluggish. Think of watching a 1 FPS movie


Conclusion
Interface is more responsive with the 0.50b Enig123 version
Download speed stability is better with 0.60b x64 Fox88 version


I would retest this if the Enig123 version changes were applied to the Fox88 version.
The Fox88 caused a lot of files to be rehashed and made a x64 version available. This makes testing these different versions a bit problematic and uneven.


Hope this helps
0

#10 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 10 March 2021 - 04:25 AM

View PostSoul--Reaver, on 09 March 2021 - 02:09 PM, said:

Alright i tested your version and these are my experiences.


0.50b Enig123 version
www . dropbox . com/s/0gjgntv5thrnxib/Untitled_0.50b_test%20version.png?dl=0
The first part of the "system information" window is file allocation so should be ignored. Did cause the rest of "system information" for disk and i/o to not be properly readable.
Actual download speed and disk usage fluctuated a lot more. eMule interface stayed decently responsive.



I've forgot to tell you that this mod will be more happy with a proper downloading limit due to the download bandwidth throttler it has.

Will be appreciated if you can test it one more time with a reasonable download speed limit.

BTW, the binary has been updated, you can download it with the same url I PMed to you.

Edit: You mentioned of the test result 'disk usage fluctuated a lot', that may caused by the strength of randomization not enough to de-couple the writing from different downloading files. I have adjusted the parameter with the new uploaded binary.

This post has been edited by Enig123: 10 March 2021 - 07:36 AM

0

#11 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 10 March 2021 - 03:11 PM

Please elaborate on a "reasonable download speed limit".
I've got 500/40 mbits internet connection.
My eMule speed limits are normally set to off/256 KBytes (with download capacity set to 1 MBytes).
I do not want to increase the upload limit beyond 256 KBytes.
Usually download speeds are low. Setting capacity higher would make the download graphs unreadable for 99% of its runtime.

For production usage (after testing). If i need to set the speed limits realistically we would also need to decouple the speed graphs from the capacity speeds
0

#12 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 10 March 2021 - 03:21 PM

View PostSoul--Reaver, on 10 March 2021 - 07:11 AM, said:

Please elaborate on a "reasonable download speed limit".
I've got 500/40 mbits internet connection.
My eMule speed limits are normally set to off/256 KBytes (with download capacity set to 1 MBytes).
I do not want to increase the upload limit beyond 256 KBytes.
Usually download speeds are low. Setting capacity higher would make the download graphs unreadable for 99% of its runtime.

For production usage (after testing). If i need to set the speed limits realistically we would also need to decouple the speed graphs from the capacity speeds


For testing purpose, of course you can try higher limit. My point is, this mod works quite well with a limit.

The latest binary has downloading improvements, I would be appreciated if you can test it's downloading stabilized. My practical use case does not involve high speed downloading, that's why your test be valuable for me to get it improved further.
0

#13 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 10 March 2021 - 09:26 PM

Just tell me with which limits you would like me to perform the tests based on earlier specified internet connection. I will test with how i like them and with your requested limits/settings.


just out of curiosity. Why don't you fork this repository https://github.com/irwir/eMule and make your changes?
Would make Fox88 work easier if he ever accepts the changes and would make testing a lot easier.
Would also make it possible to just market your changes as a mod and make it available to others if Fox88 never accepts them.
0

#14 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 10 March 2021 - 10:36 PM

View PostSoul--Reaver, on 10 March 2021 - 01:26 PM, said:

Just tell me with which limits you would like me to perform the tests based on earlier specified internet connection. I will test with how i like them and with your requested limits/settings.


just out of curiosity. Why don't you fork this repository https://github.com/irwir/eMule and make your changes?
Would make Fox88 work easier if he ever accepts the changes and would make testing a lot easier.
Would also make it possible to just market your changes as a mod and make it available to others if Fox88 never accepts them.


Start with 3000KB/s perhaps?

Just to let you know, my branch has quite a differences to fox88's code base, and I am not familiar with revision management.

This post has been edited by Enig123: 10 March 2021 - 10:36 PM

0

#15 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4973
  • Joined: 13-May 07

Posted 11 March 2021 - 10:54 AM

Interesting screenshots.
Somehow 0.50b has lower disk I/O numbers, but higher usage and response time.
Judging by these numbers, 0.60b might be more efficient.

View PostSoul--Reaver, on 09 March 2021 - 02:49 AM, said:

Pure disk read performance varies between 272-424 MB per sec depending on the location of the disk.

In that case 20 MB/s is well below 10% of the disk capabilities.

Maybe you missed that small discussion and another one where NAS usage was discussed.
0

#16 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 11 March 2021 - 09:57 PM

View Postfox88, on 11 March 2021 - 02:54 AM, said:

Interesting screenshots.
Somehow 0.50b has lower disk I/O numbers, but higher usage and response time.
Judging by these numbers, 0.60b might be more efficient.


The disk I/O of my mod is sporadical, so is the high response time, which is actually a sign of buffer system was working. And judging by the responsiveness of the UI, I'd say the situation might be on the contrary.

With tester's extremely many downloading files and so high speed, surely it can get further improvements by adjusting a few parameters, which I has already done with the new version.
0

#17 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 11 March 2021 - 11:27 PM

View Postfox88, on 11 March 2021 - 11:54 AM, said:

...
In that case 20 MB/s is well below 10% of the disk capabilities.
...


Yes it is below the capabilities but load is not decided based on pure sequential read performance. I only gave performance numbers based on "sequential pure read" and "sequential read and write at the same time".
eMule on the other hand is sequential up to ~9MB because that are the part sizes. So could cause a lot of smaller random reads and writes

View Postfox88, on 11 March 2021 - 11:54 AM, said:

...
Maybe you missed that small discussion and another one where NAS usage was discussed.


I don't follow this forum regularly. I usually only visit when i have an issue like now.
0

#18 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4973
  • Joined: 13-May 07

Posted 12 March 2021 - 01:27 PM

View PostSoul--Reaver, on 12 March 2021 - 02:27 AM, said:

Yes it is below the capabilities but load is not decided based on pure sequential read performance.

Have you tried to take a second look at your own screen shots?
Disk is reading and writing at 15 MB/s, yet response time is below 1 ms.
This is not stressing the disk system at all.
0

#19 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 553
  • Joined: 22-November 04

Posted 12 March 2021 - 11:28 PM

View Postfox88, on 12 March 2021 - 05:27 AM, said:

View PostSoul--Reaver, on 12 March 2021 - 02:27 AM, said:

Yes it is below the capabilities but load is not decided based on pure sequential read performance.

Have you tried to take a second look at your own screen shots?
Disk is reading and writing at 15 MB/s, yet response time is below 1 ms.
This is not stressing the disk system at all.


The question here is that eMule is almost ALWAYS reading/writing, may leaving actually little room for other disk i/o activities.

I don't think it's an ideal situation for user experience, even if the disk seems responsive.
0

#20 User is offline   Soul--Reaver 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 60
  • Joined: 15-February 03

Posted 05 April 2021 - 12:31 AM

Exactly Enig123
My system runs virtual machines using virtualbox (on another drive luckily)
The RAID drives run the following
- Torrents
- Donkey p2p (eMule)
- Boinc
- in-house streaming source
- H264 encoding
- Browser downloading
- Backup to cloud storage

In my opinion it would still be best if Windows could manage the read and write caching of eMule because a program cannot know what other things might take up system resources.

@fox88 I highly doubt the average response times windows task manager reports are correct. During reading it barely ever goes up even on 100% reading load. During a 100% writing load on the other hand then it goes up to at max ~1600 ms
The first Windows 10 builds reported this better but after 1 or 2 big updates i find it isn't accurate anymore. So the usage percentage is a better measure of load in my opinion.
Also this is a HDD hardware raid configuration with 1024 MB cache on the raid controller so that might skew the average response times as well.

Did some testing again
I started with a download limit of 10240 KB/s
The first red block in system information is pretty steady and good. But loads on 10240 KB/s and below were never much of an issue. Though i did find the eMule interface more responsive on high loads using the Enig123 build.
The second red block in system information is when i removed the download limit. You'll notice download speeds were all over the place. The disk usage did not change much. I'm wondering whether or not eMule is capable of finding ~9MB parts fast enough for 10+ MB/s downloads speeds. The interface became sluggish at this point but did not freeze or become stuck.
www . dropbox . com/s/0gsv2zhrzj4p2nt/eMule%20050b%20cn9%2005-04.png?dl=0

This is the system information graph a bit bigger for better reading.
The first red block is download limit at 10240 KB/s
The second red block is dwnload limit removed
The third red block is download limit at 10240 KB/s
www . dropbox . com/s/w05nvragah4ovt3/system%20informationn%2005-04.png?dl=0
0

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users