Official eMule-Board: "pre-buffer Upload" - Official eMule-Board

Jump to content


  • (3 Pages)
  • +
  • 1
  • 2
  • 3

"pre-buffer Upload" Nice to save HD... Rate Topic: -----

#1 User is offline   michelinok 

  • Member
  • PipPip
  • Group: Members
  • Posts: 42
  • Joined: 13-October 03

Post icon  Posted 01 April 2006 - 08:45 AM

First of all, sorry for my english :) :worthy:
I'm reading/studying the sourcecode and and i idea came out of the hat... why not buffering in memory much more data to upload? (one more slot for eachone in queue).
The idea was to save hard disks... i know about the filebuffer,but it only saves against the "write to disk operation", hard disks will never turn off.
I've tryed to set filebuffer to 25 meg,recompiled emule,set 1 min to shut down hard disk in the power manager of windows, but the hard disk never shut down (i've a clean windows installation on a dedicated machine).
0

#2 User is offline   qm2003 

  • V.I.P. (Volatile Indifferent Puppet)
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4441
  • Joined: 06-November 03

Posted 01 April 2006 - 09:02 AM

If your idea is to conserve harddisks, don't use any powersaving schemes.

Shutdown/powerup cycles are more stress to them than anything i can come up with right now.
Ok, maybe using them regularily in harddisk throwing contests... :angelnot:


The point is, a reading buffer to reduce disk access would not do any harm, but harddisks already have memory buffers (up to 16 MB for the newest models).

It certainly wouldn't hurt to have one in emule, if optional. At the moment i see only users with high upload speeds benefit from this, but not the regular user.
How to setup Emule. A small checklist | Schmu's MuleDoc
P2P is not piracy, it's marketing.
In fact, if your music or movie is NOT being downloaded, you should be WORRIED !
If you can't even give it away for free, how do you expect to sell it, stupid ?


I'm a bloodsucking fiend. Look at my outfit !
0

#3 User is offline   G2G 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 84
  • Joined: 14-December 04

Posted 01 April 2006 - 10:23 AM

Yes, it certainly wouldn't hurt to have it...

OK for me :+1:



P.S.
I suggest this for releasers:
http://techreport.co...am/index.x?pg=1 :D
- CPU PIII/450Mhz - RAM 256MB - HDD 40GB - Online 24h/24 - Consumption 30W - UL ~4GB/Day
X))
0

#4 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 01 April 2006 - 11:53 AM

Some mods have an improved upload buffering scheme than the original version. With such strategies the disc accesses are reduced a lot, expecially when slotfocus is used.
The problem is not the client, it's the user
0

#5 User is offline   michelinok 

  • Member
  • PipPip
  • Group: Members
  • Posts: 42
  • Joined: 13-October 03

Posted 01 April 2006 - 01:50 PM

CiccioBastardo, on Apr 1 2006, 11:53 AM, said:

Some mods have an improved upload buffering scheme than the original version. With such strategies the disc accesses are reduced a lot, expecially when slotfocus is used.
View Post


Which mods? :bounce:
0

#6 User is offline   Famerlor 

  • also known as Spike2
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 685
  • Joined: 16-October 04

Posted 01 April 2006 - 04:30 PM

not sure, but i think ionix implemented sth. like this....
Posted Image

You want a light mod with source-dropping, Powershare and WiZaRd's ClientAnalyzer ?
Try Spike2-Mod !

You rather want to stick to official eMule but don't want to miss all the new fixes and optimizations from the mods ?
Try OfFixed-Mod !


This post has been edited 1 time, the last time by God: Tomorrow, 12:74 PM
0

#7 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 02 April 2006 - 06:54 PM

I know Ionix and ZZUL Bastard [Armadillo]
The problem is not the client, it's the user
0

#8 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 03 April 2006 - 10:58 AM

@michelinok:
the filebuffer is only thought for WRITING access not for reading access.
For every client in your upload eMule allocates a buffer which is filled and checked in
void CUpDownClient::CreateNextBlockPackage()
but the cache in official client is pretty low, it allocates a max of 15secs @ 3.5kB/s - if you raise the upload speed then you should adapt that buffering, too. If you have REALLY a lot of memory then you could read in a whole chunk right on startup of each upload session (not considering CPU now!)

I changed that part for iONiX and others to

Quote

//>>> WiZaRd - caching
//hmmm official code would upload @3.5kB/s to every client... that's about 15sec cache
const uint32 cache = max(15*GetDatarate(), 150*1024);
//<<< WiZaRd - caching

    // See if we can do an early return. There may be no new blocks to load from disk and add to buffer, or buffer may be large enough already.
    if(m_BlockRequests_queue.IsEmpty() // There are no new blocks requested
  || m_addedPayloadQueueSession > GetQueueSessionPayloadUp()
//>>> WiZaRd - caching
  //&& m_addedPayloadQueueSession - GetQueueSessionPayloadUp() > 50*1024
  && m_addedPayloadQueueSession - GetQueueSessionPayloadUp() > cache
//<<< WiZaRd - caching

  ) // the buffered data is large enough already
        return;
...
...
        // Buffer new data if current buffer is less than 100 KBytes
        while (!m_BlockRequests_queue.IsEmpty()   
ReadBlockFromFileThread
//>>> WiZaRd - caching
  //&& (m_addedPayloadQueueSession <= GetQueueSessionPayloadUp() || m_addedPayloadQueueSession-GetQueueSessionPayloadUp() < 100*1024)
  && (m_addedPayloadQueueSession <= GetQueueSessionPayloadUp() || m_addedPayloadQueueSession-GetQueueSessionPayloadUp() < 2*cache)
//<<< WiZaRd - caching

    ) 

To allocate a 15sek buffer regardless of the uploadspeed.
0

#9 User is offline   Xman1 

  • Xtreme Modder
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1955
  • Joined: 21-June 03

Posted 03 April 2006 - 12:01 PM

changing the uploadbuffer makes no sence, because after sending one block you always receive one new block request. (three blocks, but only one new).

There is no way to buffer more data, until emule changes the protocol.

Edit:
you can try morph.. it buffers all requested blocks, not looking at any buffersize.

This post has been edited by Xman1: 03 April 2006 - 12:06 PM

0

#10 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 03 April 2006 - 12:31 PM

Well, changing the buffer worked quite smoothly on some of the high-speed systems that I have access to so I guess it works :flowers:
0

#11 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 03 April 2006 - 06:46 PM

It also decresed my HD accesses (I can almost sleep during the night now ;) ).
I have a simple 32KB/s line (capped at 28KB/s) and I use slotfocus (so I often upload at more then 20KB/s to a single client).
The problem is not the client, it's the user
0

#12 User is offline   FAQ 

  • Member Goofy
  • PipPipPipPipPipPipPip
  • Group: Banned
  • Posts: 1384
  • Joined: 05-May 05

Posted 03 April 2006 - 09:40 PM

CiccioBastardo, on Apr 3 2006, 07:46 PM, said:

It also decresed my HD accesses (I can almost sleep during the night now ;) ).
View Post

Works for me. :+1:


Posted Image
0

#13 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 05 April 2006 - 05:16 PM

In order to reduce disk acceses with high speed connections, you have to start buffering just before the buffers runs dry (the last cached block has begun sending) and then cache as many blocks as possible. It is the difference between the low and high-watermark of the buffer that keeps down the disk accesses, not the size of the buffer itself.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#14 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 05 April 2006 - 06:50 PM

One more idea! We could actually buffer more than requested by the client as we can make a quite safe assumption that if the last two blocks asked were in sequence and of EMBLOCKSIZE, then the next requested block will be so too. This would ofcourse require some code to create the ahead requests and to scrap them in the rare event that they are never requested.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#15 User is offline   PacoBell 

  • Professional Lurker ¬_¬ (so kyoot!)
  • PipPipPipPipPipPipPip
  • Group: Moderator
  • Posts: 7296
  • Joined: 04-February 03

Posted 05 April 2006 - 07:46 PM

FAQ, on Apr 3 2006, 01:40 PM, said:

CiccioBastardo, on Apr 3 2006, 07:46 PM, said:

It also decresed my HD accesses (I can almost sleep during the night now ;) ).
View Post
Works for me. :+1:
View Post
You guys are weird. I can't get to sleep at night without hearing fan and HDD noise. Freaks :P
Sed quis custodiet ipsos custodes
Math is delicious!
MmMm! Mauna Loa Milk Chocolate Toffee Macadamias are little drops of Heaven ^_^
Si vis pacem, para bellum DIE SPAMMERS DIE!

#16 User is offline   FAQ 

  • Member Goofy
  • PipPipPipPipPipPipPip
  • Group: Banned
  • Posts: 1384
  • Joined: 05-May 05

Posted 05 April 2006 - 08:38 PM

PacoBell, on Apr 5 2006, 08:46 PM, said:

You guys are weird. I can't get to sleep at night without hearing fan and HDD noise. Freaks :P
View Post

The worst thing is the heat generation from eight computers and one hdd rack. <_<
0

#17 User is offline   SlugFiller 

  • The one and only master slug
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 6988
  • Joined: 15-September 02

Posted 16 April 2006 - 04:38 PM

Quote

why not buffering in memory much more data to upload?

It already has a minimal buffering of 180k, since it reads file data in 180k blocks, and then compresses them(entire block must be available prior to compression).
180k is large enough, since even at that size there's a chance that the buffered data would be swapped out to disk, losing everything you were trying to save by using buffering.

Anyway, if you want more buffering, you can try starting here:

Quote

        while (!m_BlockRequests_queue.IsEmpty() &&
              (m_addedPayloadQueueSession <= GetQueueSessionPayloadUp() || m_addedPayloadQueueSession-GetQueueSessionPayloadUp() < 100*1024)) {

Still, by increasing this number, you increase the chance of memory swap, which would bring back harddrive usage.

Remember, physical memory may be fast, but it has limited size.
Why haven't you clicked yet?

SlugFiller rule #1: Unsolicited PMs is the second most efficient method to piss me off.
SlugFiller rule #2: The first most efficient method is unsolicited eMails.
SlugFiller rule #3: If it started in a thread, it should end in the same thread.
SlugFiller rule #4: There is absolutely no reason to perform the same discussion twice in parallel, especially if one side is done via PM.
SlugFiller rule #5: Does it say "Group: Moderators" under my name? No? Then stop telling me about who you want to ban! I really don't care! Go bother a moderator.
SlugFiller rule #6: I can understand English, Hebrew, and a bit of Japanese(standard) and Chinese(mandarin), but if you speak to me in anything but English, do expect to be utterly ignored, at best.
0

#18 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 19 April 2006 - 05:04 PM

I wouldn't really worry about indicating a buffer of something as 10 *180KB (divided for the uploading files) as the problem for triggering virtual memory swapping. For sure there are many areas where mods can save precious resources, not excluded the useless country flags. Slotfocus really benefits of the upload buffer increase in size, expecially when you can upload a single block in less then 10 secs with a relatively slow link as mine (28KB/s). I belive fast uploaders (50+KB/s) and really fast ones (1M+B/s) would really enjoy that.
As sais, slotfocus is the ideal companion for this feature.
The problem is not the client, it's the user
0

#19 User is offline   Semm 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 176
  • Joined: 29-May 06

Posted 04 September 2006 - 03:07 PM

I'd definitely also like a cache build into eMule, like azureus has one.

eMule is meant to be running 24/7 at best - but most standard harddisks aren't!
A big memory buffer, fitting todays RAM capacities, wouldn't only reduce noise, it would also increase your harddisks lifetime!

This post has been edited by Semm: 04 September 2006 - 03:08 PM

0

#20 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 06 September 2006 - 05:19 PM

HD are buit to spin very very long.
Motor ignition is a more critical factor (that is, if you keep starting and stopping them very often they'll die much sooner than leaving them spinning freerly).

I have a 120GB Maxtor that has been spinning 24/24 365/365 (but power cutoffs or during rare lighting storms) for the last 3 years, under the so "heavy" eMule load. Lately had some problems, I reformat it Low Level and it now seems working OK.
Keep your HDs cool and they'll enjoy spinning for years.
The problem is not the client, it's the user
0

  • Member Options

  • (3 Pages)
  • +
  • 1
  • 2
  • 3

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users