Official eMule-Board: Faster 'endgame' - Official eMule-Board

Jump to content


Page 1 of 1

Faster 'endgame' Increase the speed at end of downloads

#1 User is offline   dazzle 

  • Premium Member
  • PipPipPipPipPip
  • Group: Members
  • Posts: 277
  • Joined: 25-December 02

Posted 28 April 2005 - 09:45 PM

Emule reserves a few 180kb blocks of a file ahead for every source giving download. This means that at a certain time at the end of a file download, all blocks that still need to be downloaded are reserved. When a download source needs more blocks, it will not be able to get them and download will be aborted.

The problem with this implementation is that fast uploading sources will sooner run out of blocks to download then slow downloading blocks (because fast uploading sources request blocks more often).

The result is that at the end of the download emule is stuck with all the slow sources that only give a few bytes / secs. An experienced user knows that if emule says the download still needs 30 seconds that it can take more then 10 minutes to complete if the source only gives 100b/s.

An easy solution to this problem would be to drop the slowest source from from the download list whenever a source needs more blocks. This means emule would always stick with the fastest sources instead of the slowest sources. The code changes are very simple. I've implemented it in my mod and it easily save 10-20 minutes at the end of any file. Previously an mp3 with 400 sources easily took 20 minutes to download, no I can finish any of them within 5 minutes. It also helps at then end of other files, whenever you end with downloads from multiple sources.

These are the changes:

void CUpDownClient::CreateBlockRequests(int iMaxBlocks)
{
	ASSERT( iMaxBlocks >= 1 /*&& iMaxBlocks <= 3*/ );
	if (m_DownloadBlocks_list.IsEmpty())
	{
  //dazzle: prevent a sign error from occurring
  uint16 count, countinit;
  if (iMaxBlocks - m_PendingBlocks_list.GetCount()> 0)
  	countinit = iMaxBlocks - m_PendingBlocks_list.GetCount();
  else
  	countinit = 0;
  count = countinit;
  //end dazzle
  Requested_Block_Struct** toadd = new Requested_Block_Struct*[count];
  if (reqfile->GetNextRequestedBlock(this,toadd,&count)){
  	for (int i = 0; i < count; i++)
    m_DownloadBlocks_list.AddTail(toadd[i]);
  }
  //dazzle: if no blocks available, try to drop the slowest source, then try again
  else {
  	if (reqfile->DropSlowestSource(this)) {
    count = countinit;
    if (reqfile->GetNextRequestedBlock(this,toadd,&count)){
    	for (int i = 0; i < count; i++)
      m_DownloadBlocks_list.AddTail(toadd[i]);
    }
  	}
  }
  //end dazzle
  delete[] toadd;
	}

	while (m_PendingBlocks_list.GetCount() < iMaxBlocks && !m_DownloadBlocks_list.IsEmpty())
	{
  Pending_Block_Struct* pblock = new Pending_Block_Struct;
  pblock->block = m_DownloadBlocks_list.RemoveHead();
  m_PendingBlocks_list.AddTail(pblock);
	}
}

//dazzle: This function is used to increase endgame speed. Called whenever no blocks are available to download for a speedy source.
// this function drops the slowest source so the speedy source can download his blocks.
boolean CPartFile::DropSlowestSource(CUpDownClient* calling_source) {
	uint32 lowestspeed = 2000000000;
	CUpDownClient* slowest_source= NULL;
	for (POSITION pos = m_downloadingSourceList.GetHeadPosition(); pos != NULL;)
	{
  CUpDownClient* cur_src = srclist.GetNext(pos);
  if (cur_src->GetDownloadDatarate() < lowestspeed) {
  	lowestspeed = cur_src->GetDownloadDatarate();
  	slowest_source = cur_src;
  }
	}
	if (m_downloadingSourceList.GetCount() > 1 && slowest_source && slowest_source!=calling_source ){ // only remove source if it isn't the last one
  	ASSERT( slowest_source->socket != NULL );
  	if (slowest_source->socket != NULL)
  	{
    ASSERT( !slowest_source->socket->IsRawDataMode() );
    if (!slowest_source->socket->IsRawDataMode())
    	slowest_source->SendCancelTransfer();
  	}
  	slowest_source->SetDownloadState(DS_ONQUEUE, _T("endgame: replacing client for a faster one"));
  return true;
	}
	else
  return false;	
}
//end dazzle

0

#2 User is offline   Andu 

  • Morph Team
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 13015
  • Joined: 04-December 02

Posted 28 April 2005 - 09:58 PM

Didn't netfinity introduce something like this in his mod?
Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.


Dark Lord of the Forum


Morph your Mule

Need a little help with your MorphXT? Click here

0

#3 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 28 April 2005 - 10:38 PM

Yeah he did and he was the first pointing that problem out (code can be found in snippets section)...
0

#4 User is offline   dazzle 

  • Premium Member
  • PipPipPipPipPip
  • Group: Members
  • Posts: 277
  • Joined: 25-December 02

Posted 28 April 2005 - 11:11 PM

You mean the 'dynamic block request'? http://forum.emule-p...showtopic=63431

I don't know what it is and what it does. Unfortunately netfinity only gives code snippets without an explanation what it actually does.

Currently I'm not in the mood to read his code so I'm not sure if it accomplishes the same thing, but it does look like he uses another strategy.

Well, this is a very simple basic idea. Other people must have thought about it before and to be frank, I don't understand why this isn't in official emule yet. So well, if this has been posted before then it's here again. I think that it's an easy change and I don't see any negative side effects.
0

#5 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 29 April 2005 - 05:53 AM

Don't get me wrong, I like the idea, too!
Netfinity just started a thread in the past pointing out this problem and - to be frank, too - I also don't get why the devs won't put something like that in...
I think I'll try your version later on to check wether it helps :flowers:
0

#6 User is offline   Thievery 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 113
  • Joined: 22-November 04

Posted 29 April 2005 - 10:43 AM

It seems a great idea, going to check it out.

Goo job! :+1:
"Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us!" - Calvin & Hobbes
0

#7 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 29 April 2005 - 11:27 AM

@Moderator: this should be moved to CodeSnippets :)
0

#8 User is offline   dvader 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 10
  • Joined: 28-June 03

Posted 09 May 2005 - 04:10 PM

dazzle's snippet indeed make what it supposed to - it kills slowe sources. The only problem is that it does it all the time, not only on the last segment. Imagine you donwload from a fast source, that contain only one available segment and from some slow source, that contains many. When fast has nothing else to upload, it tries to kill a slower source... result - download stops
0

#9 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 09 May 2005 - 07:32 PM

FiXeD: *g*

Quote

boolean CPartFile::DropSlowestSource(CUpDownClient* calling_source)
{
if(GetFileSize()-GetCompletedSize()>SESSIONMAXTRANS)
  return false;

0

#10 User is offline   zz 

  • -
  • PipPipPipPipPipPipPip
  • Group: Debugger
  • Posts: 2014
  • Joined: 30-November 02

Posted 10 May 2005 - 05:13 PM

I don't think it's good to kill downloads that has started just because you have locked blocks with that source. Instead, it should be possible to request less blocks while the source is slow, to prevent it from locking blocks for too long. If the source is really really slow, it might be necessecary to shrink the size of the one block (180 KBytes?) that is requested.

I haven't checked how this would impact on the uploading side, but on the downloading side seems very possible to implement.

/zz B)
ZZUL - get control of your uploads: ZZUL Forum
0

#11 User is offline   Andu 

  • Morph Team
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 13015
  • Joined: 04-December 02

Posted 10 May 2005 - 05:56 PM

Now this really sounds like what NetF mod already has. Maybe it's time to take a look in that one?
Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.


Dark Lord of the Forum


Morph your Mule

Need a little help with your MorphXT? Click here

0

#12 User is offline   SlugFiller 

  • The one and only master slug
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 6988
  • Joined: 15-September 02

Posted 24 May 2005 - 08:13 AM

In CPartFile::DropSlowestSource when looking for the slowest source it should limit its search to sources currently locking blocks that would be possible for the new source to upload.
Why haven't you clicked yet?

SlugFiller rule #1: Unsolicited PMs is the second most efficient method to piss me off.
SlugFiller rule #2: The first most efficient method is unsolicited eMails.
SlugFiller rule #3: If it started in a thread, it should end in the same thread.
SlugFiller rule #4: There is absolutely no reason to perform the same discussion twice in parallel, especially if one side is done via PM.
SlugFiller rule #5: Does it say "Group: Moderators" under my name? No? Then stop telling me about who you want to ban! I really don't care! Go bother a moderator.
SlugFiller rule #6: I can understand English, Hebrew, and a bit of Japanese(standard) and Chinese(mandarin), but if you speak to me in anything but English, do expect to be utterly ignored, at best.
0

#13 User is offline   BlueSonicBoy 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 396
  • Joined: 26-September 04

Posted 06 November 2005 - 02:08 AM

tHeWiZaRdOfDoS, on May 9 2005, 02:32 PM, said:

FiXeD: *g*

Quote

boolean CPartFile::DropSlowestSource(CUpDownClient* calling_source)
{
if(GetFileSize()-GetCompletedSize()>SESSIONMAXTRANS)
  return false;

View Post

Just looking at the Faster 'endgame' code. :+1:

Maybe put your fix here to save the function call....? :flowers:

Quote

    //dazzle: if no blocks available, try to drop the slowest source, then try again
    else {//Wizard's fix: drop slowests source only in the last part
              if(reqfile->GetFileSize() - reqfile->GetCompletedSize() <= SESSIONMAXTRANS){
                if (reqfile->DropSlowestSource(this)){
                  count = countinit;
                  if (reqfile->GetNextRequestedBlock(this,toadd,&count)){
                    for (int i = 0; i < count; i++)
                    m_DownloadBlocks_list.AddTail(toadd[i]);
                  }
                }
              }         
            }
    //end dazzle

0

#14 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 06 November 2005 - 10:16 AM

This code is useless (IMHO) because of the latest changes in original code so I wouldn't care about it anymore... :flowers:
0

#15 User is offline   BlueSonicBoy 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 396
  • Joined: 26-September 04

Posted 07 November 2005 - 02:23 AM

tHeWiZaRdOfDoS, on Nov 6 2005, 05:16 AM, said:

This code is useless (IMHO) because of the latest changes in original code so I wouldn't care about it anymore... :flowers:
View Post

Thanks. :) I am just looking at the code after reading this thread: Slow Download Ending Solution

I don't think the Dazzle patch as is, is the best solution, but I don't think the official code fully fixes the problem either? :(
0

#16 User is offline   SlugFiller 

  • The one and only master slug
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 6988
  • Joined: 15-September 02

Posted 07 November 2005 - 04:46 PM

NetFinity's solution is the best, beating both this and, obviously, the official. It's in this very sub-forum, but it's pretty old so you'll have to look it up.
I still wonder why NetFinity's solution isn't used in the official.
Why haven't you clicked yet?

SlugFiller rule #1: Unsolicited PMs is the second most efficient method to piss me off.
SlugFiller rule #2: The first most efficient method is unsolicited eMails.
SlugFiller rule #3: If it started in a thread, it should end in the same thread.
SlugFiller rule #4: There is absolutely no reason to perform the same discussion twice in parallel, especially if one side is done via PM.
SlugFiller rule #5: Does it say "Group: Moderators" under my name? No? Then stop telling me about who you want to ban! I really don't care! Go bother a moderator.
SlugFiller rule #6: I can understand English, Hebrew, and a bit of Japanese(standard) and Chinese(mandarin), but if you speak to me in anything but English, do expect to be utterly ignored, at best.
0

#17 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 07 November 2005 - 05:56 PM

I guess it's the same reason that prevents your SafeHash to be implemented properly... :lol:
0

#18 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 07 November 2005 - 06:52 PM

Well the code I released in this subforum was quite complex even if it is a lite version of the code implemented in my Mod.

I experiment quite alot with my code so the best thing if you want it, is to look how I did it and write your own based on it!

Haven't much time for coding these days...

/netfinity
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#19 User is offline   BlueSonicBoy 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 396
  • Joined: 26-September 04

Posted 16 November 2005 - 02:43 AM

Firstly a big thank you to Dazzle for this code, which I believe is still relevent today in eMule 46c. :+1:
Secondly, I accept that block partitioning as per Netfinity's code would result in even higher completion speeds.

:respect:

Having examined the above code and I thought there may be a scenario where this code may extend end time But I was wrong! Incomplete block data,(from say a dropped source) is buffered and written to disk not lost! :angelnot:

So just a couple of small tweaks...


In DownloadClient.cpp

DownloadClient.cpp void CUpDownClient::CreateBlockRequests(int iMaxBlocks) said:

    else {//Wizard's fix: drop slowests source only in the last part only check if this client has a data rate
    if(reqfile->GetFileSize() - reqfile->GetCompletedSize() <= SESSIONMAXTRANS && GetDownloadDatarate()){
            if (reqfile->DropSlowestSource(this)){


in PartFile.cpp

PartFile.cpp boolean CPartFile::DropSlowestSource(CUpDownClient* calling_source) said:

boolean CPartFile::DropSlowestSource(CUpDownClient* calling_source) {
 
  //if there is only one transferring source return replaces the m_downloadingSourceList.GetCount() > 1 later in the code
  if(m_downloadingSourceList.GetCount()<2) return false;



[edit] more realizations more edits... ongoing thought[/edit]

This post has been edited by BlueSonicBoy: 19 November 2005 - 05:10 AM

0

  • Member Options

Page 1 of 1

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users