SiRoB, on Mar 29 2006, 02:26 PM, said:
Emule V0.47a [ese V1.0b] (2006-04-24) Updated DBR and ReadBlockFromFileThread
#41
Posted 29 March 2006 - 04:19 PM
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#42
Posted 29 March 2006 - 07:19 PM
SiRoB, on Mar 29 2006, 07:26 AM, said:
I know , I don't think there is an easy answer...
If we always send a small first request, limit it to eMule clients to avoid the 'hoard' problem; in many cases the source will transfer at an acceptable rate and we will have sent a smaller request for no gain...
The way, I've coded it, there is a chance that a very slow/new source will request a block before the threshold.... emm.. In fact maybe I should have written that as below, but still a really slow source could start with >3072000 of the part left...
Quote
uint16 PartAsked = sender->m_lastPartAsked;//so we know if it's a new source later
// Main loop
uint16 newBlockCount = 0;
while(newBlockCount != *count){
// Create a request block stucture if a chunk has been previously selected
if(sender->m_lastPartAsked != (uint16)-1){
//Reduce end of part block size for first block/slow source
if(bytesPerRequest == EMBLOCKSIZE && ((sender->GetDownloadDatarate()==0 && sender->IsEmuleClient() && DataRemainingInPart(sender->m_lastPartAsked)<3072000) || (sender->GetDownloadDatarate()<400 && PartAsked!=(uint16)-1)))
{
bytesPerRequest = 30720;
}
I think, as DBR will take care of things up to 100Mb or so, if we are outside of this range it is safe to assume we have a number of incomplete parts, so as eMule tries to spread it's requests, the chances of a new source starting a full part and then another source joining it are reduced, a little; though of course in the real world chances are two sources could only have one part we need.
edit: to fix stupid error!
This post has been edited by BlueSonicBoy: 30 March 2006 - 12:31 AM
#43
Posted 30 March 2006 - 09:29 AM
es inmoral,
o engorda...
#44
Posted 11 April 2006 - 01:06 PM
condensed anti shape code:
Quote
if (GetDownloadedTotal() < ((thePrefs.TransferFullChunks()) ? (PARTSIZE/2) : 2000000)) return 1.0F;
//END netfinity: Anti Shape
would it be reasonable to use 1677722 (1.6MB) for bit of fine tuning?
#45
Posted 11 April 2006 - 01:15 PM
moloko+, on Apr 11 2006, 02:06 PM, said:
If all mods are going to start to finetune to a hard coded value then credit shapers will just use a value depending on the modstring. Why not use a value that depends on the inclanation of mars relative to the current location. ( like 1.4 MB+ RANDOM at startup)
Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.
Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
#46
Posted 11 April 2006 - 02:07 PM
#47
Posted 11 April 2006 - 02:38 PM
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
Dark Lord of the Forum
Morph your Mule
Need a little help with your MorphXT? Click here
#48
Posted 11 April 2006 - 06:19 PM
Essentialy, it's a good idea to let this value to be unpredictable as credit-shapers would have a hard time guessing how much they need to upload to advance in the queue.
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#49
Posted 11 April 2006 - 06:24 PM
However rising the treshold to at least 1 chunk should definately finish them off - and IF a leecher uploads that much I wouldn't consider him a leecher anymore
#50
Posted 11 April 2006 - 06:27 PM
#51
Posted 11 April 2006 - 07:38 PM
I'd prefer credit shapers over real leechers, so I have not that much of a problem to give them some credit for their upload. However I don't want them to get to much of an advantage over good clients. I'll think due to the dynamics of the network, with my thresholds a credit shaper would be required to upload to the network about as much as he downloads to get a noticable boost.
There is a reason to not set the threshold too high and that is all the clients that are limited to an internet connection with 128kbit/s or less upload. Those clients have a hard time to contribute to your download with as much as an entire part, but we still want to reward them for trying.
@alpdruck
Might be an idea. Not sure if it's good or bad thought!
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#52
Posted 11 April 2006 - 09:38 PM
Anti Shape (+ Random)?
yeah these values are a bit high (just added them for test - it's working )
GetRandomUInt16() // max random value = 32767
random max = 1000KB (1024000)
initial val = 4864000
with random = 4864000 + (GetRandomUInt16() * 31) // (1024000/32767) = ~31
random max = 500KB (512000)
initial val = 2000000
with random = 2000000 + (GetRandomUInt16() * 16) // (512000/32767) = ~16
OR could use one var?
Quote
m_btransferfullchunks=ini.GetBool(L"FullChunkTransfers",true);
//BEGIN netfinity : Anti Shape (+ Random) - LoadPreferences
m_AntiShape_Rand_Full = (uint64)(GetRandomUInt16()*31);
m_AntiShape_Rand = (uint64)(GetRandomUInt16()*16);
//END
...
Quote
//BEGIN netfinity : Anti Shape (+ Random)
if (GetDownloadedTotal() < (thePrefs.TransferFullChunks() ? (4864000 + thePrefs.m_AntiShape_Rand_Full) : (2000000 + thePrefs.m_AntiShape_Rand))) return 1.0F;
//END
...
Some Leech Protection? - IDEA
formula works ok... utilizes ratio under 1.0F but...
Quote
//BEGIN Some Leech Protection?
if (DOWNLOADING_FILE){ // no effect on complete shared files - since transdown == 0
// Slow Ratio
if (transup > (SOME_THRESHOLD) && transup > (transdown * 3)) { // amount transferred up > amount transferred down * 3
float slow_ratio = 0.0F;
slow_ratio = (float)(((float)(transdown+1)/(float)(transup+3))*3.0);
if (slow_ratio < 0.1F)
return 0.1F;
//else if (slow_ratio > 1.0F) // never happens
//return 1.0F;
return slow_ratio; // < 1.0F
}
// fall through to official credit...
}
//END
//OFFICIAL CREDIT HERE...
needs work...
#53
Posted 14 April 2006 - 10:19 AM
regards gordo.
#54
Posted 23 April 2006 - 09:10 AM
Quote
{
...
// prevent locking of too many blocks when we are on a slow (probably standby/trickle) slot
int blockCount = 3;
if(IsEmuleClient() && m_byCompatibleClient==0 && reqfile->GetFileSize()-reqfile->GetCompletedSize() <= (uint64)PARTSIZE*4) {
// if there's less than two chunks left, request fewer blocks for
// slow downloads, so they don't lock blocks from faster clients.
// Only trust eMule clients to be able to handle less blocks than three
if(GetDownloadDatarate() < 18 * 1024) { // netfinity: Raised threshold so that more blocks are requested if the current ones are consumed in less than 10 seconds
blockCount = 1;
} else if(GetDownloadDatarate() < 36 * 1024) { // netfinity: Raised threshold so that more blocks are requested if the current ones are consumed in less than 10 seconds
blockCount = 2;
}
}
CreateBlockRequests(blockCount);
...
Will be fixed in ESE 1.0b.
/netfinity
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#55
Posted 23 April 2006 - 09:45 AM
#56
Posted 23 April 2006 - 10:08 AM
tHeWiZaRdOfDoS, on Apr 23 2006, 11:45 AM, said:
Note that, asking for less blocks does not increase the overhead as these block request packages are always of the same size and always sent when a block is completed. The only thing is that there will be a short delay after a block completes until the next one can be donwloaded.
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#57
Posted 23 April 2006 - 08:17 PM
netfinity, on Apr 23 2006, 04:10 AM, said:
Quote
{
...
// prevent locking of too many blocks when we are on a slow (probably standby/trickle) slot
int blockCount = 3;
if(IsEmuleClient() && m_byCompatibleClient==0 && reqfile->GetFileSize()-reqfile->GetCompletedSize() <= (uint64)PARTSIZE*4) {
...
Will be fixed in ESE 1.0b.
/netfinity
Ok, I have been touting this code shamelessly for ages.
But you might find it useful?
Using the code below you can get rid of the "eMule only" clause. Doing so IMHO has more benifits than drawbacks...
In DownClient.cpp
void CUpDownClient::CreateBlockRequests(int iMaxBlocks) said:
void CUpDownClient::CreateBlockRequests(int iMaxBlocks)
{
ASSERT( iMaxBlocks >= 1 /*&& iMaxBlocks <= 3*/ );
if (m_DownloadBlocks_list.IsEmpty())
{
uint16 count;
//START block splitting code since 1.5a only split blocks for none eMule clients save overhead
if(iMaxBlocks==1 && !m_PendingBlocks_list.GetCount() && (!IsEmuleClient() || m_byCompatibleClient!=0))
{//Block count reduction has activated convert the 1 block request to 3 blocks
count = iMaxBlocks;
Requested_Block_Struct** toadd = new Requested_Block_Struct*[1];
if(reqfile->GetNextRequestedBlock(this,toadd,&count))
{//splits 1 block into 3. generally 1 180Kb block into 3 60Kb blocks
Requested_Block_Struct* block_one = new Requested_Block_Struct;
Requested_Block_Struct* block_two = new Requested_Block_Struct;
Requested_Block_Struct* block_thr = new Requested_Block_Struct;
uint32 thirdblocksize = (toadd[0]->EndOffset - toadd[0]->StartOffset)/3;
/*First Block*/
block_one->StartOffset = toadd[0]->StartOffset;
block_one->EndOffset = toadd[0]->StartOffset + thirdblocksize;
md4cpy(block_one->FileID, toadd[0]->FileID);
block_one->transferred = 0;
/*Second Block*/
block_two->StartOffset = block_one->EndOffset + 1;
block_two->EndOffset = block_two->StartOffset + thirdblocksize;
md4cpy(block_two->FileID, toadd[0]->FileID);
block_two->transferred = 0;
/*Third Block*/
block_thr->StartOffset = block_two->EndOffset + 1;
block_thr->EndOffset = toadd[0]->EndOffset;
md4cpy( block_thr->FileID, toadd[0]->FileID);
block_thr->transferred = 0;
//add the 3 blocks
reqfile->SwitchBlocks( block_one, block_two, block_thr, toadd );
delete[] toadd;
m_DownloadBlocks_list.AddTail( block_one );
m_DownloadBlocks_list.AddTail( block_two );
m_DownloadBlocks_list.AddTail( block_thr );
while(m_PendingBlocks_list.GetCount() < 3 && !m_DownloadBlocks_list.IsEmpty())
{
Pending_Block_Struct* pblock = new Pending_Block_Struct;
pblock->block = m_DownloadBlocks_list.RemoveHead();
m_PendingBlocks_list.AddTail(pblock);
}
return;
} else
{
delete[] toadd;
return;
}
}//End block splitting code
if(iMaxBlocks > m_PendingBlocks_list.GetCount()) count = iMaxBlocks - m_PendingBlocks_list.GetCount();
else count = 0;
Requested_Block_Struct** toadd = new Requested_Block_Struct*[count];
if(reqfile->GetNextRequestedBlock(this,toadd,&count))
{
for (UINT i = 0; i < count; i++)
m_DownloadBlocks_list.AddTail(toadd[ i ]);
}
delete[] toadd;
}
while (m_PendingBlocks_list.GetCount() < iMaxBlocks && !m_DownloadBlocks_list.IsEmpty())
{
Pending_Block_Struct* pblock = new Pending_Block_Struct;
block->block = m_DownloadBlocks_list.RemoveHead();
m_PendingBlocks_list.AddTail(pblock);
}
}
In PartFile.cpp
void CPartFile::SwitchBlocks(Requested_Block_Struct* block_one, on Requested_Block_Struct* block_two,Requested_Block_Struct* block_thr,Requested_Block_Struct** toadd ), said:
void CPartFile::SwitchBlocks(Requested_Block_Struct* block_one,Requested_Block_Struct* block_two,Requested_Block_Struct* block_thr,Requested_Block_Struct** toadd )
{
POSITION pos;
POSITION posLast;
//find the 1 full size block added
posLast = pos = requestedblocks_list.GetTailPosition();
Requested_Block_Struct* block = requestedblocks_list.GetNext(pos);
if(toadd[0]->StartOffset == block->StartOffset && toadd[0]->EndOffset == block->EndOffset)
{
delete block; //delete full size block
requestedblocks_list.RemoveAt(posLast); //remove it from the list
} else
{//RemoveBlockFromList() + delete block;
for(pos = requestedblocks_list.GetHeadPosition(); pos != NULL; )
{
posLast = pos;
Requested_Block_Struct* block = requestedblocks_list.GetNext(pos);
if(block->StartOffset <= toadd[0]->StartOffset && block->EndOffset >= toadd[0]->EndOffset)
{
delete block; //delete full size block
requestedblocks_list.RemoveAt(posLast);
}
}
}
//add the 3 1/3 size blocks
requestedblocks_list.AddTail(block_one);
requestedblocks_list.AddTail(block_two);
requestedblocks_list.AddTail(block_thr);
}
Thanks for reading...
Ok, I really can't stop now can I.....?
At this point in SendBlockRequests() I am trying to do a relative comparison!
Time for more old code I've been touting for ages...
In DownClient.cpp
void CUpDownClient::SendBlockRequests() said:
// prevent locking of too many blocks when we are on a slow (probably standby/trickle) slot
int blockCount = 3;
if( reqfile->ReduceAllocatedBlocks(this ,this->m_lastPartAsked, 912) ){
//near end of part with multiply sources, share the blocks out, slower sources have blockcount reduced first.
blockCount = 1;
}
CreateBlockRequests(blockCount);
...
In PartFile.cpp
CPartFile::ReduceAllocatedBlocks(CUpDownClient* calling_source, on uint16 m_PartAsked, UINT fixComparison), said:
bool CPartFile::ReduceAllocatedBlocks(CUpDownClient* calling_source, uint16 m_PartAsked, UINT fixComparison)
{
//no 'part last asked for'
if(m_PartAsked == (uint16)-1)
{//improved for 1.5b
if(calling_source->IsEmuleClient()) return true;
else return false; //for hoard clients & prevent fragmentation from 10k DBR
}
//Quick Check, if there is only one source overall return no 'part last asked for'
if(m_downloadingSourceList.GetCount()<2)
{//if this source is the only source don't let it lock blocks if its too slow
if(calling_source->GetDownloadDatarate() < fixComparison) return true;
else return false;
}
uint16 sourcecount = 0;
uint32 remainingdata = 0;
uint32 otherstransferrate = 1;// a value of 1 to prevent 'divide by zero' (saves 1 + otherstransferrate later)
const uint32 threeblocks = EMBLOCKSIZE * 3;
/*Calculate total transfering sources for this part and overall transfer speed.*/
for(POSITION pos = m_downloadingSourceList.GetHeadPosition(); pos != NULL;)
{
//get next downloading source
CUpDownClient* cur_src = srclist.GetNext(pos);
//is this source sending data for the same part as the calling source but is not the calling source
if(cur_src->m_lastPartAsked==m_PartAsked && cur_src!=calling_source)
{
sourcecount++;//increment sources for this part
otherstransferrate += cur_src->GetDownloadDatarate();//add this clients transfer rate to the total
}
}
//if calling source is the only currently transfering source for this part use fixed reference
if(sourcecount < 1)
{//if this source is the only source don't let it lock blocks if its too slow
if(calling_source->GetDownloadDatarate() < fixComparison) return true;
else return false;
}
/* Calculate how much of the part is left in bytes (based on code from GetNextRequestedBlock() )*/
// Offsets of 'this' chunk
const uint64 uStart = (uint64)m_PartAsked * PARTSIZE;
const uint64 uEnd = (GetFileSize() - (uint64)1 < (uStart + PARTSIZE - 1)) ?
GetFileSize() - (uint64)1 : (uStart + PARTSIZE - 1);
ASSERT( uStart <= uEnd );
if(uStart >= uEnd) return false;
//gets bytes remaining
for(POSITION pos = gaplist.GetHeadPosition(); pos != NULL; )
{
const Gap_Struct* cur_gap = gaplist.GetNext(pos);
//Check if Gap is into the limit
if(cur_gap->start < uStart)
{
if(cur_gap->end > uStart && cur_gap->end < uEnd) remainingdata += cur_gap->end - uStart + 1;
else if(cur_gap->end >= uEnd) return false;
} else
if(cur_gap->start <= uEnd)
{
if(cur_gap->end < uEnd) remainingdata += cur_gap->end - cur_gap->start + 1;
else remainingdata += uEnd - cur_gap->start + 1;
}
}
if(threeblocks > remainingdata) return true;//keep up combined download speed for as long as possible
else remainingdata -= threeblocks; //remainingdata to equal the remaining data after this client gets allocated 3 blocks
//if(3 * EMBLOCKSIZE / 'calling source speed' >= 'remaining incomplete part size - 3 blocks'/'total other sources transfer rate') return true
if((uint32)(threeblocks/(1 + calling_source->GetDownloadDatarate())) >= (uint32)(remainingdata/otherstransferrate)) return true;
return false;
}
#58
Posted 24 April 2006 - 06:32 PM
Wow! That was a lot of code. A bit to much to include in the ESE mod I think, as I want to keep the code as simple as possible, but could be useful in some other mod.
You're right about getting rid of the "eMule only" would be a good thing. To be honest I'm not sure why it exists at all. As far as I know all clients can handle block requests with only one block. I've done it before and never seen any problem. I might look into that for a future mod.
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#59
Posted 24 April 2006 - 06:46 PM
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
#60
Posted 24 April 2006 - 09:24 PM