Official eMule-Board: Smart Part File Push - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Smart Part File Push *** Work in progress ***

#1 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 02 April 2006 - 05:32 PM

I'm currently working on a feature to solve the issue with part files getting to little upload when there are many complete files shared.

I would like your input on how to make the feature better and non-abusable.

Here is the code that has been applied to the CUpDownClient::GetFilePrioAsNumber() function;
	// BEGIN netfinity: Smart Part File Push
	if ( currequpfile->IsPartFile() )
	{
  if ( nGlobalUpDatarate2hr > nGlobalDownDatarate2hr && nGlobalUpDatarate2hr > (2 * nGlobalUpPartDatarate2hr) )
  	filepriority *= 2;
	}
	// END netfinity: Smart Part File Push
I think the variables is kinda self explanatory. What i do is measuring the average upload and download datarate during a 2 hour period (moving window), and for the upload I also measure the datarate for part files.

Regards,
netfinity
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#2 User is offline   leuk_he 

  • MorphXT team.
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5975
  • Joined: 11-August 04

Posted 02 April 2006 - 06:46 PM

You still do not promote leaving the complete files online.

To make it less rewarding for users to move away completed files your "2 queue, 1 for partfiel and 1 for completes" was more usefule, or is this filepriority *2 repeated nGlobalUpDatarate2hr > (2 * nGlobalUpPartDatarate2hr) )?

To promote high upload and leave complete file online i think a check the other way arround should also be fair :

pesudo cde:
if (IsPartFIle)
if nGlobalUpDatarate2hr < (2 * nGlobalUpPartDatarate2hr) )?
Filepriority=Filepriority/2


Or making the *2 factor more flexible going more in the direction of 2 queues(part/complete files) ...

Filepriority=Filepriority* nGlobalUpDatarate2hr /(nGlobalUpPartDatarate2h*2)
(with some cutoff points for extreme values..)

And then you still have to come up with something to counter the parfilepush. Else you are only promoting partfiles. (what is the point of having filediversity if those files are rarely uploaded) do I am thing about some ratio, or rare complete file push

just my 2 cents.
Download the MorphXT emule mod here: eMule Morph mod

Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.

Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
0

#3 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 02 April 2006 - 07:07 PM

Why not making something like

Quote

static int iUploadRotate = 0;
CUpDownClient* CUploadQueue::FindBestClientInQueue()
{
uint32 bestscore = 0;
uint32  bestlowscore = 0;
CUpDownClient* newclient = NULL;
CUpDownClient* lowclient = NULL;

POSITION pos1, pos2;
for (pos1 = waitinglist.GetHeadPosition();( pos2 = pos1 ) != NULL;)
{
  waitinglist.GetNext(pos1);
  CUpDownClient* cur_client = waitinglist.GetAt(pos2);
  //While we are going through this list.. Lets check if a client appears to have left the network..
  ASSERT ( cur_client->GetLastUpRequest() );
  CKnownFile* file = !theApp.sharedfiles->GetFileByID(cur_client->GetUploadFileID())
  if ((::GetTickCount() - cur_client->GetLastUpRequest() > MAX_PURGEQUEUETIME) /*|| !theApp.sharedfiles->GetFileByID(cur_client->GetUploadFileID())*/ )
  {
   //This client has either not been seen in a long time, or we no longer share the file he wanted anymore..
   cur_client->ClearWaitStartTime();
   RemoveFromWaitingQueue(pos2,true);
   continue;
  }
        //else
        {
               
              uiUploadRotate++;
              //enforce every third slot for partfiles
                                if(uiUploadRotate % 3 == 0)
                                {
                                     if(!file->IsPartFile())
                                          continue;
                                }
              //could also enforce a 1:2 scenario...
                                else
                                {
                                     if(file->IsPartFile())
                                          continue;
                                }
      


or easily counting the part/full file uploads and checking the diff there... ?


EDIT: change

This post has been edited by tHeWiZaRdOfDoS: 02 April 2006 - 07:09 PM

0

#4 User is offline   Stulle 

  • [Enter Mod] Dev
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5804
  • Joined: 07-April 04

Posted 02 April 2006 - 07:22 PM

bad. think about this scenario... 200 complete files and 2 incomplete files. we'd upload 50 times each partfile before we have uploaded every complete file. thinking a bit simple but you get the main idea!
I am an emule-web.de member and fan!

[Imagine there was a sarcasm meter right here!]

No, there will not be a new version of my mods. No, I do not want your PM. No, I am certain, use the board and quit sending PMs. No, I am not kidding, there will not be a new version of my mods just because of YOU asking for it!
0

#5 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 02 April 2006 - 07:27 PM

And...? This is EXACTLY the problem that shall be prevented: currently you upload WAY to much of complete files, you are PUNISHED for sharing - why do u think so many ppl remove their shared files so fast!?
A ratio of 2 complete against 1 part is would be way better, you could even enforce it or make it even more strict - 1:4, 1:5 - you name it...
0

#6 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 02 April 2006 - 07:41 PM

Added 'complete file push' when you download more than you upload and complete files make up to less than 50% of your upload.
	// BEGIN netfinity: Smart Part File Push
	if ( currequpfile->IsPartFile() )
	{
  if ( nGlobalUpDatarate2hr > nGlobalDownDatarate2hr && nGlobalUpDatarate2hr > (2 * nGlobalUpPartDatarate2hr) )
  filepriority *= 2;
	}
	else // For 1:1 ratio reasons!!! 
	{
  if ( nGlobalUpDatarate2hr < nGlobalDownDatarate2hr && nGlobalUpDatarate2hr < (2 * nGlobalUpPartDatarate2hr) )
  filepriority *= 2;
	}
	// END netfinity: Smart Part File Push

What I actualy want to do is to allow spending all my upload on a part file for a limited time period (1 hour). I have thought of taking file rarity into account as it's quite pointless pushing a file with 500 complete sources.

@tHeWiZaRdOfDoS
That algorithm would work too, although I wanted a more agressive but controled push.

@Stulle
As tHeWiZaRdOfDoS said, today we are getting punished for sharing many files. That is what I want to avoid.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#7 User is offline   leuk_he 

  • MorphXT team.
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5975
  • Joined: 11-August 04

Posted 02 April 2006 - 08:22 PM

netfinity, on Apr 2 2006, 08:41 PM, said:

else // For 1:1 ratio reasons!!!
{
  if ( nGlobalUpDatarate2hr < nGlobalDownDatarate2hr && nGlobalUpDatarate2hr < (2 * nGlobalUpPartDatarate2hr) )
  filepriority *= 2;
}
// END netfinity: Smart Part File Push[/code]
View Post


If you want to make a negative of the first rule the && should be || (or)

In normal words if you download more than you upload or you share less complete files than part files then push complete files? That should be a good reason to keep sharing complete files.
Download the MorphXT emule mod here: eMule Morph mod

Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.

Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
0

#8 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 02 April 2006 - 08:49 PM

@leuk_he
Actualy I meant &&, but you are right || would be a good idea! That would make eMule to stear more towards a 1:1 ratio.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#9 User is offline   BlueSonicBoy 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 396
  • Joined: 26-September 04

Posted 03 April 2006 - 12:11 AM

netfinity, on Apr 2 2006, 12:32 PM, said:

I'm currently working on a feature to solve the issue with part files getting to little upload when there are many complete files shared.

I would like your input on how to make the feature better and non-abusable.

View Post



That sounds interesting...

What about adding a completed sources check so you boost files which are more rare?
I don't know what value to set 8 was just a guess... :flowers:

CUpDownClient::GetFilePrioAsNumber() said:

// BEGIN netfinity: Smart Part File Push
if ( currequpfile->IsPartFile() )
{
  if ( currequpfile->m_nCompleteSourcesCount < 8  &&  nGlobalUpDatarate2hr > nGlobalDownDatarate2hr && nGlobalUpDatarate2hr > (2 * nGlobalUpPartDatarate2hr) )
  filepriority *= 2;
}
// END netfinity: Smart Part File Push

0

#10 User is offline   bscabral 

  • "Dream" member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 339
  • Joined: 25-December 02

Posted 03 April 2006 - 12:37 AM

BlueSonicBoy,
IMHO , i preffer witout complete source count, or with a higher (like < 50) I found a lot of files with ~10 files, with a Average queue about 5000

The logic is, if someone share a rare file, she share a lot of files and never delete his files. And because of this, have a bigger queue
IPB Image
IPB Image
0

#11 User is offline   BlueSonicBoy 

  • Magnificent Member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 396
  • Joined: 26-September 04

Posted 03 April 2006 - 02:48 AM

bscabral, on Apr 2 2006, 07:37 PM, said:

BlueSonicBoy,
IMHO , i  preffer witout complete source count, or with a higher (like < 50) I found a lot of files with ~10 files, with a Average queue about 5000

The logic is, if someone share a rare file, she share a lot of files and never delete his files. And because of this, have a bigger queue
View Post


The value 8 was just a guess/example/place holder, I have no reason to say this value is good. :flowers:
The only reason I suggest a count, is otherwise people downloading very popular files with many complete sources will have these files pushed over anything else they are sharing with no benefit to the network.

This would be less severer, (again 8 is there just as a place keeper, to be replaced with an appropriate value.)

CUpDownClient::GetFilePrioAsNumber() said:

// BEGIN netfinity: Smart Part File Push
if ( currequpfile->IsPartFile() )
{
  if (  nGlobalUpDatarate2hr > nGlobalDownDatarate2hr && nGlobalUpDatarate2hr > (2 * nGlobalUpPartDatarate2hr) )
        {
          filepriority *= 2;
          if( currequpfile->m_nCompleteSourcesCount < 8 )  filepriority += 8 - currequpfile->m_nCompleteSourcesCount; 
          }
}
// END netfinity: Smart Part File Push


This way rare files just get pushed over popular files, rather than only rare files getting pushed.

This post has been edited by BlueSonicBoy: 03 April 2006 - 02:49 AM

0

#12 User is offline   Xman1 

  • Xtreme Modder
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1955
  • Joined: 21-June 03

Posted 03 April 2006 - 06:28 AM

I think I said it thousend times... but once more: I prefer a push complete file-feature, because partfiles are already pushed by the network-desigen. (simply because you are found as source of a partfile immediately, but as source of completefile it can take hours).

But I understand why there are people complaining that complete-files are shared too much.
You have to make a difference between complete file and complete file.
a ) a complete file which was a partfile and is finished during the session
b ) a complete file which is already complete when sessions starts

for a you are known as source and netfinitys feature make sence.
for b it would be very bad to reduce the upload of this files... as I already said: it can take hours until your are found as source.

And keep in mind: the file can be a very rare file and you can be the only complete source. The automatic high uploadpriority doesn't push the file too strong. Punishing this file would be bad for the nextwork.
0

#13 User is offline   niRRity 

  • Avid Post Editor
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1229
  • Joined: 28-January 03

Posted 03 April 2006 - 08:04 AM

I agree with Xman on this, complete files should be pushed. Not part files...
0

#14 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 03 April 2006 - 09:22 AM

Well the problem is the creditsystem...
You push clients that upload to you, but those are always clients you want something from (=partfiles) - a client with a full file can't profit from the creditsys (at least not immediately) so why should he share?
He would be far better off to remove all shares, upload only parts and gain credits.
Of course one should push clients who upload full files but the real sharers also rarely download and the default users (let's call them leechers) usually only care about the download speed, not about how good their files are spreaded.
I already included a different weighting into my analyzer score for partfiles and fullfiles but as long as it isn't widely used and acknowledged the whole thing does not make any sense.


And the problem that netfinity encountered is the same for all guys with really big uploadspeed: they share and share and share and almost never get anything back! it's clear that you want to download fast (if you ever download) and you can't achieve that with the current system.
0

#15 User is offline   loulach 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 81
  • Joined: 29-December 05

Posted 03 April 2006 - 09:50 AM

The best in this case is the one pointed by bluesonicboy to push only in CompleteSourcesCount "value" but without push when too high complete source.

Loulach.
0

#16 User is offline   niRRity 

  • Avid Post Editor
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1229
  • Joined: 28-January 03

Posted 03 April 2006 - 03:12 PM

tHeWiZaRdOfDoS, on Apr 3 2006, 11:22 AM, said:

Of course one should push clients who upload full files but the real sharers also rarely download and the default users (let's call them leechers) usually only care about the download speed, not about how good their files are spreaded.
View Post


I don't think that's true. Most users share complete files AND download at the same time.The problem is that once a user understands how the CS works he realized that he should unshare complete files to gain advantage (credit shaping).
0

#17 User is offline   CiccioBastardo 

  • Doomsday Executor
  • PipPipPipPipPipPipPip
  • Group: Italian Moderators
  • Posts: 5541
  • Joined: 22-November 03

Posted 03 April 2006 - 03:38 PM

Why not using a shareratio parameter to know how much to push a file?
I mean, you make a shareratio between the size of the received data with respect to that you have uploaded.
The first parameter is capped at the file size once you have completed the download.
The second one keeps on growing as time passes. This means the ratio will decrease as a file is uploaded thus giving an advantage in the cases of partial (you try to payback to the network, independently to who you are uploading to exactly) and new file in general.
This parameter can be associated with taking care of the number of complete sources and other limtations to avoid a file priority to drop too much and so never being uploaded anymore (or too rarely).

Other kind of flags could be applied to allow the user to disable such a opinable auto-priority for each file.

Would this have a too extreme impact on file distribution?
The problem is not the client, it's the user
0

#18 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 03 April 2006 - 04:11 PM

Some ideas to modifier/limits I got from your posts

1. Average remote queue length / sources (threshold?)

2. Only allow part file push for one hour worth of upload
eg. uploaded_file < 3600 * upload_rate

One thing I thought of is to measure the part/complete file upload on a 7 day period instead. Then I can use a much lower threshold and still get a significant boost for occasional downloads.

As one of these people that only download approx. 2 hours per 16 hour period I have eMule turned on, I know that using fixed byte count thresholds are useless as they either always trigger or never. Thats why I like dynamic thresholds.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#19 User is offline   fabtar 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 880
  • Joined: 14-March 04

Posted 05 April 2006 - 09:59 AM

tHeWiZaRdOfDoS, on Apr 3 2006, 09:22 AM, said:

Well the problem is the creditsystem...

I already included a different weighting into my analyzer score for partfiles and fullfiles but as long as it isn't widely used and acknowledged the whole thing does not make any sense.

I agree about curent CS being the problem.
Your CS weightting seems a good idea.
But could an uploading client declare(to the server or kad or other way) having full file to gain advantage from your weighting?
Is your CS solution exploitable in this way? (OT: excuse me for the usage of "exploitable" but I simply think this is the right word :-( , personally I appreciate your contributions).

I have thought about this:
A client (Guyclient) is uploading "file A" to you, if the client hasn't queued in for that file in the last day (proof -> there is a high chance "guyclient" has complete "file A" and so he has no need to request to us) a very high multplier is applied for credit computation.
Trading won't be conveninet and sharing will be rewarded.

Hope you agree.
0

#20 User is offline   bscabral 

  • "Dream" member
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 339
  • Joined: 25-December 02

Posted 05 April 2006 - 03:05 PM

fabtar,
i like the idea of you won credits for anothers files, to make people share not trade.
Maybe a credit system that give a bigger credit for share a a light for trade
IPB Image
IPB Image
0

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users