[bugsan]

Currently we have a max-sources/file system.
this means if i have set max sources to 300, and i have 50 files, ideally eMule can found 15000 sources (i've said ideally).
If i have only 1 file in my list, i will change the limit to 2000 or 3000 to increase sources number (if this file have 2000 sources, it's the case for many files), because 300 is not enough.

So, the problem is : why i should have to change this setting myself, it could be done by an algorithm

I suggest max-sources should be the limit for total found sources. let say 5000, emule will search for sources until it reaches 5000, with a good distribution between all files.

This post has been edited by bugsan: 13 October 2005 - 12:07 PM

[niRRity]

You are right of course, and this feature exists in verious eMule mods. It was also requested a few times in the past and I never heard an official respose.

[SlugFiller]

What do you think max connections is for?

[leuk_he]

SlugFiller, on Oct 13 2005, 04:50 PM, said:

What do you think max connections is for?

i think the topic starter is talking about max sources not max connections.

The max sources setting is PER FILE, not a global one.

[Devil Doll]

I've been supporting this idea (total sources limit instead of hardlimit per file) since day 1 of my eMule experience, as I always considered the hardlimit "one size fits all" concept practically unusable. Hardlimit configuration errors are amongst the most popular support cases and have always been.

A total sources limit (with a default value of 3000, for those many cheap routers limited to a mere 256 simultaneous connections) instead of some random hardlimit that has no clue of the total number of downloads would be a great improvement for newbies. The eMule assistant doesn't take care of ever changing download situations, and the task of rebalancing the configuration every week or so clearly asks too much of many users.

This post has been edited by Devil Doll: 13 October 2005 - 07:34 PM

[SlugFiller]

There is no real difference between max sources and max connections.
The only difference between the settings is per file vs global.
Actually, the reason for the per-file one is that the global tends to "source-focus", rather than spread evenly. This results in some files having thousands of sources, while others having almost nothing.

[niRRity]

SlugFiller, on Oct 13 2005, 07:40 PM, said:

There is no real difference between max sources and max connections.
The only difference between the settings is per file vs global.
Actually, the reason for the per-file one is that the global tends to "source-focus", rather than spread evenly. This results in some files having thousands of sources, while others having almost nothing.

Nonsense

[Devil Doll]

SlugFiller, on Oct 13 2005, 08:40 PM, said:

There is no real difference between max sources and max connections.

Then why is there a hardlimit at all, if it can be derived from the number of connections?

SlugFiller, on Oct 13 2005, 08:40 PM, said:

The only difference between the settings is per file vs global.

That's exactly what we're talking about. Per file doesn't work because the number of files is changing but noone adapts the per-file configuration every time.

SlugFiller, on Oct 13 2005, 08:40 PM, said:

Actually, the reason for the per-file one is that the global tends to "source-focus", rather than spread evenly. This results in some files having thousands of sources, while others having almost nothing.

Let's say we never limit sources per file to fewer than 10 (as to boost rare files...) but beyond that use the global sources limit to proportionally assign sources to files based on the relative availability of sources, something we do know already (complete sources are shown on the "shares" page, incomplete sources are transferred via client source exchange).

We download 15 files, whose number of maximum available sources were 5, 5, 10, 15, 25, 40, 50, 100, 150, 200, 300, 500, 800, 1200, 1500. We set a global hardlimit of 3000 sources but we could have found 4900. So let's first assign 10 sources to every file (we'll only get 90 sources this way), and then divide (3000 - 90) / (4900 - 90) = 0,605, which we now use as ratio to scale down the sources above 10 for each file. Adapt this assignment once every 6-12 hours and the system will become a lot more stable than using a static hardlimit of 500 or whatever.
It would be easy to always use 3000 sources this way while it's impossible to achieve the same with any static hardlimit configuration - either you unnecessarily ignore sources, or you overrun your weak router.

To add more comfort to the user, make the "rare file protection parameter" configurable (with a minimum accepted value of 10), and you're done.

And if you still believe that sources for rare files would be underrepresented, then make the scaling formula proportional to the square root of available sources. I don't actually care how it is done - I just want to get a maximum of sources with a minimum of configuration effort for every user.

By the way, you could then still add relative download priorities that would halve resp. double the scaling factor for any given file. It's really easy.

This post has been edited by Devil Doll: 13 October 2005 - 08:51 PM

[SlugFiller]

Quote

Then why is there a hardlimit at all, if it can be derived from the number of connections?

Like I've already said, because sources tended to go all to the same file.

It's obviously not used as some overhead-preventing measure, or at least it wasn't created as such. I was around the time this little tid-bit was invented, and it was because a global limit caused some files to get all the sources while others didn't.

Quote

We set a global hardlimit of 3000 sources but we could have found 4900. So let's first assign 10 sources to every file

This isn't bandwidth, you can't "assign" sources to files.
Sources arrive, and you either accept or refuse them.

Accepting all would leave you without room.
Refusing to keep even-ness would have a problem if one of your files has very few sources. You can't tell apart a file with few or no sources from a file that simply hasn't found it's sources yet. How do you decide if to accept extra sources for other files?

Even if you consider dropping sources for even-ness, you still have an issue with picking a source to drop. You could end up dropping "the good source". There's no real measure for which is which.

An alternative to the per-file setting is simply having the per-file auto-set to the average of global/num_files. Though, I fail to see the big advantage from that.

[bugsan]

Quote

it was because a global limit caused some files to get all the sources while others didn't.

With a bad algorithm, it's true.
i'm sure you've encountered this problem : there are already 3000 found sources, and you add a new file with 500 unchecked sources. what to do ? => get maths knowledge then come back.

[Devil Doll]

SlugFiller, on Oct 13 2005, 10:13 PM, said:

Quote

We set a global hardlimit of 3000 sources but we could have found 4900. So let's first assign 10 sources to every file

This isn't bandwidth, you can't "assign" sources to files.
Sources arrive, and you either accept or refuse them.

I'm not talking about accepting or refusing them. I'm just calculating individual hardlimits for each file in a way they add up to 3000, that's all. Everything else works as it does in the plain vanilla eMule. Talking in eMule protocol terms I'm never refusing any source, I just treat some of these as active and others as passive (see below).

The "rare files protection treshold" might even be infinite. Which would mean that the rare files would accept as many sources as possible while the more popular files would accept as many sources as the total sources limit allows them to. The algorithm is trivial (excuse me for sounding PERLish...):

1. From your source exchange information, derive the number of potential sources for all downloading files. If their total is smaller then the total_hardlimit then {boost all these values proportionally for each file_hardlimit [$downloading_file] by the factor total_hardlimit / total_observed; we're done already}; otherwise:
2. hardlimit = 0;
3. UNTIL total_hardlimit <= 0 {
3a. hardlimit++;
3b. foreach $downloading_file {
3ba. if potential_sources ($downloading_file) > hardlimit {
file_hardlimit [$downloading_file]++; total_hardlimit--;
}}}
(This isn't even an efficient implementation, it is just easy to understand.)

Potential sources are as above: 5, 5, 10, 15, 25, 40, 50, 100, 150, 200, 300, 500, 800, 1200, 1500.
Result of the algorithm should be: 5, 5, 10, 15, 25, 40, 50, 100, 150, 200, 300, 500, 534, 534, 534.
I don't see any tendency to boost popular files - in fact the opposite is true! The result would be exactly the same as if I configured a hardlimit of 534 manually - it's just that I would have to know this value beforehand. As you see I can compute it from the 4900 "observed" sources easily.

eMule does have to keep all 4900 sources in memory (I need their user hashes to identify duplicates) but it would only treat 3002 of these sources as "active", i. e. visible in the eMule windows, contacted every 28 minutes and such. The other 1898 sources are memorized but not used (wait - we can of course shift them into "active" status if we lose sources there, sparing us asking servers and/or other clients; preferably we'd activate those passive sources that are within a 60 minutes time span so that they never know they have been passive...). Client source exchange (active as well as passive) is done depending on the computed hardlimit per file, as it is done already in mods that support hardlimits per file (sivka-ish). So far no additional network overhead.

I'm aware that compared to the static 534 hardlimit this "observation" of the 4900 sources (actually setting their hardlimit to infinite for 30 minutes) would cause additional overhead only for those files that reached their computed hardlimit, as I would lose track of their potential sources if I stopped searching for these sources forever (I do know the potential sources for all other files as I didn't stop searching for these). That's why I suggested to make this check only every 12 hours or so, maybe even less frequently. Actually it doesn't matter whether my hardlimits are off the optimum value for a couple of hours, it's still better than if they're completely wrong for months! And note that only 3 of 15 files would require the additional investigation - obviously those with most sources, unfortunately. Dropping sources from memory if they're not validated by the next "observation" cycle applies of course, as not to let the "observed" sources number grow infinitely.

The same applies for changes of any type. Completed files, added files... I don't care. Let new downloads have some standard hardlimit like 200 or so, only to fix it during the next observation cycle. All that counts is that the hardlimit per file is reasonable for the largest part of this download's life span.

Also note that a configuration with a hardlimit of 1501 would have produced a lot more overhead, and perhaps even overrun my router. But that's exactly what newbies are doing as they don't understand how to reasonably configure their hardlimit. Give them a default value of 3000 sources total and they'll be happy.

SlugFiller, on Oct 13 2005, 10:13 PM, said:

Even if you consider dropping sources for even-ness, you still have an issue with picking a source to drop. You could end up dropping "the good source". There's no real measure for which is which.

Note that I never even mentioned the word "dropping"! All that I'm doing is calculating hardlimits per file. What does eMule do if I set the existing hardlimit to a lower value manually? It drops sources, right? Fine - so that's already implemented.

This post has been edited by Devil Doll: 14 October 2005 - 12:17 AM

[poutnik]

SlugFiller, on Oct 13 2005, 09:40 PM, said:

There is no real difference between max sources and max connections.
The only difference between the settings is per file vs global.
Actually, the reason for the per-file one is that the global tends to "source-focus", rather than spread evenly. This results in some files having thousands of sources, while others having almost nothing.

No difference or no relation ?
The truth is max connections is global and max sourses is per file.
But one connection serves more sources and not all connections do that.
If there is no difference how could hundreds of connections serve thousands of sources ?

[leuk_he]

poutnik, on Oct 14 2005, 07:13 AM, said:

SlugFiller, on Oct 13 2005, 09:40 PM, said:

There is no real difference between max sources and max connections.
The only difference between the settings is per file vs global.
Actually, the reason for the per-file one is that the global tends to "source-focus", rather than spread evenly. This results in some files having thousands of sources, while others having almost nothing.

No difference or no relation ?
The truth is max connections is global and max sourses is per file.
But one connection serves more sources and not all connections do that.
If there is no difference how could hundreds of connections serve thousands of sources ?

The number of connenctions is the number of connections that is used for sources reasks (if not done by udp) and the up (not sure) & downloads. So to maintian 1000 sources you need enough connections to maintain all reasks. if you set sources to 3000 and connections to 30 you cannot reask fast enough. However if i look at my statistics the number of connections goes seldomly over 100.

The number of sources per file is i think a way to reduce the overhead for popular files. there is even an soft limit of 750 sources per file buildinto the code. Also if you have more than 1500 sources per file emule takes up far more CPU. Nothing to worry about if your emule machine is running on a 3Ghz pc, but noticeable.

[Devil Doll]

Improvement to my algorithm specified above: Actually we don't need any additional observation mechanism at all!

The result of the calculation there was a hardlimit of 534 for all files. For those files that didn't get as many sources I don't need additional information as I never rejected any source for these; for those that reach the hardlimit I don't care how many additional sources there are as all of these would be passive anyway.
So all I need is to perform said computation regularly (say, every 10-30 minutes), compute the new hardlimit and apply it - with the exact same operation as if I'd manually edit the preferences entry. If the result were to increase the hardlimit then those files previously limited would start asking for client source exchange again; if the result were to reduce the hardlimit then some sources were to be put passive, but that's the same that eMule has to do already in case I manually set this hardlimit.
As a fine-tuning it might be a good idea to not restart client source exchange for files that already have 90-95% of their allowed sources (due to a small increase in the hardlimit or having lost a handful of sources), as these files (those with the most sources, remember!) would then cause a lot of client source exchange traffic for a mere handful of additional non-passive sources. Rather let them not use their full potential instead, and leave some space for rare files to gain more sources without triggering another hardlimit change (and thus a drop of sources for other files then). It need not be exactly 3000 sources all the time. If any such file falls below its 90% threshold it will once collect another 10% of sources, only to let them gradually drop to 90% of the hardlimit again.

Given this modification I wouldn't even need a concept of "passive sources" any more - then again it would improve the network speed, i. e. reduce overhead, to memorize sources for 60 minutes (while I can still call them back and remain in the source's queue) instead of dropping them and later using source exchange to find them again. So in the end the network overhead should even be lower than with the current implementation...

[poutnik]

leuk_he, on Oct 14 2005, 12:58 PM, said:

The number of connenctions .........

Thank you, Leuk_he, you have written my thoughts in more detailed way.

[MasterJunior]

damn thats a lot for me to read now but im sure ill say something like what ive asked for

http://forum.emule-p...showtopic=53404

[McGiver]

Well, I myself suggested some time ago a method a to do this, but not by limiting sources per file, given a total sources limit (let's say 3000).

IMHO, the better solution of this is simply starting and stopping files automatically, based on its individual sources number, and its priority. If we set the total sources hard limit of 3000, and give a 5% (by example) of margin to not start/stop continually the same files, we can spare emule of a lot of trouble, by not causing a headache to the computer & his connection. Also, we can limit the attempts of restart of a file to an hour or more, to limit the bulk of reasks to the network. If we can also "remember" the average number of sources of a given file, we can prevent continually restarting and stopping a file with 600 sources when we reach the lower limit of 2850 total sources (3000 - 5%), to immediately surpass by many the upper limit (2850 + 600 = 3450 > 3150 = 3000 + 5%), and restarting it only when the total number of sources drops below 3000 - 600 = 2400, to mantain a balanced download.

It can be fine tunned controlling too the number of "Too many connections", in a given time to control if it is a wise decission to add more files downloading or not. If so, we need to control the time from the start of emule or a given file (when all sources are marked as "Too many connections" until they are reached for the first time and asked if they are available).

Doing so, we will have all the sources reachable for every file downloading, without limiting "artificially" the sources per file, IMHO.

Mc

This post has been edited by McGiver: 17 October 2005 - 12:13 PM

[Devil Doll]

McGiver, on Oct 17 2005, 01:07 PM, said:

Well, I myself suggested some time ago a method a to do this, but not by limiting sources per file, given a total sources limit (let's say 3000).

But the limitation of sources per file makes sense because it allows to stop client source exchange for those files that reached the limit, and these are the files with most sources, i. e. most overhead traffic. That's why I am trying to use the mechanism as it is, just with a more user-friendly handling. I'm fine with hardlimit per file - I just want eMule to calculate a reasonable limit by itself.

McGiver, on Oct 17 2005, 01:07 PM, said:

It can be fine tunned controlling too the number of "Too many connections"

That's a development direction I would support as well - in many cases this value is directly linked to the number of sources. Unfortunately there are situations when reducing sources won't help - such as using an unpatched WinXP SP2 or a broken router. So that's a lot less reliable than you might hope.

[SlugFiller]

You'd have to describe your algorithm in more detail, specifically how it handles the following events:
-Hardlimit set/changed
-Found x sources for file y(x may be high for SX)
-Source dropped(dead or NNS)
-File x added.
-File x cancelled(all sources dropped).

[McGiver]

Devil Doll, on Oct 18 2005, 01:55 PM, said:

McGiver, on Oct 17 2005, 01:07 PM, said:

Well, I myself suggested some time ago a method a to do this, but not by limiting sources per file, given a total sources limit (let's say 3000).

But the limitation of sources per file makes sense because it allows to stop client source exchange for those files that reached the limit, and these are the files with most sources, i. e. most overhead traffic. That's why I am trying to use the mechanism as it is, just with a more user-friendly handling. I'm fine with hardlimit per file - I just want eMule to calculate a reasonable limit by itself.

Mmmhh, yes, you're right, I guess too that global and local hard limits can coexist, givin a more flexible approach to the handling of sources, whatever the system of reaching the global limit of sources is used, your proportional one (without letting out files with very few sources), and the one I suggested, starting and stopping files.

Devil Doll, on Oct 18 2005, 01:55 PM, said:

McGiver, on Oct 17 2005, 01:07 PM, said:

It can be fine tunned controlling too the number of "Too many connections"

That's a development direction I would support as well - in many cases this value is directly linked to the number of sources. Unfortunately there are situations when reducing sources won't help - such as using an unpatched WinXP SP2 or a broken router. So that's a lot less reliable than you might hope.

True, I know you're far more proficient than myself on communications, and you're right, the problem that you point is true. And in my system, running fine, there are always a number of sources with "Too many connections" (~0.5%). Anyhow, the idea of controling the number of those sources can be ignored, as soon as it is a "fine tune" of the main idea.

But, also, the cases that you describe are obviously bad computer configuration issues, whose users will end in this forum asking for help, or repairing the systems by themselves. Otherwise, and as exposed, my idea will end stopping all files, if we use it as a zero tolerance rule. But, if we make emule "remember" the number and/or ratio of "Too many connection" sources, we can show a message of error to the user (in the logs?) pointing to that subject, if we detect an anormal behavior as you described and stopping the "automatically start/stop" routine, to prevent stopping all the files downloading based on the number of "Too many connections" sources.

Mc