[DatHebIkWeer]

I understand that an upload request is sent to a potential uploader at regular time intervals.
But if we assume the peer has queued us, why should we bother doing that?
This second request will in some cases cause us to be put back to the end of the queue so would be very disadvantageous.
The only time this would help us would be if the peer closed down and restarted, or in some other way deleted us from the queue.

Obviously there is never a good reason to delete a requester from the queue. If you do it after say, an hour you delete the ones who have waited a long time and should instead be helped with greater priority. If the requester died you will find out when the download permission is sent, and that will be soon enough.

So a requeuing request should only be sent if you are not in the queue anymore. Obviously to decide on that you need information. Therefore the next request should be replaced by a queue status request. Instead of requeuing the client should just confirm the reservation, and only if we are not queued anymore send a new request.

In that status update the peer can also renew the information about the available chunks.

[tHeWiZaRdOfDoS]

DatHebIkWeer, on 23 July 2012 - 11:15 AM, said:

But if we assume the peer has queued us, why should we bother doing that?
This second request will in some cases cause us to be put back to the end of the queue so would be very disadvantageous.

...

So a requeuing request should only be sent if you are not in the queue anymore. Obviously to decide on that you need information. Therefore the next request should be replaced by a queue status request. Instead of requeuing the client should just confirm the reservation, and only if we are not queued anymore send a new request.

In that status update the peer can also renew the information about the available chunks.

Why would you be put back to the end of the queue because of a reask?
You describe what eMule does already and I suggest that you try to read the source code if you have some coding knowledge or to write down your specific problem situations that you encountered so they can be further analyzed.

[Link64]

DatHebIkWeer, on 23 July 2012 - 11:15 AM, said:

This second request will in some cases cause us to be put back to the end of the queue so would be very disadvantageous.

We can get banned, if we send a request too often, but that should not happen with the official client or any "good" mod.

DatHebIkWeer, on 23 July 2012 - 11:15 AM, said:

Obviously there is never a good reason to delete a requester from the queue. If you do it after say, an hour you delete the ones who have waited a long time and should instead be helped with greater priority. If the requester died you will find out when the download permission is sent, and that will be soon enough.

Removing "dead" clients is useful in case of full queue, if you don't remove them, you can't insert new clients, who actually want something from you, so not removing the "dead" clients means less available sources for the "living" ones.

Also on hosts with quite long queue I could imagine, that the percentage of dead clients could be so high, that eventually at some points not the entire upload capacity would be used, simply because eMule would be trying to connect to many of them at once. I don't know what timeout eMule uses when connecting to a downloader, but the usuall HTTP timeout is 300 seconds, so if eMule uses the same or something close to that, it could collect quite many such downloaders in the active uploads list.

DatHebIkWeer, on 23 July 2012 - 11:15 AM, said:

So a requeuing request should only be sent if you are not in the queue anymore. Obviously to decide on that you need information. Therefore the next request should be replaced by a queue status request. Instead of requeuing the client should just confirm the reservation, and only if we are not queued anymore send a new request.

In that status update the peer can also renew the information about the available chunks.

And what great difference would it be between the current request and your "queue status request"? EDIT: like tHeWiZaRdOfDoS said, eMule is doing already what you request.

This post has been edited by Link64: 23 July 2012 - 09:59 AM

[DatHebIkWeer]

You 2 gave me quite a lot to reply to. But I hope I can clarify a bit.

tHeWiZaRdOfDoS, on 23 July 2012 - 10:33 AM, said:

Why would you be put back to the end of the queue because of a reask?
You describe what eMule does already and I suggest that you try to read the source code if you have some coding knowledge or to write down your specific problem situations that you encountered so they can be further analyzed.

My info may be outdated. I got it from this forum. I believe it was a description of what Stulles mod did in 2004. He said explicitly that on a new request the client would be placed back to the end of the queue.
I do have some programming knowledge. But not enough to easily read into a comprehensive program like eMule. I may try to do that one day though. I may end up making my own mod when I start doing that. And that is something I actually do not seek to do.
I rather have the official eMule and good mods use my suggestions. After all to make a meaningfull mod I would also have to look into other mods and maybe use some good functionality of theirs.
For instance if your mod already has a ‘bad uploader” blocking system that works then I would do better using that than making it myself. That would save work for me and would automatically introduce compatibility of standards.

tHeWiZaRdOfDoS, on 23 July 2012 - 10:33 AM, said:

knowledge or to write down your specific problem situations that you encountered so they can be further analyzed.

As we already discussed in the Mods section ( http://forum.emule-p...howtopic=155269 ) I am currently running 2 instances of eMule at the same time.
This seems to go quite good, except for some performance problems. Something is eating up all my memory when I run eMule. I wasn’t planning to bother people on here with that yet, since I don’t know if it is eMule or (for instance) my virus scanner going berserk on the partial downloads.
It keeps going quite stably as long as I do not touch the sort options in the transfers window. As long as I keep it sorted on a low to no list update option (like sort by filename) everything is fine. But if I try to sort by high list update options (like sort by remaining) my computer may run out of memory and just stall.

In 1 instance I run my “normal” list of shared files and downloads (currently about 500, but it used to be more than 1000 a while ago). In the other I run a download of a subset of 11 files that I want to give extra attention to. These 11 files also appear in my normal downloads list.

This configuration gives me the opportunity to watch events from 2 sides at a time.
Because my instances do not “know” they are on the same computer so they just see each other as remote peers. That means that occasionally I upload to myself. I befriended myself both ways and gave myself friend slots. Because I want myself to synchronize as good as possible to avoid unnecessary double downloads I would want to have them exchange downloaded chunks as often as possible.

Because the network traffic between me and myself does not go over the internet but is returned right on my router the transfer speed can be much higher. I reached transfer speeds of more than 1,5 MB/s at times. But that does not really help me, because of the waiting times. Every exchange always takes at least a half hour (maybe even an hour) regardless of the transfer speed, just because of the waiting time. The fact that eMule totally ignores the file download/upload priority settings does not help either. It basically means that I will (half) synchronize a file and then have to wait till the next turn to synchronize another file. But in the mean time the first file has changed so, instead of synchronizing the much needed other file, it will decide to use it’s turn to transfer the 1 freshly downloaded chunk from the first file. That way some files are quite up to date, but others haven’t been synchronized all weekend.
In this special case I see it happening from 2 sides. In normal cases exactly the same thing happens but it is not apparent to the user because he does not see the whole picture.
On the 11 file instance there is no upload queue. So there really is no need to have to wait for the next transfer, at least not in that direction.

I have to remember to request a file merge option that can merge a external part file into another part file inside eMule. I may make a separate program for that myself though, as practice before I start making my own mod.

Link64, on 23 July 2012 - 10:52 AM, said:

Removing "dead" clients is useful in case of full queue, if you don't remove them, you can't insert new clients, who actually want something from you, so not removing the "dead" clients means less available sources for the "living" ones.

Maybe the whole queue management thing will have to be thoroughly reviewed.
Maybe an hour is just too short in the life of an eMule client.
How many kilometres can a slag go in an hour? How many km does he go if he can move exactly 9,28 cm, then bumps into a barrier, then has to wait 30 minutes till he can request the barrier to be removed, than has to wait till it is his turn to have it actually removed, then move 9,28 cm to run into the next barrier, and so on? Not many.
But how many can a hare move if he can walk 9,28cm to the next barrier, , then has to wait 30 minutes till he can request the barrier to be removed, than has to wait till it is his turn to have it actually removed, then move 9,28 cm to run into the next barrier, and so on? Not many more.

eMule built in needless slack that makes everything slow. That is the reason why I requested this http://forum.emule-p...howtopic=155288 . That means that not much generally happens in an hour. It is very likely that a file of some size will still be needed an hour from now. eMule is an application that is only useful if you let it run for a considerable period of time. So the likelihood of a download request being outdated in just 1 hour is fairly moderate.

If the client is still live but has no need of the chunks anymore it can just politely thank the uploader and immediately return a denial. So the download slot will immediately be available to the next in line.

The only situation where that is impossible is when the requester closed down.
How much is the likelihood that that happens? That is an easy one.
I have my computer running day and night all week. I usually close it down for an hour on Saturday, and for an hour on Sunday. A week has 24 x 7 = 168 hours in it, so I am live on eMule 166 hours, of which 2 hours are just before close down. 1/84 of my requests are therefore useless after 1 hour.
Normal users may reboot maybe once per day, Which makes 1/24 of their requests done less than 1 hour before shut down.
Others may be on for 4 hours.
We have to estimate the average time a client will be online. From that we can deduct the likelihood of a dead requester after a set amount of time, and from that we can estimate the average number of dead requesters occupying a download slot at any given time.

Dead requesters will actually only occupy a very limited amount of network capacity. Their will not be a lot of traffic in the timeout period. They will leave their quota of upload capacity to the other downloaders, so there is nothing wasted.

Of course dead requesters will not be in the queue forever. They will be automatically removed when the uploader shuts down. And on average the uploader will shut down exactly as often as the downloader does. So requests will on average always be removed in a reasonable time period. Even without eMule actively scheduling that.

Link64, on 23 July 2012 - 10:52 AM, said:

. I don't know what timeout eMule uses when connecting to a downloader, but the usuall HTTP timeout is 300 seconds, so if eMule uses the same or something close to that, it could collect quite many such downloaders in the active uploads list.

Of course there can be freak exception situations, but usually there will not be many dead requesters high in the queue at any given moment. Regardless of the length of the queue, it will always tend to be a certain average.

Link64, on 23 July 2012 - 10:52 AM, said:

And what great difference would it be between the current request and your "queue status request"? EDIT: like tHeWiZaRdOfDoS said, eMule is doing already what you request.

It can be shorter, and will have no bearing on the queue, so it can be handled much faster. But it can be the same request, that is true, essentially only the handling would differ a bit. (Or not, if what you say is true.)

tHeWiZaRdOfDoS, on 23 July 2012 - 10:33 AM, said:

Removing "dead" clients is useful in case of full queue, if you don't remove them, you can't insert new clients, who actually want something from you, so not removing the "dead" clients means less available sources for the "living" ones.

The length of the queue is directly proportional to the number of files you share. If you have a small computer and a slow internet connection it is useless to share many files. So this problem will arise on relatively powerful computers with a fast connection. They can easily increase the queue length.
And how long do those queues actually get?
I have 500 files shared and my queue is 400 long. I used to have the queue size in the settings on the max, but when I realized how many requests were actually queued I moved it back to minimum. A lot of those files are dead, they are left over from an earlier period when I used eMule.
On my 11 files no more than 7 clients are actually downloading and there is no queue. And those files are all very much alive (though rare). There used to be 15 downloaders at a time a few days ago, but I set my upload very high, so they don’t need it anymore.

I would not be surprised if the average length of a queue would not be more than 2 x the number of shared files.

[Link64]

DatHebIkWeer, on 23 July 2012 - 03:06 PM, said:

In 1 instance I run my “normal” list of shared files and downloads (currently about 500, but it used to be more than 1000 a while ago). In the other I run a download of a subset of 11 files that I want to give extra attention to. These 11 files also appear in my normal downloads list. (...)

I don't get what you are trying to do there, but you are for sure wasting other people's upload.

DatHebIkWeer, on 23 July 2012 - 03:06 PM, said:

This configuration gives me the opportunity to watch events from 2 sides at a time.

You mean "This configuration gives me the opportunity to watch events from 2 sides get two upload/waiting slots from one source at a time."?

DatHebIkWeer, on 23 July 2012 - 03:06 PM, said:

I have to remember to request a file merge option that can merge a external part file into another part file inside eMule. I may make a separate program for that myself though, as practice before I start making my own mod.

I requested something like that, but within one eMule instance for to save some of other people's upload.

Anyway, if your issues only occur because of running two instances of eMule downloading the same files, I'd suggest you simply use just one eMule like (almost) everybody else. Or download different files in those two instances, than at least you won't waste other people's upload bandwidth.

[DatHebIkWeer]

Link64, on 23 July 2012 - 03:45 PM, said:

I don't get what you are trying to do there, but you are for sure wasting other people's upload.

And to reduce that upload I want it to synchronize quickly. But in this configuration I replace other people’s upload with my own. My upload speed on my “11 files” instance is now about 6x higher than my download.
In thread http://forum.emule-p...howtopic=155269 I described what I am trying to do. It is off topic in this thread but summarized it is: neutralizing a rogue uploader by just uploading good chunks faster. The rogue uploader makes it hard for me and others to get the good chunks, but as soon as I have them I make sure everybody has them. This can only be done with a second instance, because in my normal setup the upload bandwidth is shared among a lot of files and these particular files would not really benefit from it.

Link64, on 23 July 2012 - 03:45 PM, said:

I requested something like that, but within one eMule instance for to save some of other people's upload.

Anyway, if your issues only occur because of running two instances of eMule downloading the same files, I'd suggest you simply use just one eMule like (almost) everybody else. Or download different files in those two instances, than at least you won't waste other people's upload bandwidth.

They do not occur because of that, I just notice them because of that. Except for the performance problems of course. The high memory usage is there also when I have only 1 instance running, but in that case it won’t crash my computer.
I plan to return to using just 1 instance when this is over. That may be quite soon. It is working quite well.

I would say your request would not really work only regarding the full file hash, since a similar hash does not necessarily mean a similar file. But if the file is essentially the same the hash of most blocks will be identical. In that case it would be possible. But tricky.
Maybe something that the machine cann’t decide, but the user can. I will answer that further in your thread. That will make it clearer.

[Link64]

DatHebIkWeer, on 23 July 2012 - 05:12 PM, said:

neutralizing a rogue uploader by just uploading good chunks faster. The rogue uploader makes it hard for me and others to get the good chunks, but as soon as I have them I make sure everybody has them. This can only be done with a second instance, because in my normal setup the upload bandwidth is shared among a lot of files and these particular files would not really benefit from it.

I see... however long-term it does not change anything, the rogue uploader will probably be still there once you and most other good sources are gone (if he is doing that intentionally).

DatHebIkWeer, on 23 July 2012 - 05:12 PM, said:

I would say your request would not really work only regarding the full file hash, since a similar hash does not necessarily mean a similar file. But if the file is essentially the same the hash of most blocks will be identical. In that case it would be possible.

Not hash, hash set. With file hash only it doesn't work at all.

This post has been edited by Link64: 23 July 2012 - 04:34 PM

[DatHebIkWeer]

Link64, on 23 July 2012 - 03:45 PM, said:

You mean "This configuration gives me the opportunity to watch events from 2 sides get two upload/waiting slots from one source at a time."?

It’s off topic here but just for the record and to keep my name clear:
This is not a very smart strategy to get downloads faster.
It doesn’t only waste other people’s upload capacity but also my own download capacity. There will always be double downloads. Even if there is an excellent synchronisation. You will be competing with yourself in queues. If the file is rare that will mean that 1 of your downloads will cannibalize on the other. The source will have a limited upload speed and you share it with yourself to download something twice. Not smart.
It takes a lot of managing and monitoring to keep downloads going and to assign upload speed for synchronisations. So I wouldn’t recommend this for your regular downloads.
I do this not to speed up my own download. I don’t really care how much I have to wait for these particular files. I don’t even really want them. They are something my niece likes (yes, it’s not porn) but I don’t like very much. And she doesn’t know I am downloading it, so…
It’s a matter of honour now.
It does speed up other peoples downloads though. Of course because they get everything I have sooner.

A Good tip to speed up your downloads:

set your own upload speed higher

This will cause other people to have the chunks you share fast. That means they don’t have to waste valuable upload sources you need, to get chunks you already have. The whole effort will shift to the chunks you still need, and that you want other people to share with you. It may look like you will get behind on others because they finish earlier by your help. But they will have their hands free to help you out because of it.

Link64, on 23 July 2012 - 05:33 PM, said:

I see... however long-term it does not change anything, the rogue uploader will probably be still there once you and most other good sources are gone (if he is doing that intentionally).

If it is intentionally maybe. But:

It will not always be intentional. Though I see 2 bad uploaders here I have the impression they are both just using a wrong file. After I “killed” 1 bad file tonight (by giving it the last chunk so it had to hash) I also uploaded almost everything I had to the same person so he has a good file now. This morning he took away the other bad file. Now he is uploading good new chunks starting from the file I gave him. So maybe he (she) is not too bad after all.

The 2 original sources of these files are very slow and most other downloaders are very slow uploaders too. The “rogue” uploaders kept them at bay because of that. When more good sources become available the impact of a bad source will diminish. I may have to keep my files shared for the coming period, but if the files catch on the problems may be over.

If intentional the uploaders have to get smarter. The good guys have the advantage here, because the bad chunks do not spread. The easiest way to neutralize them is to let them finish the file. The more chunks a rogue uploader wants to give us, the closer he has to get to finishing the file. So the closer he is to decapitation.
I have some ideas how to be a smarter rogue myself. But I am not going to tell them here of course. (Might give people ideas.)

This post has been edited by DatHebIkWeer: 26 July 2012 - 10:25 AM

[Link64]

DatHebIkWeer, on 26 July 2012 - 11:59 AM, said:

It’s off topic here but just for the record and to keep my name clear:
This is not a very smart strategy to get downloads faster.

You cleared your name with the explanation above .

DatHebIkWeer, on 26 July 2012 - 11:59 AM, said:

A Good tip to speed up your downloads:

set your own upload speed higher

This will cause other people to have the chunks you share fast. That means they don’t have to waste valuable upload sources you need, to get chunks you already have. The whole effort will shift to the chunks you still need, and that you want other people to share with you. It may look like you will get behind on others because they finish earlier by your help. But they will have their hands free to help you out because of it.

That's my strategy too, but it's hard to convince others to do that. A lot of people still think there's no need to upload more than 10KB/s.

DatHebIkWeer, on 26 July 2012 - 11:59 AM, said:

It will not always be intentional. Though I see 2 bad uploaders here I have the impression they are both just using a wrong file. After I “killed” 1 bad file tonight (by giving it the last chunk so it had to hash) I also uploaded almost everything I had to the same person so he has a good file now. This morning he took away the other bad file. Now he is uploading good new chunks starting from the file I gave him. So maybe he (she) is not too bad after all.

Hmm... looks like this chunk became corrupted after the first hashing when it was completed. Maybe eMule should check completed chunks and also shared files once in a while.