[Blame]

I have close on 20,000 files offered for upload.

After a lot of checking I have fixed TV Underground as my prefered Connection. It seems to be the most reliable.

But here is the thing. It still looses knowledge of my files on an apparently daily basis and then slowly reconnects over a period of hours. Or at least it appears so according to upload demand and my searching for my own files.

Most files are in the order of 800kb and are of low demand. Collectively the demand peaks at about 10-20 per hour.

So what is going wrong and can I improve the connection?

It could be that the servers just don't like massive collections of low demand files. One option would be to bundle them up into about 2000 zipped packages. However they would be difficult to label so as to distinguish them from the mass of poor quality and (recently) fake competitors. It would also increase useless downloading as their would be no reliable way to tell if a download contains anything new. It would increase downloads per package, so there is that.

[hooligan3000]

Quote

hardLimit (integer)
If a client tries to publish more than hardLimit files, the server disconnects him (before receiving the whole list). That is to save bandwidth, because some lazy people share all their files. Default value : 4000
See softLimit parameter description.

Quote

softLimit (integer)
If a client tries to publish more than softLimit files, the server sends him a WARNING message and ignores files in excess. Default value : 1000

a server doesnt index all of your files

[fox88]

The actual limits should be checked in servers list.
Currently some servers have huge limits, but it could be that these numbers do not really work.
Also KAD would index only a small part of 20000 files.

[Blame]

fox88, on 11 December 2018 - 08:34 PM, said:

The actual limits should be checked in servers list.
Currently some servers have huge limits, but it could be that these numbers do not really work.
Also KAD would index only a small part of 20000 files.

I do check and only use servers that claim a 100k soft limit. My guess now is that number is aspirational rather than binding.
Actually KAD seems to be doing better than the servers at times.

I am getting uploads that make me think they are all available, just not all the time. Sort of like the servers purging when overloaded.

This is a real problem with no easy solution. My files are all useful, and for their size highly valuable to those who want them. However I can see no mechanism within the system to determine that. To the servers they must look very much like trash and sadly there could easily be more junk than can be tolerated.

[pier4r]

fox88, on 11 December 2018 - 09:34 PM, said:

The actual limits should be checked in servers list.
Currently some servers have huge limits, but it could be that these numbers do not really work.
Also KAD would index only a small part of 20000 files.

If I remember correctly kad publishes around 500 files per hour. So in rotation, if other clients are often online (not necessarily 24/7 even should be enough 12/5) one will find the files although they are 20k.

Then again: emulecollections helps. Long time ago I thought, and I still think - although without working out the details due to other priorities - that the idea of emulecollections plus Kad could be somewhat composed to increase the reachability of many files. Of course there is a tradeoff, one can increase the reachability of spam as well. Although Since years emule users went down so much that the spam is almost non-existent. This is the advantage of not being the 1st p2p network worldwide, though being still quite robust.

[Blame]

pier4r

Not sure what you mean by emulecollections. Is that the files available when I search "Global (Servers)" ?

A Kad search doesn't on the whole deliver as many results as a server search although I think the quality a little better. Usually I do both.

Part of the problem is I am not sure how valid searching for my own files is. However I am confident from the upload pattern that file access is intermittent with popular files more often visible.

I am now beginning to suspect the pattern is this: Server looses most files, I panic and change server then change back, files come back over a period of perhaps 2 hours or so, and the next day most of the files are lost again.

This is not necessarily a bug. Network optimisation is no easy thing with the ideal being to react to overload in the most graceful way possible. Quietly removing less popular files after a few hours connection might be just that.

If I knew the optimal connection pattern it would help. Perhaps disconnecting and reconnecting once/twice a day might be best. Then again it could be servers reward persistence.

[pier4r]

If I were you I would be mostly patient. By experience I can tell you that thousands of files shared via servers or kad are eventually picked up (if someone is interested when you are connected, otherwise bad luck!).
Only you need to wait days and weeks. That is emule.

An emulecollection (you can even search for those files) is a list of edk2 hashes to "know" the fingerprint of all files listed in a collection (if you search in the forum you find many post about them).

[Blame]

pier4r, on 18 December 2018 - 08:05 PM, said:

An emulecollection (you can even search for those files) is a list of edk2 hashes to "know" the fingerprint of all files listed in a collection (if you search in the forum you find many post about them).

Ok. Finally worked out what emulecollection is. Good idea but not I think easy to find out about. I will certainly use it in my next release (probably January). However it will be an extra feature. I will assume most won't know to use it any more than I did.

Not entirely sure it will work for a list of 20,000 files. A quick check appeared to lock up Emule. Could be it would have completed the request after a few more mins or hours or perhaps a day. Could be it just crashed.

[pier4r]

Blame, on 20 December 2018 - 02:34 AM, said:

Ok. Finally worked out what emulecollection is. Good idea but not I think easy to find out about. I will certainly use it in my next release (probably January). However it will be an extra feature. I will assume most won't know to use it any more than I did.

Not entirely sure it will work for a list of 20,000 files. A quick check appeared to lock up Emule. Could be it would have completed the request after a few more mins or hours or perhaps a day. Could be it just crashed.

Yes unfrotunately the emulecollection is a powerful feature not advertised or used enough. I myself took years before appreciating it. It would make things easier, way better than an archive that is large and with unknown content for a while. Anyway the emule userbase, now and then, never used this enough. But some use them and it is nice.

Emulecollection are not meant, I guess, to be very large. I recently got a collection with 40k files and it works (it is 4 MB in size), only it takes a while to open or edit.


I would suggest to break the collection inf 5k or 10k files, then you can still do a collection of collections. Or I am quite sure that your files could be grouped. Say: video collection about X, video collection about Y, documents about Z, etc...

So far I the collctions I have are:
this one with close to 40K entries.
Another with 1k entries.

Many under 1K entries. As I said, I think that they work good up to 5k entries.

Also emulecollections are great because if one knows that the files are in the network, if the system crashes, with a backup of only emulecollections (and with an emulecollection of all ones files) one can start to get them from the network again.

This post has been edited by pier4r: 23 December 2018 - 08:04 PM

[Blame]

20k library now up a few days. ALL the files processed to gain about 25% average compression and thus got new hash tags. I am considering processing to a 2'nd format but it's days of my computer crawling under the load. Plus I would expect failed conversions to take up more time.

So far visibility of individual files looks very good and seems to be staying that way, which surprised me.

There are two differences. Last time I had the files in about 1600 sub directories and I updated or accessed them often. This time I have put them all in a single directory which I have not yet fiddled with. Fiddling is happening with another copy. Update release not planned till next spring. I am not saying that I have found a bug but perhaps the new plan stresses the system less.

The files are still in sub directories when distributed as a library. Just broken down into four 3GB downloads because I find really big downloads tend to go bad.

This post has been edited by Blame: 18 February 2019 - 02:20 PM

[Blame]

Nope lost visibility to all files again today. And, I think everybody else's. Came back after a few hours. New theory: The servers are overloading or just crashing and re-upping the hash files every few days. Can't surprise. They are a networking system and those things tend to need maintenance.

Oh. I have got Emulecollections to work with 20,000 files. No problem once I figured out the right way to do it. Be interesting to see if anybody downloads it.

This post has been edited by Blame: 19 February 2019 - 03:03 PM

[Blame]

Been stable the last few days. Could be just luck of the draw but I wonders. No loss since I set a few critical files to release priority. Including the Emulecollections file. It has been only downloaded twice but could it be somehow encouraging the server to hold my hashtags?

[pier4r]

No emulecollections do not encourage servers or the like. Plus they need "appealing" names (i.e: with easy to search keywords) to be downloaded. Or you need to share the ed2k link somewhere, as it is not illegal to share the link of a collection.

I have one and with the proper keywords it is downloaded every now and then. Whether the contents are interesting for the downloaders I don't know. I hope yes.