20K Of Files - Upload Links Keep Disappearing.
#1
Posted 11 December 2018 - 11:49 AM
After a lot of checking I have fixed TV Underground as my prefered Connection. It seems to be the most reliable.
But here is the thing. It still looses knowledge of my files on an apparently daily basis and then slowly reconnects over a period of hours. Or at least it appears so according to upload demand and my searching for my own files.
Most files are in the order of 800kb and are of low demand. Collectively the demand peaks at about 10-20 per hour.
So what is going wrong and can I improve the connection?
It could be that the servers just don't like massive collections of low demand files. One option would be to bundle them up into about 2000 zipped packages. However they would be difficult to label so as to distinguish them from the mass of poor quality and (recently) fake competitors. It would also increase useless downloading as their would be no reliable way to tell if a download contains anything new. It would increase downloads per package, so there is that.
#2
Posted 11 December 2018 - 04:13 PM
Quote
If a client tries to publish more than hardLimit files, the server disconnects him (before receiving the whole list). That is to save bandwidth, because some lazy people share all their files. Default value : 4000
See softLimit parameter description.
Quote
If a client tries to publish more than softLimit files, the server sends him a WARNING message and ignores files in excess. Default value : 1000
a server doesnt index all of your files
ed2k://|server|91.208.162.87|4232|/
ed2k://|server|85.239.33.123|4232|/
ed2k://|server|91.208.162.55|4232|/
SD - Telegram
Air VPN - The air to breathe the real Internet
BTC
bc1qdrk0ld07jtg99ym2zg68cpqhqj34qnf2txm93n
XMR
48ja6xJ2NyPMNzmY1pA3ZZPpX5yTaw9Ym28jrDPCL7Y7L7pr5wXFdpeK4WqBbvVY5qEa6VDfhFKTnHWef3EPC4zgQNTnAwg
#3
Posted 11 December 2018 - 08:34 PM
Currently some servers have huge limits, but it could be that these numbers do not really work.
Also KAD would index only a small part of 20000 files.
#4
Posted 12 December 2018 - 01:16 PM
fox88, on 11 December 2018 - 08:34 PM, said:
Currently some servers have huge limits, but it could be that these numbers do not really work.
Also KAD would index only a small part of 20000 files.
I do check and only use servers that claim a 100k soft limit. My guess now is that number is aspirational rather than binding.
Actually KAD seems to be doing better than the servers at times.
I am getting uploads that make me think they are all available, just not all the time. Sort of like the servers purging when overloaded.
This is a real problem with no easy solution. My files are all useful, and for their size highly valuable to those who want them. However I can see no mechanism within the system to determine that. To the servers they must look very much like trash and sadly there could easily be more junk than can be tolerated.
#5
Posted 16 December 2018 - 12:46 PM
fox88, on 11 December 2018 - 09:34 PM, said:
Currently some servers have huge limits, but it could be that these numbers do not really work.
Also KAD would index only a small part of 20000 files.
If I remember correctly kad publishes around 500 files per hour. So in rotation, if other clients are often online (not necessarily 24/7 even should be enough 12/5) one will find the files although they are 20k.
Then again: emulecollections helps. Long time ago I thought, and I still think - although without working out the details due to other priorities - that the idea of emulecollections plus Kad could be somewhat composed to increase the reachability of many files. Of course there is a tradeoff, one can increase the reachability of spam as well. Although Since years emule users went down so much that the spam is almost non-existent. This is the advantage of not being the 1st p2p network worldwide, though being still quite robust.
Extended signature: click.
#6
Posted 16 December 2018 - 07:52 PM
Not sure what you mean by emulecollections. Is that the files available when I search "Global (Servers)" ?
A Kad search doesn't on the whole deliver as many results as a server search although I think the quality a little better. Usually I do both.
Part of the problem is I am not sure how valid searching for my own files is. However I am confident from the upload pattern that file access is intermittent with popular files more often visible.
I am now beginning to suspect the pattern is this: Server looses most files, I panic and change server then change back, files come back over a period of perhaps 2 hours or so, and the next day most of the files are lost again.
This is not necessarily a bug. Network optimisation is no easy thing with the ideal being to react to overload in the most graceful way possible. Quietly removing less popular files after a few hours connection might be just that.
If I knew the optimal connection pattern it would help. Perhaps disconnecting and reconnecting once/twice a day might be best. Then again it could be servers reward persistence.
#7
Posted 18 December 2018 - 08:05 PM
Only you need to wait days and weeks. That is emule.
An emulecollection (you can even search for those files) is a list of edk2 hashes to "know" the fingerprint of all files listed in a collection (if you search in the forum you find many post about them).
Extended signature: click.
#8
Posted 20 December 2018 - 01:34 AM
pier4r, on 18 December 2018 - 08:05 PM, said:
Ok. Finally worked out what emulecollection is. Good idea but not I think easy to find out about. I will certainly use it in my next release (probably January). However it will be an extra feature. I will assume most won't know to use it any more than I did.
Not entirely sure it will work for a list of 20,000 files. A quick check appeared to lock up Emule. Could be it would have completed the request after a few more mins or hours or perhaps a day. Could be it just crashed.
#9
Posted 23 December 2018 - 07:54 PM
Blame, on 20 December 2018 - 02:34 AM, said:
Not entirely sure it will work for a list of 20,000 files. A quick check appeared to lock up Emule. Could be it would have completed the request after a few more mins or hours or perhaps a day. Could be it just crashed.
Yes unfrotunately the emulecollection is a powerful feature not advertised or used enough. I myself took years before appreciating it. It would make things easier, way better than an archive that is large and with unknown content for a while. Anyway the emule userbase, now and then, never used this enough. But some use them and it is nice.
Emulecollection are not meant, I guess, to be very large. I recently got a collection with 40k files and it works (it is 4 MB in size), only it takes a while to open or edit.
I would suggest to break the collection inf 5k or 10k files, then you can still do a collection of collections. Or I am quite sure that your files could be grouped. Say: video collection about X, video collection about Y, documents about Z, etc...
So far I the collctions I have are:
this one with close to 40K entries.
Another with 1k entries.
Many under 1K entries. As I said, I think that they work good up to 5k entries.
Also emulecollections are great because if one knows that the files are in the network, if the system crashes, with a backup of only emulecollections (and with an emulecollection of all ones files) one can start to get them from the network again.
This post has been edited by pier4r: 23 December 2018 - 08:04 PM
Extended signature: click.
#10
Posted 18 February 2019 - 02:14 PM
So far visibility of individual files looks very good and seems to be staying that way, which surprised me.
There are two differences. Last time I had the files in about 1600 sub directories and I updated or accessed them often. This time I have put them all in a single directory which I have not yet fiddled with. Fiddling is happening with another copy. Update release not planned till next spring. I am not saying that I have found a bug but perhaps the new plan stresses the system less.
The files are still in sub directories when distributed as a library. Just broken down into four 3GB downloads because I find really big downloads tend to go bad.
This post has been edited by Blame: 18 February 2019 - 02:20 PM
#11
Posted 19 February 2019 - 02:54 PM
Oh. I have got Emulecollections to work with 20,000 files. No problem once I figured out the right way to do it. Be interesting to see if anybody downloads it.
This post has been edited by Blame: 19 February 2019 - 03:03 PM
#12
Posted 07 March 2019 - 05:57 PM
#13
Posted 31 March 2019 - 09:25 PM
I have one and with the proper keywords it is downloaded every now and then. Whether the contents are interesting for the downloaders I don't know. I hope yes.
Extended signature: click.