[aleste81]

Long time user, usually sharing a few thousand files.

Recently I have begun first downloading then sharing 20 000 files, and I quickly noticed that eMule isn't capable of handling such numbers.

eMule is extremely sluggish since. Not the download/upload part, which is ok, but the overall delay between user mouse/keyboard input.

This is a problem, as Elon would say.

[BuyukBang]

aleste81, on 02 January 2024 - 11:08 PM, said:

Long time user, usually sharing a few thousand files.

Recently I have begun first downloading then sharing 20 000 files, and I quickly noticed that eMule isn't capable of handling such numbers.

eMule is extremely sluggish since. Not the download/upload part, which is ok, but the overall delay between user mouse/keyboard input.

This is a problem, as Elon would say.

Original/community realeses don't care much on sharing or downloading "big" number of files. There are some post from the developers explaining why, with their point of view.

So, I've decided to improve this part of eMule myself. I'm working on an eMule mod based on v0.70a (actually started with 0.60d but migrated to v0.70a code base) since 8 months on my free times and I think I'll need a few months to finish it before the realese. I've already completed the support for big number of files and now my mod is capable of big numbers without being sluggish on UI.

I've added a lot of other nice features, too. Some of them are from v0.50a mods while some are unique and developed by me for my needs. Below you can find screenshots of the preferences what I've already completed (all seen in these are fully tested and working stable). There are also a lot of features working under the hood without any trace on the preferences screenshots, like NAT-T (I decided to implement it as a "always on" feature without on/off toggle to make it widespread). I've also implemented a lot of performance improvements and some bug fixes, too.

There are a lot of other things in my To Do list. I really want to make it near perfect (in my point of view, ofc) before the release, so need more time.

It will be released as fully compliant to GPL (opensource, as it should be) and also to eMule community rules (no leeching features, community rewards etc).

Forum doesn't allow me to add links so i've removed https part of links:
i.ibb.co/JQJsLg6/1.png
i.ibb.co/Chm5M0z/2.png
i.ibb.co/JRvzJLc/3.png

This post has been edited by BuyukBang: 11 January 2024 - 09:33 AM

[aleste81]

BuyukBang, on 06 January 2024 - 10:58 PM, said:

There are some post from the developers explaining why, with their point of view.
[...]
my mod is capable of big numbers without being sluggish on UI.

This is great news. 100 000 files is a little number compared to the number of interesting stuff to I want share.
Could you point to the posts with the point of view of Merkur ?

How did you fix the sluggish code ?
Kudos dude

[BuyukBang]

aleste81, on 10 January 2024 - 01:00 AM, said:

This is great news. 100 000 files is a little number compared to the number of interesting stuff to I want share.

I've tested with 130k files, but I believe my mod can easily do more. Though, there still will be a server limitation.

aleste81, on 10 January 2024 - 01:00 AM, said:

Could you point to the posts with the point of view of Merkur ?

Sorry I have't saved links for the posts. Btw, I can't confirm it is written by Mercur, but I remember some posts from the developers, which are now hiding in the forum history.

aleste81, on 10 January 2024 - 01:00 AM, said:

How did you fix the sluggish code ?

I'll write two most important ones among the other things:
1- I've rewritten SharedFileList module which is responsible for displaying shared files (and also downloaded history/known.met in my mod).

I've a test result for the history part: For 130k files (SSD + 13900k + 64GB RAM)
XMod: Loading/reloding on SharedFileList takes more than 5 minutes.
My Mod: Loading/reloding completes under 1 second.

As you can see there is an extremely big difference. That's because my implementation uses a virtual list and holds about only 150 files, which are what you see on the screen at a specific moment. When you scroll down/up it just updates. This works much faster than the current implementation what loads everyting at start and hold them on the listbox all time.

2- When sharing/downloading lots of files, on startup of emule and also on every reconnection, some specific code blocks are executed. These code blocks are resposible to create a package which will be sent to the eservers and kad nodes. But these codes are so slow and causing long UI freezes. I've used a workaround to fix this and now reconnects work without UI freezes.

This post has been edited by BuyukBang: 10 January 2024 - 11:57 PM

[fox88]

Long ago 500 shared files, including downloads was a resonable number; and most servers would not even allow more.
Now with bigger RAM, higher connection speeds and smaller count of users in the net this is less critical; so the servers increase limit for shared files quite voluntarily.
However, there are still limits that should not be ignored or changed without risks of damaging eMule network.
Probably sharing a thousand files, maybe 2000 or even 3000 would be considered fine.
For that number of files performance of the existing code should be sufficient and does not require changes.
So if you feel a slow down, then safe limits were exceeded.

[BuyukBang]

fox88, on 10 January 2024 - 11:23 PM, said:

So if you feel a slow down, then safe limits were exceeded.

This is not true when the eservers and kad network have already their own limitations. Especially on these days with less users & files on the network and still with much powerful PC's, internet connections.

This post has been edited by BuyukBang: 11 January 2024 - 12:03 AM

[fox88]

BuyukBang, on 11 January 2024 - 02:39 AM, said:

This is not true when the eservers and kad network have already their own limitations.

THe phrase contradicts itself. There is no reason for stunning performance with 130K files while a server sets limit of 7500 only.
Better understanding of the eMule network and balanced approach is what developers need, not advetising of yet unreleased mod.
Besides, mod discusisons belong to a different subforum.

This post has been edited by fox88: 11 January 2024 - 08:07 AM

[BuyukBang]

fox88, on 11 January 2024 - 11:06 AM, said:

BuyukBang, on 11 January 2024 - 02:39 AM, said:

This is not true when the eservers and kad network have already their own limitations.

THe phrase contradicts itself. There is no reason for stunning performance with 130K files while a server sets limit of 7500 only.
Better understanding of the eMule network and balanced approach is what developers need, not advetising of yet unreleased mod.
Besides, mod discusisons belong to a different subforum.

Current soft file limits from eMule Security server.met:
8 servers: 1M
3 servers: 100k
1 server: 50k
1 server: 7500

There are limits, yes. But it is above where eMule GUI slow down.

There are also other use cases for mod users (ie. Xmod and several others) like history search/management, which are out of scope for official realeases.

I will not buy into the idea of having a "GUI freezing feature" for whatever reason. This just sounds lazy to me.

This post has been edited by BuyukBang: 11 January 2024 - 09:36 AM

[panzo]

salve sono nuovo ho istallato emule v50a lo aperto lo istallato.
e ho messo la lista. pero non so piu andare avanti se qualcuno mi aiuta grazie

[aleste81]

fox88, on 10 January 2024 - 09:23 PM, said:

Probably sharing a thousand files, maybe 2000 or even 3000 would be considered fine.

Are you kidding me ?
I have stuff to share by the million.
Besides, eMule advantage over BitTorrent is precisely the slow sharing of many files over extendend / infinity time period.

[fox88]

The network was never designed for this.
Unlike torrents, eMule is not that great in sharing large quantity of files, or contents of whole directory trees with many small files (archives should be used instead).

[BuyukBang]

fox88, on 13 January 2024 - 09:35 PM, said:

The network was never designed for this.
Unlike torrents, eMule is not that great in sharing large quantity of files, or contents of whole directory trees with many small files (archives should be used instead).

eServer: Admins are reponsible to configure soft limits according to the target level of the server performance.
Kad: There is already logical limits, nothing can be done in terms of sharing big quantity of files, except creating multiple intances.
Client: Performance is the responsibility of the developers and eMule is not the only client of ed2k network. One client can be slow, another can be better. For example, in this particular subject, aMule is superior to eMule.

This post has been edited by BuyukBang: 17 January 2024 - 02:08 PM

[fox88]

KAD should be functional as a search engine, and this becomes an issue for huge file counts.
In case of proper configuration the time from "Starting eMule..." to "eMule... ready" is about 2 seconds without any SSD.

[Enig123]

To be honest, in my opinion the request is basically requested to make eMule scalable regarding some specific aspects. As we know eMule is quite an old project, started when 'scalable' was not a thing in program design/implementation.

There are quite a few functions with n^n or n^m complexity can be optimized, an example is reload shared files would take quite some time, due to inefficient implementation of CKnownFileList::FindKnownFile that will be invoked in a loop during reloading.

aMule uses a temp multimap sorted with lower 32-bit of file size in that routine to accelerate reloading a lot. Although reloading is user-triggered function and only used occasionally, it's still worth optimizing for the user point of view.

[dfpmola]

BuyukBang, on 06 January 2024 - 09:58 PM, said:

aleste81, on 02 January 2024 - 11:08 PM, said:

Long time user, usually sharing a few thousand files.

Recently I have begun first downloading then sharing 20 000 files, and I quickly noticed that eMule isn't capable of handling such numbers.

eMule is extremely sluggish since. Not the download/upload part, which is ok, but the overall delay between user mouse/keyboard input.

This is a problem, as Elon would say.

Original/community realeses don't care much on sharing or downloading "big" number of files. There are some post from the developers explaining why, with their point of view.

So, I've decided to improve this part of eMule myself. I'm working on an eMule mod based on v0.70a (actually started with 0.60d but migrated to v0.70a code base) since 8 months on my free times and I think I'll need a few months to finish it before the realese. I've already completed the support for big number of files and now my mod is capable of big numbers without being sluggish on UI.

I've added a lot of other nice features, too. Some of them are from v0.50a mods while some are unique and developed by me for my needs. Below you can find screenshots of the preferences what I've already completed (all seen in these are fully tested and working stable). There are also a lot of features working under the hood without any trace on the preferences screenshots, like NAT-T (I decided to implement it as a "always on" feature without on/off toggle to make it widespread). I've also implemented a lot of performance improvements and some bug fixes, too.

There are a lot of other things in my To Do list. I really want to make it near perfect (in my point of view, ofc) before the release, so need more time.

It will be released as fully compliant to GPL (opensource, as it should be) and also to eMule community rules (no leeching features, community rewards etc).

Forum doesn't allow me to add links so i've removed https part of links:
i.ibb.co/JQJsLg6/1.png
i.ibb.co/Chm5M0z/2.png
i.ibb.co/JRvzJLc/3.png

Hello,

First of all thank you very much for the work with the mod, it looks amazing. Could I make a request? I am developing a program to connect emule to services like radarr/sonarr or jackett. And I've noticed the instability of the emule web interface (accessing certain parts of the web interface makes emule crash, especially in the transfer area), could you take a look at it. Looking at the code I have seen that accessing the main thread data is a danger, because of the way it does it. Could it be possible that you could revise it to add it in this mod?

I have made this thread https://forum.emule-...8&#entry1108468

[BuyukBang]

dfpmola, on 30 January 2024 - 08:16 PM, said:

Hello,

First of all thank you very much for the work with the mod, it looks amazing. Could I make a request? I am developing a program to connect emule to services like radarr/sonarr or jackett. And I've noticed the instability of the emule web interface (accessing certain parts of the web interface makes emule crash, especially in the transfer area), could you take a look at it. Looking at the code I have seen that accessing the main thread data is a danger, because of the way it does it. Could it be possible that you could revise it to add it in this mod?

I have made this thread https://forum.emule-...8&#entry1108468

Hello,

I've just tested this. On v0.70 community release clicking on Transfer tab immediately crashes. 3 eMule sessions and 3 crashes on first click on Web Interface Transfer Tab. So I can confirm this problem is very easily reproducible.

Now the good news:
When I test same scenario on my mod, it never crashed. On Web Interface, I searched and added 5-6 downloads, clicked Transfer Tab contiously, deleted downloads, restarted eMule, did same things, then one more restart and doing same things. It never crashed.
I've intensively changed all of the Download*.cpp/h for feature additions, bug fixes and performance optimisation. So I strongly believe I've already fixed the root cause of the Web UI Download Tab bug with one of these code changes, but without noticing its positive impact on Web UI

To be honest, I'm not using Web UI much. But I'm still aware that it is a good feature to integrate other programs with eMule. I've even planned to write an application to use Web UI in the past, but then changed my mind and started this project.

This post has been edited by BuyukBang: 30 January 2024 - 08:28 PM

[dfpmola]

BuyukBang, on 30 January 2024 - 08:16 PM, said:

dfpmola, on 30 January 2024 - 08:16 PM, said:

Hello,

First of all thank you very much for the work with the mod, it looks amazing. Could I make a request? I am developing a program to connect emule to services like radarr/sonarr or jackett. And I've noticed the instability of the emule web interface (accessing certain parts of the web interface makes emule crash, especially in the transfer area), could you take a look at it. Looking at the code I have seen that accessing the main thread data is a danger, because of the way it does it. Could it be possible that you could revise it to add it in this mod?

I have made this thread https://forum.emule-...8&#entry1108468

Hello,

I've just tested this. On v0.70 community release clicking on Transfer tab immediately crashes. 3 eMule sessions and 3 crashes on first click on Web Interface Transfer Tab. So I can confirm this problem is very easily reproducible.

Now the good news:
When I test same scenario on my mod, it never crashed. On Web Interface, I searched and added 5-6 downloads, clicked Transfer Tab contiously, deleted downloads, restarted eMule, did same things, then one more restart and doing same things. It never crashed.
I've intensively changed all of the Download*.cpp/h for feature additions, bug fixes and performance optimisation. So I strongly believe I've already fixed the root cause of the Web UI Download Tab bug with one of these code changes, but without noticing its positive impact on Web UI

To be honest, I'm not using Web UI much. But I'm still aware that it is a good feature to integrate other programs with eMule. I've even planned to write an application to use Web UI in the past, but then changed my mind and started this project.

You are my Hero, unbelievable.

It would be very useful to be able to access all the information of the GUI from the WEB UI to be able to integrate it in more places, like creating categories, see all the information of a file (download directory among other things) the peers. But these are suggestions, in any case thank you very much for looking at this.