Official eMule-Board: New Emule Compatible Client In The Making - Official eMule-Board

Jump to content


  • (3 Pages)
  • +
  • 1
  • 2
  • 3

New Emule Compatible Client In The Making

#21 User is offline   MrIceBHCla 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 10
  • Joined: 08-July 09

Posted 13 July 2020 - 08:20 PM

Hello All,
g8 news about a new/modern client!
I can tested and will certainly use it.
If I can humbly suggest to look around radarr/sonarr integration, because I was eMule solo user for a very long time, but I am now also using qBittorrent because of that integration.
It is a double efford as I am not sure sonarr/radarr can already be integrated with eMule, but I had at least mention as an idea.
About translation, I can surely also assist with Brazilian Portuguese.
Thanks!
0

#22 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 519
  • Joined: 22-November 04

Posted 22 July 2020 - 10:54 PM

Very excited for your project to be running on ARM platform, since I probably intend to build an ARM based desktop recently, it's better to have a state of the art ed2k client out there.
0

#23 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 23 July 2020 - 11:50 AM

View PostEnig123, on 22 July 2020 - 10:54 PM, said:

Very excited for your project to be running on ARM platform, since I probably intend to build an ARM based desktop recently, it's better to have a state of the art ed2k client out there.

Absolutly, and not to forget that Apple will change its processor to ARM based as well (very soon?).
Further there are a lot of dedicated machines such as NAS which are more feasable to leave them powered on (because you have them already anyways),
or smaller Raspi /(Raspi clones) based systems.
0

#24 User is offline   omeringen 

  • l÷l
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 978
  • Joined: 01-January 06

Posted 24 July 2020 - 08:44 AM

@megaT,
Any github/gitlab page available ?
0

#25 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 24 July 2020 - 10:39 AM

View Postomeringen, on 24 July 2020 - 08:44 AM, said:

@megaT,
Any github/gitlab page available ?

Not yet, in fact I dont use github at all - I prefer my own server with my own files on it.
If I release the source I'd first like to document it, because undocumented code is horrible and (almost) useless.
However Im always looking for people helping with coding... B)
Most likely I'll make the code available on one of my own boxes thru git then.
0

#26 User is offline   wariete 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 5
  • Joined: 06-March 07

Posted 12 August 2020 - 11:26 AM

@megaT I wish I could help at some stage of the project. I don't have strong programming skills, I'm just a daily python and R user in my data science job.
I could also help with the Spanish translation.
And I've been using eMule for more than 16 years and this project makes me especially excited.
A better management of the chunks I think would be desirable. Also increase the buffer limit before writing to disk, or make it dependent on whether it is a solid state disk or not.
Anyway don't take my opinion on technical too serious hehe
0

#27 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 13 August 2020 - 12:08 AM

Hey ;)

Quote

I could also help with the Spanish translation.

Will definetly come useful at some point in time.

Quote

A better management of the chunks I think would be desirable. Also increase the buffer limit before writing to disk, or make it dependent on whether it is a solid state disk or not.
Anyway don't take my opinion on technical too serious hehe

Well what I can just tell right now is that Im not doing any kind of buffer limitation or fiddling with OS internals here,
the reason is that I find this handling pretty much obsolete nowadays.
The OS' and drivers are so well optimized, we've a huge range of mass storage devices including very good HDDs and SSDs alltogether.
Furthermore it could be a network storage like a windows share or attached iSCSI etc.
My best guess is that its much better to leave those decisions to the OS and not fiddling with buffer sizes,
for good handling one could introduce a limitation bound to the network throughput which is oriented by how much data you could actually write out - so you dont fill up all your memory
if you really push data to a slow device/connection.
0

#28 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 519
  • Joined: 22-November 04

Posted 18 August 2020 - 12:04 AM

View PostmegaT, on 12 August 2020 - 05:08 PM, said:

My best guess is that its much better to leave those decisions to the OS and not fiddling with buffer sizes,


That is not entirely true, at least for eMule. In order to maintain data consistency eMule would issue an order to flush buffered data to disk, which actually bypassed the possibility for OS to handle buffer stuff.

Merely adjusting buffer size apparently is not a good idea neither.

But the buffer system is not unable to get huge improvement, with some algorithms and tricks, I have developed a scalable buffer system, with which not only can reducing disk activity, but also with only controllable/limited increase of memory usage.

The changes are not trivial, which need quite some code changes and debugging. This is one that I am willing to adopt to the hopefully new ed2k compatible client codes.
0

#29 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 18 August 2020 - 08:33 AM

Quote

In order to maintain data consistency eMule would issue an order to flush buffered data to disk,

Which is the business of the OS, a forced flush will interrupt the cycle of the OS anyways - how do you know when its a good time to flush?
Why wouldn't you not leave the issue to the OS? Because once you write it out it is not anymore the business of the application.

Quote

I have developed a scalable buffer system, with which not only can reducing disk activity,

I wouldn't bother because I don't want to write an operating system or virtual file system here... I leave the OS internals to the OS (whatever it is, because I'll target at least 3 different ones).

Quote

The changes are not trivial, which need quite some code changes and debugging.

Then its not worth implementing, I'd require that I can only change the underlying interface calls - nothing else of the code base.

Feel free to give some performance details with and without these changes, real measuring data etc. :)
0

#30 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 519
  • Joined: 22-November 04

Posted 19 August 2020 - 01:03 AM

It's not what you think it is, no virtual file system nor OS internal stuff. The buffering system only changes the way eMule use memory as downloading file buffer, which utilized nature of eMule downloading: the downloading data is actually consecutive locally.

And I don't think the changes are OS related neither, I am hoping it can benefit other OS other than windows.

BTW, the buffering system was designed with the help of windows Task Manager visualization tool. Not sure what's the proper way to measure the performance improvements, any recommendation?

Edit1: The buffer system's original design goal is, to minimize 'Average Response Time' spikes caused by eMule downloading.

Edit2: And, for someone who's interested in testing proof of concept of my buffering system, please PM me for it. If he/she is able to get a proper comparison measurement to show the improvement of my buffering system to the original default one, that would be nice and appreciated.

This post has been edited by Enig123: 19 August 2020 - 01:12 AM

0

#31 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 19 August 2020 - 12:05 PM

Quote

It's not what you think it is, no virtual file system nor OS internal stuff. The buffering system only changes the way eMule use memory as downloading file buffer, which utilized nature of eMule downloading: the downloading data is actually consecutive locally.

I see, well it might have been relevent 20y ago but Im not sure if it's still relevant today as on modern systems.
What Im doing is that each uploading/downloading happens in its own thread, I've simply allocated memory for block data on the stack.
This wont change for the whole lifetime of the upload/download, so quite some time.
The blocks will get commited in some (other) dedicated memory for the file, which also wont change for a very long time - in the end
full parts get commited and also write out to disk.

So there is no real issue with allocation response time, I simply dont allocate very often... because its always the same amount of memory required.

Edit:
What I personally find more relevant is the uploading/handling of shared files (file parts) and how long requested parts remain in memory.
While others will become deallocated (because nobody needs them)

This post has been edited by megaT: 19 August 2020 - 12:17 PM

0

#32 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 519
  • Joined: 22-November 04

Posted 20 August 2020 - 05:02 AM

Just keep in mind that ed2k clients can have multiple downloading files, each one can have multiple concurrent downloading clients, from each downloading client, we could ask for a random part.

I wonder how're you going to handle this, and how many memory will it needs to be allocated with your mechanism? If you're going to keep the whole part in memory, I am afraid memory will be bloat real quick.

This post has been edited by Enig123: 20 August 2020 - 05:04 AM

0

#33 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 20 August 2020 - 10:20 AM

Hi,
sure but we've to mind the time eMule originally was developed in - back then files were a lot smaller (so was end-user hardware).
But Im developing a new thing here, these days people have like 16GB+ RAM or more normally.

Quote

If you're going to keep the whole part in memory, I am afraid memory will be bloat real quick.

Yes and no, but that is also kind-of business of the OS (memory mapping etc.) -
its well possible that I'll have to optimize here.

So you're saying your loading and shoving KB chunks in/out memory?
I personally didnt find it feasable to bother with <1M chunks and bothering with disk I/O for this,
but I'll have to test that out.

I mean developing a real 64bit application doesnt only mean to shove it thru the compiler and hope that a 64bit binary falls out,
but to also make use of the 64bit address space and being able (due more RAM) to store more data and use it in the same time (without having to raise disk I/O here)

This post has been edited by megaT: 20 August 2020 - 10:26 AM

0

#34 User is offline   hooligan3000 

  • European Community
  • PipPipPipPipPip
  • Group: Members
  • Posts: 295
  • Joined: 19-December 09

Posted 20 August 2020 - 01:26 PM

View PostmegaT, on 20 August 2020 - 12:20 PM, said:

But Im developing a new thing here, these days people have like 16GB+ RAM or more normally.



sorry but no. 4-8gb ram is normal...8-16gb for gamers...

ed2k://|server|91.208.184.143|4232|/

SD - Telegram

Air VPN - The air to breathe the real Internet

BTC
bc1qdrk0ld07jtg99ym2zg68cpqhqj34qnf2txm93n
XMR
48ja6xJ2NyPMNzmY1pA3ZZPpX5yTaw9Ym28jrDPCL7Y7L7pr5wXFdpeK4WqBbvVY5qEa6VDfhFKTnHWef3EPC4zgQNTnAwg
1

#35 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 20 August 2020 - 01:48 PM

View Posthooligan3000, on 20 August 2020 - 01:26 PM, said:

View PostmegaT, on 20 August 2020 - 12:20 PM, said:

But Im developing a new thing here, these days people have like 16GB+ RAM or more normally.



sorry but no. 4-8gb ram is normal...8-16gb for gamers...

4GB is pretty much low end, RAM is super cheap...
And I think most people have 8GB nowadays, I dont even know if PCs with <8GB are being sold any longer??
If that wont convience you then still mind that for the future... we wont go back to 4GB times. :)

I myself had 32GB since a couple of years already... then got back to main laptop usage which had 16GB -
now Im having one with 32GB again.
So in summary, for the last 6years (I think) I never had less than 8GB.
Even my NAS has 16GB.

Edit:
It makes perfect sense to me, if you'd have a low-end box like ~4GB why would you handle 36GB big files or use p2p for this?!
If you do this I can expect that your box is not the lowest of lowest setup.
But again I have no real life data so far how my program performs... right now Im cleaning up a lot of code
and prepare for the very first Windows build.

This post has been edited by megaT: 20 August 2020 - 02:08 PM

0

#36 User is offline   hooligan3000 

  • European Community
  • PipPipPipPipPip
  • Group: Members
  • Posts: 295
  • Joined: 19-December 09

Posted 20 August 2020 - 05:36 PM

i have just 16gb ram boxes also my 4k high end gaming machine...but a normal office desktop pc for example from dell comes with just 4gb ram for 400-500€
don't argue with how much you use...

a lot of users have an old cheap notebook with 4gb ram and use emule on that machine just to save power.

This post has been edited by hooligan3000: 20 August 2020 - 05:37 PM


ed2k://|server|91.208.184.143|4232|/

SD - Telegram

Air VPN - The air to breathe the real Internet

BTC
bc1qdrk0ld07jtg99ym2zg68cpqhqj34qnf2txm93n
XMR
48ja6xJ2NyPMNzmY1pA3ZZPpX5yTaw9Ym28jrDPCL7Y7L7pr5wXFdpeK4WqBbvVY5qEa6VDfhFKTnHWef3EPC4zgQNTnAwg
0

#37 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 20 August 2020 - 05:45 PM

Quote

a lot of users have an old cheap notebook with 4gb ram...

Maybe, I know win10 runs like crap on those machines...

Quote

and use emule on that machine just to save power.

You dont save power with old hardware.

I will develop software for the future, not for the past.
But I can still look into conservative memory behaviour once Im at that point.

16GB is fine, 8GB is also okay... everything below on a modern machine is just horrible.
That said, I dont think you can get around memory mapping for larger files anyways...

This post has been edited by megaT: 20 August 2020 - 05:50 PM

0

#38 User is offline   Enig123 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 519
  • Joined: 22-November 04

Posted 20 August 2020 - 07:03 PM

View PostmegaT, on 20 August 2020 - 03:20 AM, said:

So you're saying your loading and shoving KB chunks in/out memory?
I personally didnt find it feasable to bother with <1M chunks and bothering with disk I/O for this,
but I'll have to test that out.


I cannot agree you more here.

Actually my buffering system has similar mechanism to accumulate consecutive data until enough to flush into disk, in order to reduce unnecessary disk I/O. In respect to maintain minimum 'Average response time', flush too much data at a time is not optimized choice, the disk I/O calling needs to be sporadic (less number of calling) and swift (not flush too much data at a time).

Also, if you'd like your client to run on ARM machines, which as server may don't have much memory to spend, you'll have to keep in mind to manage the memory occupation of your program. Yes?
0

#39 User is offline   megaT 

  • Member
  • PipPip
  • Group: Members
  • Posts: 41
  • Joined: 09-May 20

Posted 21 August 2020 - 11:20 AM

Quote

Also, if you'd like your client to run on ARM machines, which as server may don't have much memory to spend,

Well thats more or less being an implication that ARM based machines must be weak machines.
I come from the embedded world, I agree there are ARM in handheld devices like ARMv5 which are quite old and have not much mem.. ;)
But you can also get an Raspi4 with 8GB, there are machines were 64bit ARM (Aarch64) is used in super computers. Or in servers.
It doesnt necessarily mean there is less than 4GB available...

It really depends on the use case, if you have less RAM available then you have to maintain a low footprint anyways - but this also mean
to be conservative as an admin. Maybe not share very large files anyways...
Maybe one can implement to ways of caching behaviour one for memory constraint devices, which have then thus load more from disk...
0

#40 User is offline   Vanad 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 1
  • Joined: 22-August 20

Posted 22 August 2020 - 05:07 PM

Good news!

My best wishes.
0

  • Member Options

  • (3 Pages)
  • +
  • 1
  • 2
  • 3

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users