[netfinity]

NetF WARP 0.2b

WARNING!
Several features in this Mod is very beta and might cause unexpected results, this is especially true for the corruption and KAD related features. Use on your own risk!

Features:
Faster Endgame
- Dynamic Block Requests, tries to make all sources to complete at the same time
- Delayed NNP, keeps downloading sources a little bit longer in case a part turns out corrupt
- Drop Stalled Sources, cancels stalled or slow downloads when file is about to complete to give way for faster sources
- Redownload only data of banned clients on corruption
Anti-P2P Protection
- Safe KAD (slows down the spreading of bad KAD nodes)
Credits and Payback
- VIP "Payback" Queue, as the Payback queue but with different bandwidth management
- Balanced Ratio Payback, make clients, who have a ratio higher than your inverse ratio, VIP's
- Smart Part File Push, rebalances between complete and part files based on previous upload
All the features of the ESE Mod

Known Bugs:
- Redownload only data of banned clients might cause looping downloads or complete stalls if corruption occurs and NetF fail the recovery process.

Changelog:
*** NetF WARP 0.2b
- Based on eMule 0.47c
- Added disconnection of very slow upload sessions
- Removed some 'Safe KAD' code that become redundant with the 0.47c release
- Removed the 'Fixed TCP port' feature as it caused problem for some
- Changed 'VIP Queue' so VIP upload is always allowed upto 10% of your upload bandwidth even if you have a bad ratio
- Changed, files in release cause clients to automatically become VIP's
- Fixed a small bug in the server list
- Fixed some parts in the upload handling

- Note! There was an unmentioned change in NetF WARP 0.2a that disables the friend auto VIP, this change is still in place.

*** NetF WARP 0.2a
- Based on eMule 0.47b
- Added 'Safe KAD' (filters KAD from unrequested responses (anti-P2P spam), and also limits processed nodes by IP)
- Added 'Drop Stalled Sources' (speeds up file completion)
- Added 'Anti Fragmenting' (splits KAD search responses so they (hopefully) fit into one IP packet)
- Added 'CorruptionBlackBoxFix' (ensure all data been processed before banning) [by Xanatos]
- Added 'Dynamic Corruption Threshold' (to avoid mass banning when using Dynamic Block Requests)
- Added 'Only Recover Banned Data' (only redownload data that is "known" to be corrupt)
- Added 'Fixed TCP port' (may help some clients behind NAT/Firewalls to get HighID)
- Added 'Smart Part File Push' (shifts focus between partial and complete files depending on actual UL/DL ratio)
- Added 'Slot Limiting' (tries to reduce the number of upload slots)
- Added a lot of small optimizations and fixes from the NetF Mod. [by various modders]
- Updated 'Flush Thread' to do more of the work in the actual thread

Changelog:
*** nAM 0.1a BETA
- Based on eMule 0.47a
- Added 'Delayed NNP' (speeds up file completion when there is corruption or problem with the source finalizing)
- Added 'VIP Queue' (good uploaders and friends get their own lane)
- Added 'Known Client Bonus' (gives a small bonus to known clients not seen for 12 hours)
- Added 'Transfer Rate BlackBox' (hides datarate calculations from programmer)
- Added 'Dynamic Block Requests' (speeds up file completion)
- Added 'Active Ratio'(must have when using 'Dynamic Block Requests' to prevent excessive downloading)
- Added 'Safe Hash' (part hashing is done in separate thread) [by SLUGFILLER]
- Added 'ReadBlockFromFileThread' (prevents GUI to become sluggish when uploading) [by SiRoB / MorphXT]
- Added 'Flush Thread' (may help a little when doing heavy downloading) [by SiRoB / MorphXT]
- Added 'Mod version' (makes it possible to distinguish the client from the official one)
- Added 'Anti Shape' (makes credit shaping less profitable)
- Fixed memleak! Deletion in lastcommonroutefinder. [by tHeWiZaRdOfDoS] 

Download:
ED2K Links
Binary: ed2k://|file|WARP_0.2b_binary.zip|2445871|131FED770872570ADE5770D2EDEE07CA| |h=M3FW744NKZTFUQEXLSUHVEXYZTTBZJHW|/
Source: ed2k://|file|WARP_0.2b_source.zip|4018937|696EC07C92305EC169B5636A66E8A6EE| |h=XM6QMZRBPLLZDD34LHCBOUB7CMVEF3GD|/
HTTP Links
Binary (DDoS Isle of Tortuga)
Source(DDoS Isle of Tortuga)

[niRRity]

Testing...

[Firnus]

Take it and testing !

[leexgx]

clicked

[JulGer]

With this, I only get a LowID and firewalled Kad. Same with the latest Zzul build. The only 47c that works for me, is the offical...

[Bero0571]

Hi!
First of all sorry for my english
So, I'm testing and after 23 hours my impression is good!
I used USS setted in this way: max upload limit = upload capacity = 36 KB/s , while lowest allowed upload speed = 26 KB/s ( I've a 640/256 adsl)...I have to say USS works very good , upload is high with stable speeds so no problem with this.
Overhead is ok ...download is stable too,very few failed sessions (~9.1%) no corruptions (~0%)...


But on my PC I've found a high ram usage : from 130 even to 150 MB! and sometime I've seen that active connections are a bit high but only for a small period!
Maybe is to fix:

[charlesdewei]

Laptop Sempron 2800+
512 Mb


Cable 4 Mb/300 kb

TCP=1755
UDP=1765

Hard limit = 1000

Upload limit=28 kb/s
DL=disabled

Max Connections=500

ed2k & KAD= ON

USS enabled, max ping=100

Max 5 conn= 40
Max half = 30 (tcpip.sys patched to 50)

Obfuscation=ON

eMule v0.47c [NetF WARP v0.2b] Statistics [http://emule-project.net]

Transfer
   Session UL:DL Ratio: 4.91 : 1
   Session UL:DL Ratio (Friends UL excluded): 4.91 : 1
   Cumulative UL:DL Ratio: 5.73 : 1
   Uploads
	  Session
		 Uploaded Data: 4.01 GB
			Clients
			   eMule: 3.80 GB (94.8%)
			   eD Hybrid: 64.94 MB (1.6%)
			   eDonkey: 0 Bytes (0.0%)
			   aMule: 17.51 MB (0.4%)
			   MLdonkey: 0 Bytes (0.0%)
			   Shareaza: 0 Bytes (0.0%)
			   eM Compat: 130.36 MB (3.2%)
			Port
			   Default: 2.12 GB (52.8%)
			   Other: 1.89 GB (47.2%)
			   Other: 0 Bytes (0.0%)
			Data Source
			   Complete File: 3.64 GB (90.8%)
			   Part File: 375.86 MB (9.2%)
		 Uploaded Data to Friend Slots (Session): 0 Bytes
		 Active Uploads/Needed to fill Bandwidth: 8
		 Total Uploads: 8
		 Waiting Uploads: 2178
		 Upload Sessions: 2112
			Total successful upload sessions: 1897 (89.82%)
			Total failed upload sessions: 215 (10.18%)
			Average Uploaded Per Session: 2.17 MB
			Average upload time: 11:59 Minutes
		 Total Overhead (Packets): 101.58 MB (1.71 M)
			File Request Overhead (Packets): 22.94 MB (754.99 k)
			Source Exchange Overhead (Packets): 2.13 MB (6.15 k)
			Server Overhead (Packets): 72.97 KB (3.31 k)
			Kad Overhead (Packets): 59.89 MB (753.64 k)
	  Cumulative
   Downloads
	  Session
		 Downloaded Data: 837.22 MB
			Clients
			   eMule: 797.38 MB (95.2%)
			   eD Hybrid: 100.52 KB (0.0%)
			   eDonkey: 0 Bytes (0.0%)
			   aMule: 0 Bytes (0.0%)
			   MLdonkey: 0 Bytes (0.0%)
			   Shareaza: 0 Bytes (0.0%)
			   eM Compat: 39.74 MB (4.7%)
			   URL: 0 Bytes (0.0%)
			Port
			   Default: 531.43 MB (63.5%)
			   Other: 305.79 MB (36.5%)
			   Other: 0 Bytes (0.0%)
		 Completed Downloads: 4
		 Active Downloads (chunks): 2
		 Found Sources: 842
			On Queue: 695
			Queue Full: 115
			No needed parts: 30
			Asking: 0
			Receiving hashset: 0
			Connecting: 0
			Connecting via server: 0
			Too many connections: 0
			Cannot connect LowID to LowID: 0
			Problematic: 0
			Banned: 0
			Asked for another file: 1
			Unknown: 0
			via eD2K Server: 54
			via Kad: 19
			via Source Exchange: 701
			via Passive: 68
			eD2K: 825 (98.0%)
			Kad: 612 (72.7%)
			eD2K/Kad: 604 (71.7%)
			UDP File Reasks: 6.10 k, Failed: 777 (12.7%)
			Dead Sources: 72 (56 + 16)
		 Download Sessions: 170
			Successful Download Sessions: 151 (88.8%)
			Failed Download Sessions: 19 (11.2%)
			Average Downloaded Per Session: 5.54 MB
			Average Download Time: 26:05 Minutes
		 Gained Due To Compression: 11.12 MB (1.3%)
		 Lost Due To Corruption: 0 Bytes (0.0%)
		 Parts Saved Due To I.C.H: 0
		 Total Overhead (Packets): 78.66 MB (1.33 M)
			File Request Overhead (Packets): 10.66 MB (431.56 k)
			Source Exchange Overhead (Packets): 185.25 KB (1.61 k)
			Server Overhead (Packets): 93.38 KB (1.90 k)
			Kad Overhead (Packets): 53.38 MB (706.74 k)
	  Cumulative
Connection
   Session
	  General
		 Reconnects: 1
		 Active Connections (estimate): 49 (Half:0 | Compl:6 | Other:43)
		 Average Connections (estimate): 51
		 Peak Connections (estimate): 452
		 Max Connection Limit Reached: 0
	  Upload
		 Upload-Speed: 28.02 KB/s
		 Average Uploadrate: 23.43 KB/s
		 Max Upload Rate: 30.04 KB/s
		 Max Average Upload Rate: 23.43 KB/s
	  Download
		 Download-Speed: 7.21 KB/s
		 Average Downloadrate: 4.78 KB/s
		 max. Downloadrate: 39.38 KB/s
		 Max Downloadrate Average: 4.78 KB/s
   Cumulative
Time Statistics
   Statistics last reset: 11/09/2006 09:17:36 p.m.
   Time Since Last Reset: 8 Days 14:46 Hours
   Session
	  Runtime: 2 Days 1:51 Hours
	  Transfer Time: 2 Days 1:51 Hours (100.0%)
	  Current Server Duration: 21:55 Hours (44.0%)
	  Total Server Duration: 1 Days 23:14 Hours (94.8%)
   Cumulative
   Projected Averages
Clients
Servers
   Working Servers: 58
	  Users on Working Servers: 3.55 M; Low ID: 1.65 M (46.5%)
	  Files on Working Servers: 552.11 M
	  Server Occupation: 13.39%
   Failed Servers: 1
   Deleted Servers: 1
   Total: 59
   Total Users: 3.67 M
   Total Files: 566.68 M
   Records
Shared Files
Disk Space

Mem usage = 113 Mb

This post has been edited by charlesdewei: 20 September 2006 - 10:06 AM

[Rapid_Mule]

I tested WARP for a couple of days on a relative slow connection (128/64 Kbps).
* Noticed the following:
1. After a connection loss/reconnect LowID most of the time.
2. WARP does not obey the DOWNLOAD limit set.
3. When WARP reaches the maximum LINE SPEED it displays weird values in the transfer window (like 0 B/s or even sometimes speeds higher than my connection).
4. The status bar also does show very high download speed for short periods of time.
5. After the LINE is saturated with downloads the upload drops dramatically and shows lots of spikes (monitored with a bandwidth meter) but not in statistics graphs.

Can someone please confirm my findings?

Regards

This post has been edited by Rapid_Mule: 20 September 2006 - 04:14 PM

[CiccioBastardo]

Bero0571, on Sep 19 2006, 05:51 PM, said:

Maybe is to fix:

It seems a job for the extremely powerful extended tooltip feature

[Mr.Blonde]

I have loads of failed DLs and 0KB DLs with 0.2b (maybe a ratio-problem)
...but anyway, a nice statistic

btw: the release-vip-feature is very nice!

eMule v0.47c [NetF WARP v0.2b] Statistics [*******]

Transfer
   Session UL:DL Ratio: 1 : 2.87
   Session UL:DL Ratio (Friends UL excluded): 1 : 2.87
   Cumulative UL:DL Ratio: 1 : 2.90
   Uploads
	  Session
		 Uploaded Data: 1.25 GB
		 Uploaded Data to Friend Slots (Session): 0 Bytes
		 Active Uploads/Needed to fill Bandwidth: 4
		 Total Uploads: 4
		 Waiting Uploads: 5032
		 Upload Sessions: 1157
			Total successful upload sessions: 1022 (88.33%)
			Total failed upload sessions: 135 (11.67%)
			Average Uploaded Per Session: 1.26 MB
			Average upload time: 7:41 Minutes
		 Total Overhead (Packets): 76.93 MB (1.43 M)
	  Cumulative
   Downloads
	  Session
		 Downloaded Data: 3.59 GB
		 Completed Downloads: 1
		 Active Downloads (chunks): 12
		 Found Sources: 2419
		 Download Sessions: 1400
			Successful Download Sessions: 916 (65.4%)
			Failed Download Sessions: 484 (34.6%)
			Average Downloaded Per Session: 4.02 MB
			Average Download Time: 24:10 Minutes
		 Gained Due To Compression: 13.86 MB (0.4%)
		 Lost Due To Corruption: 0 Bytes (0.0%)
		 Parts Saved Due To I.C.H: 3
		 Total Overhead (Packets): 70.26 MB (1.59 M)
	  Cumulative
Connection
   Session
	  General
		 Reconnects: 1
		 Active Connections (estimate): 109 (Half:3 | Compl:19 | Other:87)
		 Average Connections (estimate): 106
		 Peak Connections (estimate): 502
		 Waiting...
	  Upload
		 Upload-Speed: 12.05 KB/s
		 Average Uploadrate: 11.23 KB/s
		 Max Upload Rate: 12.50 KB/s
		 Max Average Upload Rate: 11.23 KB/s
	  Download
		 Download-Speed: 38.29 KB/s
		 Average Downloadrate: 32.22 KB/s
		 max. Downloadrate: 63.38 KB/s
		 Max Downloadrate Average: 32.22 KB/s
   Cumulative
Time Statistics
   Statistics last reset: 13.09.2006 21:09:31
   Time Since Last Reset: 6 Days 23:13 Hours
   Session
	  Runtime: 1 Days 8:30 Hours
	  Transfer Time: 1 Days 8:30 Hours (100.0%)
	  Current Server Duration: 8:30 Hours (26.2%)
	  Total Server Duration: 1 Days 8:29 Hours (99.9%)

[LorenzoC]

I get the same percentage of failed downloads but It seems It happens also with other clients.
I am not sure what the cause could be. Any hint ?

[netfinity]

@Rapid_Mule
You are absolutly right. The download throttler SUXX!!!!
The download oscillates really heavy when the limit is reached. I wonder if that happens with the official client too!?

@Mr.Blonde
Yes, you have an unusual high amount of failed downloads considering the size of the file you're downloading. However it's very hard to tell because failed downloads often just means that you became NNP before getting the slot.

I like the VIP-release too! Made it much easier to release.

@Bero0571 + CiccioBastardo
Yep, this one always been an issue. Maybe I should cut the eMule off as that is a bit redundant!

[Rapid_Mule]

netfinity, on Sep 20 2006, 09:55 PM, said:

@Rapid_Mule
You are absolutly right. The download throttler SUXX!!!!
The download oscillates really heavy when the limit is reached. I wonder if that happens with the official client too!?

The Official client v0.47c download behaviour is alot more stable on my slow connection than WARP.

Netfinity I really like your work. A few hours ago I managed to remote launch WARP on my office machine with a dedicated bandwidth of 2048/512 Kbps. WARP behaves really good when given enough bandwidth for it to squeeze every last bit of data. To bad I don't have a high speed connection at home.

I would like to ask about your future plans for the mod (what do you plan to implement/improve/tweak, etc.)?
Do you consider introducing user tweak-able features in the mod or all future work/settings well be hidden behind the official client UI?

Regards.

This post has been edited by Rapid_Mule: 20 September 2006 - 09:30 PM

[Bero0571]

eMule v0.47c [NetF WARP v0.2b] Statistics [*******]

Transfer
   Session UL:DL Ratio: 1 : 1.97
   Session UL:DL Ratio (Friends UL excluded): 1 : 1.97
   Cumulative UL:DL Ratio: 1 : 1.97
   Uploads
	  Session
		 Uploaded Data: 5.18 GB
		 Uploaded Data to Friend Slots (Session): 0 Bytes
		 Active Uploads/Needed to fill Bandwidth: 6
		 Total Uploads: 6
		 Waiting Uploads: 5034
		 Upload Sessions: 999
			Total successful upload sessions: 875 (87.59%)
			Total failed upload sessions: 124 (12.41%)
			Average Uploaded Per Session: 6.06 MB
			Average upload time: 25:28 Minutes
		 Total Overhead (Packets): 187.55 MB (3.80 M)
	  Cumulative
   Downloads
	  Session
		 Downloaded Data: 10.19 GB
		 Completed Downloads: 15
		 Active Downloads (chunks): 19
		 Found Sources: 1670
		 Download Sessions: 2621
			Successful Download Sessions: 2310 (88.1%)
			Failed Download Sessions: 311 (11.9%)
			Average Downloaded Per Session: 4.52 MB
			Average Download Time: 23:59 Minutes
		 Gained Due To Compression: 114.83 MB (1.1%)
		 Lost Due To Corruption: 6.06 MB (0.1%)
		 Parts Saved Due To I.C.H: 1
		 Total Overhead (Packets): 186.53 MB (4.27 M)
	  Cumulative
Connection
   Session
	  General
		 Reconnects: 2
		 Active Connections (estimate): 169 (Half:0 | Compl:15 | Other:154)
		 Average Connections (estimate): 193
		 Peak Connections (estimate): 501
		 Max Connection Limit Reached: 6 : 20/09/2006 17.05.41
	  Upload
		 Upload-Speed: 29.94 KB/s
		 Average Uploadrate: 28.65 KB/s
		 Max Upload Rate: 36.08 KB/s
		 Max Average Upload Rate: 33.76 KB/s
	  Download
		 Download-Speed: 68.65 KB/s
		 Average Downloadrate: 56.42 KB/s
		 max. Downloadrate: 77.84 KB/s
		 Max Downloadrate Average: 56.42 KB/s
   Cumulative
Time Statistics
   Statistics last reset: 18/09/2006 18.17.39
   Time Since Last Reset: 2 Days 5:06 Hours
   Session
	  Runtime: 2 Days 4:42 Hours
	  Transfer Time: 2 Days 4:37 Hours (99.9%)
	  Current Server Duration: 2 Days 2:26 Hours (95.7%)
	  Total Server Duration: 2 Days 4:37 Hours (99.8%)
   Cumulative
   Projected Averages
Clients
Servers
Shared Files
Disk Space

Really good job with this mod!...and VIP-release is a good idea I like it!
The only negative point ( on my pc) is a high ram usage...and maybe ( but not so important ) high active connections values and overhead...but I repeat I'm surprise , really good job netfinity!

This post has been edited by Bero0571: 20 September 2006 - 10:10 PM

[Aireoreion]

netfinity, on Sep 20 2006, 08:55 PM, said:

@Rapid_Mule
You are absolutly right. The download throttler SUXX!!!!
The download oscillates really heavy when the limit is reached. I wonder if that happens with the official client too!?

I do think so. One of the iONiX Betatesters reported the same problem - and i havn't made any changes on the throttler

Greetings
---Aireoreion

[netfinity]

Aireoreion, on Sep 21 2006, 12:07 AM, said:

netfinity, on Sep 20 2006, 08:55 PM, said:

@Rapid_Mule
You are absolutly right. The download throttler SUXX!!!!
The download oscillates really heavy when the limit is reached. I wonder if that happens with the official client too!?

I do think so. One of the iONiX Betatesters reported the same problem - and i havn't made any changes on the throttler

Greetings
---Aireoreion

Possibly the problem is made worse by ZZ ratios and such, cause adding a ratio is the only change I made to my Mod related to download limits (well, the bandwidth calculation method has been changed too).

Have to look into this!

Quote

Really good job with this mod!...and VIP-release is a good idea I like it!
The only negative point ( on my pc) is a high ram usage...and maybe ( but not so important ) high active connections values and overhead...but I repeat I'm surprise , really good job netfinity!

The ram usage bugs me too, but it only grows to a certain point then it stops so it appears not to be a memory leak. I however think I should be able to shrink the number of active connections to the next release.


Regards,
netfinity

[leexgx]

do not seee the 000000 in KAD any more

unless i download 50 files at once i cant realy max my download out to test the limiter (i test later on mp3 files seem to start fast)

[leexgx]

spamming my download 101



looks like the limter has kicked in (not sure but why is the download limter kicking in at 1:2 but it seems to be kicked in when it was not close to) 1:2 ratio seem an bit limited to me (no flames you know who you are)



stoped all other downloads



done (if it was not for the limiter it been done in 1 min)

next



looks like we got an staller here (official way) yep it droped have to wait upto 30 mins now for reask

This post has been edited by leexgx: 21 September 2006 - 05:58 PM

[netfinity]

Did some massive downloading myself. And noticed that GUI became very unresponsive when the download hit 300 kB/s (statistics actually said it was 350 kB/s, but the display was not updated for a long period so I never saw it in realtime). Obviously there is still alot to do before we will see any serious download rates.

When it comes to the 0000... KAD ID's you where probably lucky. I still get those and the Safe KAD feature doesn't really prevent only making it slightly less probable in best case.

I will look into the limiter to see what the problem might be.

[netfinity]

Seems that removing some security checks from the disk flush function as well as forcing disk allocation and the actually flushing to always take place in a separate thread is making downloads to become much more stable and as such provide much better download rates. Need to investigate this further. Maybe I will be able to saturate my connection one day!