Official eMule-Board: Netf Warp Svn-121 (2013-08-23) - Official eMule-Board

Jump to content


  • (22 Pages)
  • +
  • 1
  • 2
  • 3
  • Last »

Netf Warp Svn-121 (2013-08-23) Better handling of for rare files through Sub-Chunk Transfers

#1 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 10 September 2006 - 09:58 PM

Latest untested source: http://svn.goran.se....etF-WARP/trunk/
Latest generated doxygen documentation: http://www.goran.se....-WARP/doc/html/ (currently, shows wrong version number)

Note: building directly from source requires at least Visual C++ 2010, although it has only been tested with Visual C++ 2012 (SP3).

NetF WARP SVN-121

WARNING!
Several features in this Mod is very beta and are expected to blow up into your face!

Features:
Sub-Chunk Transfer
- Enables sharing of not yet completed parts which allows for clients with this feature to more efficiently collaborate on downloading rare files
Faster Endgame
- Dynamic Block Requests, tries to make all sources to complete at the same time
- Delayed NNP, keeps downloading sources a little bit longer in case a part turns out corrupt
- Drop Stalled Sources, cancels stalled or slow downloads when file is about to complete to give way for faster sources
- Redownload only data of banned clients on corruption
Anti-P2P Protection
- Safe KAD (slows down the spreading of bad KAD nodes)
Credits and Payback
- VIP "Payback" Queue, as the Payback queue but with different bandwidth management
- Balanced Ratio Payback, make clients, who have a ratio higher than your inverse ratio, VIP's
- Smart Part File Push, rebalances between complete and part files based on previous upload
QOS (Vista/7/8 only!)
- Allow Windows and compatible routers to prioritize surfing and VoIP before eMule (upstream only)
Bandwidth efficient
- Tries to maximize the amount of data sent in each IP packet by identifying the maximum segment size (Vista/7/8 only!)
All the features of the ESE Mod
Multi Threaded
- Many tasks have its separate thread to make WARP more responsive

Platforms supported:

Quote

Platform | 32 bit | 64 bit
Windows 2000 | No | N/A
Windows XP | Yes | No
Windows Vista | Yes | Yes
Windows 7 | Yes | Yes
Windows 8 | Yes | Yes
Wine 1.5.31 (Ubuntu 12.04) | Yes | N/A


Known Bugs and limitations:
'Only Recover Banned Data' might cause (in theory) looping downloads or complete stalls if corruption occurs and NetF fail the recovery process.
FastKAD ICMP unreachable feature seems not to work with personal firewalls.
Sub-Chunk Transfers is currently only enabled for files smaller than ~1.4GB.

Condensed changelog: (see eMule-WARP_changelog.txt for complete set of changes)
Fix FindExistingHash method
Perform validation of sub chunks immediatly after we loaded the part file, if possible
Save AICH recovery data to part met file
Make UPnP work on interfaces with secondary IPs
Don't send file not found when no parts are available and client is SCT aware (behaviour stolen from eDonkeyHybrids)
Fix A4AF looping on NNP
Allow downloaded AICH recovery data to be forwarded to other clients with same master hash
Improve stability and fix crashes related to hashing and flushing.
Make 32 bit version to use v110_xp toolset so it runs on Windows XP
Extend SCT feature to share and use AICH sub chunk vectors in extent to crumbs
Only share sub chunks that been verified with AICH hash
Move AICH recovery hashing into part hashing thread
Improve compability with other clients using crumbs
Fix broken CAICHMap
Fix error not found in QOS when socket list got emptied
Fix crash when up parts status is not available
Do searching of shared files in separate thread
Improve MiniMule dialog stability (no deleting of a dialog being created)
Fix one liner tooltips only showing the first two character


Download: (Note: Due to experienced stability issues in the 64bit version, I recommend using the 32bit instead)

ED2K Links
Binary x86: ed2k://|file|WARP_Svn121_x86_binary.zip|3779565|46720CF378446B242AE3A8EF90E8EAED|h=QWBHDHKVFZNEDK4I5VOHHHAYLKDM7TXP|/
Binary x64: ed2k://|file|WARP_Svn121_x64_binary.zip|5156741|07BAA3DDDE0ED55BB4636D8CD35A974C|h=U25JSVAUDXIKDGRVJKYOHCDCBKH7JIKJ|/
Source: ed2k://|file|WARP_Svn121_source.zip|7986189|D54AD5800123DAE8076E7DFC5D5F6CBC|h=WFYSBH4N536QDZOG3RKE5EL7HN6GFNQU|/
HTTP Links
Binary x86: http://www.goran.se...._x86_binary.zip
Binary x64: http://www.goran.se...._x64_binary.zip
Source: http://www.goran.se....n121_source.zip

*** Older releases! ***

NetF WARP 0.3a (RELEASE)

Changelog:
*** NetF WARP 0.3a (RELEASE)
Some performance and binary size improvements
Make unit test to work for real
Fix upload slot handling to better cope with 100Mbit connections
Optimize upload file caching
Sligtly more effecient looping in RC4 algorithm
Added unit tests for Fakealyzer and RC4Crypt
Disabled saturation checks in ForceNewClient.
Removed commented-out code.
Corrected some comments.
Increased upload slot limit to 500 (to support Giga bit connections)
Removed contact info in version as it's not valid anymore
Reactivated unreachable code in CUploadQueue::ForceNewClient
Added check for memory availability (if we need to conserve or not)
Changed Fakealyzer algorithm slightly to reduce false positives
Rearranged code and made some const to improve code size and performance
Changed some == and && operators with ^ and |, to improve CPU pipelining
Added throw() to some methods to reduce executable image size (should be replaced with nothrow in C++11)
Updated code for use with latest SDKs (removed undocumented calls needed for W2K or earlier)
Rearranged code in hope to improve CPU branch predictions and cache hits
Changed reask times to be more aggressive for files that recently got NNP, when we have few sources
Fixed some compile warnings
Fixed so upload sessions aren't terminated while we need more slots or just added a new one (<10 secs)
Changed so CUpDownClient::CreateNextBlockPackage only tries to reduce memory usage when available memory is really of concern
Implemented inlined memcpy's for fixed size operations
Optimized CBarShader::FillRect to use cached bitmaps for rendering and reduced conditional statements to improve performance
Upgraded project files to VC2012 backwards compatible format
Fixed some warnings and a VC2012 compile error
Changed EPartFileStatus enum to avoid conflict with recent SDKs.
Changed request block trimming
Set doxygen to generate call references in documentation
Implemented AICH recovery data retries and use of CorruptionBlackBox even when no AICH data is available
Use background thread and io priority when hashing files
Fixed type warnings and disabled some logging
Changed ntohl to htonl for consistency
Improved CurruptionBlackBox when no AICH data available
Fixed problem with part hashing when only one part
Added AICH hashing to the SafeHASH thread
Fixed PacketRecovery function
Improved FastKad
Made connection state (byConnected) volatile
Removed hardcoded sendbuffers code.
Small fix in CreateStandardPackets code.
Preparations for using SCT with AICH blocks rather than CRUMBS.
Increased hashing working buffer from 8kB to 128kB.
Fixed refreshing of download list window.
Added passing of multimedia timer value to socket send functions from UploadBandwidthThrottler, to reduce overhead of calling the timer function at each send.
Fixed performance problem in SafeKAD contact tracking cleanup routines.
Added partstatus maps for AICH to replace crumbs.
Added copying of upload partstatus to download partstatus when same file and more pieces available in the upload partstatus. Added even more consts.
Replaced some recurring calls to time/clock functions with a single call and a temp variable
Added L1 cache prefetch in RC4Crypt for x64 builds


Download:
ED2K Links
Binary x86: ed2k://|file|WARP_0.3a_binary_x86.zip|3763919|6B9DAF49D85CF841DF92FA77E986B497|h=YWM6QAXD5W3PNIB3GQLFUM2ZVFH7L6XK|/
Binary x64: ed2k://|file|WARP_0.3a_binary_x64.zip|4187463|559D57853571A4C19ED520B3ABF63FDA|h=4PN6FBQKIQTYZ3YY6BT5CGYZKL4GZY55|/
Source: ed2k://|file|WARP_0.3a_source.zip|8069415|E7B9DC04B60CA416F5746F1E1EF1BA8D|h=YM7MCHRQIRQ2EHTDDJROCECOO66MMVQN|/


Note! Use the x64 version on 64bit Windows, as the x86 version tends to be a bit sluggish (for reasons unknown); especially at startup.

NetF WARP 0.3a.14 BETA

Changelog:
*** NetF WARP 0.3a.14 BETA
- Fixed the Fakealyzer code so it works again. Also improved it by taking account the number of Kad publishers.
- Improved performance of FastKAD to be comparable in speed to a server search. (optimum performance is met after KAD has been running for a couple of minutes)
- Improved TCP-MSS Awareness; Now also checks the local interface in case your MTU is lower than the remote end (fixes problem with anonymizing VPN services)
- Added a missing security check in AsyncSocketEx code (thanks dolphinX for pointing that out)
- Removed the obsolete HTTP 1.0 choice in proxy settings (thanks dolphinX for pointing that out)
- Added new SafeKAD2 feature; Ignore problematic contacts (kind of banning but only for a short time) (doesn't seem to work at the moment)
- Added a security check to AsyncSocketEx code and fixed som type casts too
- Changed a lot of DWORD in the source to the proper clock_t, time_t and size_t types. To get rid of some typecast warnings, and to make the purpose of a data member more clear
- Added a lot of const specifiers in the source to help the compiler generate better code, and maybe to catch some bugs too!
- Reordered some code to allow the compiler to generate more compact data structures, which should improve the CPU cache hit ratio
- Replaced alot of C-style type casts to the proper C++ ones (this is safer and marks more clearly were there is code that might need fixing)
- Fixed alot of compiler warnings
- Fixed a rare deadlock in CKnownFile::CreateHash when hashing data in a CMemFile object
- Changed the file IO cache mode to random access for most operations to allow the OS to better utilize its resources
- Added new part selection algorithm that now also takes the rarity among the current transferring sources into account (currently choses parts on the left when same priority, but that is quite rare)  
- Added progress and status notification into Windows 7 taskbar (progress status need some polishing)
- Attempt to start eMule a second time will now bring the already running eMule client to front


ED2K Links
Binary x86: ed2k://|file|WARP_0.3a.14_binary_x86.zip|3873140|BC6CB01541A98459F1483D2D3F613B03|h=SJBJ6EAIBZF7V5RHEBC2SZO2Z7XNMSUT|/
Binary x64: ed2k://|file|WARP_0.3a.14_binary_x64.zip|4339397|6C4269C2D16CDCD47EC9CC2D15B63A03|h=Q2MYM4JB2Z74BISXYCVXXIUBNNYUDFTQ|/
Source: ed2k://|file|WARP_0.3a.14_source.zip|9133003|8B30C7AA88C188C86DB4858E702ADED1|h=BDLHM3V5TM64QD7VMU2G4IBBKDYIDV2Y|/


NetF WARP 0.3a.13 BETA


Changelog:
*** NetF WARP 0.3a.13 BETA
- Based on eMule 0.49c
- Replaced the new download security feature in beta 12 with the official implementation
- Replaced SafeKAD2 feature with official; Only accept as many contacts as was being asked for

*** NetF WARP 0.3a.12 BETA
- Based on eMule 0.49b
- Added security feature to make eMule downloads to be treated like IE downloads on Windows XP/Vista (NTFS only)
- Added qWave QOS support (Vista only) 
- Added TCP-MSS Awareness; Upload bandwidth throttler uses fragment size gotten from the TCP socket rather than the fixed 1300 bytes value (Vista only!)
- Added FastKAD feature; Kad searches progress imediatly when all contacts responded
- Added FastKAD feature; ICMP unreachable response from a contact cause a serach to stop waiting for it to respond
- Added new SafeKAD2 feature; Only accept as many contacts as was being asked for
- Added new SafeKAD2 feature; Search popular contacts first (increases hit ratio, but can technically introduce a weekness that has to be evaluated)
- Removed old SafeKAD code
- Lots of debugging and optimization


ED2K Links
Binary x86: ed2k://|file|WARP_0.3a.13_binary_x86.zip|3706450|14001B674AAF480D14288030549BA2BC|/
Binary x64: ed2k://|file|WARP_0.3a.13_binary_x64.zip|4074033|73EF506358875814BFB116FE44A3E01E|/
Source: ed2k://|file|WARP_0.3a.13_source.zip|7706572|DEFC86471C5F650034D575F88F3F10D0|/


NetF WARP 0.3a.11 BETA

Changelog:
*** NetF WARP 0.3a.11 BETA
- Added support to launch files directly with WINE Gnome & KDE desktops from eMule.
- Fixed a problem with checkin in the code to my private VSN server.

*** NetF WARP 0.3a.10 BETA
- Dropped PeerCache due to problematic code and that it is rarely/never used.
- Switched to SECURE CRT functions in a lot of places.
- Made Fakealyzer a separate module and added more color levels.
- Changed CrumbMaps to pack 8 status bits per byte. (less memory, more CPU)
- Changed UpdatePartsInfo related code to use much less CPU but with a little bit less precision.
- Fixed an issue with files >4GB in KAD.
- Fixed AsyncSocketEx deadlocks and memory corruption.
- Fixed deadlock in debug mode when opening dialog windows.
- Fixed a lot of warnings!


ED2K Links
Binary X86: ed2k://|file|WARP_0.3a.11_binary_x86.zip|3695317|BF090FFB74D082D93544F275EEEFEEE3|/
Binary AMD64: ed2k://|file|WARP_0.3a.11_binary_amd64.zip|4623991|1E681E44741FEFCD06EC4CD960CDC807|/
Source: ed2k://|file|WARP_0.3a.11_source.zip|6751250|FB63D118E6DD2EC1F5364FD626A818D2|/


NetF WARP 0.3a.9 BETA

Changelog:
*** NetF WARP 0.3a.9 BETA
- Updated code to compile with Visual C++ 2005 and in 64 bit mode using Vista SDK
- Updated to latest AsyncSocketEx code
- Changed library set to use the ones compiled by Howe
- Added CPU load limiter to main thread
- Changed compression levels to better reflect the higher performance of 64 bit machines
- Changed KAD search algorithm to better deal with rouge nodes

*** NetF WARP 0.3a.8 BETA
- Changed KAD to only add nodes that appears useful
- Fixed source distribution with some files that wasn't checked in 

Changelog:
*** NetF WARP 0.3a.7 BETA
- Added "Track and ban" functionality to KAD 
- Added message bounce protection to KAD
- Changed KAD to only add contacts received throught searches after succesfully managed to receive new contacts throught them
- Changed KAD NODE searches to choose a contact at a random distance from the target (gives better distribution of contacts)
- Optimized KAD searches to always attempt to find the 8 best contacts
- Fixed a bug that could cause NetF WARP to crash when a part is corrupted
- Fixed a bug causing some corrupt parts to download forever

- Note! There was an unmentioned change in NetF WARP 0.3a.6 BETA that makes the download throttling more stable.


ED2K Links
Binary X86: ed2k://|file|WARP_0.3a.9_binary_x86.zip|3355934|0E48935A64AFCAF74546755C8B40F054|/
Binary AMD64: ed2k://|file|WARP_0.3a.9_binary_amd64.zip|3638173|E12157F334C4A33B7C3B51A3D833C37D|/
Source: ed2k://|file|WARP_0.3a.9_source.zip|4056761|2CD268684908164E4A905EF8A4BCE266|/


NetF WARP 0.3a.6 BETA

Changelog:
*** NetF WARP 0.3a.6 BETA
- Based on eMule 0.48a
- Added filtering of KAD nodes based on if the ID xor difference between own / source node have more than 6 bytes identical
- Added recalculation of own KAD ID if more than 6 bytes are identical
- Added modified NetF WARP icon [thanks Typoteus for the original one]
- Fixed lookup when not receiving valid KAD recovery data
- Fixed some messed up type casts
- Fixed a bug with the long term (2hrs) datarate calculation for clients
- Fixed some crashes related to incorrect pointer references
- Optimized CUInt128 class code (made inline)
- Put some often used code inlined


ED2K Links
Binary: ed2k://|file|WARP_0.3a.6_binary.zip|2549754|B96265FFF7F9031DFDE3A5B579031C25|/
Source: ed2k://|file|WARP_0.3a.6_source.zip|4426873|8A37F29B6805BCD24397F7E602D0258F|/


NetF WARP 0.3a.5 BETA

Changelog:
*** NetF WARP 0.3a.5 BETA
- Added 'Fakealyzer' (Colour guiding on search results)
- Fixed a few bugs around the new part/crumb map handling code

Changelog:
*** NetF WARP 0.3a.4 BETA
- Some changes to improve download throttling stability
- Fixed the issue with clients getting slots without having any waiting time / score
- Fixed a rollover issue with the datarate calculations (Only occured on very fast links)
- Fixed a numerous bugs with files larger than 4GB in the code introduced with the Crumbs support
- Fixed an issue causing part files to not load after restarting the application
- Fixed some issues with downloading incomplete files smaller than one part


ED2K Links
Binary: ed2k://|file|WARP_0.3a.5_binary.zip|2414415|00E74AB6F0A8398B59FECF05BA0A3149| |h=BFSWOTENHRY2OWSIK3BZRQERGL7TX7KE|/
Source: ed2k://|file|WARP_0.3a.5_source.zip|4106704|F71E301662E4E791E476805F917230A3| |h=5EEMVGDJL3RIDJQUCZXKPKA5334422HL|/


NetF WARP 0.3a.3 BETA

Changelog:
*** NetF WARP 0.3a.3 BETA
- Added basic Crumbs support (Only publishing and download. No verification!)
- Added the 'Fixed TCP port' feature as configurable through the 'NetF.ini' file (set SymmetricTCP=1 in [NetF] section)
- Added a NetF logo in the splash screen [thanks Typoteus]
- Added additional IP checks in 'Safe KAD' code before storing a node
- Fixed a few merging errors that made some lines go missing in the KAD code


Download:
ED2K Links
Binary: ed2k://|file|WARP_0.3a.3_binary.zip|2413238|E2563482609BE7969D03AD7CE49FFFC4| |h=GGSACHCIQGA4QAHWWGCOZ6G5RMX5VD5P|/
Source: ed2k://|file|WARP_0.3a.3_source.zip|4103485|495878A2750B38FABA6FBD2A96615BB4| |h=FO427UQS3ZBIXKNTQKGJXVB52EWWQHIE|/


NetF WARP 0.2b

Changelog:
*** NetF WARP 0.2b
- Based on eMule 0.47c
- Added disconnection of very slow upload sessions
- Removed some 'Safe KAD' code that become redundant with the 0.47c release
- Removed the 'Fixed TCP port' feature as it caused problem for some
- Changed 'VIP Queue' so VIP upload is always allowed upto 10% of your upload bandwidth even if you have a bad ratio
- Changed, files in release cause clients to automatically become VIP's
- Fixed a small bug in the server list
- Fixed some parts in the upload handling

- Note! There was an unmentioned change in NetF WARP 0.2a that disables the friend auto VIP, this change is still in place.


ED2K Links
Binary: ed2k://|file|WARP_0.2b_binary.zip|2445871|131FED770872570ADE5770D2EDEE07CA| |h=M3FW744NKZTFUQEXLSUHVEXYZTTBZJHW|/
Source: ed2k://|file|WARP_0.2b_source.zip|4018937|696EC07C92305EC169B5636A66E8A6EE| |h=XM6QMZRBPLLZDD34LHCBOUB7CMVEF3GD|/


NetF WARP 0.2a

Changelog:
*** NetF WARP 0.2a
- Based on eMule 0.47b
- Added 'Safe KAD' (filters KAD from unrequested responses (anti-P2P spam), and also limits processed nodes by IP)
- Added 'Drop Stalled Sources' (speeds up file completion)
- Added 'Anti Fragmenting' (splits KAD search responses so they (hopefully) fit into one IP packet)
- Added 'CorruptionBlackBoxFix' (ensure all data been processed before banning) [by Xanatos]
- Added 'Dynamic Corruption Threshold' (to avoid mass banning when using Dynamic Block Requests)
- Added 'Only Recover Banned Data' (only redownload data that is "known" to be corrupt)
- Added 'Fixed TCP port' (may help some clients behind NAT/Firewalls to get HighID)
- Added 'Smart Part File Push' (shifts focus between partial and complete files depending on actual UL/DL ratio)
- Added 'Slot Limiting' (tries to reduce the number of upload slots)
- Added a lot of small optimizations and fixes from the NetF Mod. [by various modders]
- Updated 'Flush Thread' to do more of the work in the actual thread

Changelog:
*** nAM 0.1a BETA
- Based on eMule 0.47a
- Added 'Delayed NNP' (speeds up file completion when there is corruption or problem with the source finalizing)
- Added 'VIP Queue' (good uploaders and friends get their own lane)
- Added 'Known Client Bonus' (gives a small bonus to known clients not seen for 12 hours)
- Added 'Transfer Rate BlackBox' (hides datarate calculations from programmer)
- Added 'Dynamic Block Requests' (speeds up file completion)
- Added 'Active Ratio'(must have when using 'Dynamic Block Requests' to prevent excessive downloading)
- Added 'Safe Hash' (part hashing is done in separate thread) [by SLUGFILLER]
- Added 'ReadBlockFromFileThread' (prevents GUI to become sluggish when uploading) [by SiRoB / MorphXT]
- Added 'Flush Thread' (may help a little when doing heavy downloading) [by SiRoB / MorphXT]
- Added 'Mod version' (makes it possible to distinguish the client from the official one)
- Added 'Anti Shape' (makes credit shaping less profitable)
- Fixed memleak! Deletion in lastcommonroutefinder. [by tHeWiZaRdOfDoS]


ED2K Links
Binary: ed2k://|file|WARP_0.2a_binary.zip|2442282|4F4BFC8E20318C2DE37A62A5D6DEB8F1| |h=LTRAXDAFAABVC2CVQXIAGTI2NFG54ZQY|/
Source: ed2k://|file|WARP_0.2a_source.zip|4001445|B76799A215EF8B26C4AFF716A768E98E| |h=FPMW3466QZN4MRRQCUHUSFYQVTQPUOKM|/

This post has been edited by netfinity: 27 August 2013 - 08:10 PM

eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
2

#2 User is offline   leexgx 

  • UK MAD FOR LESS
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2533
  • Joined: 04-November 02

Posted 11 September 2006 - 12:23 AM

nice testing :)
in and around
0

#3 User is offline   leexgx 

  • UK MAD FOR LESS
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2533
  • Joined: 04-November 02

Posted 11 September 2006 - 12:47 AM

seems to be an problem with it been able to keep upload slots

11/09/2006 01:43:35: Removing client from upload list: Uploading to client without socket? (CUploadQueue::Process

its doing that alot

This post has been edited by leexgx: 11 September 2006 - 12:47 AM

in and around
0

#4 User is offline   white lightning 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1311
  • Joined: 18-April 06

Posted 11 September 2006 - 01:26 AM

Finally here !
Thanks netfinity :+1:
rEally lOve eMule & tHis fOrum
0

#5 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 04:55 AM

View Postleexgx, on Sep 11 2006, 02:47 AM, said:

seems to be an problem with it been able to keep upload slots

11/09/2006 01:43:35: Removing client from upload list: Uploading to client without socket? (CUploadQueue::Process

its doing that alot
Strange, can't remember I ever seen that.

What OS and service pack are you using?
Is protocol obfuscation enabled?
Do you use Peer Cache?

Regards,
netfinity
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#6 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 11 September 2006 - 05:15 AM

Hmmm Netfinity I'm just wondering about the KAD part - shouldn't you discuss KAD related changes first before applying them?
Have to download the sources and check it out, though... keep it up ;)
0

#7 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 06:33 AM

View PosttHeWiZaRdOfDoS, on Sep 11 2006, 07:15 AM, said:

Hmmm Netfinity I'm just wondering about the KAD part - shouldn't you discuss KAD related changes first before applying them?
Have to download the sources and check it out, though... keep it up ;)
Yes, I know. But, considering the activity in the development forum I doubt I would have gotten much response. So, I thought better release the solution and force a discussion.

There is one thing about the KAD changes I'm a bit unsure how to best deal with and that is when a KAD node does an IP change as we can't really trust a node doing that without some sort of validation. The benefit of not allowing IP changes is that nodes that are NAT'ed but believes they are HighID will be dropped as they fail to respond. They might pop in again once dropped, but it gives a chance a better node might take it's place instead.

/netfinity
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#8 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 11 September 2006 - 09:03 AM

Just dug a bit in your code...

Questions:
why don't you simply use that code:

Quote

if (!bTrim || iBit != 0)
{
pstr->AppendFormat(_T("%d"), iBit);
bTrim = false;
}

in the UInt128.cpp - it should do the same without the additional check (or is that where you save the CPU!?)

Instead of the additional bool bAdd can't you simply check for bUpdate being false? I *think* this should do the same, no?


Have to check the rest of your code - many changes this time :lol:
0

#9 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 09:50 AM

View PosttHeWiZaRdOfDoS, on Sep 11 2006, 11:03 AM, said:

Just dug a bit in your code...

Questions:
why don't you simply use that code:

Quote

if (!bTrim || iBit != 0)
{
pstr->AppendFormat(_T("%d"), iBit);
bTrim = false;
}
in the UInt128.cpp - it should do the same without the additional check (or is that where you save the CPU!?)
Now I don't have the code in front of me, but if I recall correctly I did append a static string being "0" or "1" depending on the state of iBit. Yes, it is the formatting (%d) operation that is time consuming.

View PosttHeWiZaRdOfDoS, on Sep 11 2006, 11:03 AM, said:

Instead of the additional bool bAdd can't you simply check for bUpdate being false? I *think* this should do the same, no?
Yes, I could. However there might be cases where you might want to allow both operations depending on if the node already exist or not. Actually I did allow both operations on KADEMLIA_HELLO_REQ from the beginning until I decided it was a major sequrity risk, as it would allow for an attacker to take over the ID of good nodes.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#10 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 11 September 2006 - 04:44 PM

Hmmm netfinity do you think it's such a good idea to wrap the seeking/writing of the partfile handle into the separate thread? Other threads (AICH) might access it, too and cause trouble by changing the position (or vice versa)
0

#11 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 06:27 PM

View PosttHeWiZaRdOfDoS, on Sep 11 2006, 06:44 PM, said:

Hmmm netfinity do you think it's such a good idea to wrap the seeking/writing of the partfile handle into the separate thread? Other threads (AICH) might access it, too and cause trouble by changing the position (or vice versa)
Isn't that what Slugfillers CReadWriteLock is for?
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#12 User is offline   tHeWiZaRdOfDoS 

  • Man, what a bunch of jokers...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5630
  • Joined: 28-December 02

Posted 11 September 2006 - 06:55 PM

Hmmm I'm not absolutely sure... just thought of some issues I encountered when I wrote a threaded Import/Export feature.

It would be great to get it out of the main thread... I have an additional proposal, what do you think of this:
  • use a CArray instead of the CList
  • make that array a member of the FlushThread instead of CPartFile
  • in Run() quicksort that array
  • parse the array, it's sorted so we only have to move nextitem->startpos - curitem->startpos from the current pos

0

#13 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 07:03 PM

Has anyone tested this build yet?

It would be nice to hear your opinions. Is it better than X or did I screw up completly? :D

I need to say one thing about Safe KAD. The results of Safe KAD isn't instant but should be noticable after a couple of hours when the KAD nodes have been refreshed. I have noticed between 50-200% increase in search results myself even thought I use IP filter. Ofcourse it could just be that I have a lucky week!

Regards,
netfinity

This post has been edited by netfinity: 11 September 2006 - 07:04 PM

eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#14 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 07:19 PM

View PosttHeWiZaRdOfDoS, on Sep 11 2006, 08:55 PM, said:

Hmmm I'm not absolutely sure... just thought of some issues I encountered when I wrote a threaded Import/Export feature.

It would be great to get it out of the main thread... I have an additional proposal, what do you think of this:
  • use a CArray instead of the CList
  • make that array a member of the FlushThread instead of CPartFile
  • in Run() quicksort that array
  • parse the array, it's sorted so we only have to move nextitem->startpos - curitem->startpos from the current pos
Yes, I'd like that idea! Shouldn't really make that much difference but it's a more logical design. The list is supposed to be sorted already, but it might be more CPU efficient to do it in the Flush Thread once than everytime a new buffered entry is added.

The problem I've encountered was related to completing the file where you have to ensure that Flush Buffers doesn't return until the buffers had actually been written to the file when calling from the hash thread. I think there was some other issue too, but I don't remember. It was some sort of race condition!
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#15 User is offline   Derbet 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 14
  • Joined: 14-March 06

Posted 11 September 2006 - 07:48 PM

I have tested this build about 10 hours, uploadspeed stablely, but 5 hours later, LowID more and more, about 80% on upload queue, same the wait queue, because "safe kad"? i don't like that, can fix?

Sorry my bad English. -_-!

Regards Derbet
0

#16 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 08:21 PM

View PostDerbet, on Sep 11 2006, 09:48 PM, said:

I have tested this build about 10 hours, uploadspeed stablely, but 5 hours later, LowID more and more, about 80% on upload queue, same the wait queue, because "safe kad"? i don't like that, can fix?
Very strange! Safe KAD should not have anything to do with your upload queue as it is only used for finding files. One possibility is that you have uploaded to most of the HighID clients and that there is only LowID clients left that wants anything from you.

How many clients do you have on the waiting queue?
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#17 User is offline   Derbet 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 14
  • Joined: 14-March 06

Posted 11 September 2006 - 08:48 PM

View Postnetfinity, on Sep 12 2006, 04:21 AM, said:

View PostDerbet, on Sep 11 2006, 09:48 PM, said:

I have tested this build about 10 hours, uploadspeed stablely, but 5 hours later, LowID more and more, about 80% on upload queue, same the wait queue, because "safe kad"? i don't like that, can fix?
Very strange! Safe KAD should not have anything to do with your upload queue as it is only used for finding files. One possibility is that you have uploaded to most of the HighID clients and that there is only LowID clients left that wants anything from you.

How many clients do you have on the waiting queue?


Maybe the cause like you said, this is my problem, I'll go on testing until your next build.

About 1000 clients on waiting queue.

thx your response :+1:

regards
0

#18 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 11 September 2006 - 09:01 PM

Hmm, 1000 should be fine! I had problems before when the queue drops below 100, but that doesn't happen often. There is however alot of changes in my code so I'm not too suprised if there is something wrong with it.
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

#19 User is offline   leexgx 

  • UK MAD FOR LESS
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2533
  • Joined: 04-November 02

Posted 12 September 2006 - 01:09 AM

sorry about fast post
----------------------------------------------------------
every one testing turn on verbose and turn on log upload download events then you see the socket errors report if you get them or not (i get Lots of them and i only have 200 in my upload que)
----------------------------------------------------------
OS is win3k > IPcop Linux 2.4 router > internet
Is protocol obfuscation enabled? no (default options)
Do you use Peer Cache? (UK has none > default options)

the upload socket errors are booting High ID users

low ID are only in your upload que (responding to the other users says he is ONLY seeing LOW id in he upload que) that statment is right

me and that other user reporting same problem

This post has been edited by leexgx: 12 September 2006 - 01:15 AM

in and around
0

#20 User is offline   netfinity 

  • Master of WARP
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1658
  • Joined: 23-April 04

Posted 12 September 2006 - 07:47 AM

Considering that the problem is related to uploading to HighID clients, my guess is that the problem is with creating outgoing sockets. So, it must be related to the code changes I made to make TCP NAT traversal possible. The question is why it happens for you and not for me!?

Well, guess I'll have to make a build with it disabled.

/netfinity
eMule v0.50a [NetF WARP v0.3a]
- Compiled for 32 and 64 bit Windows versions
- Optimized for fast (100Mbit/s) Internet connections
- Faster file completion via Dynamic Block Requests and dropping of stalling sources
- Faster searching via KAD with equal or reduced overhead
- Less GUI lockups through multi-threaded disk IO operations
- VIP "Payback" queue
- Fakealyzer (helps you chosing the right files)
- Quality Of Service to keep eMule from disturbing VoIP and other important applications (Vista/7/8 only!)
0

  • Member Options

  • (22 Pages)
  • +
  • 1
  • 2
  • 3
  • Last »

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users