Official eMule-Board: Bootstrapping Kad - Official eMule-Board

Jump to content


Page 1 of 1

Bootstrapping Kad

#1 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4580
  • Joined: 13-May 07

Posted 18 January 2013 - 02:34 PM

Recently I see quite a few posts with the subject KAD is not connecting.

To check the situation I use kind of clean install: no shared files, no downloads, no servers, just a nodes.dat from eMule's version 0.50a installation files.
I did such tests shortly after the release of 0.50a, then a few months later. It worked, KAD was connecting.
But not today.

The verbose log shows:
18.01.2013 12:13:07: Loaded Bootstrap nodes.dat, selected 50 out of 1200 valid contacts

Later:
18.01.2013 12:14:58: Trying to Bootstrap Kad from 59.54.220.*, Distance: 0A88DF7869490EEBBD713449339CFCA2, Version: 7, 0 Contacts left
18.01.2013 12:15:02: No valid client for requesting external port available

Stopping KAD at this point overwrites nodes.dat and leaves in it 0 contacts; all other 1150 contacts are lost.
Copying again the same nodes.dat does not help because exactly the same IPs would be tried, even in the same order.

Luckily, RoutingZone.cpp contains sufficiently clear comments: bootstrapping code selects the first 50 closest nodes, the rest is dropped.
I decided that this could be fixed by selecting all available nodes, sorted by the distance. That would use all possible contacts while avoiding overload for certain IPs the same way as it was designed.
It was enough to comment out just 5 lines (with //--- sequence) in RoutingZone.cpp, starting at line 318
//---	if (CKademlia::s_liBootstapList.GetCount() < 50 || CKademlia::s_liBootstapList.GetTail()->GetDistance() > uDistance){
		// look were to put this contact into the proper position
		bool bInserted = false;
		CContact* pContact = new CContact(uID, uIP, uUDPPort, uTCPPort, uMe, uContactVersion, 0, false);
		for (POSITION pos = CKademlia::s_liBootstapList.GetHeadPosition(); pos != NULL; CKademlia::s_liBootstapList.GetNext(pos)){
			if (CKademlia::s_liBootstapList.GetAt(pos)->GetDistance() > uDistance){
				CKademlia::s_liBootstapList.InsertBefore(pos, pContact);
				bInserted = true;
				break;
			}
		}
		if (!bInserted){
//---			ASSERT( CKademlia::s_liBootstapList.GetCount() < 50 );
			CKademlia::s_liBootstapList.AddTail(pContact);
		}
//---		else if (CKademlia::s_liBootstapList.GetCount() > 50)
//---			delete CKademlia::s_liBootstapList.RemoveTail();
//---	}


Started compiled eMule again with the same nodes.dat. I had to wait, but finally, after about 13 minutes and 372 attempts:
18.01.2013 14:04:50: Trying to Bootstrap Kad from 125.110.65.*, Distance: 008A1C684F92BC7B83916203969A3E2C, Version: 4, 1199 Contacts left
. . .
18.01.2013 14:17:15: Trying to Bootstrap Kad from 88.161.93.*, Distance: 411F02B75688B4AB4FA235450E8EAB92, Version: 8, 838 Contacts left
18.01.2013 14:17:15: Starting NODEFWCHECKUDP Kad Search
18.01.2013 14:17:15: Inc Kad2 Bootstrap Packet from 88.161.93.*


Which means, there are very few active contacts in the old file: 1200/372=3.2
Guess the chances for connection with only 50 contacts selected now.

What should be changed in my opinion:

1. If there are no big changes in code, there could be minor fixes and updated third party libraries. I do not see why devs could not compile versions 0.50b, c, d etc. and refresh nodes.dat too. And server.met.
2. There is no clear indication of the bootstrapping progress (except verbose log), so most users do not know when to keep waiting, and when it's time to try something else.

This post has been edited by fox88: 19 January 2013 - 02:58 PM

2

#2 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3634
  • Joined: 27-June 03

Posted 03 April 2015 - 09:20 AM

I looked into this now. After I wrote in the other thread it was a good idea, I actually really only meant not overwriting the existing bootstrap nodes.dat with an empty one while we still try to bootstrap (which is already in however). The reason why only 50 nodes are selected is indeed not bombarding IPs with packets.

If you have 15 000 new installs every day (which is about the state right now) with an outdated nodes.dat which 99% contains dead nodes (4 years old), then the difference for each IP is whether they receive 15000 unwanted packets or 625. If none of the 50 nodes are active then the nodes.dat is just too old to be used. Plus trying all all 1200 nodes would take 40 minutes.

The GUI indicator idea is good however and added to the code already.

#3 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4580
  • Joined: 13-May 07

Posted 04 April 2015 - 01:08 PM

View PostSome Support, on 03 April 2015 - 12:20 PM, said:

The reason why only 50 nodes are selected is indeed not bombarding IPs with packets.

I did not try to estimate the potential scale of the problem.
The reasoning is correct; though the best way to avoid the issue would be regular updates.

The question is: will the current limit of 50 be good enough to get connected in a year (or even in half of that time)? Forum search is not helpful here, because now it starts at 2009; but from previous example it is known that 1200 contacts are nearly all gone in 3 years.
For the reference: time to try 50 nodes was less than 2 minutes, so from this point of view 100 (and even 200) are possible.

View PostSome Support, on 03 April 2015 - 12:20 PM, said:

The GUI indicator idea is good however and added to the code already.

I made a test with old nodes.dat.
Default window size makes visible only about 15 nodes; so for a while all activity was not evident.
Later red icons appeared, and finally the list gets cleared in one move when the all nodes were tried.

My idea was simpler implementation without the list of already tried nodes.
Nodes are immediately deleted instead of being copied to another list (as it was with delete pContact;).
In that case the number above the list counts down remaining nodes.
That variant could be made by deleting all code referencing s_liTriedBootstapList. The only problem would be repeating messages in the log: "Failed to bootstrap Kad....", but to stop that a simple local flag variable should suffice.

Edit.
There is a typo in list names - missing character r:
_ContactList CKademlia::s_liBootstrapList;
_ContactList CKademlia::s_liTriedBootstrapList;

This post has been edited by fox88: 04 April 2015 - 06:06 PM

0

#4 User is offline   fox88 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4580
  • Joined: 13-May 07

Posted 26 August 2015 - 09:57 PM

View Postfox88, on 04 April 2015 - 04:08 PM, said:

My idea was simpler implementation without the list of already tried nodes.
Nodes are immediately deleted instead of being copied to another list (as it was with delete pContact;).

Implementation details.

Those lines are not needed in Contact.h
//bool		IsBootstrapFailed() const						{return m_bBootstrapFailed;}
//void		SetBootstrapFailed()							{m_bBootstrapFailed = true;}
//bool m_bBootstrapFailed;

and in Contact.cpp
//m_bBootstrapFailed = fromContact.m_bBootstrapFailed;
//m_bBootstrapFailed = false;


In KadContactListCtrl.cpp[/code] replace the line
nImageShown = contact->IsBootstrapFailed() ? 4 : 5;

with
nImageShown = 5;


Three changes in Kademlia.cpp.
Remove list, add boolean flag:
//_ContactList CKademlia::s_liTriedBootstrapList;
bool		CKademlia::m_bootstrapping = false;

Next, loop for list cleanup is replaced with assignment:
m_bootstrapping = false;
//while (!s_liTriedBootstrapList.IsEmpty())
//	delete s_liTriedBootstrapList.RemoveHead();

Finally, in void CKademlia::Process() conditional operator (which deals with bootstrapping) should be:
if(!IsConnected() && !s_liBootstrapList.IsEmpty()
	&& (tNow - m_tBootstrap > 15 || (GetRoutingZone()->GetNumContacts() == 0 && tNow - m_tBootstrap >= 2)))
{
	CContact* pContact = s_liBootstrapList.RemoveHead();
	m_tBootstrap = tNow;
	m_bootstrapping = true;
	DebugLog(_T("Trying to Bootstrap Kad from %s, Distance: %s, Version: %u, %u Contacts left"), ipstr(ntohl(pContact->GetIPAddress())), pContact->GetDistance().ToHexString(), pContact->GetVersion(), s_liBootstrapList.GetCount());
	m_pInstance->m_pUDPListener->Bootstrap(pContact->GetIPAddress(), pContact->GetUDPPort(), pContact->GetVersion(), &pContact->GetClientID());
	delete pContact;
	theApp.emuledlg->kademliawnd->StartUpdateContacts();
} else if (m_bootstrapping) {
	// failed to bootstrap
	m_bootstrapping = false;
	AddLogLine(true, GetResString(IDS_BOOTSTRAPFAILED));
	theApp.emuledlg->kademliawnd->StopUpdateContacts();
}


As said, unresponsive nodes would be removed from the list and nodes count decremented.

This post has been edited by fox88: 26 August 2015 - 10:01 PM

0

  • Member Options

Page 1 of 1

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users