[Nissenice]

Hi

2009-10-30 02:06:04: Wrote 2 contacts to file.
2009-10-30 02:06:04: Wrote 2 contacts to file.
2009-10-30 02:06:04: Kad merged 1 Zones

^^ I've been noticing for quite a time that a nodes.dat containing 200 nodes is overwritten with just 1 or 2 nodes occasionally. The mule is running 24/7 and this has been happening after approximately 24h. Recently I've discovered it can happen a bit earlier too. Normally this is not a problem as a more functional nodes.dat will be saved when KAD is stopped or eMule is closed. But if something happens in between so that eMule is not closed properly, you'll end up with a bit short nodes.dat. With just one or two nodes the chances is high that you can't connect to the network and IMO it's better to not touch the old one as it can be used for many months. Especially the special nodes.dat.

After checking the KAD source I think I've found what's causing this behaviour.

The root zone (CRoutingZone::CRoutingZone()) is initialized when Kademlia is started with it's m_pSuperZone set to NULL (Init(NULL, 0, CUInt128((ULONG)0))), which is also one of the conditions for reading the nodes.dat (CRoutingZone::Init) and written to it (CRoutingZone::~CRoutingZone())

CRoutingZone::~CRoutingZone()
{
	// Root node is processed first so that we can write our contact list and delete all branches.
	if ((m_pSuperZone == NULL) && (m_sFilename.GetLength() > 0))
	{
		// Hide contacts in the GUI
		theApp.emuledlg->kademliawnd->HideContacts();
		WriteFile();
	}
	// If this zone is a leaf, delete our contact bin.
	if (IsLeaf())
		delete m_pBin;
	else
(..)
}

This makes me believe that the intention is that nodes.dat should be read when the root zone is created and written to when the root zone is destroyed.

But in (CRoutingZone::Consolidate()), which is called from CKademlia::Process() every 45 minutes, under a certain condition two subzones's superzone's are set to Null before the subzones are destructed which if I have got it right explains why only a few nodes is saved in nodes.dat.

uint32 CRoutingZone::Consolidate()
{
(..)
	if( m_pSubZones[0]->IsLeaf() && m_pSubZones[1]->IsLeaf() && GetNumContacts() < K/2 )
	{
(..)
		m_pSubZones[0]->m_pSuperZone = NULL;	// <-- Can safely be removed ?
		m_pSubZones[1]->m_pSuperZone = NULL;	// <-- ditto
		m_pSubZones[0]->m_pBin->m_bDontDeleteContacts = true;
		m_pSubZones[1]->m_pBin->m_bDontDeleteContacts = true;
		delete m_pSubZones[0];			// <-- destroyed when it's m_pSuperZone == NULL
		delete m_pSubZones[1];			// <-- destroyed when it's m_pSuperZone == NULL
		m_pSubZones[0] = NULL;
		m_pSubZones[1] = NULL;

(..)
}

I'm running a test now that will confirm my thoughts - I hope..

This post has been edited by Nissenice: 15 November 2009 - 06:50 PM

[Enig123]

Good catch!

I've noticed this problem for quite some time but with no further investigations. Now I wonder how can we fix this issue. This may become annoying when more and more people rely on kad. Imagine nodes.dat be written and electric power is down, eMule may not working unless there's a copy of nodes.dat or there's a working server available.

[Nissenice]

Thanks!


I run that test (trace) and it convinced me that the suspicion was correct.

2009-11-16 02:45:35: Removed 0 keyword out of 2508 and 238 source out of 3022
2009-11-16 02:46:00: Cleaned up Kad Incoming Requests Tracklist, entries before: 462, after 15
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()
...
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()   ( m_pSubZones[0]->IsLeaf() && m_pSubZones[1]->IsLeaf() && GetNumContacts() < K/2 ) == True
2009-11-16 02:47:35: ---- CRoutingZone::~CRoutingZone()
2009-11-16 02:47:35: ---- CRoutingZone::WriteFile()
2009-11-16 02:47:35: Wrote 1 contact to file.
2009-11-16 02:47:35: ---- CRoutingZone::~CRoutingZone()
2009-11-16 02:47:35: ---- CRoutingZone::WriteFile()
2009-11-16 02:47:35: Wrote 3 contacts to file.
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()   ( m_pSubZones[0]->IsLeaf() && m_pSubZones[1]->IsLeaf() && GetNumContacts() < K/2 ) == True
2009-11-16 02:47:35: ---- CRoutingZone::~CRoutingZone()
2009-11-16 02:47:35: ---- CRoutingZone::WriteFile()
2009-11-16 02:47:35: Wrote 2 contacts to file.
2009-11-16 02:47:35: ---- CRoutingZone::~CRoutingZone()
2009-11-16 02:47:35: ---- CRoutingZone::WriteFile()
2009-11-16 02:47:35: Wrote 1 contact to file.
2009-11-16 02:47:35: ---- CRoutingZone::Consolidate()
...
2009-11-16 02:47:36: ---- CRoutingZone::Consolidate()
2009-11-16 02:47:36: Kad merged 2 Zones
2009-11-16 02:47:55: Ignored kad contact (IP=211.xx.xx.180:4803) - IP filter (Keimyung University)

Properties of nodes.dat afterwards: size 46 byte, last changed 16 november 2009, 02:47:35.

Enig123, on 16 November 2009 - 02:07 AM, said:

Now I wonder how can we fix this issue. This may become annoying when more and more people rely on kad. Imagine nodes.dat be written and electric power is down, eMule may not working unless there's a copy of nodes.dat or there's a working server available.

Another issues could be computer crash or hang. Not sure what's happening if you begin to lose the connection to Kad, dropping nodes because of router issues for example.

I think the fix is simple. Just remove

		m_pSubZones[0]->m_pSuperZone = NULL;
		m_pSubZones[1]->m_pSuperZone = NULL;

before the deletions of the zones in CRoutingZone::Consolidate()

		delete m_pSubZones[0];
		delete m_pSubZones[1];

so the only part that will be executed in CRoutingZone::~CRoutingZone() is

	// If this zone is a leaf, delete our contact bin.
	if (IsLeaf())
		delete m_pBin;

This post has been edited by Nissenice: 16 November 2009 - 09:03 AM

[Some Support]

Might be indeed an issue, thanks for pointing out.

[CarlosMC]

Some Support, on 16 November 2009 - 09:37 PM, said:

Might be indeed an issue, thanks for pointing out.

Hi all,

I had this issue happen to me twice, recently - I'm having router related dns issues (timeouts), so don't know if this was the cause, but for the second time in a few days I lost all my nodes (except for a couple, in the first time) and wasn't able to reload the list any other way but by importing a new file. I'm using x-ray v1.7 (0.49c) under win7 x64.

Cheers,

Carlos