Official eMule-Board: Observation Of The Network - Official eMule-Board

Jump to content


  • (2 Pages)
  • +
  • 1
  • 2

Observation Of The Network Spreading of popular files, fake servers

#1 User is offline   sebasto 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 14
  • Joined: 15-February 03

Posted 22 August 2005 - 10:56 AM

Hi,

I am trying to devellop some crawlers linked to a database and a web site to make some charts about p2p networks, beginning with the edonkey network. It includes some graphs, mainly similar to the ed2khistory ones, but including more servers.

My work is still at its begining. I still have a lot of things to do, but the first graphs are begining to talk, I can follow the files spreading, see which files are the most shared, and in a more general way, study the dynamics of the network.

The site is p2p-top50, and I'll try here to say in a few words what this way of watching the network allowed me to 'discover'.

First, the most obvious, laying around since the begining, a little file (22 bytes), certainly a virus, is present on the network and replicates very well. Since two month at least, it can be seen on 6000 to 16000 computers, taking a great variety of appealing names (see a graph and the list of filenames it can take).
The ed2khistory for this file confirm the activity (on razorback servers). It is difficult to estimate the spreading rate, but as the file is extremely small, I guess it is downloaded instantly, so the difference between nb sources and nb complete sources, shows us that the file is spreading activelly.

Second - but you all know that I guess - it shows that this network is mainly a european one. I am working on an indicator to have more precise estimator of location of downloaders by file, but clearly the top 50 speaks for itself, amongst the most downloaded files, there are french, german, spanish, ... We do not observe a great US domination, as we do in bittorrent. To me, this explains why bittorrent is said to be the most popular p2p client in the world, and little word is said about emule; just because the US media are much more represented on the web than any other, and US ISPs mainly see bittorrent traffic. I would be curious to see the results of a european study.

This conducts me to a third aspect, I wanted to have an estimation of the number of times a popular file is downloaded and the amount of bandwidth that represented. I will take the most popular file of the moment as an example. It is copyrighted material, I know, but I can't do anything about that. Perhaps this could even be an obvious deduction: the most downloaded files are very oftenly copyrighted material, but this is an other debate ...
Back to our example, we can see on the graph that the spreading of the file began 08/august 2005, I have comments on the first two days of spreading, but that will be an other post. This file is clearly in a spreading state, the number of sources as well as the number of complete sources is increasing. The estimated number of downloads is 130000, representing 87 Tb of bandwidth in approx. 2 weeks (I make this estimations with an average download rate of 10kb/s , I still have no clear idea of what is the 'real' average download rate, but I think this one is a minimum on popular files.) This shows the great efficiency of the network. I have great variability, but I think this is because of my crawlers, not because of he network. A look at ed2khistory confirm this assumption - I still have place for improvements :).
If I take an older popular file which spreading began sooner, I have an estimate of nearly 300000 downloads, representing 216 Tb. I think this is a good order of magnitude of the number of download for a popular file. I did not have any idea of the number of downloads before doing those calculation. To me these a new kind of statistics available about the network, interresting ones ....

And last but not least, since the begining of august, I can clearly see the activity of the fake servers, for example on this file or this one. We can see on those graphs that the dynamics of the downloads are completely false, the nb of sources beeing almost always the same than the the nb of complete sources, it take no time to download those files !! Of course, this is due to servers falsely claiming they have a lot of sources, if we compare with dynamics on razorback servers , we can see that those files are 'rare' on 'true' servers, and poorly exchanged, even if we can see the effect of the fake servers (increase in download at 8/aug)
To me, the explanation is clear, their goal is not to log your transactions - we've seen that the number of download by files is something like hundred thousands, what would they do with those logs ? Instead, they are trying the same strategy that worked on gnutella, put junk on the network ...
I think they tryed that without fake servers first, but it didn't worked. Now they are counting on the people making global search and sorting by avaibility to spread their fakes, we will see if this strategy is 'better', to me it seems that it's not working (exept on my charts, but that is an other story, I said I still have some work ... ;) )

Just a final word to say that the devellopment of this site already took me a lot of time, but it is a real pleasure. I hope this kind of informations which were not available before will be usefull to the community, and that I will be able to improve them thanks to your comments.

Excuse me for my bad english but I'm french ...

Sebasto.

This post has been edited by Some Support: 22 August 2005 - 11:26 AM

0

#2 User is offline   rwolf969 

  • Wolf inside
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4313
  • Joined: 25-September 02

Posted 22 August 2005 - 11:09 AM

:goodpost:
0

#3 User is offline   wud03 

  • imapwnu
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 322
  • Joined: 24-March 04

Posted 22 August 2005 - 11:59 AM

Great post indeed, especially about why eMule is rarely mentioned in US media. I hope your research goes well for the rest of the way. Do drop a link when it's done and ask any questions that might help you.
0

#4 User is offline   lugdunummaster 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Member_D
  • Posts: 1040
  • Joined: 19-September 02

Posted 22 August 2005 - 12:05 PM

Quote

First, the most obvious, laying around since the begining, a little file (22 bytes), certainly a virus, is present on the network and replicates very well. Since two month at least, it can be seen on 6000 to 16000 computers, taking a great variety of appealing names (see a graph and the list of filenames it can take).
The ed2khistory for this file confirm the activity (on razorback servers). It is difficult to estimate the spreading rate, but as the file is extremely small, I guess it is downloaded instantly, so the difference between nb sources and nb complete sources, shows us that the file is spreading activelly.


I have yet to see a 22 bytes virus :)

If you refer to the file 397BA1563373D7F0E5CF915764F2A480 , this is not a virus, and a valid vile (not copyrighted as well)

You seem to think that (#AVAIL - #COMPLETE) gives you the number of people downloading a file. This assumption is wrong, because old clients dont know the #COMPLETE stuff, so they just publish a file, without saying it is complete or not.
0

#5 User is offline   sebasto 

  • Newbie
  • Pip
  • Group: Members
  • Posts: 14
  • Joined: 15-February 03

Posted 22 August 2005 - 12:52 PM

I know a 22 byte virus seems strange, but it's my guess, and the only explanation I have to the number of names it has (perhaps it's compressed). Try to download, execute, and tell me what's happening :)

The file I refer to is 3D221CF97333893FF0E434F1940F1810, It took me a lot of time to document what I say, including links to my site and ed2khistory. To my surprise, all those links (source of the information, allowing people to verify what I say and/or make their own opinion) are gone, moderated I suppose, leading to misinformation as your post shows. I really don't understand why. If people can't see this information, the post is much less informative, what is the problem with saying what's happening on the network ?

And yes, I assume (#AVAIL - #COMPLETE) = nb people downloading. When I see the stats on my mule, it seems to be a good approximation, as the great majority of clients are recent. In some rare cases, I guess we can't make this assumption, for example, the file shareaza_2.1.0.0.exe shows a strange behavior, which could be explained if shareazaa doesn't know the #COMPLETE stuff ...
I also think this approximation can only be done on large files ...


To me, the average download rate is much more problematic, I estimated it to 10kb/s, but it could be 20,30, ... I have no idea and it has great repercution on the estimated number of downloads.

This post has been edited by sebasto: 22 August 2005 - 12:54 PM

0

#6 User is offline   xnorf 

  • eMule King
  • PipPipPipPipPipPipPip
  • Group: Validating
  • Posts: 4790
  • Joined: 30-December 02

Posted 22 August 2005 - 02:22 PM

Sebasto, why do you open a new thread instead of posting in your old thread?
Oh, wait: it was binned...
And about that 22 bytes virus: It's just a empty zip file. (Create a zip, delete all files in that zip and you get that file)
0

#7 User is offline   lugdunummaster 

  • Golden eMule
  • PipPipPipPipPipPipPip
  • Group: Member_D
  • Posts: 1040
  • Joined: 19-September 02

Posted 22 August 2005 - 02:26 PM

Quote

I know a 22 byte virus seems strange, but it's my guess, and the only explanation I have to the number of names it has (perhaps it's compressed). Try to download, execute, and tell me what's happening smile.gif

The file I refer to is 3D221CF97333893FF0E434F1940F1810, It took me a lot of time to document what I say, including links to my site and ed2khistory


Congratulations ! This file is an empty zip file.

Try for yourself to create an empty zip file, with whatever filename you want.

Calling an empty zip file (or an empty rar file as in my previous posting) a virus is not very good for you if you want us to read your (long) posts.
0

#8 User is offline   coluche 

  • hm ?
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 2274
  • Joined: 02-May 05

Posted 22 August 2005 - 02:32 PM

Hej, on your site it says: "in the emule/donkey network". Is this supposed to mean kademlia and overnet included, or just ed2k-network ?
- and yes, interesting topic, so far.

first post edited? well, it says edited by some support, and he seems to have edited very nicely. I read it before and after, and everything is still understandable. but when your first post got it's links removed, you shouldn't put links in your second post. this could be seen as - errm, stubborn or whatever.

btw - get rid of that popupcrap on your page, it will keep lots of people from going there twice.
It's Screamin' Jay Hawkins and he's a Wild Man, so bug off!
0

#9 User is offline   leuk_he 

  • MorphXT team.
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5975
  • Joined: 11-August 04

Posted 22 August 2005 - 03:17 PM

Quote

to my surprise, all those links (source of the information, allowing people to verify what I say and/or make their own opinion) are gone, moderated I suppose


were they posted here, on this board, or where? (i am impressed with these results so far.)

by the way, how do you spider the ed2k network. Do you use a server that gives you links to hashes? or are you using the kadnetwork? It is quite clear to me how you try to find the # of sources for a file if you have a hash, but how do you get those initial hashes?

and that hash you name is also interresting(obviously a fake.)

/Edit: now i understand: some mod here modded your first post. I wih that they moderated more transparent here. Print a clear ruel , a plicy and show other users when that policy was violated.

This post has been edited by leuk_he: 22 August 2005 - 03:20 PM

Download the MorphXT emule mod here: eMule Morph mod

Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.

Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
0

#10 User is offline   Some Support 

  • Last eMule
  • PipPipPipPipPipPipPip
  • Group: Yes
  • Posts: 3667
  • Joined: 27-June 03

Posted 22 August 2005 - 05:00 PM

Quote

/Edit: now i understand: some mod here modded your first post. I wih that they moderated more transparent here. Print a clear ruel , a plicy and show other users when that policy was violated.


There are clear rules and one of the easiest to understand is that there are no links to copyrighted material allowed on this board (regardless if it turns otu to be a fake or not, since im not going to check every link).

#11 User is offline   Timwi 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 123
  • Joined: 17-January 05

Posted 22 August 2005 - 06:34 PM

I keep seeing references to "copyrighted material", but I challenge you to give me a link to anything even close to popular on ed2k that isn't actually copyrighted. Hint: free software (Mozilla, Shareaza, Linux) is copyrighted. "Copyrighted" doesn't mean "illegal to download".

Edit: Posting links to material that is actually illegal to download isn't illegal.

This post has been edited by Timwi: 22 August 2005 - 06:36 PM

0

#12 User is offline   Timwi 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 123
  • Joined: 17-January 05

Posted 22 August 2005 - 06:40 PM

What is the best way to find out the size of a file given only its hash? <bleep!> appears to be bigger than 22 bytes, but eMule doesn't seem to ask the sources how big it really is. Typing the hash into an eMule search, into FileDonkey or indeed into Google doesn't yield anything...

This post has been edited by PacoBell: 24 May 2006 - 10:03 AM

0

#13 User is offline   qm2003 

  • V.I.P. (Volatile Indifferent Puppet)
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 4441
  • Joined: 06-November 03

Posted 22 August 2005 - 06:42 PM

Who cares ?

It's the forumrule, which by the way you read and accepted when you registered here. And it's been handled exactly like this for quite some time.

And since we are all just guests here to enjoy the stay, we're better off playing by the houserules ...

Aren't we ?
;)

This post has been edited by qm2003: 22 August 2005 - 06:43 PM

How to setup Emule. A small checklist | Schmu's MuleDoc
P2P is not piracy, it's marketing.
In fact, if your music or movie is NOT being downloaded, you should be WORRIED !
If you can't even give it away for free, how do you expect to sell it, stupid ?


I'm a bloodsucking fiend. Look at my outfit !
0

#14 User is offline   xnorf 

  • eMule King
  • PipPipPipPipPipPipPip
  • Group: Validating
  • Posts: 4790
  • Joined: 30-December 02

Posted 22 August 2005 - 06:52 PM

Timwi, on Aug 22 2005, 08:40 PM, said:

What is the best way to find out the size of a file given only its hash?View Post
Either search for ed2k::<bleep!> or go <bleep!>.

Timwi, on Aug 22 2005, 08:40 PM, said:

<bleep!> appears to be bigger than 22 bytes...View Post
Says who? Every source I know says it's exactly 20 bytes big.

And why isn't this thread closed yet? :-k

This post has been edited by PacoBell: 24 May 2006 - 10:04 AM

0

#15 User is offline   Timwi 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 123
  • Joined: 17-January 05

Posted 22 August 2005 - 07:39 PM

Quote

Who cares ?
Uhm... maybe I didn't make myself clear. The rules state that references to copyrighted material will be removed, but the forums and even the main website contain loads of references to copyrighted material (such as free software). The term should be changed to "illegal material", and people should be made aware that copyrighted doesn't mean illegal, and that free software is copyrighted.
0

#16 User is offline   Timwi 

  • Splendid Member
  • PipPipPipPip
  • Group: Members
  • Posts: 123
  • Joined: 17-January 05

Posted 22 August 2005 - 07:48 PM

xnorf, on Aug 22 2005, 07:52 PM, said:

Timwi, on Aug 22 2005, 08:40 PM, said:

What is the best way to find out the size of a file given only its hash?View Post
Either search for ed2k::<bleep!> ...
Thanks! That was what I was looking for.

xnorf, on Aug 22 2005, 07:52 PM, said:

Timwi, on Aug 22 2005, 08:40 PM, said:

<bleep!> appears to be bigger than 22 bytes...View Post
Says who? Every source I know says it's exactly 20 bytes big.
Oops, I meant "different from exactly 22 bytes" :)

This post has been edited by PacoBell: 24 May 2006 - 10:05 AM

0

#17 User is offline   Andu 

  • Morph Team
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 13015
  • Joined: 04-December 02

Posted 22 August 2005 - 08:00 PM

Timwi, on Aug 22 2005, 09:39 PM, said:

Quote

Who cares ?
Uhm... maybe I didn't make myself clear. The rules state that references to copyrighted material will be removed, but the forums and even the main website contain loads of references to copyrighted material (such as free software). The term should be changed to "illegal material", and people should be made aware that copyrighted doesn't mean illegal, and that free software is copyrighted.
View Post


We know that tho I would call that nitpicking. You also would have to differ between copyright and copyleft.
Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.


Dark Lord of the Forum


Morph your Mule

Need a little help with your MorphXT? Click here

0

#18 User is offline   niclights 

  • lost in space
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 10288
  • Joined: 01-November 04

Posted 22 August 2005 - 09:38 PM

Indeed! :lol:

I'm sure the experienced here will agree that picking holes in the forum rules is a surefire way to annoy. It is not the details, but the reason for the rules that is important. To me the reasons seem obvious and just common sense.

Oh bugger. Another one goes offtopic....

OMG. Am I really listening to a big band version of Nirvana/Teen Spirit? :shock: hmmm... actually not too bad... da da dum di da da... :)
0

#19 User is offline   moloko+ 

  • ...
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 1209
  • Joined: 18-August 05

Posted 23 August 2005 - 02:57 AM

interesting.
it is our desire that makes it a virus or some_app.zip, and for the folks above empty.zip... hence it (we) replicate it hoping it will be what.we.want... being the generic empty.zip it assumes many names, and then under hasty preparations is left in a shared folder... and so, uploaded with LOL, cos u want it, it appears to behave like a virus... i think the inverse is true.

0b can't be good but 22b is plausible (to noob, me)...

also great stats, have uses (more work) below:

couldn't the stats provide a way of identifying junk like known "empty" file types (that incidently have a polymorphic nature [renamed]) and stop them from being uploaded at all. or included in searches at the client level. a candidate would be this file empty.zip...

on the other hand this file is a creature of the network, a successful and unique one, and around because we wanted it. family foto hxxp://www.p2p-top50.com/ :respect:
0

#20 User is offline   leuk_he 

  • MorphXT team.
  • PipPipPipPipPipPipPip
  • Group: Members
  • Posts: 5975
  • Joined: 11-August 04

Posted 23 August 2005 - 10:32 AM

Some Support, on Aug 22 2005, 07:00 PM, said:

Quote

/Edit: now i understand: some mod here modded your first post. I wih that they moderated more transparent here. Print a clear ruel , a plicy and show other users when that policy was violated.


There are clear rules and one of the easiest to understand is that there are no links to copyrighted material allowed on this board (regardless if it turns otu to be a fake or not, since im not going to check every link).
View Post



checking the rules:

Quote

Violation of this rule will result in the references being removed and the user warned and/or immediately banned.


You managed to do the first part (remove refererences). But failed to warn the user, since he seems not to understand what was. After checking the history of the user i can understand since his only prior post was a binned post to a doubtful site.

I am not against removing these references, but i am all for making more effort to warn the users. Most binned messages are just binned/edited without any expanation whatsoever. This is going to make users make the same mistake 10 minutes later. (also a better definition of illegal could be made, emule thrives on the content you call illegal.)

I hope i made my opinion clear.

This post has been edited by leuk_he: 23 August 2005 - 11:04 AM

Download the MorphXT emule mod here: eMule Morph mod

Trouble connecting to a server? Use kad and /or refresh your server list
Strange search results? Check for fake servers! Or download morph, enable obfuscated server required, and far less fake server seen.

Looking for morphXT translators. If you want to translate the morph strings please come here (you only need to be able to write, no coding required. ) Covered now: cn,pt(br),it,es_t,fr.,pl Update needed:de,nl
-Morph FAQ [English wiki]--Het grote emule topic deel 13 [Nederlands]
if you want to send a message i will tell you to open op a topic in the forum. Other forum lurkers might be helped as well.
0

  • Member Options

  • (2 Pages)
  • +
  • 1
  • 2

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users