[BigRedBrent]

I added this because I want a collection file that is able to be fully functional in plain text, is able to be easily compressed to save disk space, and is able to be easily backwards compatible for clients that do not want to add any support for any of the more advanced features that could be added to the format.

I want to include that the main thing that made me come up with this was the limited way emule translates its very own collection file that it has implemented. It does not allow the ability to add more information to the links such as the full hash set, and even if you add the information to the links manually in the collection file, when you load the file into emule it ignores the extra information in the link such as the full hash set.

And most of all I want it to be made so that it is easily adopted by all other ed2k compatible clients.

Here is a layout of the protocol that could be used for such a file system:








Before reading the file it will always be decompressed first if it is compressed.

A compressed ed2k file will have a line added to the top of the file containing text that will denote that the file is compressed and what compression method was used, so that it may be detected as compressed and able to be decompressed with whatever method that was used to compress it. When decompressing the file the first line will be read to determine the compression method used, then the plain text ed2k file will be extracted from the compression archive to be used.

A compressed ed2k file will simply be an ed2k file that has been added to a compressed archive file with the extension of that archive file also using *.ed2k. After the file is made a line will be added to the top of the file denoting that it is a compressed file and what compression method was used.

If you wanted to manually compress a plain text ed2k file, all you would need to do is compress it with a compression method that can be decompressed by the desired clients, then change the extension of the archive file to ed2k (or just add .ed2k at the end of the file name after the extension) and then add a line to the begging of the compressed file in the same way as the examples given below.

Example of compressed ed2k archive file with ZIP compression:

-c zip
Here is where the compressed archive data line(s) would be.

Example of compressed ed2k archive file with RAR compression:

-c rar
Here is where the compressed archive data line(s) would be.

Example of compressed ed2k archive file with 7-Zip compression:

-c 7z
Here is where the compressed archive data line(s) would be.

The files may also be manually decompressed by simply extracting the ed2k file out of the archive file. The added line to the archive file does not change the functionality of any current archivers that I know of, so removing the added line is not required to decompress the ed2k files from most archive file formats.

More then one compression format may be used for future support. Compression formats that are not supported by clients would need to be decompressed into plain text format with another utility before that client would be able to use the file, but this would be necessary to make the compression format possibly upgradeable in the future.





When not compressed the file would be in plain text format.

All spaces and tabs in the file that are in front of or behind a line will be striped away when the file is processed like most other code formats are handled.


Lines that begin with anything that is not recognized by the client will be ignored.
All lines that begin with a ; will also be ignored by clients.

All abilitys would be available and backwards compatible in plain text format.

No line but a line with only an ed2k link on it may begin with:
ed2k://

In plain text format the first line may never begin with:
-c

All the information that can be added to a ed2k link may be included with the link in the ed2k file and may be used by the client when the information is read from the file if supported by the client.



A directory structure may be added to each link but is not required to make the file nor required to read the file unless you want to use the directory structure.

Here is an example of the directory structure format with 10 ed2k example links added to it:

ed2k://|file|FileName1|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
<f="Folder Name">
	ed2k://|file|FileName2|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	ed2k://|file|FileName3|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	ed2k://|file|FileName4|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	<f="SubFolder Name">
		ed2k://|file|FileName5|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
		<f="SubSubFolder Name">
			ed2k://|file|FileName6|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
			ed2k://|file|FileName7|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
		</f>
	</f>
	<f="Another SubFolder Name">
		ed2k://|file|FileName8|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
		ed2k://|file|FileName9|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	</f>
</f>
<f="Another Folder Name">
	ed2k://|file|FileName10|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
</f>

As you can see the first ed2k link in this example is not inside of the file tree and would not be placed inside of a created folder.

The folders would be created if not already present inside of your main download folder after the files have completed downloading, so that the files could be placed inside of the created folders.

I think if you want to add directory structure support to the client you may want to add the ability to add a download destination to each file instead of just to each category. If a file specific download destination is set it should supersede the category download destination. Personally I think this ability to add a per file download destination independent of categories should be implemented regardless of how it may be used.





The author of the file may sign the file as well as add a name, comment, and title to the file.
The signature, name, comment, and title will be placed on a single line each.

The signature will be a hash that is encrypted by your private key.
This will be a hash of the entire contents of the file with the line that the signature is on excluded. This would make it so that anyone with your public key would be able to know for sure that the contents of the file had not been tampered with before they received it, as well as be able to know that this file did in fact come from you.

Here is an example of the signature, name, title, & comment line:

-signature EncryptedHashOfFileContents
-author TheGivenAuthorName
-title This would be the author's given title of the ed2k file.
-comment You would be able to place a limited length comment here.

The limited length of the comment would depend on how much the reading client would display and would cut off any excess at the end of the comment.









This file system could easily be adopted by all other ed2k clients. Each author of each ed2k client could easily decide for themselves what features they want to add support for.

If an ed2k client wants to just add the limited most basic support they could just add support for stripping away or ignoring all spaces and tabs in the file that are in front of or behind a line, and then only reading lines that begin with:
ed2k://

When the file is read by the client then all of the information in the link(s) should be used such as the full hash set if present, just as if you had manually copy and pasted the link into the ed2k client.

If other ed2k clients don't even want to add support for compression they would not need to add it since no links would be found in such a file if read and would simply be skipped.

This would be very simple to implement for other clients that just want basic reading support.











I am going to add a bunch of example files below to better illustrate the file format.


Here is an example of a full ed2k file with added functionality such as signature and a directory structure:

-signature EncryptedHashOfFileContents
-author TheGivenAuthorName
-title This would be the author's given title of the ed2k file.
-comment You would be able to place a limited length comment here.
ed2k://|file|FileName1|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
<f="Folder Name">
	ed2k://|file|FileName2|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	ed2k://|file|FileName3|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	ed2k://|file|FileName4|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	<f="SubFolder Name">
		ed2k://|file|FileName5|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
		<f="SubSubFolder Name">
			ed2k://|file|FileName6|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
			ed2k://|file|FileName7|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
		</f>
	</f>
	<f="Another SubFolder Name">
		ed2k://|file|FileName8|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
		ed2k://|file|FileName9|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
	</f>
</f>
<f="Another Folder Name">
	ed2k://|file|FileName10|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
</f>

Here is an example of a full ed2k file without a directory structure added:

-signature EncryptedHashOfFileContents
-author TheGivenAuthorName
-title This would be the author's given title of the ed2k file.
-comment You would be able to place a limited length comment here.
ed2k://|file|FileName1|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName2|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName3|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName4|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName5|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName6|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName7|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName8|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName9|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName10|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/

Here is an example of a full ed2k file without any functionality added:

ed2k://|file|FileName1|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName2|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName3|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName4|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName5|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName6|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName7|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName8|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName9|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/
ed2k://|file|FileName10|FileSize|FileID|h=AICHHash|p=CompleteHashSet|/

This post has been edited by BigRedBrent: 14 August 2007 - 10:55 PM

[CiccioBastardo]

I think the compression option can be completely skipped as eMule already compress sent data, and ASCII files usually compress well.

[BigRedBrent]

CiccioBastardo, on Jul 30 2007, 02:49 PM, said:

I think the compression option can be completely skipped as eMule already compress sent data, and ASCII files usually compress well.

I was thinking that as well but some people may store the collections on web sights or a plethora of other useful locations that may be useful to have the files compressed on if they are distributed to many people. It may be useful for the files to be as small as possible. The compression ability is however on the bottom of the list of things I am interested in, but I think it should still be included as part of the standardized format for future use. Better to add it now then to wish it had been added later but then can't without breaking backwards compatibility. It would be very simple to add as well as just as easy to add support for other clients who may wish to implement support for the file system. I mean many compression options are readily available to use for this support without giving it a second thought.

This post has been edited by BigRedBrent: 31 July 2007 - 01:43 AM

[Andu]

Erm if it's compressed you can't just edit the files. You'd have to use the client to decompress them and then edit them from within the client or with an external tool. I don't think that's a good solution.

[BigRedBrent]

Andu, on Jul 30 2007, 03:21 PM, said:

Erm if it's compressed you can't just edit the files. You'd have to use the client to decompress them and then edit them from within the client or with an external tool. I don't think that's a good solution.

Well yeah, but that would be the sacrifice of using compression. That is however why plain text format would be optional. If you want to guarantee backwards compatibility you would not use compression. That would be the choice of the author of the file though.

Man, I remember all you guys from this forum way back when I would post here. You guys never left did you?

This post has been edited by BigRedBrent: 30 July 2007 - 07:30 PM

[Andu]

Well the devs kind of bought me and now I'm just forum inventory.

This post has been edited by Andu: 30 July 2007 - 07:41 PM

[BigRedBrent]

Andu, on Jul 30 2007, 03:41 PM, said:

Well the devs kind of bought me and now I'm just forum inventory.

[basti2k]

The files in the ed2k-file are listed, can one into a large file combine also internally (The files one behind the other-hang and then as a file with hash treat) and the ED2K-file contains also information around these again into individual files to divide (reduces files on the server, in addition one can select individual files from this Stream and then only this download)

and I would replace the tags "SUBFOLDER" by "RELATIVEPATH"

Example for *.ed2k-File:

<ED2K-FILE>
   <Stream-Link>ed2k://|Stream|StreamSize|StreamID(MD4)|h=AICHHash|/</Stream-Link>
	  <FILE>
		 <Relative-Path>/Subfolder1/SubSubFolder</Relative-Path>
		 <Stream-Start>0</Stream-Start>		// Offset of Filestart in the Complete Stream
		 <Stream-End>1512454</Stream-End>
		 <File-Hash>ed2k://|file|FileName1|FileSize1|FileID1|h=AICHHash1|p=CompleteHashSet1|/</File-Hash>
	  </FILE>
	  <FILE>
		 <Relative-Path>/Subfolder1/</Relative-Path>
		 <Stream-Start>1512455</Stream-Start>
		 <Stream-End>1512460</Stream-End>
		 <File-Hash>ed2k://|file|FileName2|FileSize2|FileID2|h=AICHHash2|p=CompleteHashSet2|/</File-Hash>
	  </FILE>
</ED2K-FILE>

EDIT:
the idea for this format comes from the Torrent

Translated with Babelfish

This post has been edited by basti2k: 31 July 2007 - 08:09 AM

[BigRedBrent]

basti2k, on Jul 30 2007, 05:03 PM, said:

The files in the ed2k-file are listed, can one into a large file combine also internally (simply files to one behind the other-hang and check total more drueber form) and the ED2K-file contains also information around these again into individual files to divide (reduces files on the server, in addition one can select individual files from this Stream and then only this download)

and I would replace the tags "SUBFOLDER" by "RELATIVEPATH"

I must admit that I have no idea what you are trying to say and the example did not help me to understand.

I did notice that your example broke one of the main features of the file system that I would like to be used.

The main thing that I want in the ed2k file is that nothing is to ever be in front of or behind the ed2k links on the same line. This is to make it so a simple file can be easily created by just throwing a bunch of links in a file with one link per line if one wanted to do that, as well as the ability to add much more advanced features while still using the clear text format.

This post has been edited by BigRedBrent: 09 August 2007 - 04:51 AM

[derJörg]

Hello,

BigRedBrent, on Jul 31 2007, 01:38 AM, said:

The main thing that I want in the ed2k file is that nothing is to ever be in front of or behind the ed2k links on the same line. This is to make it so a simple file can be easily created by just throwing a bunch of links in a file with one link per line if one wanted to do that, as well as the ability to add much more advanced features while still using the clear text format.

I am going to guess that you are talking about something to do with making a collection searchable on the network without publishing the contents of the files within the collection on the network, and this would make the files unable to be found on the network unless you was able to find the collection first. If this is the case then it would have nothing to do with my request and should be discussed in a different topic post. If this is not the case then I still do not see what this has to do with my idea since I do not understand what you are proposing.

...what basti2k tried to introduce is mainly an imitation of .torrent files. I don't think this is very useful. He would be better off to create an archive and simply share it.

Collections are most useful (compared to static archives) if their contents change over time (files added, renamed, etc.), just like a subscription of certain contents. So your way of describing a collection would be a huge step forward to structured downloads of contents. I appreciate your original suggestion. However, to take real advantage of collections (here I'm having the dynamic character of collections in mind, as opposed to static archives), two more things would have to be implemented:

• A mechanism to automatically rename and move(!) files to match newer versions of a particular collection, even if the folder structure changed. This is really easy, if a collection gets its own subfolder within some incoming folder.
  
• A mechanism to resolve ambiguities (mainly file dupes, i.e. one file is referenced by two or more collections). This can easily be done with prioritized collections.

Regards,
derJörg

This post has been edited by derJörg: 31 July 2007 - 08:21 AM

[BigRedBrent]

derJörg, on Jul 31 2007, 04:18 AM, said:

Hello,..what basti2k tried to introduce is mainly an imitation of .torrent files. I don't think this is very useful. He would be better off to create an archive and simply share it.

Well yeah when it comes to wanting almost exact functionality of BitTorrent an archive file would be necessary. I just want a collection file that could be easily implemented in all ed2k clients that also has some advanced features in mind such as directory structure and full hash set storage.

I don't really see duplicates as a problem. I guess for the most part files you already have downloaded or downloading would be ignored. Anything more then that would get into complications.

I suppose the most that could be done is have a prompt come up with every instance of such situation asking if you want to add already downloaded or downloading files to the collections file structure if it is not already in the desired directory. maybe an option to have it always ask this or never ask and either not move the file or automatically always move the file if needed.

Again I don't feel that this is necessary to add right away and could be added later as an additional feature that is purely the decision of the client developers to add or not.

This post has been edited by BigRedBrent: 09 August 2007 - 04:55 AM

[BigRedBrent]

I have updated this post some to fill in a few gaps to refine this conceptual format.

I wonder if any developer will ever give this any weight. I mean this feature would be very useful to those who want this functionality. And the implementation of this feature should be done in this way so that it gains more popularity and is used wide spread.

I remember when posts here would get like 50 replies in a weeks time. I guess this old engine has lost some of its steam.

This post has been edited by BigRedBrent: 09 August 2007 - 04:52 AM

[CiccioBastardo]

I think many of us have learnt how to deal with this kind of threads

[BigRedBrent]

CiccioBastardo, on Aug 8 2007, 05:29 PM, said:

I think many of us have learnt how to deal with this kind of threads

What do you mean by this kind? Do you mean a suggestion? I mean heaven forbid anything ever gets asked for, and more so that request ever gets implemented.

This post has been edited by BigRedBrent: 09 August 2007 - 04:57 AM

[leuk_he]

BigRedBrent, on Jul 30 2007, 08:28 PM, said:

I want to include that the main thing that made me come up with this was the limited way emule translates its very own collection file that it has implemented. It does not allow the ability to add more information to the links such as the full hash set, and even if you add the information to the links manually in the collection file, when you load the file into emule it ignores the extra information in the link such as the full hash set.

I think that is a bug IMHO. I think all that information should be saved.

BigRedBrent, on Aug 9 2007, 06:44 AM, said:

CiccioBastardo, on Aug 8 2007, 05:29 PM, said:

I think many of us have learnt how to deal with this kind of threads

What do you mean by this kind? Do you mean a suggestion? I mean heaven forbid anything ever gets asked for, and more so that request ever gets implemented.

It is easy to come up with all kind of eMule extensions. To program this is hard and time consuming.

This post has been edited by leuk_he: 13 August 2007 - 08:17 AM