[Stulle]

Hi guys,

Just wanted to report that there is something wrong with downloading files from SF via eMule. This is most unfortunate because still a considerable number of people rely on SF for downloading files like the IPFilter via eMule. I have no idea what exactly is failing because it looks as if anythings works just smoothly but we don't receive the file but we still think we did...

Well, so I went to the SF support channel, explained it and they asked me to file a support ticket which you can find using this link. I hope the issue will soon be resolved. I'll keep you briefed. If you got any idea what could be causing this and how to fix it, please let me know!

Greetings,
Stulle

EDIT:
Due to the very effective help from the SF staff we got that sorted out fairly quickly. Once again, thanks to those guys!

Click here! if you like to know how we fix this.

This post has been edited by Stulle: 21 July 2010 - 08:32 PM

[tHeWiZaRdOfDoS]

Interesting... did you check the contents of the file that's being downloaded instead? It's a HTML file that's basically showing the default project page (and which btw. freezes the current Opera build I'm using). Did you try to change the useragent?

[Stulle]

nope, i really didn't try anything much. i just traced it in the code and it was like really weird because it did not fail and yet did not work either...

[tHeWiZaRdOfDoS]

It IS strange but I think it's somehow the fault of SF as they seem to redirect the request internally though I stay to be corrected

[Stulle]

okay, so i got a fix from SF and it's so easy, it does actually hurt! so for the log:

Quote

<@ctsai-sf> Stulle: okay, so it looks like you're using a browser-like user agent, which means the download will be treated as a browser and not as a download script (like wget, curl, etc). So, we recommend changing that. As far as I know, we don't cap downloads for unknown UAs.

the fix:

	if(StrStr(m_sURLToDownload,_T("sourceforge"))==0)
		HttpAddRequestHeaders(m_hHttpFile, _T("User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1)\r\n"), (DWORD)-1L, HTTP_ADDREQ_FLAG_ADD);

a little inconvenient because we loose backward compatibility for now but nothing we could do about that now without getting SF play along.

edit:
Wiz, could you prolly provide us with a bit of code that allows us to enter a series of url's that exclude this particular line of code so we can easily append the list whenever there is a need to without having to recompile eMule? i wasted enough time already for a single day...

This post has been edited by Stulle: 21 July 2010 - 08:41 PM

[Ki3r]

Stulle, on 21 July 2010 - 09:31 PM, said:

the fix:

	if(StrStr(m_sURLToDownload,_T("sourceforge"))==0)
		HttpAddRequestHeaders(m_hHttpFile, _T("User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1)\r\n"), (DWORD)-1L, HTTP_ADDREQ_FLAG_ADD);

Shouldn't be NULL?

[Stulle]

yes, because we only want to set a user agent if it's not sourceforge. see this: http://www.cplusplus...cstring/strstr/

[tHeWiZaRdOfDoS]

Stulle, on 21 July 2010 - 10:31 PM, said:

okay, so i got a fix from SF and it's so easy, it does actually hurt! so for the log:

Quote

<@ctsai-sf> Stulle: okay, so it looks like you're using a browser-like user agent, which means the download will be treated as a browser and not as a download script (like wget, curl, etc). So, we recommend changing that. As far as I know, we don't cap downloads for unknown UAs.

Aha, so it WAS SFs fault... however, the answer is rather wrong because I thought that'd be the issue and entered a browser agent and tried again - to no avail - and I can directly click&download the mentioned link with my browser - the redirect happens only with eMule... maybe you could tell them?

Quote

edit:
Wiz, could you prolly provide us with a bit of code that allows us to enter a series of url's that exclude this particular line of code so we can easily append the list whenever there is a need to without having to recompile eMule? i wasted enough time already for a single day...

Should be no problem, I'd add that to preferences.ini, so we could easily change the entries and maybe expand the list when necessary.

[Stulle]

yeah, i was thinking about something like that.

anyway, i assume that SF is doing things differently as soon as you send a user agent.

[Ki3r]

Stulle, on 22 July 2010 - 07:33 AM, said:

yes, because we only want to set a user agent if it's not sourceforge. see this: http://www.cplusplus...cstring/strstr/

I got that. Let me rephrase my question: Shouldn't that "==0" be "==NULL"? It's the same thing in C++?

[tHeWiZaRdOfDoS]

NULL is 0 - you can use both though you might want to use the appropriate one due to better coding style

@Stulle: checked in in Morph GIT (hope it's correct this time)

[Stulle]

looks good, thank you for that. just posting the diff here, though, so anyone can make use of that:

srchybrid/HttpDownloadDlg.cpp

    // some sites give unacceptable low download speed if they don't see a well known user agent in the headers... 
//>>> WiZaRd::Fix broken HTTP downloads 
    int curPos = 0; 
    bool skipAgent = false; 
    CString strBroken = thePrefs.GetBrokenURLs(); 
    CString cur = strBroken.Tokenize(L"|", curPos); 
    while (!cur.IsEmpty() && !skipAgent) 
    { 
        cur.Trim(); 
        if (!cur.IsEmpty())  
        { 
            if(StrStr(m_sURLToDownload, cur)) 
                skipAgent = true; 
        } 
        cur = strBroken.Tokenize(L"|", curPos); 
    } 
    if(!skipAgent) 
//<<< WiZaRd::Fix broken HTTP downloads 
    HttpAddRequestHeaders(m_hHttpFile, _T("User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1)\r\n"), (DWORD)-1L, HTTP_ADDREQ_FLAG_ADD); 

srchybrid/Preferences.cpp (just the lines you need... figure out where to add them yourself! :-P)

	CString CPreferences::m_strBrokenURLs; //>>> WiZaRd::Fix broken HTTP downloads

ini.WriteString(L"BrokenURLs", m_strBrokenURLs); //>>> WiZaRd::Fix broken HTTP downloads

m_strBrokenURLs = ini.GetStringLong(L"BrokenURLs", L"sourceforge"); //>>> WiZaRd::Fix broken HTTP downloads

srchybrid/Preferences.h (same as above)

//>>> WiZaRd::Fix broken HTTP downloads 
private: 
    static    CString    m_strBrokenURLs; 
public: 
    static    CString    GetBrokenURLs()                        {return m_strBrokenURLs;} 
    static    void    SetBrokenURLs(const CString& str)    {m_strBrokenURLs = str;} 
//<<< WiZaRd::Fix broken HTTP downloads

[Ki3r]

Thanks wiz! I've stumbled some times with that type or "errors" in other programming languages.

Here is how SF behave without agent:

--2010-07-22 08:29:14--  http://emulepawcio.sourceforge.net/nieuwe_site/Ipfilter_fakes/fakes.zip
Resolving emulepawcio.sourceforge.net... 216.34.181.96
Connecting to emulepawcio.sourceforge.net|216.34.181.96|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://downloads.sourceforge.net/emulepawcio/fakes.zip?use_mirror= [following]
--2010-07-22 08:29:15--  http://downloads.sourceforge.net/emulepawcio/fakes.zip?use_mirror=
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://downloads.sourceforge.net/project/emulepawcio/Fakes/Fakes/fakes.zip [following]
--2010-07-22 08:29:16--  http://downloads.sourceforge.net/project/emulepawcio/Fakes/Fakes/fakes.zip
Reusing existing connection to downloads.sourceforge.net:80.
HTTP request sent, awaiting response... 302 Found
Location: http://heanet.dl.sourceforge.net/project/emulepawcio/Fakes/Fakes/fakes.zip [following]
--2010-07-22 08:29:16--  http://heanet.dl.sourceforge.net/project/emulepawcio/Fakes/Fakes/fakes.zip
Resolving heanet.dl.sourceforge.net... 193.1.193.66, 2001:770:18:aa40::c101:c142
Connecting to heanet.dl.sourceforge.net|193.1.193.66|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2681045 (2.6M) [application/zip]
Saving to: `fakes.zip'

Here is how SF behave with a known browser agent:

--2010-07-22 08:36:36--  http://emulepawcio.sourceforge.net/nieuwe_site/Ipfilter_fakes/fakes.zip
Resolving emulepawcio.sourceforge.net... 216.34.181.96
Connecting to emulepawcio.sourceforge.net|216.34.181.96|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://downloads.sourceforge.net/emulepawcio/fakes.zip?use_mirror= [following]
--2010-07-22 08:36:42--  http://downloads.sourceforge.net/emulepawcio/fakes.zip?use_mirror=
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://sourceforge.net/project/downloading.php?groupname=emulepawcio&filename=fakes.zip&use_mirror=freefr [following]
--2010-07-22 08:36:43--  http://sourceforge.net/project/downloading.php?groupname=emulepawcio&filename=fakes.zip&use_mirror=freefr
Resolving sourceforge.net... 216.34.181.60
Connecting to sourceforge.net|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://sourceforge.net/projects/emulepawcio/files/Fakes/Fakes/fakes.zip/download [following]
--2010-07-22 08:36:43--  http://sourceforge.net/projects/emulepawcio/files/Fakes/Fakes/fakes.zip/download
Connecting to sourceforge.net|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16705 (16K) [text/html]
Saving to: `download'

[Famerlor]

Just for interest:

What benefit do we get using

if(StrStr(m_sURLToDownload,_T("sourceforge"))==0)

instead of

if(!StrStr(m_sURLToDownload,_T("sourceforge"))

?

The latter is IMO better readable...

[tHeWiZaRdOfDoS]

It's the same and basically both aren't "correct". The proper way would be to check for !=/== NULL. 0 is the same in C++ but it isn't really "correct". !StrStr works only because C++ interprets a value != 0 (also pointers != NULL) as "true" and == 0/NULL as false.
The latter is more readable... all versions work - do as you wish

[Stulle]

well, i don't think adding ! infront is actually more readable. i really don't like the usage of ! unless we are comparing a bool type function or a local bool type expression/variable. anyway, you are right about the 0 vs. NULL... i just copied that line from somewhere else and exchanged the params so i wouldn't have to bother.

[fox88]

Famerlor, on 22 July 2010 - 08:06 PM, said:

The latter is IMO better readable...

The first version assumes that NULL has the same type (there's default conversion rule) as pointer.
The second version implies: 1) the pointer is equivalent to one of the integer types (there's default conversion rule) 2) the converted value of unassigned pointer is equal to 0.

In eMule's code and majority of C++ programs both version work well.
But in theory it's quite simple to define a custom pointer type where these conditions (all or some of them) would not work. Then your two version would not be the same.

PS. I prefer ! for shortness too.

[Ki3r]

With the fix I'm seeing very low transfer rates. And connection resets. I was unable to get the ipfilter...

Can anyone confirm this?

EDIT: Don't bother. It finally downloaded the ipfilter and at good speed. Maybe I was getting slow mirrors.

This post has been edited by Ki3r: 26 July 2010 - 04:32 PM

[Stulle]

yeah. according to the support guys from SF they don't throttle!

[azzz]

Why dont't you use the sourceforge .jp mirror homesite?
For example: http://sourceforge.j...et_emulepawcio/
it should also work for countries which are blocked by sorcefourge.net.