is there any web crawler can get access to users region for download?


Actually I am using Httrack as a web crawler, can it use my credentials to access members area and download the zip files because they are restricted from public access.

Thank you in advance.


After all the problem was with my IPV6
it must be disabled; with (Windows 8 x64), I created a registry Dword32 "DisabledComponents" with the value of 0xffffffff
restart the system
httrack works with capture url, because with ipv6 it gives me a proxy adress like xxxx::xxxx:xxxx:xxxx%19 and this is not useful with proxy settings in windows

Thank you for the answers

asked on Server Fault Nov 17, 2013 by M. A. • edited Nov 17, 2013 by M. A.

1 Answer


It's a little difficult to fully understand what you are actually asking due to the way you have worded your question. I'm guessing you are trying to access a restricted part of a website and download files from that section?

If so, yes it can be done but its not easy to achieve and it will fail if the website provides a logout link which undoubtedly it will. As the crawler follows the logout link you will no longer have a valid session/cookie with the website and the crawl will come to a halt.

You can view everything you need to know here:

answered on Server Fault Nov 17, 2013 by twigg

User contributions licensed under CC BY-SA 3.0