Download a WHOLE site? AllData, Toyota TIS etc

Status
Not open for further replies.
Joined
Feb 11, 2014
Messages
1,980
Location
Upper Midwest by the Lakes USA
Anybody know how to download a whole site with many clicks in it such as AllData or Toyota TIS? I've tried what I could find on searches for programs and they get stumped pretty easily, even the best rated ones.

FOr now I've resorted to Google Chrome "capture WHOLE window" at a time with automated little apps but that's alooooot of clicking into sub menus with cute little plus boxes and submenus of submenus.


I found this "how to" (doing it the hard way non-automated kinda) but I cant make sense of it about halfway down the page of instructions

http://caphector.com/atoyota/TableOfContents/TISSearchandReplaceInstructions.txt
 
Originally Posted By: 97K15004WD
Black Widow Site Ripper.....



I havnt tried that one. I have tried the famous "HTTrack Mirror" but it is defeated instantly and reports its results simply as a "log in" screen for AllData. Disappointing. It's like it isn't smart enough to start archive the site I AM ALREADY logged into. It wants to open a fresh page but then it isn't human so it doesn't login in before proceeding.

So I need a ripper that can rip on a window I am already logged in upon.
 
The site relies on financial support from advertising to keep afloat.

Downloading would defeat it. Not cool.
 
Originally Posted By: DrRoughneck
The site relies on financial support from advertising to keep afloat.

Downloading would defeat it. Not cool.


AlData paid subscriber relies on web ads !?! Toyota subscriber TIS repair manuals rely on web ads?!?

I really don't think so.

Maybe you thought I was try to download BITOG? LOL
 
Haven't had any use for such a program in quite a while (like, 15 years), so I don't have one installed on the machine. I used it when I was self-studying Web design and website creation with text tools to write the HTML code. Still have the text editors installed, but they have other uses beyond simple text editing.

But the site-wide downloaders definitely exist. For example, SiteSucker for OSX.

http://ricks-apps.com/osx/sitesucker/
 
well, despite your objection on a technicality, the theme of DrRoughneck's point is still valid.

The subscription and setup is intended as a daily or 3day thing for you to solve your immediate issue; and they set up the site to obstruct your ability to do bulk downloading for the same reason.

It's not hard for them to have offered click this one button to download all documents (as the print or I believe there's other electronic version is basically that).

But for $15 or $10 or whatever it is, they're not giving you the whole store.
 
Last edited:
Originally Posted By: SumpChump
Originally Posted By: DrRoughneck
The site relies on financial support from advertising to keep afloat.

Downloading would defeat it. Not cool.


AlData paid subscriber relies on web ads !?! Toyota subscriber TIS repair manuals rely on web ads?!?

I really don't think so.

Maybe you thought I was try to download BITOG? LOL

You're right I thought he meant BITOG. My bad.

Download wget and try different options as explained here: http://stackoverflow.com/questions/17282915/download-an-entire-directory-using-wget

For example: wget -r http://yoursite/directory/
 
So if he pays $15, then goes and finds the torrent of the whole site, that's ethical in your guy's books?
 
HTTrack, wget, whatever, those all work great on "web 1.0" or "static" types of sites. AllData and TIS are interactive, full of scripts, have a login, et cetera. You can't recursively download them because those recursive downloading tools rely on the idea that a website is a bunch of html pages sitting on a server waiting to be transmitted. Modern webpages are generated on-the-fly, if that makes sense.
 
Originally Posted By: raytseng
So if he pays $15, then goes and finds the torrent of the whole site, that's ethical in your guy's books?

A membership site probably has a "terms and conditions" agreement that restricts or forbids bulk downloads. You should read the fine prints.
 
I have full access to AllData at my local public library. They have a complete subscription. I can download to PDF or print. Seems like less of a hassle even though I have to drive a little bit.
 
Originally Posted By: Michael_P
I have full access to AllData at my local public library. They have a complete subscription. I can download to PDF or print. Seems like less of a hassle even though I have to drive a little bit.


Some libraries let you do it through their website.
 
Originally Posted By: eljefino
Originally Posted By: Michael_P
I have full access to AllData at my local public library. They have a complete subscription. I can download to PDF or print. Seems like less of a hassle even though I have to drive a little bit.


Some libraries let you do it through their website.


Now THAT would be a heck of a library to find. One of those "some" libraries.
 
Older toyota docs are pdf. newer ones are dynamic and cannot be downloaded with any kind of scripting tool. You can do it via a print to pdf driver. it is still long and cumbersome but works.
 
If the website is compatible with javascript disabled a consumption tool would work to spider the pages. Old school straight HTML is quite easy to process, javascript/dynamic sites are quite difficult for the processing software to work with.
 
Status
Not open for further replies.
Back
Top