How To Scrape Website Content (*complex* Iframe, Javascript Submission)

I've done web scraping before but it was never this complex. I want to grab course information from a school website. However all the course information is displayed in a web scrap

Solution 1:

If you need to scrape a site with heavy JS / AJAX usage - you need something more powerful than php ;)

First - it must be full browser with capability to execute JS, and second - there must be some api for auto-browsing.

Assuming that you are a kid (who else would need to parse a school) - try Firefox with iMacros. If you are more seasoned veteran - look towards Selenium.

Solution 2:

I used to scrap a lot of pages with JS, iframes and all kinds of that stuff. I used PhantomJS as a headless browser, that later I wrapped with PhantomCurl wrapper. The wrapper is a python script that can be run from command line or imported as a module

Solution 3:

Are you sure you are allowed to scrape the site?

If yes, then they could just give you a simple REST api?

In rare case when they would allow you to get to the data, but would not provide API, my advice would be to install some software to record your HTTP interaction with web site, maybe wireshark, or some HTTP proxy, but it is important that you get all details of http requests recorded. After you have that, analyze it, and try to replay it up to the latest bit.

Among possible chores, it might be that at some point in time server sends you generated javascript, that needs to be executed by the client browser in order to get to the next step. In this case you would need to figure how to parse received javascript, and figure out how to move next.

An also good idea would be not to fire all your http requests in burst mode, put put some random delays so that it appears to the server more "human" like.

But in the end you need to figure out if all this is worth the trouble? Since almost any road block to scraping can be worked around, but it can get quite involved and time consuming.