cURL on site that redirects to disclaimer, updated

[Megathan]

Hello,

This is my first ever post, so I apologize in advance if I am wrong in approach or etiquette, and thank you for any assistance despite my faults.
I know this topic was covered back in March

cURL on site that redirects to disclaimer

Hello, I am trying to scrape some data from Environment Canada about water levels. For example: http://www.wateroffice.ec.gc.ca/graph/graph_e.html?stn=08KA007&prm1=6&mode=text When you first visit this site it redirects you to a disclaimer page with an “I Agree” button that must be clicked in order to proceed to the data I want. What I need to do is write a PHP script that follows the redirect, submits “I Agree” and then outputs the data from the actual page I need to view. I have tried to…

However I am looking for assistance. The accepted solution of that topic is no longer viable as the Water Survey of Canada page has updated. I have tried every possible variation I can think of / find, but I admit I am neither proficient in cURL nor PHP.
Say for example I am trying to scrape data from http://wateroffice.ec.gc.ca/report/report_e.html?type=realTime&stn=08KA007 - how might I go about doing that and getting around the ever-present disclaimer, now that the website has been updated and the finely-crafted script by @rpkamp does not work anymore?

Thank you all very much!

[davidtsadler]

See if the following works. The contents of the page should be in the $res variable. How you scrape it will be up to you.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://wateroffice.ec.gc.ca/report/report_e.html?type=realTime&stn=08KA007');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); 
curl_setopt($ch, CURLOPT_COOKIE, 'disclaimer=agree');
$res = curl_exec($ch); 
curl_close($ch);  
echo $res;

It appears that the site removes the disclaimer if it detects the presence of a disclaimer cookie so you just need to ensure that you send it with the curl request.

[StarLion]

Have to say i’m a bit disappointed in the site not making the data more easily available (like a CSV form, for example) if it’s going to be publicly distributed data.

For those that actually care: Normally I throw up the standardized “Dont scrape data without permission”, but this site specifically says the information MAY be copied and distributed as long as you’re not selling the data.

[Megathan]

Thanks! This worked perfectly!!

[Megathan]

Hello. I totally agree. I am only looking to get the river data for research purposes. USGS makes it exceptionally easy - but Canada does not seem as user-friendly. Thanks again for the support!