Web Scraping Fisheries and Oceans Data

[mjones]

How difficult is it to web scrape and have the data go into WISKI using KiDAT. I am trying to get the St. Lawrence River Water Levels from this site to go into WISKI. How difficult would this be based on your experience @cmcguire ? Is there any help documents that you can send me to?

http://www.waterlevels.gc.ca/eng/Station/Month?type=1&sid=14400&tz=EST&pres=1&date=2018%2F01%2F31

[SteveWKISTERS]

Hi Michael,

I have a sample KiDAT task that scrapes data similar (but more complicated) from a website in Alberta. There's a decent amount of coding that goes into this, so it really depends on how comfortable you are working with webpages: you have to be able to identify the position of the information you're looking to collect and iterate through it. Script attached below.

Steve with KISTERS

[mjones]

Thanks Steve!

Would anyone happen to have the most updated version of KiDAT that  I can download onto my computer (not the WISKI Computer)? I can import some tasks but not others which makes me think I don't have the latest version.

Thanks!

[SteveWKISTERS]

Hi Michael,

I know I updated the KiDAT version for Curtis at the annual hub meeting last year, but I'm not sure if this change has been rolled out to everyone. I've forwarded Curtis this email thread to get his take on it.

Steve with Kisters

[cmcguire]

@cvance

[clairemp]

Unable to import the task @SteveWKISTERS  shared. Has anyone else had that issue?

[SteveWKISTERS]

Hi Claire,

If you're unable to import the task its because you still have access to the "free" version of KiDAT - you'll need to update to the version that was installed at Quinte last year. I'll speak with Curtis about the best path forward on updating everyone's system.

Steve with KISTERS

[SteveWKISTERS]

Morning everyone,

I believe we completed the DFO task that you were looking for during an informal KiDAT training with the MNRF. This task is pulling data from www.tides.gc.ca/eng and can rotate through the stations to scrape and format any of the data there.

Cheers,

Steve

[SteveWKISTERS]

Hi Everyone,

Here's the actual waterlevel.gc web scraper task that you might be looking for. Brandon and I worked on this with Stacey Van Opstal of Hamilton CA last week. Thought it might be useful.

Steve with KISTERS