retrieving ad urls using scrapy and selenium

Vote count:

0

I am trying to retrieve the ad URLs for this website: http://ift.tt/OnqfiM

The ad URLs are loaded using javascript so a standard crawlspider does not work. The ads also changes as you refresh the page.

I found this question here and what I gathered is that we need to first use selenium to load a page in the browser then use Scrapy to retrieve the url. I have some experiences with scrapy but none at all in using Selenium. Can anyone show/point me to resource on how I can write a script to do that?

Thank you very much!

asked 1 min ago

Onyi Lam

8

retrieving ad urls using scrapy and selenium