#173 Screen Scraping with ScrAPI

Aug 03, 2009 | 15 minutes | Plugins

Screen scraping is not pretty, but sometimes it's your only option to extract content from an external site. In this episode I show you how to fetch product prices using ScrAPI.

Click to Play Video ▶

Download:
source codeProject Files in Zip (97.4 KB)
mp4Full Size H.264 Video (27 MB)
m4vSmaller H.264 Video (17.2 MB)
webmFull Size VP8 Video (44.7 MB)
ogvFull Size Theora Video (36.8 MB)

Browse Source Code

Resources

          bash
        
sudo gem install scrapi

          config/environment.rb
        
config.gem "scrapi"

          models/product.rb
        
def self.fetch_prices
  scraper = Scraper.define do
    process "div.firstRow div.priceAvail>div>div.PriceCompare>div.BodyS", :price => :text
    result :price
  end
  
  find_all_by_price(nil).each do |product|
    uri = URI.parse("http://www.walmart.com/search/search-ng.do?search_constraint=0&ic=48_0&search_query=" + CGI.escape(product.name) + "&Find.x=0&Find.y=0&Find=Find")
    product.update_attribute :price, scraper.scrape(uri)[/[.0-9]+/]
  end
end

          scrapitest.rb
        
require 'rubygems'
require 'scrapi'

scraper = Scraper.define do
  array :items
  process "div.item", :items => Scraper.define {
    process "a.prodLink", :title => :text, :link => "@href"
    process "div.priceAvail>div>div.PriceCompare>div.BodyS", :price => :text
    result :price, :title, :link
  }
  result :items
end

uri = URI.parse("http://www.walmart.com/search/search-ng.do?search_constraint=0&ic=48_0&search_query=lost+third+season&Find.x=0&Find.y=0&Find=Find")
scraper.scrape(uri).each do |product|
  puts product.title
  puts product.price
  puts product.link
  puts
end