Login is restricted to DCN Publisher Members. If you are a DCN Member and don't have an account, register here.

Digital Content Next


Research / Insights on current and emerging industry topics

The growing economy of web scraping

November 2, 2016 | By Rande Price, Research VP – DCN @Randeloo

Web scraping is a technique that allows people to easily extract large amounts of information from around the web – for legal and illegal uses. Distil Networks’ 2016 Economics of Web Scraping Report notes that bots, which make up approximately 46% of web traffic, often conduct the web scraping at a faster rate than humans. The Report found about two percent of online revenue is lost to web scraping.

Key findings also noted in Distil Networks Report:

  • 38% of companies engaged in web scraping to take content.
  • Web scraping is inexpensive costing about $3.33 per hour.
  • The average web scraping project costs approximately $135.
  • Real estate sites are the #1 web scraping target.
  • The average web scraper salary ranges from $58,000 to $128,000 per year.

There are six primary reasons for web scraping:

  1. Content scraping: stealing original content from one website and posting it on another website without the knowledge or permission from the publisher;
  2. Research: gaining marketplace intelligence;
  3. Contact scraping: acquiring customer emails and contact information for marketing and lead-generation;
  4. Price comparison: competitors in the real estate and travel industries see a lot of this activity;
  5. Weather data monitoring; and
  6. Website change detection: notifying users about changes made to specific websites.

It’s important for publishers to understand the extensive web scraping economy so they can take steps to protect their proprietary information.

Print Friendly and PDF