Just for the visual and auditory learners — and/or those of you who prefer their web crawling with the dulcet tones of yours truly — a couple of Crawlbot tutorials to help you get up and running:
Crawlbot Basics
A quick overview of Crawlbot using the Analyze API to automatically identify and extract products from an e-commerce site.
Advanced Usage
This tutorial discusses some of the methods for narrowing your crawl within a site, and setting up a repeat or recurring crawl.
Related links:
- Various Ways to Control Your Crawlbot Crawls for Web Data (blog.diffbot.com)
- Crawlbot Support