From the Changelog: Crawlbot Updates

Another year almost down, but we’re sneaking out some last-minute updates in the dregs of 2015.

The latest highlights from our Changelog include a host of updates for our intelligent crawler, Crawlbot:

  • Crawl and Bulk jobs pending delete are now identified within the Developer Dashboard. We store data from completed or paused jobs for 15 to 30 days depending on your Diffbot plan
  • We’ve improved our ability to crawl Ajax-heavy sites, particularly those where most of a site’s links are delivered via Javascript. See our support topic on crawling sites for Javascript-generated links.
  • We’ve improved Crawlbot’s data download options — including the ability to download only the most recent 500 URLs in the URL Report

Plenty more where that came from: check out the complete changelog!