New Crawlbot Features: API Parameters and Product Crawl CSVs

We added a couple of frequently requested features to Crawlbot this week: the ability to pass in Diffbot API parameters to tailor the output of your crawl extractions; and the option to download a comma-separated-values (CSV) file of product crawl data.

API Parameters

Crawlbot API Parameters
Pass in API parameters just as you would with a querystring.

To pass in Diffbot API parameters, simply enter a querystring in the new “API Querystring” field. This allows you to augment your Crawlbot-extracted data with tags, additional product ID fields, meta information, or anything else you’d like to specify. Check out our Automatic API docs for more details on the fields you can control, or Crawlbot in the Developer Dashboard.

Download Product Data as a CSV

Crawlbot download optionsWe’ve added CSV as a format for downloading your Product API crawl data. This makes it a snap to review Crawlbot data without needing to enlist the help of your IDE, a JSON formatter, or a developer.

The CSV Download option will appear next to every Product API crawl you run, and is also available via the Crawlbot API.

Keep crawling!