From the Changelog: Product API Improvements, Custom API Management, Article Categorization
We’ve had a busy start to 2016. Here are some of the highlights from our January Changelog:
From the Changelog: Crawlbot Updates
Another year almost down, but we’re sneaking out some last-minute updates in the dregs of 2015. The latest highlights from our Changelog include a host… Keep Reading
Video: Crawling Basics and Advanced Techniques for Web Site Data Extraction
Just for the visual and auditory learners — and/or those of you who prefer their web crawling with the dulcet tones of yours truly —… Keep Reading
Various Ways to Control Your Crawlbot Crawls for Web Data
In 2013 we welcomed Matt Wells, founder of Gigablast (and henceforth known as our grand search poobah) aboard to head up our burgeoning crawl and search… Keep Reading
Create a Searchable Archive with the Bulk Service and Search API
A common use for Diffbot APIs: build an index of structured content for easy and precise searching. This post walks through the most simple way… Keep Reading
Crawling News Sites for New Articles and Extracting Clean Text
One of the more common uses of Crawlbot and our article extraction API: monitoring news sites to identify the latest articles, and then extracting clean… Keep Reading
Article API: Returning Clean and Consistent HTML
We’ve long offered HTML as a response element in our Article API (as an alternative to our plain-text text field). This is useful for maintaining… Keep Reading
New API Features: Authentication and Content POSTing
One of our most common feature requests: can Diffbot APIs access content behind a login or firewall? Until recently, the answer was mostly “no.” But… Keep Reading
How we spent $2500 and got 36 libraries and thousands of new developers
We just released Diffbot API clients in 36 different programming languages, ranging from general purpose languages (Ruby/Python/Java), to systems languages (Go/C), to scripting languages (Bash),… Keep Reading
Crawlbot Updates: Webhooks and Preventing Duplicate Content
We added a couple of frequently requested Crawlbot features this week: webhook notifications and much smarter content de-duplication.