Content Harvesting

Stories By DQL: Tracking the Sentiment of a City


The story: sentiment of news mentions of Gaza fluctuate by as much as 2000% a week. 90% of news mentions about Minneapolis have had negative sentiment through the first week in June 2020 (they’re typically about 50% negative). Positive sentiment news mentions about New York City have steadily increased week by week through the pandemic.

Locations are important. They help form our identities. They bring us together or apart. Governance organizations, journalists, and scholars routinely need to track how one location perceives another. From threat detection to product launches, news monitoring in Diffbot’s Knowledge Graph makes it easy to take a truly global news feed and dissect how entities being talked about.

In this story by DQL discover ways to query millions of articles that feature location data (towns, cities, regions, nations).

How we got there: One of the most valuable aspects of Diffbot’s Knowledge Graph is the ability to utilize the relationships between different entity types. You can look for news mentions (article entities) related to people, products, brands, and more. You can look for what skills (skill or people entities) are held by which companies. You can look for discussions on specific products.
(more…)

Read More

Merrill Cook

Stories By DQL: George Floyd, Police, and Donald Trump

We will get justice. We will get it. We will not let this door close.

– Philonise Floyd, Brother of George Floyd

News coverage this week centered on George Floyd, police, and Donald Trump. COVID-19 related news continue to dominate globally.
That’s the macro story from all Knowledge Graph article published in the last week. But Knowledge Graph article entities provide users with many ways to traverse and dissect breaking news. By facet searching for the most common phrases in articles tagged “George Floyd” you see a nuanced view of the voices being heard.

In this story hopefully you can begin to see the power of global news mentions that can be sliced and diced on so many levels. Wondering how to gain these insights for yourself? Below we’ll work through how to perform these queries in detail.

(more…)

Read More

Merrill Cook

How Diffbot’s Automatic APIs Helped Topic’s Content Marketing App Get To Market Faster

The entrepreneurs at Topic saw many of their customers struggle with creating trustworthy SEO content that ranks high in search engine results.

They realized that while many writers may be experts at crafting a compelling narrative, most are not experts at optimizing content for search. Drawing on their years of SEO expertise, this two-person team came up with an idea that would fill that gap.

They came up with Topic, an app that helps users create better SEO content and drive more organic search traffic.They had a great idea. They had a fitting name. The next step was figuring out the best way to get their product to market.

(more…)

Read More

Merrill Cook

Comparison of Web Data Providers: Alexa vs. Ahrefs vs. Diffbot

Use cases for three of the largest commercially-available “databases of the web”

Many cornerstone providers of martech bill themselves out as “databases of the web.” In a sense, any marketing analytics or news monitoring platform that can provide data on long tail queries has a solid basis for such a claim. There are countless applications for many of these web databases. But what many new users or those early in their buying process aren’t exposed to is the fact that web-wide crawlers can crawl the exact same pages and pull out extensively different data.

(more…)

Read More

Merrill Cook

Why You Need to Crawl Your Own Site

How much usable data is on your own website?

Chances are that there is plenty of data already available to you that could help power your business, but it’s not organized in a way that’s practical for analyzing.

This is where web crawlers can help.

One of the main functions of a crawler is to categorize, organize and make sense of data sets from websites so that you can use that data in valuable ways.

And while the majority of businesses who use web scrapers will crawl other sites to do this, many stop short at gathering this data from their own sites.

But it’s equally important to crawl your own site on a regular basis.

Not only can your own data sets give you a glimpse into how you stack up against your competitors, but it can give you valuable insights into how your customers think, what they want, and how you should market to them.

Here are a few examples of why having your own website crawled is good for business.

Free Download: 5 Things Ecommerce Sites Can Do With Crawled Data
Click Here

Improve Your Ecommerce Store Sales

One of the most common uses for web crawling is for product price comparisons.

You can scrape almost any ecommerce site for product descriptions, prices and images to get data for analysis, affiliation, and comparison to your own site.

While you can scrape other sites to compare this data against, you can (and should) also scrape your own store for this data, too.

This will not only help keep your data organized – ensuring all of your data is there (nothing is missing), everything is where it should be, prices are correct, and so on – but also help you see where you are falling short compared to your competitors.

Amazon, for example, frequently crawls their own sites to ensure that they have products that people are looking for.

They look to see if there are gaps in their product listings to see where competitors are selling products they don’t have yet.

Amazon’s ultimate goal is to have every product sold to their store. But they can only accomplish that if they understand what products they have and don’t have. Thus, scraping their own site allows them to fill in those gaps.

If you’re looking to grow an ecommerce store, you can do the same thing by frequently crawling your own product data to see how it measures up to other stores selling similar items.

You can also see how your shipping times, product availability, and recommended products stack up to competitors as well.

Repackaging Your Data into Something New

You can also use data from your own site (and competitor sites) to create new product price comparison sites if you wish.

But there are plenty of other ways to repackage your own data for other business purposes.

For example, a healthcare practice could scrape the data about physicians, doctors and other practitioners listed on their site to create a catalog of available doctors.

You could even include specializations and regions, or other specifications that could form an online directory for potential patients.

If you ran a blog, online publication or media site, you could scrape your site for related stories that could be used to create a content hub or resource center for specific topics.

You could even repackage specific articles into ebooks or other downloadable resources.

If you were building a mobile app, for instance, you could extract the title, author, date, text, images, videos, captions, categories, entities, and other metadata from your article pages to enhance mobile readability.

Taking the data you already have and turning it into something new allows you to offer something of value to site visitors without the extra work.

The data already exists. You just need to find a new way to use it.

Click To Tweet

Being able to scrape your own website for this data can help you see what resources you already have available that can be shared with visitors or customers for a new user experience.

This will also help you see which content is the most popular so you can target future content to improve engagement for your readers.

Monitor Public Opinion About Your Brand

You can (and should) scrape other websites for mentions of your brand. But you can also monitor your own website (and social media sites) for mentions, comments and reviews.

If you have product reviews on your own site, you can gather information about how certain products are perceived, what buying behaviors accompany which products, or spot fraudulent reviews and remove them quickly.

Comparing your onsite reviews to those from third-party review sites can also help you analyze customer loyalty, product perceptions, brand perceptions, and other potential issues that might prevent sales.

Maybe customers are happy buying from your site, but they don’t like buying your products that are sold on other sites (or vice versa).

You can also scrape information from your social media company pages to see potential interactions you might have otherwise missed.

If you have a LinkedIn company page, for instance, you could gather information about the business profile, address, email, phone, products/services, working hours, and Geocodes of those who have clicked on your profile.

This can help your sales team narrow down leads and reach out to those who might be interested in your products or services.

Other Ways Crawling Your Own Site Helps

There are many other instances where crawling your own site or sites can help.

Regularly crawling your site can help you detect malicious or fraudulent activity, missing data or metadata, and other gaps that might affect your sales, for instance.

You can also use web crawling for use with predictive analysis tools. U.S. retailer Target, for example, once used analytic data to predict when customers were pregnant (and send them related ads).

You can also use your data to predict churn, which products might sell, or how well your customers are enjoying your products.

This type of market research can be helpful for businesses in every industry. So even if you don’t run an ecommerce site, frequently crawling your data can have positive benefits to your business.

And it doesn’t matter how big or small your brand is, you can still use the data from your own page. You can scrape multiple sites or pages to pull this information, or gather it from one specific page.

Either way, the end results are the same. The more data you have access to, the better your decision making process will be when it comes to your business, your products, and your customers.

Here are 5 ways ecommerce sites can use their own crawled data for sales
Click Here

Final Thoughts

What would you do if you were able to see all of the data about your business in an easily accessible way?

Hopefully, you would use that data to outpace your competitors and improve your offerings for your customers and site visitors.

With data extraction – web crawling – you can do that. While the majority of people will crawl competitor sites or other sites around the web (and that’s something we recommend) it’s equally important to gather data from your own site so you can see how it compares.

Without insight into your own business, you can’t make decisions that will put you ahead of the game. So while you’re busy scraping the web, make sure you include your own site on that list, too.

Read More

Dru Wynings

Why You Need Custom Brand-Monitoring Software

How do you convince a potential customer to buy from you if they’ve never bought from you before?

There are a few tried and true sales strategies you could try, of course, like having a well-designed website, writing sales copy that pitches your product as the solution to their problems. You could develop a strong value proposition that sets you apart from your competitors, and even split test your site to ensure visitors see a version of your brand that works for them.

But in the end, it might not be enough. That’s because 81% of buyers make purchasing decisions not based on sales gimmicks, but on what they hear about your brand on the Web.

The number one selling point for customers is still reputation, and what people say about you online matters to your bottom line.

While there are plenty of tools out there that can help you check your reputation, what you need to truly sell is a custom brand monitoring solution. Here’s why…

What people say about you online matters to your bottom line.

Click To Tweet

Free download: Checklist for Better Social Listening
Click Here

Why You Should Care About Brand Monitoring

By learning about a customer’s experience with your site, you can discover what you’re doing right and in which areas your sales pitch is falling short.

But following your potential customers around the Internet isn’t always easy, and deciphering information beyond data points and figures can be equally complicated. You can’t just look at a spreadsheet of statistics; you have to practice what’s known as “social listening.”

Social listening is the process of monitoring digital channels – social media sites, review sites, blogs, forums and comment sections, for example – for mentions of your brand, competitors, products, or relevant themes to your business.

But while traditional monitoring is focused on metrics (engagement rates, number of mentions, and so on), social listening looks beyond the numbers at the overall mood behind the social media posts — how people actually feel about you, your competitors, and your industry.

Most brand-monitoring software will find mentions of your brand, but not all software will help you truly listen to what’s being said. This is why having custom brand monitoring software may be a better option.

Why Custom Brand Monitoring Is Essential

Most brand-monitoring software will allow you to track and report mentions based on keywords or groups of related keywords, as well as alert you when mentions appear on any given site or channel.

But custom brand-monitoring software goes a step further by allowing you to collect, analyze and manage your mentions, apply sentiments, and even compare your sentiments to those of your competitors.

Applying sentiment is especially important in brand monitoring.

Most brand monitoring software attempts to assign a sentiment to a mention – positive, negative or neutral – but without knowing your audience or your intention, it doesn’t always have the capacity to assign those sentiments properly.

Sentiment is important when you’re trying to decipher whether or not 100 mentions of your brand on Twitter were for something great (people loved your product) or because you made a marketing faux pas that’s gone viral.

Custom software allows you to better measure sentiment by tracking trends, identify and amplify positive interactions, and respond appropriately when negative responses are flagged.

It can help you see trends that occur over time so you can develop marketing tactics aligned with your audience’s perception of your brand, not just static metrics and mentions.

How Web Scraping Can Help with Brand Monitoring

Part of the way that custom software can better help applying sentiment is through the use of structured data, gathered through web scraping.

By using web crawling tools that are preconfigured to collect and store only certain kinds of data, you can monitor relevant information with more targeted sentiment from thousands of different web sources.

Not only does this give you a detailed idea about your brand sentiment, it also makes things easier for brand marketers to formulate strategy, target new marketing campaigns and generate new leads and sales, leading to better revenue.

In addition to sentiment and social media mentions, you can also do things like:

  • Develop a more competitive pricing strategy – You can crawl price comparison sites for pricing data, product descriptions, as well as images to receive data for comparison, affiliation, or analytics.
  • Track reviews and industry trends – Scraping reviews and profiles from social media channels and review sites can give you a clearer picture of product performance, customer behavior, and interactions.
  • Detect fraudulent reviews – Web scraping can help identify opinion spamming, fake reviews, and other deceptive marketing strategies that may be used to harm your brand’s reputation on review sites or social media.
  • Create highly targeted ads – A good web crawler can identify opinions by demographics such as age group, gender, sentiments, and GEO location, which can be used to create highly targeted campaigns and advertisements.
  • Perform social media analytics – You can extract data from social media channels for better social analytics, aiding in the social listening and response process.

This process can be largely automated as well, allowing you to pull data whenever you need. You can also engage with customers in real time and use the information you gather to make your brand more credible.

Can your custom software check off these social listening boxes?
Click Here

Final Thoughts

While simple brand monitoring tools can provide you with metrics that can help you monitor your online reputation, in order to get the most accurate results, you really need a custom solution.

Custom brand-monitoring software takes your social listening to the next level, allowing you to customize the data you pull and apply sentiment so that you can make better marketing decisions.

It can also help you with other practicalities, like finding reviews from many different sites, detecting fraudulent reviews that could harm your brand, and help you create targeted ads to niche demographics.

At the end of the day, it matters what people say about you online. But in order to address your online reputation, you need better metrics and you need to be able to listen with all your digital ears open.

Custom monitoring software allows you to do all that and more, which makes it an essential tool to have when creating a robust marketing strategy for your business.

Read More

Dru Wynings

Why Data-Driven Content Marketing Doesn’t Always Mean Blogging

94% of small businesses, 93% of B2Bs, and 77% of B2Cs use content marketing in some form or another.

Yet many are still struggling to create content that actually boost their rankings. Why? Well, for one, they’re not using data to power their content.

Data offers insights into what customers are reading about, where they’re going to get their information, and what type of content they’re consuming (and believe it or not, it’s more than just blogs). Without understanding these variables, any content strategy is merely a guessing game at best.

If you’re serious about improving your SEO using data-driven content marketing, or you just want to understand how to use data to drive better marketing results, here’s what you need to know.

Download your copy of our Data-Driven Content Marketing Roadmap
Click Here

Why Data-Driven Content Is Important

The amount of content being published on the web each day is staggering. It’s reported that each second around 1,400 blogs are posted, 277,000 tweets are sent, 2,460,000 Facebook posts are shared, 26,380 users review something on Yelp, and 4,000,000 search queries are received by Google.

But how much of this content is actually impacting its intended audience?

Most businesses that dip their toes into content marketing do a decent job of creating content, but few actually track their content’s metrics or assess their overall ROI. This is partly due to a lack of understanding about how data affects content.

Being able to monitor trends, collect data about platform engagement, and track review or news sites for pertinent information is essential to developing content that actually reaches people.

Data can help answer questions like, “Who are we selling to?” “What topics resonate with them?” and “Who and where are they getting their information from?”

Popular retailer Kohl’s, for example, uses both internal and external data to track customer behavior in-store and then creates targeted email campaigns based on their findings. They also analyze their online data to find relevant content to send to customers in multiple mediums.

Without data, marketing to a mass audience is a shot in the dark.

Click To Tweet

Without data, marketing to a mass audience is a shot in the dark. You may hit on relevant content, but you’ll never be able to repeat your results. Data helps pinpoint real-time user activity so you can narrow down your marketing efforts in meaningful ways.

But just because data is essential to seeing results doesn’t mean every company knows how to successfully use it to drive their content..

How Companies Use Data-Driven Content Marketing

One of the biggest driving factors for content creation is SEO. Search engines like Google and Bing often reward websites that publish dynamic content with better rankings. This is partially why so many companies are now using frequent, fresh content to improve their online visibility.

So where does data fit into the process?

Some digital marketers use web crawlers and other data scraping techniques to scan competitors’ sites and analyze trending topics. Typically, they’ll scrape specific company pages or even social media networks to see what people are saying about certain topics, companies and/or products and then create content relevant to those trends.

If they find a mention related to their business, industry or product, they might extract data like related posts, comments, URLs, hashtags, and even tweets and number of likes. They will then use that data to generate relevant content.

They will even use data to perform content audits on their own sites for better ROI. Kissmetrics actually uses data scraping to create a list of all of their own blog’s content to determine which topics rank the best and which titles and keywords have the highest click-through rates. They also use that data to build a list of the leading influencers in their industry.

Data scraping can also be used for content related things like:

  • Getting product reviews from retailers to determine customer pain points
  • Finding news sites to curate content for your own website visitors
  • Crawling sites for statistics for general marketing research

While the application of data-driven marketing is vast, the real question for many businesses still remains the same: How will we use this data to create content and what type of content should we create?

For the majority of businesses, the output of this data is blogging. Surveys show that 76% of B2C businesses have a blog and/or participate in frequent writing-related content marketing efforts. While blogging is a significant generator of SEO and inbound leads, it’s not the only thing that can reach customers.

In fact, limiting yourself to only blogging means you restrict the power of data to drive your content, and in turn, cap your ability to see real results.

Why Data Needs More Than Blogs

Over half of all B2C businesses consider creating engaging content a significant obstacle to growing their business, but the content itself might not be the problem – it could be the way content is being presented.

82% of B2C marketers produce videos as part of their content marketing strategy, 61% rate digital newsletters as an effective tool parallel to blogging, 40% of small businesses use webinars and/or webcasts as a part of their marketing strategy, and 90% of B2C marketers use social media in their content marketing programs, making it the most popular platform to publish content.

Studies show that when B2B buyers do research to make buying decisions, they actually look for white papers, case studies, and webinars more often than blogs. And retail customers more often use online reviews and social media than blogs to determine whether or not they’ll buy from you.

Some companies are taking advantage of this by incorporating other forms of data-driven content on their sites to generate leads.

Intuit, for example, tracks data about the financial habits of users in different age ranges in order to create shareable infographics, like this one comparing the financial habits of Millennials versus Gen Xers.

True data-driven content marketing should give you a solid understanding of your target customers, and it’s very possible that your customers are using social media to share videos and infographics instead of reading your blog. It’s also very possible that your target customers are looking at review sites instead of browsing Google for related articles.

That’s why the data collection process is so important to content creation. Without understanding data, you can’t predict which content or medium will have the most impact on your ROI. But you can use data to pinpoint your audience’s habits and prioritize your content development accordingly.

And if you’re looking to stay ahead of your competitors, you’ll need that data to tell you what’s working and what’s not.

How can data drive your content? Grab your free roadmap here
Click Here

Final Thoughts

Gathering as much information as possible about your customers’ online and offline behaviors, reading patterns, and even which platforms they use is essential to your content development.

If you’re focusing on creating content in order to improve your SEO or to generate leads, you need data to drive the creation process, or at the very least, to evaluate the success of your content’s performance so you know the real value of your investment.

Data should also inform your chosen delivery method, and it’s important to remember that blogging is not the be-all and end-all for content marketing. If your data shows you that another method is more effective, trust those results. Don’t follow the trends just because everyone else is doing it. Do it because the data tells you it works.

Read More

Dru Wynings