Total addressable market (TAM) is the — hopefully — large figure that represents potential revenue for a given product or service. These figures are useful for fundraising, assessing market saturation, and the prioritization of opportunities.
In our recently published guide to writing a market intelligence report with the Knowledge Graph we worked through creating a report for a fictitious Acme Energy. Acme Energy provides backup energy services and energy disruption mitigation for hospitals. In this guide we’ll work through finding and visualizing three useful TAM-related datasets with Diffbot’s Knowledge Graph.
In particular, we’ll look at how you can quickly surface the datasets needed for the following three visualizations:
Google Sheets (or equivalent spreadsheet software)
I’ll use Infogram to visualize the data. Feel free to use any charting tool with mapping capabilities.
Step One: Define Service Set
There are three ways to calculate TAM, one of the most straightforward (if you have existing products or services) is as follows:
(# of potential customers) x (annual contract value)
In our case let’s look at a hypothetical in which Acme Energy sells two service sets.
$5,000 ACV deals to hospitals with less than 500 employees
$100,000 ACV deals to hospitals with greater than 500 employees
Because we have two distinct sets of customers here, we’ll need to calculate both TAMs separately and add them together. In particular, we’ll need to calculate the following:
(# of hospitals with less than 500 employees) x $5,000
(# of hospitals with more than 500 employees) x $100,000
In the next step we’ll find our figures for the first portion of these formulas.
Step Two: Calculate Total Addressable Market
In Diffbot’s Knowledge Graph we can query for organizations based on specific firmographics. Both industries and number of employees are attached to organizations, which makes it easy to return the number of hospitals needed for our calculation. Below I’ll show two routes to obtaining your data. The first will utilize the visual query builder, which allows you to craft basic search queries in a beginner-friendly way. The second involves using Diffbot Query Language (DQL), which is slightly more involved, but allows for greater control over your query. New to using DQL? Start by simply pasting in the queries typed out below or check out our DQL Quick Start guide.
Using the Visual Query Builder
We can form an initial hospital query using a few fields: industries, nbEmployees, and location. Start by choosing the type of entity you want returned (organization). Then simply toggle the location to United States, the industry name to hospitals, and the nbEmployees to <=500.
One quick query returns over 100,000 results! To obtain the second group of hospitals (with greater than 500 employees), simply alter the nbEmployees field. Also of note to the right of the screen is the preview of your query. This shows you the DQL version of your query and is a great way to start familiarizing yourself with what this query language looks like.
Using Diffbot Query Language
While this visual query is a great starting point, this particular data set could use some more work. As I looked through the returned organizations I saw some veterinary hospitals, optometric clinics, and home health businesses returned. While these may in some senses be “hospitals,” they aren’t what we’re looking for here. This is an instance in which DQL comes in handy.
The eventual query I settled on specifies that we don’t want organizations who are in sometimes related industries to hospitals, and that “hospital” should be in the name of the organization returned. This seemed to provide the most reliable dataset.
This query returns 1,244 results, the number of large hospitals for one half of our TAM equation. By changing the nbEmployees to nbEmployees<=500 we can find our other number. Plugged into the equation this means that our TAM is as follows.
(1,244 x $100,000) + (11,151 x $5,000) = $180,155,000
While we could export all of this data, using DQL enables facet queries, which are a useful way to quickly summarize the results of a specific field. In this case we can use this to return a summary of which states provide the most TAM.
To obtain the complete dataset we'll yet again need to alter the nbEmployees field and then download the results. I ended up pulling both datasets into the same spreadsheet to perform the simple TAM arithmetic to all states at once.
After converting the number of large and small hospitals per state into state-by-state TAM, we can analyze the data as we wish. In my case I pulled the numbers into a data visualization tool to see which regions have the largest opportunities.
What we've done here is quickly survey the number of hospitals by location and size across the United States. This search wouldn't have been possible in consumer search engines. And it's a good starting point. But the general trend above is still similar to a population density map. Perhaps there's more we can do to surface where opportunity lies for our fictitious Acme Energy.
Step Three: Analyze Competitors
In case our initial query of small hospitals didn't show this to be the case, the Knowledge Graph excels at long tail (SMB and MMKT) information. We have over 250M organizations in total, with solid coverage worldwide and across many, many industries.
To show this at work, let's surface a dataset of Acme Energy's competitors and plot it on a similar map to our TAM by state graphic.
Using the Visual Query Builder
After several exploratory queries, the query that yielded the best results for competitors for Acme Energy relied on the description field. This field is a few sentence summary of what an organization does. While we can look at energy companies from an industry level, this is a much more general query. What we're after here are American companies who provide services related to backup power.
Our visual query builder results return 327 backup energy providers across the United States. Clicking through some of the organization's profiles, they offer the precise service set of Acme Energy. The only downside to using the visual query builder is that there is not presently the ability to facet (provide a summary view). This means that you would need to export the data to csv and do a small bit of data wrangling to determine the number of competitors by state.
Using Diffbot Query Language
With Diffbot Query Language we can use the same query as we generated with the visual query builder and simply add a facet statement to the end (similarly to how we faceted to gain TAM by state).
After exporting our facet view, we can move straight to visualization or analysis.
Step Four: Analyze Competitors By TAM
While our competitors map largely also follows population density (with the exception of New York), with some simple arithmetic we can gain an even clearer view of where opportunity may lie.
Using our datasets for TAM by state and competitors by state, we can simply divide the two to provide a general view of how much unclaimed market there is.
Loading the resulting data into the same format provides the following visualization:
While state-by-state location may not matter for some industries (say, SAAS), many market intelligence analyses go to great depth to obtain state-by-state data. In this case we've surfaced relative opportunity in North Dakota and Iowa that wasn't present in our initial data set.
Our Knowledge Graph is based on web-wide crawls that update our organization database every few days. Want to see what coverage is like for your industry? Try out a free two-week trial or contact sales for a customized demo!
Market intelligence is the tracking and analysis of all important parties within a given market. In particular, market intelligence commonly looks at competitors, suppliers, governmental agencies, product offerings, customers, and broader trends.
Market intelligence can inform a range of tasks including (but not limited to):
Minimizing risk of new investments
Identifying new markets to enter
Increasing market share
Informing (or updating) ideal customer segments
Developing brand positioning
Assessing risks or opportunities in supply chain and production
In this quick guide we’ll work through reasons why the following market intelligence metrics are important, as well as how to gain market intelligence insights with Diffbot’s Knowledge Graph.
Calculating Total Addressable Market (TAM) Using Diffbot’s Knowledge Graph
Smart investors and management teams lean back on total addressable market (TAM) and related measurements to discern what level of opportunity a given service set has. A total addressable market is a measurement depicting the total potential sales given complete market saturation and with no monetary (or otherwise) constraints in providing this many services. Accurate TAM assessments can provide an early guidepost for product market fit as well as where opportunities are.
There are three primary routes to determine TAM, each with a set of trade-offs.
The “top down” method, looks at a well established industry as a whole. This form of research typically relies on analyst firms as middle men, and can enable you to say something like “Gartner estimates solar panel sales could reach X by Y.” This is a fine starting point and a bit of a gut check, but this method typically relies on the trust in a particular research firm and doesn’t provide a ton of detail about how results were created (or the underlying data set).
The “bottom up” method is a great choice for organizations who have already sold some of their products. It enables you to do your own research and understand the nuances of the underlying data. In the bottom up method you’ll take your annual contract value and multiply it by the number of organizations who fit a specific firmographic profile. This can enable you to gain a set of granular and related data points. For example, the TAM of solar energy in Texas (versus, say, Arizona).
The “value-theory” method adjusts the annual account value input to a TAM by providing an educated guess as to what individuals “could” be willing to spend for the value of your product. This can be accomplished by looking at competitors, or combining the value of multiple markets in the event your service is creating a new category.
For our purposes here, we’ll jump into the “bottom up” method, which provides the most underlying data and can be constructed “in house.”
Diffbot’s Knowledge Graph has unrivaled longtail and midmarket coverage for organizations through our web-wide fact extraction. The inclusion of a range of firmographics, technographics, and employee demographics allows for uniquely granular and accurate calculation of TAM values.
In our hypothetical, let’s calculate TAM for a company that makes backup energy sources for hospitals. They serve two primary industry segments. For community and mid-sized city hospitals that tend to have 500 or less employees, they provide backup energy monitoring and maintenance for a price of $5,000 a year. For larger hospitals that can have thousands of employees, their average annual contract value is $100,000 a year.
Within the Knowledge Graph we can start by assessing the underlying data on hospitals. Our initial query returns over 26,000 organizations who have been tagged as operating in the “hospitals” industry. This seems a bit high, and upon some perusal we can see some optometric, physical therapy, and related industries that are to some degree “hospitals” but not what we’re looking for. We then exclude organizations with these industries and provide a summary view of the number of employees of each one of these organizations.
As we can see, the lion’s share of the market aligns with our hypothetical energy provider’s customer profile with less than 500 employees. Though there are several thousand hospitals at their higher price point.
At this point we can facet (summary view) our results to provide total counts for both categories.
Here an initial take on TAM is simple. Simply multiply your two annual contract values by the number of organizations who could sign up.
$5,000 x 10,312 = $51,560,000
$100,000 x 2,121 = $212,100,000
Add the above to find a total addressable market of $263,660,000. Interestingly, the potential value for much smaller subset of larger hospitals vastly outstrips potential earnings for the many small hospitals.
One aspect in which the Knowledge Graph can provide unrivaled granularity is in the ability to quickly provide views of different portions of a TAM calculation. These additional calculations may take the form of your total reachable market or related numbers.
For example, let’s say the above TAM number is solid. But for now you only have legal approval to sell your services in the state of Texas. A quick adjustment to our Diffbot Query Language query can provide us with a TAM bounded by Texas.
The TAM calculable from the above states rounds out at $10.5MM. You can likely begin to see how differing views of segments of TAM can become valuable for discerning opportunity and direction.
Extrapolating From Lists of Customers, Competitors, or Suppliers
A common blocker when entering a new market is the ability to gain a circumspect (and global) view of customers, competitors, and suppliers. Manual research can quickly yield a handful of names. But the ability to extend a dataset can yield datasets of meaningful scale for analysis.
All of the 240MM+ organizations within the Knowledge Graph have a machine learning-computed similarTo score for every other organization. This field looks at a wide range of firmographics to determine what organizations are similar to one another.
Presently the input for similarTo queries can include one or two organizations, so it’s a great way to start with a very small number of example organizations and gain a wider list. To utilize similarTo, you’ll need the DiffbotURI (unique identifier) for the organizations you’re interested in. You can gain this simply by searching by name if you already know of an organization. The final portion of the URL attached to the entity will be your unique identifier.
💡 Tip: have a moderately-sized list of competitors, customers, or suppliers you want to extrapolate from? Use Diffbot’s Google Sheets or Excel Integrations to perform multiple similarTo queries at once.
A second method by which to grow lists of competitors, customers, or suppliers for further analysis takes a top-down approach. There are a range of filters to create lists of companies by industry, size, revenue, location, and more.
One catch-all approach often utilized in market intelligence queries is to utilize the description field. For example, let’s say you’re looking for suppliers of citric acid within a specific region. Citric acid in-and-of-itself is more granular than typical NAICs industry codes, but we can start from a broader industry and use the description field to find a more targeted list.
The below query looks for chemical manufacturing companies in China for whom citric acid is central enough to their offerings to be included in their description.
At 56 China-based citric acid manufacturer results, you’re well on your way to a comprehensive review of suppliers of interest.
Calculating Market Share And Saturation With Diffbot’s Knowledge Graph
Now that we have a list of competitors as well as TAM-related metrics, we can begin to look at potential market share and saturation rates.
Of the many fact types that our Knowledge Graph extracts from the public web, revenue (or estimated revenue) is one of the most prominent. For organizations that must publicly disclose revenue, this information is almost always online. For organizations who don’t have to publicly disclose, DIffbot provides a machine learning-computed estimated revenue field. This field looks at scores of firmographics to provide a best guess for what present revenue is.
Again we can approach these measurements from a top-down or bottom-up approach. With a discrete list of competitors we can simply enrich data using Diffbot’s Enhance product. Enhance provides Knowledge Graph data by searching for precise matches of organizations or people. Rather than search using a large OR query, Enhance let’s us enrich organizations in bulk.
Alternatively, if you can find a top-down query specific enough to only provide competitors, you can calculate revenue from what is likely an even larger list. If your competition can be defined by clear cut firmographics, then this is a good route. For example, let’s say all alternative energy providers with less than 50 employees in Georgia are competitors.
While 150 results is likely a majority of the market segment you’re looking for in this case, you should be aware of data points surrounding your specifications. For example, perhaps 50 employees is a bit arbitrary. And perhaps some competitors you would be remiss to exclude have around 55 employees. A quick facet query can gut check the distribution of data to ensure you aren’t missing out on data slightly beyond the specifications of your search.
In this summary view of employee counts for renewable energy companies in Georgia you would likely need to rely on industry insight. You could likely exclude 100-500 employee companies as a different segment. But are your competitors truly in the 1-50 employee range (e.g. largely 10-20 employee companies)? Or are your true competitors somewhere in the 50-100 employee bin?
Let’s be safe and export revenue for all companies with less than 80 employees. In the upper right corner of the results screen you can select CSV export, then on the following screen ensure that “revenue value” is toggled.
From this point calculating market share is simple arithmetic.
Ranking Competition With Net Income Per Employee
Ranking competition by net income per employee can point you in the direction of the most mature organizations within your market. This can provide valuable insight into who you should watch, what organizations you can learn from, and what’s working within your market.
We’ve already shown how to export revenue for a range of organizations meeting a specific criteria. The only difference here is you’ll want to export the nbEmployees, nbEmployeesMin, or nbEmployessMax fields to divide by total revenue.
Gauging Organization Sentiment
Thus far we’ve only touched on firmographic-related searches. But a great deal of market intelligence involves analyzing the overall operating environment, future trends, and pending events. This is where news monitoring can come into play. Diffbot’s article index is several times the size of Google News, augmented with natural language processing-enabled fields, and not siloed by location or language.
There are several routes to gaining an article feed of interest. These include:
Searching articles by AI-generated topical tags (e.g. “show me all articles about Apple Inc”)
Searching articles by AI-generated categories (e.g. “show me all articles about mergers and acquisitions”)
Searching articles by publishing location or region (e.g. “show me all articles from these 10 sites” or “show me all articles mentioning petroleum published in Russia”)
In the end, multiple feeds may be consolidated, or portions of the above searches may be combined. Let’s take a look at a feed of mergers and acquisitions related to fintech companies.
💡 Tip: want to find a list of all categories we track across our article index? Be sure to check out our list of categories in our documentation.
Once you have a collection of articles you’re interested in, two useful metrics to track include the velocity of publication as well as the sentiment. In some cases you may want to highlight only positive or negative sentiment, or to showcase a trend surrounding a topic over time.
A facet query can give you a quick distribution of sentiment around a topic. It’s worth noting that there are two “levels” of sentiment within the Knowledge Graph. The first is document-level sentiment, which is visible from within the results page. The second is entity level sentiment. Entity level sentiment provides a view of sentiment pointed at a specific entity within it’s context in an article. While both are valuable, entity-level sentiment is a stronger signal about a precise portion of a story.
One technique to generate a view into the velocity of positive or negative news over a period of time is to facet by publication date for positive (or negative) articles on a topic. A sample query for this looks like the following:
As with many facet queries within the Knowledge Graph, the resulting chart is immediately insightful and points to data ranges that might be worth looking into more. In the above example we look at articles tagged with Apple Inc and that are negative sentiment, clustered by week of the last year.
Need to track a custom event across a specific group of articles? You can pass Knowledge Graph or extracted articles to our Natural Language API, which we can quickly help to train to identify custom fact types and entity mentions.
Tracking Shifts In Talent
On top of the 240MM+ organizations in the Knowledge Graph, over 750MM person entities enable detailed employment, skill, and hiring records. There are a few useful market intelligence lenses to evaluate. One simply starts by looking at new or leaving employees at an organization within a time period. Summary views of these individuals can provide a glimpse into what skills, seniority levels, or locations are being hired at.
To begin this type of inquiry, we can use nested queries to ensure not only that a person has an employer we’re looking for, but ALSO that that is their present employer. A query like this looking at individuals working at Meta presently who were hired after the start of 2020 could look like the following.
A quick facet by skills, locations, or job categories can give a high level view of what transitions are happening across organizations.
Market Intelligence Dashboards With Diffbot
While the techniques covered above can help you to quickly generate a static market intelligence report, many market intelligence users want data that updates in real time. We’re constantly crawling the web and update the entirety of our Knowledge Graph every few days. Additionally, the use of our Automatic Extraction APIs can enable you to extract facts as often as you like from a predefined set of domains.
Customer built or custom solutions provided on our end often center around finding a set of Knowledge Graph queries that you truly care about. Datasets that you want to know the moment they change. And feeds that draw from both custom sets of domain, internal documents put through NLP, and the structured article and organization entities of the Knowledge Graph.
Above is a demo dashboard (filled with non-demo data) for a fitness software startup. We pull in many queries similar to those we have worked through in this guide as well as custom crawling of domains and additional parsing via NLP. Together this provides a nearly-live view of topics, discussions, and firmographic changes of competitors and customers within a market! For more information on custom build market monitoring dashboards using Diffbot’s structured web data projects, reach out to sales.
The primary Knowledge Graph use cases we see center around market intelligence, ecommerce, news monitoring, and machine learning. With that said, similar datasets and analysis techniques can yield a different set of organizations and individuals: investors.
The bedrock of investigating investments and potential investors within the Knowledge Graph is the investments field attached to organization entities. This field has a few components, all of which can yield useful data for both market intelligence, investing, or funding searches. In particular, the following sub-fields can be useful:
Investment amount
Investment currency
Investment date
Names and DiffbotUri’s for investing orgs
“Importance” of investing orgs
What series of funding rounds were raised
There are three basic motions that can yield insights for fundraising.
Look at the specifics of investments in orgs similar to your own (e.g. ‘who invests in battery tech companies who are expanding in Asia?’)
If orgs similar to your own don’t have many investments, look for orgs your org could be similar to in the future. Who invested in these orgs?
Once you have a set of investing organizations, can you discern actionable intel? Who might you reach out to? What do these organizations write about? What are their focus areas? How would you pitch them?
Investors today operate globally, and to answer the above questions on this scale you’ll need a tool that can aggregate relationships between global organizations as well as monitor news from around the world (and potentially in many languages). Our Knowledge Graph is a cinch in both of these instances.
Who Invests In Companies Like Mine?
To show how the Knowledge Graph might be used in fundraising scenarios, let’s start with a hypothetical scenario. You’re a alternative energy company based on Arizona, and you want to expand throughout the region.
First off, let’s get a list of regional alternative energy companies. If you aren’t concerned with the specific state or nation, you can utilize the near parameter to look within a specific radius.
This query returns over 2,400 renewable energy companies within 500 miles of Phoenix, Arizona. This is likely too many companies to manually look through. So you’ll likely want to perform some facet searches to get a summary view of what is in this dataset.
Adding facet:nbEmployeesMax provides a summary view of the number of employees of these organizations. It looks like this specific set of organizations primarily fall into three sizes: 100-500 FTEs, 10-20 FTEs, or 50-100 FTEs. While these clusters could be explained by the type of renewable energy product each company makes (e.g. software vs. large physical installations), these clusters also align with common headcounts associated with particular funding rounds. 10-20 FTEs, may be a bootstrap, seed, or angel round company. 50-100 FTEs may have raised a series A funding round, with 100-500 FTEs may be multiple funding rounds in.
In this hypothetical you have 20 employees, and need funding to expand your operations and grow into the slightly larger renewable energy companies. So let’s mine into the 50-100 FTE cluster.
The above query yields 236 organizations. A decent sample size from which to investigate past funding trends.
From here we can look at a summary of the organizations that invested in these organizations by adding facet:investments.investors.name to the end of the query. For this group of companies, only three investors have invested in multiple renewable energy companies in our list. 9 total investing organizations are present. If you need a larger list for outreach, you could try altering or removing the nbEmployeesMax fields. (Removing nbEmployeesMax returns >25 results, with 9 organizations who have invested in multiple of this set of renewable energy companies.)
This list of 9 investors could be your jumping off point for the third stage of this inquiry below. Or you could continue investigating to explore other angles for generating a list of potential investors.
What Similar Orgs Receive Investments?
Jumping to the second angle of inquiry we outlined in the intro, we can begin to look at the characteristics of organizations who gain investment in this industry. But first, let’s gain some insight into what types of investments have been attained by our similar organizations.
The above query returns sizable groupings for “series unknown,” “post IPO equity,” “seed,” “debt financing,” and “grant.” In our hypothetical our organization isn’t close to an IPO and is perhaps beyond seed funding stage. So let’s exclude organizations at these stages. One way to do this is to check what funding stage an organization is currently in. As organizations in series B have already gone through series A. This means organizations in the Knowledge Graph in series B would show up for searches looking for both series A and series B funding round recipient organizations. By using the isCurrent we can look at organizations currently in a given stage of funding.
The above query returns 16 companies, a nice middle ground for some aggregation of values with the potential to deep dive into each.
By looking at results on our map view, we can see two clusters of activity. As in many industries, investment is higher in specific locales. In this case, Henderson/Las Vegas, Nevada and Phoenix, Arizona.
Two useful fields to obtain summary views of for a group of organizations include descriptors as well as industries.
Within the industries facet query, we predictably see that these organizations are both “energy” and “renewable energy companies.” We can also see that solar power– in particular — as well as manufacturing tend to be most commonly invested in.
Within descriptors we can jump to specifics that are more granular than entire industry. In this case perhaps our hypothetical organization is already involved in building or energy storage (or are considering an expansion in these areas). Below they can find validation that similar organizations have been invested in, and surface an even more targeted list of organizations to deep dive into.
In order to shorten this list of organizations to only those who are described as working in energy storage and building, we could add a descriptors filter to our query.
The above query surface 8 organizations who are beyond seed funding, not yet to close to an IPO, provide energy storage and building services within the renewable energy industry and who are regional to Phoenix, Arizona. With a targeted list this size we can begin to look at each and every investor manually.
Investigating A Targeted List Of Investors
Now that we have a targeted list of organizations we can grab a list of all their investors. One route to quickly generate the list of investors is to simply add facet:investments.investors.diffbotURI to the end of the query. Another route is to export the investor fields into CSV.
The fields we may find of interest include investments_amount and investments_investor_diffbotUri. Also referencing the size and summary of the invested-in organizations to verify they are similar enough to your current firmographics.
DiffbotURIs are unique identifiers for entities in the Knowledge Graph. In the event entities have similar or identical names, DiffbotURIs are a more precise way to reference the actual organization of interest and disambiguate.
Once you have this list of DiffbotURIs, we can string them together into an “or” statement for organization, article, and person entity analysis. In our case there are 18 investors, 11 of which are unique. If you were looking for a serial investor in this space, this would also be promising by mining in to which of these organizations have invested in multiple of our 8 company target list.
We can start by simply returning the list of investors with the following query:
A quick view of the entities mapped shows that few of these organizations are regional. Meaning you may not need to limit your investor search by region.
A second search we can perform is to look at all organizations who have been invested in by these 11 investors to surface their broader interests. We can then facet through location and industry.
The largest industry clusters for investments from these organizations include software, energy, manufacturing, renewable energy, solar, and computer hardware.
By clicking through any one of these facet results, you can see a list a companies invested in with that specific industry. For example, clicking through solar energy companies yields over 200 companies invested in by this cohort. This can be used to provide another view of the types of observations surfaced in the first and second sections of this guide.
A second facet query around location of invested-in organizations can be useful to start focusing on which investors tend to invest within the region. We can filter by organizations in states located in the Southwest and then facet by investor to get a view of which of these investors invests the most in Texas and Arizona. While the below query is quite lengthy, the basics are simple, passing in the DiffbotURI of specific investors and then bounding (the DiffbotURIs inside of the square brackets) our facet query at the end to only return results about the same set of investors.
This final view shows a clear winner, a DiffbotURI we identified as a investor within our targeted list of renewable energy companies in an earlier section and who can see has invested in 70 companies in Texas and Arizona from this view.
This DiffbotURI resolves to the New York State Energy Research and Development Authority, a public benefit corporation that may be a great candidate to look into for potential investment.
Armed with a single (or handful) of DiffbotURIs we can look for news coverage of these entities, key individuals to reach out to, and more.
DiffbotURIs can show up as topical tags mentioned in articles. Tags are natural language processing-generated topics found in articles within our article index. They are available in content of every language and are presented in English.
The following query looks at articles we’ve identified as mentioning the New York State Energy Research and Development Authority. At present over 260 results are returned.
Using an ‘or’ statement similarly to prior queries we’ve worked through, we could also return a larger newsfeed of all of the investors we’re interested in. An alterative route to expanding your list of organizations is to utilize our similarTo query. Our machine learning computed similarity scores are present for every unique pairing of Knowledge Graph organizations. The syntax for expanding your list of interesting orgs for news monitoring via similarTo would look like the following.
The above returns 25 organizations most similar to our investor of interest.
Jumping back to useful article queries that start from a list of organizations, the sentiment field can be a powerful way to quickly surface actionable data. By adding sentiment>0 date<365d to our article query above we can see positive news about an entity over the last year. This can be used to quickly assess where industry successes and expansions are occurring.
Finally, we can use the name(s) of our investor of interest to search through person entities connected to this entity. In this case, this could involve looking at hiring trends (e.g. an entity is expanding in the southwest, or with analysts related to a specific technology). It can also be used to discern the proper contacts in a use case like we’re describing in this guide. In our case, some of the useful fields we may wish to look at include:
Skills
Seniority
Role
New Hires
New Locations
Details Related To Personalization of Outreach
Among Others
While fundraising isn’t one of the most common uses for the Knowledge Graph we see, many organizations that understand the basic strengths of Knowledge Graph data do go on to use our data for a variety of uses. On one level, most tasks that require manually gathering information from the web for further analysis can be completed at a much larger scale within the Knowledge Graph.
Disclaimer: this article is about a very mundane consumer search. With this said, how knowledge work and fact accumulation are often performed have wide-reaching implications for knowledge work flows.
The other day I was searching for coworking spaces.
As in many domains of knowledge, data coverage online was largely human curated. Lists with some undisclosed methodology provided the writer’s favorite coworking spots by city.
Sure, search engines will return a list plotted to a map in any major search engine. But I’m sure we’ve all run into the following.
Load map…
Pan slightly to surface more results…
Zoom slightly to surface more results…
Pan the opposite direction to try and find a result that had caught our eye…
Try to recall the name that caught our eye in a new search…
Five steps to seek further data points on a single search result. Devoid of context, data provenance, and the ability to analyze at scale.
Sure, consumer search works in many, many cases. So do phone books.
If you’re a power user, a data hoarder, or a productivity buff, you can likely see the appeal of a search that actually returns comprehensive data. If you’re building an intelligent application or performing market intelligence, using search that won’t let you explore the underlying data is just a waste of time.
So after this predictable foray in which I ignored the advice of several articles, scrolled around a map, and got sidetracked once or twice, I decided to resort to a different sort of search: Diffbot’s Knowledge Graph.
Prerequisites
The title of our article may not make much sense if you haven’t been acquainted with Diffy, Diffbot’s web-reading bot
You see the promise of external web data for many applications… if it were structured (or at least felt disappointment at consumer search engines keeping you from public web data)
Knowledge Graph Search
Opening the Knowledge Graph, it took all of 20 seconds to return data on over 4,000 coworking spaces. And sure, unless you’re selling a service to coworking space, you may wonder why anyone would need all this data as a personal consumer…
4000+ coworking space entities in ~20s
Maybe it’s simple curiosity. Maybe it’s the principle of it all; the fact that all of this information is publicly available online, but not in a structured format. Maybe this is just an analogy for non-consumer searches that also can’t be performed on major search engines. Any way you take it, search of the present is flawed for many uses, and it’s still our primary collective data source.
So what does search in the Knowledge Graph look like?
Well it starts with entities.
Knowledge graphs are built around entities (think people, places, or things) and relationships between entities. The types of relationships that can occur between entities, and the types of facts attached to entities are prescribed by a schema. One of the major “selling points” for knowledge graphs is that they have flexible schemas. That is — more so than other types of databases — they can adapt to what types of facts matter out in the world.
The Importance of Structured Web Data
At their core knowledge graphs (the category of graphs) can be built from any underlying data set. In the case of Diffbot’s Knowledge Graph, it’s the world’s largest structured feed of web data. Diffbot is one of only a handful of organizations to crawl the web. And using machine vision and natural language processing we’re able to pull out mentions of entities as well as infer facts and relationships.
Why is this important?
The web is largely made up of unstructured or semi-structured data. This means you can’t easily filter, sort, or manipulate this data at scale. While the internet is our largest collective source of knowledge, it’s not organized for modern knowledge work.
Diffbot’s products center around organizing the world’s information, whether through our AI-enabled web scrapers, our Knowledge Graph, or our Natural Language API. The ability to source the information from the web in a structured way provides the bedrock for machine learning initiatives, market intelligence, news monitoring, as well as the monitoring of large ecommerce datasets.
The State of Coworking Spaces As Told By AI
So what can you learn from a coworking space dataset that’s much more explorable than consumer search?
It turns out a lot.
While each individual data point is all available online, it’s not aggregated anywhere else in quite as explorable of a format.
In our case we can start with a simple facet query. Faceted search provides a summary view of the value of one fact type attached to a set of entities. So with this sort of query we can quickly discover what locations have the most coworking spaces.
By simply adding facet:locations.city.name we can turn over 4,000 unique results into an observation. While data found about these coworking spaces across the web would be in many different formats (and in many languages), knowledge graphs help to consolidate similar entities around standard fields.
An additional strength of knowledge graphs is that data points can be consolidated from many different sources with data provenance and then built off of. Using natural language processing and machine learning, fields can be computed or inferred from many underlying data sources. Our original query looked at organization entities with “coworking spaces” as part of their description. But an AI-generated field of “descriptors” allows for additional granularity. Let’s look at a facet view of the most common services offered by coworking spaces.
Depending on your experience with a range of coworking spaces, descriptors such as “expat,” “civil & social organization,” or “self improvement” may be novel. By amalgamating tens of thousands of online mentions, articles, and entries into this subset of org entities, the Knowledge Graph dramatically cuts down on time of fact accumulation.
One final area in which consumer search is severely lacking (or just in practice unpractical) is that of market research. Industry-specific events such as funding rounds, openings of new offices, key executive hires or leavings, or clues as to private organization revenue can be hard to pinpoint across the web. Softer signals like sentiment around topics or velocity of news coverage can also be informative.
Diffbot’s article index is roughly 50x the size of Google News. Unlike traditional content channels, you aren’t presented with content that’s gamed the system or paid to get your attention. Additionally, where consumer search engines are siloed by language or location, Diffbot’s article index is pan-lingual. With articles augmented by additional filterable fields underlying articles can become unique observations on sentiment, key happenings, and more. All underlying article data is returned as well, supporting the ability to mine in once you’ve found an interesting angle.
For a deeper dive into creating custom news feeds around organizations and events be sure to check out our Knowledge Graph news monitoring test drive.
Takeaways
Maybe you don’t buy the segue from what really is a consumer search (“coworking spaces near me”) and the copious coworking data available in the Knowledge Graph. But the fact of the matter is that a great deal of knowledge work still relies on human fact accumulation. Without automated ways to structure unstructured data, there’s a definite floor to the cost per fact.
Knowledge graphs provide a bedrock for knowledge workflows reengineered from the ground up. In particular:
Knowledge graphs mirror what we care about “in the world” (entities and relationships)
Knowledge graphs provide flexible schemas allowing for fact types attached to entities to change over time (as the world changes)
Automated knowledge graphs provide one of the only feasible ways to structure market intel and news monitoring data that can be spread across the web
Knowledge graphs that don’t expose their underlying data aren’t suitable for use in intelligent applications or machine learning use cases
Knowledge graphs that provide additionally computed fields (sentiment, tags, inferences on revenue or events) provide additional value for market intelligence and news monitoring
Ever have the feeling that news used to be more objective? That news organizations — now media empires — have moved into the realm of entertainment? Or that a cluster of news “across the aisle” from your beliefs is completely outrageous?
Many have these feelings, and coverage is rampant on bias and even straight up “fake” facts in news reporting.
With this in mind, we wanted to see if these hunches are valid. Has news gotten more negative over time? Is it a portion of the political spectrum driving this change? Or is it simply that bad things happen in the world and later get reported on?
To jump into this inquiry we utilized Diffbot’s Knowledge Graph. Diffbot is one of the few North American organizations to crawl the entire web. We apply AI-enabled web scrapers to pages that are publicly available to extract entities — think people, places, or things — and facts — think job titles, topics, and funding rounds.
We started our inquiry with some external coverage on bias in journalism provided by AllSides Media Bias Ratings.
While Diffbot’s Knowledge Graph has historically offered revenue values for publicly-held companies, we recently computed an estimated revenue value for 99.7% of the 250M+ organizations in the KG.
What does this mean?
Most organizations are privately-held, and thus have no public revenue reporting requirement. Diffbot has utilized our unrivaled long-tail organization coverage to create a machine learning-enabled estimated revenue field. This field looks at the myriad fact types we’ve extracted and structured from the public web and infers a revenue from a range of signals.
Estimated revenue is just that… a machine learning-enabled estimate. But with a training set the size of our Knowledge Graph, we’ve found that a great majority of our revenue values are actually quite accurate.
How can I use estimated revenue?
Revenue — even if estimated — is a huge marker for determining size and valuation. In it’s absence it’s hard to effectively segment organizations. We see this field used in market intelligence, finance, and investing use cases. And it’s as simple as filtering organizations using the revenue.value field.
Where Does Diffbot Get It’s Data?
Diffbot is one of only a handful of organizations to crawl the entire web. We apply NLP and machine vision to crawled web pages to find entities and facts about them. These entities are consolidated in the world’s largest Knowledge Graph along with data provenance, linkages between entities, and additional computed fields (like sentiment, or estimated revenue). In this ranking we looked at organization entities. But organization entities are just the “tip of the iceberg” for Diffbot data, which comprises articles, products, people, events, and many other entity types.
Last week we took a look at the top universities for female founders. In our results, we noted that our web-reading AI associates tech bootcamp attendance with education, and a large cluster of founders attended specific universities in conjunction with bootcamps.
New to the Knowledge Graph? Diffbot’s Knowledge Graph is constructed by crawling a vast majority of the web and structuring data on pages using NLP and machine vision. The end result is one of the world’s largest databases of organizations, people, articles, products and more, all linked and with data provenance.
To return results from the Knowledge Graph, you submit queries which filter which entities to return. In this case we queried the Knowledge Graph to return individuals who:
Attended an educational institution with the name of a top bootcamp
Have held a job title including “CEO,” “chief executive officer,” or “founder”
We then returned a facet (summary) view of how many of these individuals attended each bootcamp.
Upon seeing Crunchbase’s annual ranking of the best schools for graduating entrepreneurs, we wanted to see how our Knowledge Graph results stack up.
The Diffbot Knowledge Graph is sourced from crawling a majority of the web and extracting entities and facts using NLP and machine vision.
Two prominent entity types are person and organization entities. When paired together powerful observations sourced from across the web are possible. In this exploration we returned all person entities within the Knowledge Graph who are currently founders and who are female. We filtered to make sure each organization had at least some publicly disclosed funding, and then we took a look at a summary view of which schools these founders had attended. You can check out the Knowledge Graph query here with a free trial.
While the top schools for female founders were consistent with Crunchbase’s coverage, you may wonder why the numbers vary so dramatically. Crunchbase’s ranking this year was looking at 2019-2020 graduates, and Crunchbase’s data is centered around tech and startup firmographics. While Diffbot’s Knowledge Graph certainly has firmographic details on tech-centered companies, our database of organizations is much wider ranging (over 250M+ orgs at last count). This means our list includes founders of all sorts of endeavors: non-profits, artistic organizations, medical organizations, and tech companies to name a few.
A few weeks ago we published a view into Big Tech investments by industry. In this post we’ll take a similar look at the largest food retailers.
Panning out a bit, there are over 250M organizations within the Knowledge Graph. To obtain this list of large food retailers we first narrowed our search to food retailers with more than 1,000 employees. This query surfaces more than 7,000 fact-rich entities.
From there we simply sorted the results by number of employees to gain the largest food retailers including Walmart, Target, Tesco, Kroger, Carrefour, and Safeway.
With this list in mind, we looked for a list of organizations who had been invested in by one of these organizations. Bounded by calendar years, we then returned a summary view that looked at which industries the invested-in companies represented. If you have a subscription or free trial feel free to check out the resulting query. Continue reading →
What can you do with billions of web-sourced facts on hundreds of millions of organizations? Beyond analyzing the facts themselves, you (or a machine of your choice) can learn a lot. Historically, our Knowledge Graph has had one of the largest collections of publicly-disclosed organization revenue. Recently, we’ve applied machine learning processes across many org fields to estimate revenue for private organizations as well.
