Data Dictionary Home

 

Overview

Socialgist is the privacy-compliant social data access layer for commercial and enterprise applications. Through partnerships with key social platforms like Reddit, Quora, VK and Wordpress, our products allow you to understand consumer opinion as it evolves in real time across a wide variety of topics. Socialgist offers the broadest available set of social media data sources for market research and other approved use cases.

This document contains field definitions and message examples for Socialgist data sources.

Data Collection

The data is gathered in a variety of ways including:

  • licensed “firehose” integrations with top social platforms

  • partnerships with data aggregators

  • public Web scraping

  • RSS feeds

  • public social platform APIs

Data Types

Our platform delivers several types of social datasets:

  • Text content for natural language processing (NLP) and meaning extraction

  • Compliance events notifying customers when content is edited or deleted on social platforms

  • Engagement metadata including updated counts of comments, votes, likes and shares

Data Enrichments

Socialgist performs transformations and enrichments such as:

  • Normalizing data from hundreds of thousands of websites into a common schema

  • Language detection in more than 170 languages

  • Links resolution, or gathering metadata about URLs that are shared in social media text

Data Delivery

Most datasets are available via 4 delivery methods:

  1. Full Streams of every message and activity on a particular social platform, delivered as a stream of JSON over a continuous HTTP streaming connection.

  2. Filtered Streams of messages that match your keywords and other filtering criteria, also delivered as streaming JSON.

  3. Search APIs which allow you to perform complex queries over a rolling 25-month history and page through results via a RESTful API.

  4. Archived data, available back to 2007 for some sources, delivered as files via Amazon Web Services S3 storage.

Please note that all of the message examples contained in this Data Dictionary are from Full Stream delivery. The format and structure of the same messages delivered via Filtered Streams or Search APIs is slightly different, but generally contains all of the same data elements shown in the Full Stream example.

Services

In addition to providing streams, APIs and archive files which empower you to explore public attitudes on any organization, industry or issue, Socialgist can provide expertise and development services to help with:

  • Custom data collection

  • Website change detection

  • Data delivery integration

  • Insights & data visualization services