Videos

Source Vitals

Source Attributes	Description

Source Attributes	Description
Data Collection Method	public Web scraping & public APIs
Geographic Coverage	global; coverage includes YouTube, Dailymotion, Acfun, Vimeo, Rumble and many more.
Key Use Cases	product & market research; company reputation analysis; influencer identification
Delivery Methods	Full stream; filtered stream; search APIs

Data Dictionary

Videos field name	Field description	Data example
video	The parent element that contains information about an individual video	"video": {
id	An ID generated by the crawler to uniquely identify each video.	yt.FauOVXgJ0zU
siteid	A unique text value that identifies the video site (e.g. youtube).	youtube
site_videoid	An identifier created by the video site to uniquely identify each video	FauOVXgJ0zU
title	The title of the video.	maaf jelek
description	The description of the video which is not always included.
thumb_url_code	The image of the screenshot of the actual video.	https://i.ytimg.com/vi/FauOVXgJ0zU/default.jpg
tags	Tags that describe the content of the video. Not always included.	[]
category	The category the video was assigned	People and Blogs
published	The date/time the video was published (UTC)	2024-08-13T13:11:01Z
crawled	The date/time the video was harvested (UTC)	2024-08-16T14:30:11Z
duration	The length of the video in seconds	17
videourl	The URL where the video was found.	https://www.youtube.com/watch?v=FauOVXgJ0zU
author	The parent element that contains author information	author: {
author/name	The name of the author who submitted the video	Prinsca gaming
author/site_authorid	An identifier created by the video site to uniquely identify each author	UCwmB2IO7xlu4fmTdH6Ul36A
author/authorurl	The URL to the author's profile page	http://youtube.com/channel/UCwmB2IO7xlu4fmTdH6Ul36A
lang	The language code to identify the language used in comment.	id8
langid	The socialgist internal mapped id for this language	40
comment	The parent element that contains information about an individual comment	comment: {
id	An ID generated by the crawler to uniquely identify each video or comment.	yt.Ugxb8X5GgmyTo0M24FF4AaABAg
videoid	The ID of the Video that the comment is associated with. This is used by BoardReader as the Thread ID.	yt.9IHwqdz8Xhw
siteid	A unique text value that identifies the video site (e.g. youtube).	youtube
site_commentid	An identifier created by the video site to uniquely identify each comment	UgxPGSMDXNWXumpJw8d4AaABAg
title	Some video sites include a title for each comment. Most cases we just add part of the first sentence of the comment text.	অতীত এর ভুল থেকে শিক্ষা নিবেন। এ দেশ হতে আ লিগ শব্দ টা চিরতরে
content	The text of the comment.	অতীত এর ভুল থেকে শিক্ষা নিবেন। এ দেশ হতে আ লিগ শব্দ টা চিরতরে কবর দিতে হবে জনগনের কাছে আইডল হতে হলে পরানতিক পর্যায়ে রিদয়ের ভিতর ডুকতে হলে জিয়াউর রহমানের আদর্শ উপলব্ধি করবেন। নির্বাচন নিয়ে মাথা…..
published	The date/time the video was published (UTC)	2024-08-15T18:56:03Z
crawled	The date/time the video was harvested (UTC)	2024-08-16T13:50:42Z
videourl	The URL where the video was found.	https://www.youtube.com/watch?v=PH-sbPSv3M8
commentsurl	A URL that displays all the comments for the video. For YouTube this is limited to the most recent 1,000 comments.	https://www.youtube.com/watch?v=PH-sbPSv3M8&lc=Ugxb8X5GgmyTo0M24FF4AaABAg
author	The author object element that contains information about the author of the comment	"author": {
author/name	The comment author’s name or video account name	@kingtexzone5051
author/site_authorid	The video site specific id for the author	UCUb5_lzoVTJ67OienqU5FLQ
author/authorurl	The url for the author of the comment. In some cases it can also be the author of the video.	http://www.youtube.com/@kingtexzone5051
author/profile_picture	The author profile avatar or picture.	https://yt3.ggpht.com/ytc/AIdro_kvrMWUD2Ptl9SASYrYT9RvDS24j2RSUw7ieikxndaDiHo=s48-c-k-c0x00ffffff-no-rj
lang	The language code to identify the language used in comment.	bn
langid	The socialgist internal mapped id for this language	8

Example Messages

Video

{
  "video": {
    "id": "dm.x7vh6r3",
    "siteid": "dailymotion",
    "site_videoid": "x7vh6r3",
    "title": "Barbie cooking toys- Barbie make a cake-",
    "description": "Barbie cooking toys- Barbie make a cake-",
    "thumb_url_code": "https://s1.dmcdn.net/v/SOJTl1VC2czVCsHjr",
    "tags": [
      "Barbie cooking toys- Barbie make a cake-"
    ],
    "category": "kids",
    "published": "2020-08-09T15:25:06Z",
    "crawled": "2020-08-11T12:50:36Z",
    "duration": "229",
    "videourl": "https://www.dailymotion.com/video/x7vh6r3",
    "author": {
      "name": "bookidstoys",
      "site_authorid": "x2a5wbk",
      "authorurl": "https://www.dailymotion.com/bookidstoys"
    },
    "lang": "en",
    "langid": "22"
  }
}

Comment

{
  "comment": {
    "match_info": {
      "rule": "https://www.youtube.com/channel/UC3IZKseVpdzPSBaWxBxundA",
      "rule_type": "channel",
      "source_rule_id": "UC3IZKseVpdzPSBaWxBxundA",
      "rule_id": 109524,
      "datastream_id": 988
    },
    "id": "yt.UgxPGSMDXNWXumpJw8d4AaABAg",
    "videoid": "yt.9IHwqdz8Xhw",
    "siteid": "youtube",
    "site_commentid": "UgxPGSMDXNWXumpJw8d4AaABAg",
    "title": "Bts you’re my world forever Bts stay Gold 💜💜💜💜💜💜💜",
    "content": "Bts you’re my world forever Bts stay Gold 💜💜💜??💜💜💜",
    "published": "2020-08-04T16:22:27Z",
    "crawled": "2020-08-11T12:50:36Z",
    "videourl": "https://www.youtube.com/watch?v=9IHwqdz8Xhw",
    "commentsurl": "http://www.youtube.com/all_comments?v=9IHwqdz8Xhw",
    "author": {
      "name": "Abdullah Haytham",
      "site_authorid": "UCeqdyLPFlq7sZJcdtBjTIew",
      "authorurl": "http://www.youtube.com/channel/UCeqdyLPFlq7sZJcdtBjTIew",
      "profile_picture": "https://yt3.ggpht.com/a/AATXAJyKDFYyLpJDmLKqmQayHAXgI8p5PMewyR-E9Q=s48-c-k-c0xffffffff-no-rj-mo"
    },
    "lang": "en",
    "langid": "22"
  }
}

Videos

Source Vitals

Data Dictionary

Example Messages

Related content