How I Used Data Journalism and NLP to Analyze Public Opinion on a Viral Political Debate 我如何利用数据新闻和自然语言处理分析热门政治辩论的公众意见

TL;DR

• The project analyzes 8,062 YouTube comments regarding Indonesian political figure Dedi Mulyadi (KDM) using a data journalism workflow to uncover public sentiment and narrative themes.
• Natural Language Processing (NLP) is utilized as a tool for evidence-based investigation rather than an end goal, focusing on extracting stories from large-scale text data.
• Data collection was automated via the YouTube Data API v3, capturing metadata such as usernames, timestamps, and engagement metrics alongside comment text.
• Text preprocessing involved rigorous cleaning including deduplication, lowercasing, tokenization, slang normalization, stopword removal, and stemming using the Sastrawi library for Indonesian language processing.
• The analysis aims to determine if public discourse focuses on policies, personalities, or media narratives without taking a political stance on the correctness of the subjects involved.

Why It Matters

This case study demonstrates the practical application of NLP in social science and political analysis, showing how practitioners can move beyond simple accuracy metrics to investigate complex human behaviors and societal trends at scale. It highlights the importance of domain-specific text preprocessing (such as handling Indonesian slang and morphology) which is critical for effective sentiment analysis in non-English contexts. Furthermore, it reinforces the value of the data journalism methodology—starting with a specific investigative question—to ensure that technical efforts yield actionable insights about public opinion rather than just model performance.

Technical Details

• Dataset: A collection of 8,062 comments generated by 7,409 unique users from a Tempo YouTube video discussing Governor Dedi Mulyadi.
• Data Acquisition: Utilized the YouTube Data API version 3 to programmatically retrieve comment threads, user handles, like counts, and publication timestamps, ensuring reproducibility compared to manual scraping.
• Preprocessing Pipeline: Implemented a multi-step cleaning process specifically tailored for informal social media text, including removing duplicates, converting to lowercase, tokenizing strings, normalizing repetitive characters and internet slang, filtering stopwords, and applying lemmatization/stemming via the Sastrawi library alongside NLTK utilities.
• Analytical Framework: Followed a structured data journalism workflow involving story ideation, data collection, preparation, analysis, visualization, and narrative construction to answer questions regarding sentiment polarity (supportive vs. critical), dominant emotions, and recurring topics within the discourse.

Industry Insight

AI professionals should recognize that successful deployment of NLP in real-world scenarios often requires significant investment in domain-specific data cleaning and linguistic adaptation, particularly when dealing with colloquial languages or regional dialects where standard models may fail. Adopting a "question-first" approach, similar to data journalism, helps align technical capabilities with business or societal goals, ensuring that machine learning projects deliver meaningful interpretability and impact rather than merely optimizing abstract performance scores. Additionally, leveraging public APIs for data extraction offers a scalable and ethical method for gathering large volumes of conversational data while maintaining compliance with platform terms of service.

TL;DR

• 项目利用数据新闻学与自然语言处理（NLP）分析印尼政治人物KDM相关YouTube视频的8,062条评论，揭示公众舆论情绪与叙事结构。
• 采用YouTube Data API v3采集数据，结合Sastrawi和NLTK进行文本清洗、去重、小写化、停用词移除及词干提取，实现非结构化评论的标准化处理。
• 强调“问题驱动”的数据新闻工作流：从提出社会议题出发，通过自动化手段挖掘隐藏叙事，而非单纯追求模型精度或技术炫技。
• 展示NLP在跨语言（印尼语）、非正式文本（网络用语、表情符号）场景下的实际应用价值，为政治舆情监测提供可复现方法论。
• 核心贡献在于将技术工具服务于公共讨论理解，体现AI赋能人文社科研究的范式转变——从“预测”转向“解释”。

为什么值得看

本文对AI从业者而言，是NLP在非英语、高噪声社交媒体语境下落地实践的典型案例，尤其适合关注多语言情感分析与内容治理的研究者；对行业而言，它揭示了数据新闻中“人机协作”的新路径：算法不替代记者，而是放大其洞察能力，推动舆论分析从主观印象走向证据驱动。

技术解析

• 数据采集：使用YouTube Data API v3批量获取公开评论字段（文本、用户名、点赞数、时间戳），覆盖7,409名独立用户生成8,062条原始记录，确保样本代表性与可重复性。
• 文本预处理流程：执行五步清洗链——删除重复项 → 统一转为小写 → 分词（tokenization）→ 标准化 slang/重复字母（如“mantappppp”→“mantap”）→ 移除停用词 + Sastrawi词干提取 + NLTK通用清理，有效解决印尼语口语化表达干扰问题。
• 工具栈选择：优先选用开源本地化工具Sastrawi处理印尼语词形还原，辅以NLTK完成基础文本操作，避免依赖商业API或预训练英文模型带来的语义偏差。
• 分析目标导向设计：未构建复杂分类器或生成式模型，而是聚焦于描述性统计与主题归纳，符合数据新闻“以故事为中心”的原则，降低工程复杂度同时提升结果可读性。
• 局限性说明：文中未提及后续情感极性建模或话题聚类方法，仅停留在数据准备阶段，暗示完整分析需进一步引入词典法或轻量级监督学习模块。

行业启示

• 政治舆情监测应转向“过程可视化”：传统民调易受抽样偏差影响，而基于真实平台交互数据的动态追踪更能反映即时社会心态，建议媒体机构建立常态化自动采集管道。
• 多语言NLP产品需重视方言与亚文化适配：印尼语网络表达高度灵活且缺乏标准规范，通用模型效果有限，开发领域特定预处理组件成为提升准确性的关键突破口。
• 技术团队应与内容创作者深度耦合：成功的AI应用往往诞生于明确的社会疑问而非算法优化竞赛，推动工程师参与选题定义环节，有助于产出兼具学术价值与社会影响力的成果。