December 3, 2025 • 6 min read

Multimodal Information Credibility Classification Using CNN-BiGRU with Particle Swarm Optimization

For my undergraduate thesis, I developed a multimodal deep learning pipeline combining text, images, and user metadata with Particle Swarm Optimization, which was published at IEEE ICICyTA.

Multimodal Information Credibility Classification Using CNN-BiGRU with Particle Swarm Optimization

Why I Built This

When I started planning my undergraduate thesis at Telkom University, I wanted to tackle misinformation on social media, focusing on Indonesian economic discourse. Topics like IHSG index movements, Rupiah depreciation, and the launch of the sovereign wealth fund Danantara Indonesia frequently triggered polarized commentary and misleading claims.

Most credibility detection studies I read looked only at tweet text. In practice, false claims on platforms like X (formerly Twitter) rarely rely on text alone. People share misleading screenshots, edited charts, and memes alongside their captions. At the same time, account history (follower count, verification status, and posting behavior) provides subtle cues about authenticity that text alone cannot capture.

The core question behind my thesis was: can fusing text, image features, and user metadata into a single deep learning pipeline improve credibility classification accuracy over standard text-only baselines?

I later refined this work into a paper and published it at IEEE ICICyTA 2025.

Multimodal data streams combining text, images, and social metadata

How I Built the Pipeline

1. Data Collection and Labelling

I collected 23,564 tweets from X between January 2024 and September 2025 using the Tweet-Harvest library, targeting keywords like “IHSG,” “Pelemahan Rupiah,” “Investasi,” “#EkonomiIndonesia,” and “#Danantara.” Each tweet came bundled with its text, attached image, and user metadata, making it a naturally multimodal dataset.

Three annotators labeled each tweet as credible or non-credible using majority voting to reduce bias, following Castillo et al.’s definition of credible content as factual, informative, and sourced from verified accounts. The final dataset was evenly balanced:

LabelCountPercentage
Credible11,78250%
Non-credible11,78250%

2. Text Representation: TF-IDF + Custom GloVe

Because standard pre-trained embeddings often struggle with Indonesian social media slang and finance-specific jargon, I trained a custom GloVe embedding model directly on domain data instead of relying on off-the-shelf vectors.

I built a domain-specific corpus of 62,274 records by combining the 23,564 collected tweets with 38,710 articles scraped from IndoNews, pairing informal social vocabulary with formal news reporting:

Corpus SourceTotal Records
Tweets23,564
IndoNews Articles38,710
Combined Corpus62,274

These embeddings expanded the TF-IDF feature matrix by substituting zero-weight terms with semantically similar words identified by GloVe to reduce sparsity. For example, the top semantic neighbors GloVe found for “Danantara” were investasi, ekonomi, indonesia, kelola, and hadir.

3. Multimodal Fusion Architecture

I combined three feature streams into a single classification network:

  • Text stream: TF-IDF for lexical frequency fused with GloVe for semantic expansion
  • Visual stream: 1024-dimensional image feature maps extracted using MobileNetV1
  • Metadata stream: follower count, verification status, account age, and engagement metrics (quotes, replies, retweets, and favorites)

These fused features were fed into a hybrid CNN + BiGRU network, tested in two configurations:

  • CNN + BiGRU: convolution and pooling first to extract local patterns, then BiGRU for broader contextual modeling
  • BiGRU + CNN: BiGRU first for contextual dependencies, then Conv1D and Global Max Pooling to condense local patterns

Deep neural network architecture for multimodal feature extraction

4. Automated Tuning with Particle Swarm Optimization

Instead of manually tuning hyperparameters through trial and error, I implemented Particle Swarm Optimization (PSO). Each particle in the swarm represents a candidate hyperparameter configuration, updating its position based on its own best-known result and the swarm’s global best.

The search space covered six parameters:

HyperparameterRange
GRU units32 to 256
Filters32 to 256
Kernel size2 to 5
Dropout0.2 to 0.6
Dense units32 to 128
Learning rate1e-5 to 1e-3

Experimental Journey

The thesis ran through five sequential experiment scenarios, each narrowing down the best configuration for the next stage:

ScenarioGoalKey Result
1. Data split ratioFind best train/test split90:10 split won
2. TF-IDF feature sizeFind optimal feature dimensionalityCNN: 3,000 features / BiGRU: 1,000 features
3. GloVe expansion + hybrid modelsReduce TF-IDF sparsity, compare CNN-BiGRU vs BiGRU-CNNBiGRU-CNN: 75.92% / CNN-BiGRU: 75.78%
4. Multimodal fusionAdd image and user-based featuresCNN-BiGRU jumped to 78.43%
5. PSO hyperparameter tuningFine-tune the best multimodal modelCNN-BiGRU reached 79.09%

Evaluation metrics and model performance across experimental scenarios

Findings and Results

Model ConfigurationAccuracy
CNN-BiGRU + PSO (proposed)79.09%
BiGRU-CNN + PSO77.93%
CNN-BiGRU baseline (text only, TF-IDF)73.48%

Integrating visual and account metadata produced a +4.95% accuracy gain over the text-only TF-IDF baseline (73.48% to 78.43%, before PSO tuning).

Looking at the best model’s confusion behavior:

ModelLabelPrecisionRecallF1-Score
CNN-BiGRUNon-credible (0)78.14%80.97%79.45%
CNN-BiGRUCredible (1)80.38%77.22%78.68%
BiGRU-CNNNon-credible (0)79.22%75.76%77.43%
BiGRU-CNNCredible (1)76.79%80.10%78.40%

CNN + BiGRU turned out to be the more cautious model. It caught non-credible content reliably (80.97% recall) and its “credible” predictions were solid (80.38% precision). In credibility detection, keeping false positives low is essential: mistakenly labeling misleading content as trustworthy does more harm than flagging an ambiguous post for review.

Limitations and Future Work

  • The dataset only covers Indonesian text, without regional dialects or multilingual test cases.
  • The multimodal component is limited to static images; video and audio were not explored.
  • TF-IDF remains non-contextual compared to modern transformer-based embeddings like IndoBERT.

Next steps include testing transformer-based embeddings, expanding the domain vocabulary corpus, improving annotation quality through crowdsourcing, and building a real-time credibility detection system for practical deployment.

Publication

This research was accepted and published in the peer-reviewed proceedings of the IEEE International Conference on ICT for Smart Society (ICICyTA) 2025:

  • Title: Multimodal Information Credibility on Social Media X Using a Hybrid CNN-BiGRU with Particle Swarm Optimization
  • DOI: 10.1109/ICICyTA68677.2025.11362759
  • Authors: Gede Bagus Krishnanditya Merta, Erwin Budi Setiawan
  • Institution: Telkom University, School of Computing
  • Academic Honors: Summa Cum Laude (GPA 3.96/4.00)
  • Tech stack: Python, PyTorch, TensorFlow, Scikit-learn, CNN, BiGRU, GloVe, TF-IDF, MobileNetV1, Particle Swarm Optimization

IEEE ICICyTA research publication and academic defense