I like to have tangible metrics assigned to my goals. Ideally, these would be automatically tracked and there would be a mechanism to hold me accountable.

One of my aims this year is to publish more, be more open about how I work and what I find interesting in my areas of expertise. What metrics can I attach to it? One is certainly a number of posts, the other could be how many people find them interesting enough to follow me.

To see how these metrics change over time, I decided to create a small dashboard that would track their historical values. I decided to start with X/Twitter.

Check out the dashboard created in this tutorial here: https://horosin.github.io/metrics-dashboard/

Full code: https://github.com/horosin/metrics-dashboard

You may have heard that X restricted access to their API last year. They did, but they still allow us to access our own basic metrics (contrary to platforms like LinkedIn - shame on you Microsoft, I have to scrape in order to access my data).

What we’re going to build

There will be a few pieces of software to write/configure:

The code for fetching the data from X
A script saving the data somewhere (in this case in the JSON file in GitHub repository)
Schedule to run the code periodically - every day at a given time
Dashboard to present the data (simple single HTML file using chart.js)
GitHub Pages to host the dashboard

The best part is that we can do all of that for free (including compute).

Set up

Twitter application

Setting up a Twitter application in the developer section is a prerequisite for accessing Twitter's API, which is essential for fetching data like follower counts, posting tweets, or accessing other Twitter resources programmatically. Here's a step-by-step guide to get you started.

Navigate to the Twitter Developer Site: Go to Twitter Developer and sign in with your Twitter account. If you don't have a Twitter account, you'll need to create one. Complete the application/sign up.
Go to the Developer Dashboard: Access your Twitter Developer Dashboard.
Create a Project: Click on "Create Project". You will be asked to provide a project name, description, and use case. Fill these out according to your project's needs.
Create an App within Your Project: After creating your project, you'll have the option to create an app within this project. Click on "Create App" and fill in the necessary details like the App name.
Get Your API Keys and Tokens: Once your app is created, you will be directed to a page with your app's details, including the API Key, API Secret Key, Access Token, and Access Token Secret. Save these credentials securely; you'll need them to authenticate your requests to the Twitter API.

Project

Now let’s get to coding. Create a new directory on your system and open a console there.

mkdir metrics-dashboard
cd metrics-dashboard

Make sure to initialize a Git repository, and later connect it to a GitHub project.

Initialise node.js project and install some packages that we’re going to need to authenticate with the API.

npm init
npm i dotenv oauth-1.0a crypto

Create a .env file with all the keys acquired from X before. DO NOT commit this to the repository. This is only to test the script locally.

TWITTER_API_KEY=''
TWITTER_API_SECRET=''
TWITTER_ACCESS_TOKEN=''
TWITTER_ACCESS_SECRET=''

Create the .gitignore file to avoid this. Sample below contains other paths we’d want to ignore.

node_modules/
.env
.DS_Store
dashboard/data/

Fetching the data

First, we’ll write a Node.js script called to fetch follower statistics from your platform's API. We'll use standard fetch library to make the API calls and oauth-1.0a to authenticate with X. After fetching the data, we will write results to a JSON file that will serve as our database. This will be handled by a separate script. To make the output accessible to it, we will write it to an environment file available in GitHub Actions.

I am using node 20.

Create a file called x_fetch_data.js in the root of our project.

require('dotenv').config();
const OAuth = require('oauth-1.0a');
const crypto = require('crypto');
const fs = require('fs');

// Initialize OAuth 1.0a
const oauth = OAuth({
  consumer: {
    key: process.env.TWITTER_API_KEY, // Read from environment variable
    secret: process.env.TWITTER_API_SECRET // Read from environment variable
  },
  signature_method: 'HMAC-SHA1',
  hash_function(base_string, key) {
    return crypto.createHmac('sha1', key).update(base_string).digest('base64');
  }
});

const token = {
  key: process.env.TWITTER_ACCESS_TOKEN, // Read from environment variable
  secret: process.env.TWITTER_ACCESS_SECRET // Read from environment variable
};

const url = 'https://api.twitter.com/2/users/me?user.fields=public_metrics';

const fetchTwitterFollowerCount = async () => {
  const requestData = {
    url,
    method: 'GET',
  };

  // OAuth header
  const headers = oauth.toHeader(oauth.authorize(requestData, token));
  headers['User-Agent'] = 'v2UserLookupJS';

  const response = await fetch(url, {
    method: 'GET',
    headers
  });

  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }

  const data = await response.json();
  console.log(data);

  // Extract the metrics
  const metrics = data?.data?.public_metrics;

  // Write the metrics to the environment file
  fs.appendFileSync(process.env.GITHUB_OUTPUT, `METRICS=${JSON.stringify(metrics)}\n`);
};

fetchTwitterFollowerCount().catch(err => console.error(err));

To test the script, you can run:

GITHUB_OUTPUT=testoutput node x_fetch_data.js

You should see your X metrics in the output, as well as in the testoutput file:

metrics={"followers_count":288,"following_count":302,"tweet_count":1381,"listed_count":0,"like_count":591}

Saving the data

To save the data, create another script in a file x_save_data.js . It will take the output from the environment and append it to the ./data/x.json .

Make sure to create this file first and commit it to the git repository. It should have an empty array as its content.

[]

The script also doesn’t add a duplicate record if the data was already fetched that day. It overwrites the old one instead.

const fs = require('fs');

// Parse the JSON string from the environment variable
const metrics = JSON.parse(process.env.METRICS);
const path = './data/x.json';
const now = new Date();
const today = `${now.getFullYear()}-${String(now.getMonth() + 1).padStart(2, '0')}-${String(now.getDate()).padStart(2, '0')}`;

let data = [];
if (fs.existsSync(path)) {
    data = JSON.parse(fs.readFileSync(path));
}

const todayIndex = data.findIndex(entry => entry.date === today);
if (todayIndex > -1) {
    data[todayIndex] = { date: today, ...metrics };
} else {
    data.push({ date: today, ...metrics });
}

fs.writeFileSync(path, JSON.stringify(data, null, 2));

You can test the script by editing testouput file by adding single quotes around the JSON and then running the following. File edit I necessary, as GitHub Actions environment behaves differently and doesn't need the quotes.

# load output from the previous script
set -a; source testoutput; set +a;
node x_save_data.js

Scheduled GitHub Action

Now, let’s create a file with GitHub action code. It will run every day at a specified time and fetch our metrics. It will then save them and commit to the repository.

Save the following code under .github/workflows/fetch_x_data.yml .

name: Fetch X Data

on:
  schedule:
    # Runs at 4 AM UTC
    - cron: '0 4 * * *'
  workflow_dispatch: # This line enables manual triggering of the action

jobs:
  fetch_data:
    runs-on: ubuntu-latest

    permissions:
      contents: write
      pull-requests: write

    steps:
      - name: Check out the repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm install

      - name: Fetch Data from Platform X
        id: fetch_data
        run: node x_fetch_data.js
        env:
          TWITTER_API_KEY: ${{ secrets.TWITTER_API_KEY }}
          TWITTER_API_SECRET: ${{ secrets.TWITTER_API_SECRET }}
          TWITTER_ACCESS_TOKEN: ${{ secrets.TWITTER_ACCESS_TOKEN }}
          TWITTER_ACCESS_SECRET: ${{ secrets.TWITTER_ACCESS_SECRET }}

      - name: Save data
        run: node x_save_data.js
        env:
          METRICS: ${{ steps.fetch_data.outputs.METRICS }}

      - name: Commit and push if there's changes
        run: |
          git config --global user.email "action@github.com"
          git config --global user.name "GitHub Action"
          git add data/x.json
          git commit -m "Update data for Platform X" || exit 0  # exit 0 if no changes
          git push

Run the action manually by committing the code and then going to “Actions” section of your project on GitHub and triggering it from there.

Dashboard

Okay, how about presenting the data? I didn’t want to mess around with simple HTML, so I asked ChatGPT to generate it for me.

Create an index.html file in the dashboard folder. We’re not using the main directory of our project in order to avoid hosting the data fetching code alongside the HTML.

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Twitter Dashboard</title>
  <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
  <style>
    body {
      font-family: Arial, sans-serif;
      margin: 0;
      padding: 0;
      display: flex;
      flex-direction: column;
      align-items: center;
    }

    .chart-container {
      width: 80%;
      max-width: 1000px;
    }

    canvas {
      max-width: 100%;
    }

    h1 {
      text-align: center;
      margin-top: 20px;
    }

    h2 {
      text-align: center;
      margin-top: 20px;
    }
  </style>
</head>

<body>
  <h1>Twitter Dashboard</h1>

  <h2>Number of Followers</h2>
  <div class="chart-container">
    <canvas id="followersChart"></canvas>
  </div>

  <h2>Number of Tweets</h2>
  <div class="chart-container">
    <canvas id="tweetsChart"></canvas>
  </div>

  <script>
    fetch('data/x.json')
      .then(response => response.json())
      .then(data => {
        const dates = data.map(item => item.date);
        const followers = data.map(item => item.followers_count);
        const tweets = data.map(item => item.tweet_count);

        const minFollowers = Math.min(...followers) - 100;
        const minTweets = Math.min(...tweets) - 100;

        const followersCtx = document.getElementById('followersChart').getContext('2d');
        const tweetsCtx = document.getElementById('tweetsChart').getContext('2d');

        new Chart(followersCtx, {
          type: 'line',
          data: {
            labels: dates,
            datasets: [{
              label: 'Followers',
              data: followers,
              backgroundColor: 'rgba(54, 162, 235, 0.2)',
              borderColor: 'rgba(54, 162, 235, 1)',
              borderWidth: 1
            }]
          },
          options: {
            scales: {
              y: {
                beginAtZero: false,
                min: minFollowers
              }
            }
          }
        });

        new Chart(tweetsCtx, {
          type: 'line',
          data: {
            labels: dates,
            datasets: [{
              label: 'Tweets',
              data: tweets,
              backgroundColor: 'rgba(255, 99, 132, 0.2)',
              borderColor: 'rgba(255, 99, 132, 1)',
              borderWidth: 1
            }]
          },
          options: {
            scales: {
              y: {
                beginAtZero: false,
                min: minTweets
              }
            }
          }
        });
      });
  </script>
</body>

</html>

Commit it to the repository.

(optional) If you want to test it locally, do so by copying the data folder to the dashboard folder and launching a simple server inside it.

cp -r data dashboard/
cd dashboard
# start server with Python if you have it installed (version 3)
# otherwise, use other way e. g. https://gist.github.com/willurd/5720255
python -m http.server 8000

Dashboard deployment to GitHub Pages

Now that we have our dashboard, it’s time to present it to the web!

If you are using a free account on GitHub, your page needs to be public, as well as the whole repository.

Create a .github/workflows/deploy_dashboard.yml file.

name: Deploy to GitHub Pages

on:
  schedule:
    # redeploy after data update
    - cron: '0 5 * * *'
  push:
    branches:
      - main
  workflow_dispatch: # This line enables manual triggering of the action

permissions:
  contents: read
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  deploy:
    permissions:
      pages: write      # to deploy to Pages
      id-token: write   # to verify the deployment originates from an appropriate source

    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}

    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Pages
        uses: actions/configure-pages@v4

      - name: Copy data to dashboard folder
        run: cp -r data dashboard/

      - name: Update pages artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: dashboard/

      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4 # or specific "vX.X.X" version tag for this action

The action should deploy the page. You can find the URL in your Github project settings or in the Actions section in the workflow output.

Again, you can find mine here: https://horosin.github.io/metrics-dashboard/.

Summary

And there you have it! A complete, automated system to track your social media (X) metrics, automate data fetching, save historical data, and visualize the trends. With this setup, you can extend the functionality to other platforms and metrics, creating a comprehensive dashboard for all your analytical needs. Let me know if it’s something you’d like to read about.

If you liked this content, please support me by sharing this post and subscribing to my Newsletter.

You can also find me here:

x/twitter: horosin_
linkedin

Create a dashboard to track your twitter/X follower stats with APIs and GitHub Actions

Comments

More from this blog

Building for myself: simple tools, real use

AI made my game while I waited for coffee

The AI coding tool I use the most

Navigating Large Codebases in VS Code

How to set up free, local coding AI assistant for VS Code

What we’re going to build

Set up

Twitter application

Project

Fetching the data

Saving the data

Scheduled GitHub Action

Dashboard

Dashboard deployment to GitHub Pages

Summary

Command Palette

Comments

More from this blog

What we’re going to build

Set up

Twitter application

Project

Fetching the data

Saving the data

Scheduled GitHub Action

Dashboard

Dashboard deployment to GitHub Pages

Summary