I Built a Tool to Stop Wasting Time on Toxic Open Source Projects

The Motivation

After contributing to several open source projects, I realized some of them have serious issues. Many maintainers don't provide help when you submit pull requests, and you end up wrestling with automated code reviews just to show one commit on your GitHub profile. So this time, instead of building more personal projects (which I've written about in my blogs like Building My Own HTTP Server in TypeScript and Building a CLI Tool That Made My Life Easier), contributing to random open source repositories, or grinding LeetCode problems, I wanted to create something impactful for the open source community. I decided to build repo-health (live demo) to help contributors choose projects they can successfully contribute to, and learn what works and what doesn't in open source collaboration.

My Tech Stack

• Frontend: Next.js 16, React 19, Chakra UI
• Backend: tRPC, Octokit, Zod
• Data: MySQL (Prisma), Redis

I chose this stack to get familiar with current industry-standard tools and see how they work together in a real application(at the end of the day, they are just tools to build something cool).

First Challenge - What am I building?

When I started, I was just showing some data from GitHub and thinking that this is cool until I show my friends and college students. I realized that even though you spent a lot of time building one feature or solving one big bug it does not really matter if it doesn't solve the real world problem. So, I realized that before writing anything it is better to sit and think why it is needed to write that. So, I challenged myself by putting all my efforts to this product into 2 weeks.

Narrowing the Project Scope

I decided to focus on helping open source contributors with issues I've personally experienced and seen my classmates face in college: toxic communication environments on GitHub and wrestling with automated code reviews without proper guidance from maintainers.

First Feature: Overall Score

My system uses a Hybrid Approach that combines a deterministic formula grounded in industry standards with a qualitative Language Model Judge to account for real-world context.

The Algorithm (0-100 Score)

The base health score is calculated using a custom weighted average that I designed, inspired by standardized CHAOSS Metrics. I tuned the weights myself based on what I believe indicates a healthy modern project:

Score = (0.3 × Activity) + (0.25 × Maintenance) + (0.2 × Community) + (0.25 × Docs)

• Activity (30%): Frequency of commits + Recency of updates + Unique authors.
• Maintenance (25%): Issue response time + Open issue ratio + Repository age.
• Community (20%): Logarithmic scale of Stars & Forks.
• Documentation (25%): Existence of README, LICENSE, and CONTRIBUTING files.

The Language Model Adjustment

Standard formulas often misjudge "Feature-Complete" projects as "Dead." To solve this, I added a Judge Layer using a language model.

My Contribution:

I implemented a secondary logic layer where a language model analyzes the repository's purpose (through README content and file structure). I explicitly allow the model to override the algorithmic score by ±20 points if it detects that the metrics are misleading. This was my addition to bridge the gap between raw numbers and real-world context. For the MVP (minimum viable product), I'm currently using GPT-4 Mini due to its cost-effectiveness and fast response times. This allows me to validate the approach before potentially scaling to other models like Claude Sonnet or coming up with stronger ideas (I expect your ideas :)).

• Example: A stable utility library with 0 commits in 6 months.
• Algorithm: Penalizes it as "Old/Abandoned."
• Language Model Judge: Recognizes it as "Completed/Stable" and awards a +20 Stability Bonus.

Implementation Snippet:

// I feed the calculated score into the AI prompt and ask for an adjustment:

prompt += `
  "scoreInsights": {
    "adjustment": {
       "shouldAdjust": true, 
       "amount": 20, // Range: -20 to +20
       "reason": "This is a stable utility library in maintenance mode. Low activity is expected and healthy.", 
       "confidence": "high"
    }
  }
`;

Checking PR Metrics

I built the PR Metrics Analysis to solve the lack of communication in open source. Before contributing, you need to know:

1. Speed: Is the average merge time hours or months?
2. Humanity: Are you dealing with real people or just fighting bot reviews?
3. Growth: Do new contributors actually stick around?

1. Handling Data Efficiently (Backend Concurrency)

To get these stats fast, I couldn't fetch everything one by one. I used Promise.all to fetch Open PRs, Closed PRs, and Template checks in parallel, cutting load time significantly.I dived into this topic in my http server blog post about event loop, here, the longest operation defines the total execution time.

// Efficiently fetching Open PRs, Closed PRs, and Template checks simultaneously
const [openPRs, closedPRs, template] = await Promise.all([
  fetchPRs(octokit, { owner, repo, state: "open" }),
  fetchPRs(octokit, { owner, repo, state: "closed" }),
  checkPRTemplate(octokit, { owner, repo }),
]);

Keeping contributors around matters more than total count. I used a Sankey Diagram to visualize the flow from "First-time" to "Core Team," making it easy to see if contributors stay or leave immediately.

// Visualizing the contributor flow
const data = {
  nodes: [
    { id: "First PR", color: "#58a6ff" },
    { id: "2nd Contribution", color: "#3fb950" },
    { id: "Regular (3-9)", color: "#a371f7" },
    { id: "Core Team (10+)", color: "#f0883e" },
  ],
  links: [
    {
      source: "First PR",
      target: "2nd Contribution",
      value: funnel.secondContribution + funnel.regular + funnel.coreTeam,
    },
    // ... logic to map flows for Regular and Core contributors
  ].filter((link) => link.value > 0),
};

Where I Made the Wrong Choice: The Security Scanner

I initially built a full Secrets Detection to catch exposed API keys, inspired by Gitleaks and TruffleHog.

How it worked:

• Regex Pattern Matching: I used ~22 industry-standard patterns to catch known secrets (AWS keys, GitHub tokens, Stripe keys).
• Randomness Detection: I implemented mathematics to detect highly random strings that "look" like secrets even if they don't match a pattern.

Why I removed it:

While building a security scanner was a great engineering challenge, I realized it drifted from my core mission. I decided to cut the feature to keep the project focused on community metrics rather than security auditing. Believe me, deleting it was painful, but that is just the way it is. Software Engineering is not about solving hard problems, it is about solving existing real life issues. Thanks to my friends' wake-up calls I opened my eyes and saw how I wasted my time.

Intelligent Issue Analysis

Analyzing Issues is the most effective way to understand a project's activity. I implemented several specific metrics to provide real insight to contributors:

• Average Close Time: This measures the project's true speed. A repository with many open issues is still healthy if the average close time is short (e.g., 2 days vs. 6 months). I track both Average and Middle Value to filter out unusual cases (like issues that took 2 years to close).

• Hot Issues: To help contributors find active discussions, I use a custom algorithm that prioritizes recent updates (last 48h), high engagement (comments/reactions), and security-related keywords.

• Hidden Gems: This highlights "Old" but "High Impact" issues (like ignored feature requests). These are often ideal first contributions because they provide value without the conflict of highly active discussions.

• Crackability Score: A calculated difficulty rating (0-100) based on documentation quality, file scope, and testing requirements. This filters complex issue lists into tasks that are feasible for new contributors to complete.

Project Overview & File-Issue Mapping

I have never been a frontend guy, but this journey pushed me to dive in. One problem I wanted to solve was reducing the complexity of exploring a new project. When you land on a repository with 500 files, where do you even start?

My Solution: Language Model-Powered Structure Analysis

I built a system that recursively fetches the entire file tree and feeds the structure to LLM . The LLM then tells you the entry points, key files, and which folders are responsible for which features. This way, the LLM gives you a proper project tour.

// Recursively fetch the entire file tree from GitHub
const { data } = await octokit.git.getTree({
  owner,
  repo,
  tree_sha: "HEAD",
  recursive: "true",  // Fetches entire tree structure at once
});

File-Issue Mapping: Connecting Problems to Code

On top of that, I added File-Issue Mapping. Before reaching the LLM, I use regex to scan all issue descriptions for file paths. If an issue mentions src/components/Button.tsx, I link that issue directly to that file in the overview. This way, a user can click on a file and immediately see if this file has any open issues and whether this is a single-file fix or if the issue affects multiple files.

// Extract file paths mentioned in issue text using regex
const FILE_PATTERN = /[\w\-\/\.]+\.(ts|tsx|js|jsx|py|go|rs|java|cpp|c)/gi;
function extractFilePaths(text: string): string[] {
  const matches = text.match(FILE_PATTERN) || [];
  return [...new Set(matches)]; // Deduplicate
}

Project Tree Visualization (Frontend)

For drawing the project structure, I got inspiration from the repo-visualizer project. On the frontend, I implemented a recursive function that builds a hierarchy from the flat file list. This function traverses each file path, splits it by /, and creates nested parent/child relationships to form a tree.

// Recursively build hierarchy from flat file paths
function buildHierarchy(files: FileNode[], repoName: string, maxDepth = 3): HierarchyNode {
  const root: HierarchyNode = { name: repoName, path: "", children: [] };
  files.forEach((file) => {
    const parts = file.path.split("/");
    let current = root;
    // Traverse and build tree structure
    parts.slice(0, maxDepth + 1).forEach((part, index) => {
      let child = current.children?.find((c) => c.name === part);
      if (!child) {
        child = { name: part, children: [] };
        current.children!.push(child);
      }
      current = child;
    });
  });
  return root;
}

I also added collision reduction by limiting depth and file count. This keeps the visualization readable even for large repositories.

Known Limitation: I tried to add zoom and pan functionality, but the sensitivity was off and it broke normal scrolling (PRs are welcome). I decided to keep the visualization simple and stable rather than ship a broken interactive version. This feature needs more polish in future iterations.

Activity Pattern Detection

As you know, especially during Hacktoberfest, some people make commits and PRs just for the sake of doing it. Spam contributions, bulk deletions, and suspicious patterns are everywhere. To detect these activities, I built a Pattern Detection system based on commit metrics from the GitHub API.

What is a Suspicious Pattern?

A suspicious pattern is any commit activity that looks very different from normal development work. I track several types:

Mass Deletion Detection

"Deletion Rate" measures the ratio of code deleted vs total changes. A commit that deletes 90% of what it touches with 100+ lines removed is suspicious. It could be a cleanup, or it could be vandalism.

// Detect unusual deletion patterns
function detectChurnAnomalies(commits: CommitWithStats[]): PatternAnomaly[] {
  for (const commit of commits) {
    const total = commit.additions + commit.deletions;
    const churnRatio = commit.deletions / total;

    // Flag commits that delete >80% of touched code
    if (churnRatio > 0.8 && commit.deletions > 100) {
      anomalies.push({
        type: "churn",
        severity: churnRatio > 0.9 ? "critical" : "warning",
        description: `Deleted ${Math.round(churnRatio * 100)}% of code (${commit.deletions} lines)`,
      });
    }
  }
}

Rapid-Fire Commits Detection

This catches contribution bombarding. When someone makes 10+ commits in under 10 minutes. Real development doesn't work that way.

// Detect rapid-fire commits (likely spam or farming)
function detectBurstActivity(commits: CommitWithStats[]): PatternAnomaly[] {
  for (let i = 0; i < sorted.length - 4; i++) {
    const windowStart = new Date(sorted[i].date).getTime();
    const windowEnd = new Date(sorted[i + 4].date).getTime();
    const diffMinutes = (windowEnd - windowStart) / (1000 * 60);

    // 5+ commits in under 10 minutes = suspicious
    if (diffMinutes <= 10) {
      anomalies.push({
        type: "velocity",
        severity: count > 10 ? "critical" : "warning",
        description: `Burst: ${count} commits in ${Math.round(diffMinutes)} minutes`,
      });
    }
  }
}

Risk Grades

Pivot: Checking from Maintainer's Perspective

When I started this project, I was only thinking about contributor's perspective, but reality hit hard. I read blog posts and articles such as :

• Burn out Story from Maintainer
• No Money In Open Source
• Extraordinary requirements from users

From these blogs I understood how wild open source can be. Basically, users and companies can ask for features that don't suit the project's goals, or big companies use open source projects without sponsoring them. Also, there's toxicity from users, people who just add their name to the README and feel proud even though that's not a real contribution. So I decided to look at the bigger picture and examine projects from the maintainer's perspective.

Contribution Insights

I built a feature that analyzes rejected PRs and shows why they failed, helping future contributors avoid the same mistakes.

Spam Detection

First, I filter out obvious spam PRs that just add a name to the README:

const SPAM_TITLE_PATTERNS = [
  /add(ed|ing)?\s+(my\s+)?name/i,
  /update(d)?\s+readme/i,
  /hacktoberfest/i,
];

function detectSpam(pr, files): { isSpam: boolean; reason: string } {
  const isReadmeOnly = files.length === 1 && 
    files[0].filename.toLowerCase().includes("readme");

  if (isReadmeOnly && files[0].additions < 5) {
    return { isSpam: true, reason: "Trivial README change" };
  }
  return { isSpam: false, reason: "" };
}

Automated Failure Analysis

For legitimate rejected PRs, I send the code diff and reviewer comments to a language model. It categorizes each failure:

type PitfallAnalysis = {
  prNumber: number;
  mistake: string;
  reviewFeedback: string;
  advice: string;
  category: "tests" | "style" | "scope" | "setup" | "breaking" | "docs";
};

This turns rejected PRs into a learning resource for the community.

Weird Bugs

Cache Security Vulnerability

I wasn't familiar with the stack when I started, and I made a mistake by using the same cache key for both public and private repositories. The cache was making the user experience much faster and smoother, so I was feeling proud of myself until I saw my friend's private project showing up in my account.

Out of curiosity, I asked my friend to tell me their private repo name, and in the fuzzy search and analysis, I saw the project. Boom! Of course, GitHub doesn't allow you to visit someone's private repo, but this was still a security vulnerability. I realized I wasn't creating different cache keys for each user.

The Fix: Token-Based Cache Isolation

I implemented a function that creates a unique hash from the user's access token:

import crypto from "crypto";

export function getTokenHash(token?: string | null): string {
  if (!token) return "public";
  return crypto.createHash("sha256").update(token