Add README.md

2026-10-04 22:34:30 +00:00
commit c84e37ec3f
1 changed files with 78 additions and 0 deletions
@@ -0,0 +1,78 @@
+## Overview
+
+For my personal self-hosted infrastructure I found the typical Prometheus and Grafana set up to be too bloated, involving too much network traffic and overhead, for what I actually want to monitor
+
+Consequently I decided to build my own lightweight tooling with Python, Linux commands, SSH, Cron, JSON, HTML/CSS and Javascript
+
+It is capable of sending offline/healing alerts via email, as well as scraping relevant metrics, (CPU, RAM, Disk Space etc), to display on a centralized webpage
+
+The metrics page, as well as the generated JSON files and Python script is hosted on my [Apache-Intranet LXC Container](https://git.tobiastime.xyz/Tobias/Apache-LXC-Intranet)
+
+## High Level Architecture Overview
+
+```
+Cron starts scraper.py every 15 minutes
+                ↓
+       For each monitored server
+                ↓
+     Attempt to reach via ICMP 
+          /             \
+     SUCCESS             FAILURE
+        ↓                   ↓
+Check previous state      Check previous state
+        ↓                   ↓
+ Previously offline?      Previously online?
+   /          \              /            \
+ YES           NO           YES            NO
+  ↓             ↓            ↓              ↓
+Send "back      Continue    Send           Continue
+online" email              "offline" email
+     \         /              \           /
+      └───────┬───────────────┴──────────┘
+                  ↓
+         If server is pingable
+                 ↓
+   Connect via dedicated SSH key
+        as restricted user
+                 ↓
+   Run collector.py to scrape metrics
+                 ↓
+       Collector generates JSON
+                 ↓
+        Copy JSON back via SFTP
+                 ↓
+Store in appropriate Apache server directory
+                 ↓
+          index.html reads JSON
+                 ↓
+       Metrics displayed in web portal
+                 ↓
+Save current online/offline server state to JSON
+```
+
+## Monitoring Failover
+
+I have a secondary Debian LXC container on a separate hypervisor host to provide monitoring redundancy
+
+It is scheduled via Cron to attempt to ping Apache-Intranet every 5 minutes
+
+If Apache-Intranet, (the primary), is unreachable it takes over monitoring/alerting duties, otherwise it lies dormant
+
+Its purpose is to provide email alerts to me informing that we are currently in the midst of an outage and which servers are offline
+
+```
+┌──────────────────────────────────────────┐
+│       Secondary Monitoring LXC           │
+│    Runs ping.py every 5 minutes which    │
+│ checks Apache-Intranet's health via ICMP │
+└──────────────────────────────────────────┘
+                │
+      Is primary reachable?
+      /          \
+    YES            NO
+     ↓              ↓
+  Do nothing      Take over
+                    ↓
+             Begin monitoring servers' reachability
+                   and sending alerts
+```