Add README.md
This commit is contained in:
@@ -0,0 +1,78 @@
|
|||||||
|
## Overview
|
||||||
|
|
||||||
|
For my personal self-hosted infrastructure I found the typical Prometheus and Grafana set up to be too bloated, involving too much network traffic and overhead, for what I actually want to monitor
|
||||||
|
|
||||||
|
Consequently I decided to build my own lightweight tooling with Python, Linux commands, SSH, Cron, JSON, HTML/CSS and Javascript
|
||||||
|
|
||||||
|
It is capable of sending offline/healing alerts via email, as well as scraping relevant metrics, (CPU, RAM, Disk Space etc), to display on a centralized webpage
|
||||||
|
|
||||||
|
The metrics page, as well as the generated JSON files and Python script is hosted on my [Apache-Intranet LXC Container](https://git.tobiastime.xyz/Tobias/Apache-LXC-Intranet)
|
||||||
|
|
||||||
|
## High Level Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Cron starts scraper.py every 15 minutes
|
||||||
|
↓
|
||||||
|
For each monitored server
|
||||||
|
↓
|
||||||
|
Attempt to reach via ICMP
|
||||||
|
/ \
|
||||||
|
SUCCESS FAILURE
|
||||||
|
↓ ↓
|
||||||
|
Check previous state Check previous state
|
||||||
|
↓ ↓
|
||||||
|
Previously offline? Previously online?
|
||||||
|
/ \ / \
|
||||||
|
YES NO YES NO
|
||||||
|
↓ ↓ ↓ ↓
|
||||||
|
Send "back Continue Send Continue
|
||||||
|
online" email "offline" email
|
||||||
|
\ / \ /
|
||||||
|
└───────┬───────────────┴──────────┘
|
||||||
|
↓
|
||||||
|
If server is pingable
|
||||||
|
↓
|
||||||
|
Connect via dedicated SSH key
|
||||||
|
as restricted user
|
||||||
|
↓
|
||||||
|
Run collector.py to scrape metrics
|
||||||
|
↓
|
||||||
|
Collector generates JSON
|
||||||
|
↓
|
||||||
|
Copy JSON back via SFTP
|
||||||
|
↓
|
||||||
|
Store in appropriate Apache server directory
|
||||||
|
↓
|
||||||
|
index.html reads JSON
|
||||||
|
↓
|
||||||
|
Metrics displayed in web portal
|
||||||
|
↓
|
||||||
|
Save current online/offline server state to JSON
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring Failover
|
||||||
|
|
||||||
|
I have a secondary Debian LXC container on a separate hypervisor host to provide monitoring redundancy
|
||||||
|
|
||||||
|
It is scheduled via Cron to attempt to ping Apache-Intranet every 5 minutes
|
||||||
|
|
||||||
|
If Apache-Intranet, (the primary), is unreachable it takes over monitoring/alerting duties, otherwise it lies dormant
|
||||||
|
|
||||||
|
Its purpose is to provide email alerts to me informing that we are currently in the midst of an outage and which servers are offline
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────┐
|
||||||
|
│ Secondary Monitoring LXC │
|
||||||
|
│ Runs ping.py every 5 minutes which │
|
||||||
|
│ checks Apache-Intranet's health via ICMP │
|
||||||
|
└──────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
Is primary reachable?
|
||||||
|
/ \
|
||||||
|
YES NO
|
||||||
|
↓ ↓
|
||||||
|
Do nothing Take over
|
||||||
|
↓
|
||||||
|
Begin monitoring servers' reachability
|
||||||
|
and sending alerts
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user