traveler/Netgrimoire
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: create Gremlin/Morning_Briefing

Browse source
This commit is contained in:
Administrator 2026-04-19 04:27:24 +00:00 • committed by John Smith
parent 7fd2f0c91e
commit 8e5b76103a
1 changed files with 639 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

639
Gremlin/Morning_Briefing.md Normal file
View file

@ -0,0 +1,639 @@
---
title: Consolidated Morning Briefing
description:
published: true
date: 2026-04-19T04:27:14.348Z
tags:
editor: markdown
dateCreated: 2026-04-19T04:27:14.348Z
---
# Gremlin's Morning Briefing System
**Complete automated daily briefing integrating calendars, work taskings, and personal email**
---
## Overview
The Gremlin Morning Briefing is an automated system that generates a unified HTML dashboard displaying:
- **SIL Calendar** (Google Calendar synced to Nextcloud)
- **Work Email Taskings** (exported from Outlook as Markdown)
- **Personal Email Tasks** (parsed from MailCow IMAP via Ollama LLM)
- **Federal Holidays**
- **Future:** NetGrimoire stack status, Vikunja tasks
The briefing organizes everything into three priority zones:
- 🔥 **TODAY** - Overdue tasks, today's events, high-priority items
- 📆 **THIS WEEK** - Next 7 days planning horizon
- 📬 **LATER** - Days 8-14, backlog items
---
## Architecture
### Data Sources
1. **Google Calendar (SIL)** → Service Account API
2. **Nextcloud** → Work briefing MD file via WebDAV
3. **MailCow IMAP** → Personal email scanning
4. **Ollama (llama3.2:3b)** → Email parsing for tasks/due dates
5. **Federal Holidays API** → date.nager.at
### Processing Pipeline
```
Google Calendar ──┐
Work Email MD ────┼──> Python Script ──> HTML Briefing ──> Apache ──> Glance iframe
MailCow IMAP ─────┤ ↓
Federal Holidays ─┘ Nextcloud CalDAV
(SIL + Taskings calendars)
```
### Key Optimizations
- **Email Pre-screening:** Regex filters reduce Ollama calls from 285 → ~30
- **Caching:** Processed email IDs prevent re-processing
- **Zone-based bucketing:** Smart categorization by urgency and date
---
## Installation
### Prerequisites
**System Requirements:**
- Python 3.9+
- Docker Swarm (for Glance)
- Apache web server
- Nextcloud instance
- MailCow mail server
- Ollama with `llama3.2:3b` model
**Python Dependencies:**
```bash
pip3 install --break-system-packages \
google-auth google-auth-oauthlib google-auth-httplib2 \
google-api-python-client \
caldav icalendar pytz requests
```
---
### 1. Google Calendar Setup
**Create Service Account:**
1. Go to [Google Cloud Console](https://console.cloud.google.com)
2. Create project: `NetGrimoire`
3. Enable **Google Calendar API**
4. Create Service Account: `n8n-calendar-reader@netgrimoire.iam.gserviceaccount.com`
5. Generate JSON key → save to `/home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json`
6. Set permissions: `chmod 600 /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json`
**Share Calendar:**
1. Open SIL calendar in Google Calendar
2. Settings → Share with specific people
3. Add service account email (read-only)
---
### 2. Nextcloud Setup
**Create App Password:**
1. Login to Nextcloud: `https://cloud.netgrimoire.com`
2. Settings → Security → Devices & sessions
3. Create new app password: `Gremlin Briefing Script`
4. Save password for script configuration
**Create Folders:**
```
/Briefings/
└── work-briefing.md
```
**Upload Work Briefing:**
- Export daily work tasking email as Markdown
- Upload to `/Briefings/work-briefing.md` (overwrite daily)
---
### 3. Script Installation
**Deploy Script:**
```bash
# Create directory structure
mkdir -p /home/gremlin/scripts
mkdir -p /home/gremlin/.config/gcloud
mkdir -p /home/gremlin/logs
# Copy script
cp gremlin-briefing-final.py /home/gremlin/scripts/
chmod 600 /home/gremlin/scripts/gremlin-briefing-final.py
chmod +x /home/gremlin/scripts/gremlin-briefing-final.py
# Edit configuration
nano /home/gremlin/scripts/gremlin-briefing-final.py
```
**Configuration (lines 24-48):**
```python
# Nextcloud
NEXTCLOUD_PASSWORD = 'YOUR_APP_PASSWORD_HERE'
# MailCow IMAP
MAILCOW_PASSWORD = 'YOUR_MAILCOW_PASSWORD_HERE'
# Email Keywords (flagged for attention)
EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy']
# VIP Senders (always surface in TODAY)
EMAIL_VIP_SENDERS = ['cindy']
```
**Test Run:**
```bash
/home/gremlin/scripts/gremlin-briefing-final.py
```
Expected output:
```
============================================================
Gremlin Briefing Generator - Optimized
============================================================
Fetching Google Calendar events...
Found 15 SIL events
Fetching work briefing from Nextcloud...
Found work briefing (12453 chars)
Parsed 18 work events, 24 action items
Connecting to MailCow IMAP...
Found 285 emails in last 14 days
Fetched 8 new emails
Pre-screening 8 emails...
✓ 3 emails need processing (filtered 5 junk)
Processing 3 emails with Ollama...
Parsing: Need to get Harvey a bath on april 19th...
✓ Successfully parsed 3 emails
Syncing SIL calendar to Nextcloud...
✓ SIL sync complete
Syncing email tasks to Nextcloud...
✓ Added 2 email tasks to calendar
Fetching federal holidays...
Found 11 federal holidays
Organizing items into zones...
Generating HTML briefing...
Writing to /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html...
============================================================
✓ SUCCESS!
TODAY: 3 events, 2 tasks, 1 flagged, 0 VIP
THIS WEEK: 5 days, 1 tasks
LATER: 2 days, 0 backlog
============================================================
```
---
### 4. Cron Job Setup
**Daily Execution (6:00 AM):**
```bash
crontab -e
```
Add:
```cron
0 6 * * * /usr/bin/python3 /home/gremlin/scripts/gremlin-briefing-final.py >> /home/gremlin/logs/briefing.log 2>&1
```
**Manual Trigger:**
```bash
# Run now
/home/gremlin/scripts/gremlin-briefing-final.py
# Clear email cache (force re-process all emails)
rm /home/gremlin/.gremlin-email-cache.json
```
---
### 5. Glance Integration
**Update Glance Configuration:**
```bash
nano /data/nfs/znas/Docker/glance/glance.yml
```
**Add iframe widget:**
```yaml
- type: iframe
title: "☠️ Gremlin's Morning Briefing"
source: "https://www.netgrimoire.com/briefing.html"
height: 1200
```
**Restart Glance:**
```bash
docker service update --force glance_glance
```
**Access:**
- Direct: `https://www.netgrimoire.com/briefing.html`
- Via Glance: `https://home.netgrimoire.com`
---
## Configuration Guide
### Adding Email Keywords
Email keywords flag specific topics for your attention. Emails matching these keywords appear in the **🚨 Flagged Emails** section of TODAY.
**Edit Script:**
```bash
nano /home/gremlin/scripts/gremlin-briefing-final.py
```
**Modify Line 45:**
```python
EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy']
```
**Add keywords** (lowercase, no spaces):
```python
EMAIL_FLAG_KEYWORDS = [
'mxroute', # Hosting provider mentions
'travel', # Travel-related emails
'dts', # Defense Travel System
'cititravel', # Travel booking system
'.mil', # Military emails
'cindy', # From/mentions Cindy
'vikunja', # Task management mentions
'backup', # Backup-related alerts
'downtime', # Service outage mentions
]
```
**Keywords are case-insensitive** and match anywhere in subject or first 500 chars of body.
---
### Adding VIP Senders
VIP senders always appear in TODAY's **💕 VIP** section, regardless of content.
**Edit Line 46:**
```python
EMAIL_VIP_SENDERS = ['cindy']
```
**Add multiple VIPs:**
```python
EMAIL_VIP_SENDERS = [
'cindy', # Wife
'boss@company', # Boss's email
'urgent@', # Any address containing "urgent"
]
```
**Matching is case-insensitive substring match** on the `From:` field.
---
### Adjusting Email Search Window
**Default:** 14 days (last 2 weeks)
**Edit Line 44:**
```python
EMAIL_SEARCH_DAYS = 14
```
**Change to 7 days:**
```python
EMAIL_SEARCH_DAYS = 7
```
**Warning:** Larger windows = more emails to process. Pre-screening mitigates this, but 30+ days may be slow.
---
### Customizing Ollama Model
**Default:** `llama3.2:3b` (fast, good quality for email parsing)
**Edit Line 49:**
```python
OLLAMA_MODEL = 'llama3.2:3b'
```
**Alternative models:**
```python
OLLAMA_MODEL = 'qwen2.5-coder:7b' # Better date extraction, slower
OLLAMA_MODEL = 'llama3.1:8b' # More accurate, much slower
```
**Test model availability:**
```bash
curl http://docker4:11434/api/tags
```
---
## Email Cache Management
The script tracks processed emails in `/home/gremlin/.gremlin-email-cache.json` to avoid re-processing.
**View cache:**
```bash
cat /home/gremlin/.gremlin-email-cache.json
```
**Clear cache (force re-process all emails):**
```bash
rm /home/gremlin/.gremlin-email-cache.json
```
**Cache grows indefinitely.** Periodically clear to prevent bloat:
```bash
# Monthly cache reset (add to crontab)
0 0 1 * * rm /home/gremlin/.gremlin-email-cache.json
```
---
## Work Email Export Process
**Daily Workflow:**
1. **Export from Outlook:**
- Select "Daily Executive Briefing" email
- Save as `.md` file
2. **Upload to Nextcloud:**
- Navigate to `/Briefings/`
- Upload/replace `work-briefing.md`
3. **Script automatically fetches** on next run
**Expected MD Format:**
```markdown
## Daily Executive Briefing: April 18 - April 24, 2026
### Next Week's Schedule
| Date | Time | Subject | Location |
|:---|:---|:---|:---|
| Mon, Apr 20 | 08:30 - 09:30 | Meeting Name | Conference Room |
### Prioritized Action Items
| Priority | From | Subject | Action / Deadline |
|:---|:---|:---|:---|
| **High** | Boss Name | Task Subject | Action required: Details here |
| Normal | Colleague | Task Subject | Action required: Details here |
```
**Script parses:**
- Schedule table → merges into calendar view
- Action items table → displays in **📧 Work Taskings**
- Filters out "Canceled:" meetings
---
## Troubleshooting
### Script Fails: Google Calendar Auth Error
**Error:**
```
Error fetching Google Calendar: invalid_grant
```
**Fix:**
1. Verify service account JSON file exists:
```bash
ls -l /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json
```
2. Check calendar sharing in Google Calendar settings
3. Regenerate service account key if needed
---
### Script Fails: Nextcloud Connection Error
**Error:**
```
Error fetching work briefing: HTTP 401
```
**Fix:**
1. Verify Nextcloud app password
2. Test WebDAV manually:
```bash
curl -u graymutt:YOUR_PASSWORD \
https://cloud.netgrimoire.com/remote.php/dav/files/graymutt/Briefings/work-briefing.md
```
3. Regenerate app password if needed
---
### Script Fails: MailCow IMAP Error
**Error:**
```
Error fetching emails: Login failed
```
**Fix:**
1. Verify MailCow credentials
2. Test IMAP manually:
```bash
telnet 192.168.5.16 143
# Type: a001 LOGIN phil@pncharris.com YOUR_PASSWORD
```
3. Check MailCow IMAP is enabled for user
---
### Ollama Parsing Returns Null
**Symptom:** Emails found, but `Successfully parsed 0 emails`
**Causes:**
1. Ollama not running:
```bash
curl http://docker4:11434/api/tags
```
2. Model not pulled:
```bash
docker exec -it ollama ollama pull llama3.2:3b
```
3. Timeout issues (increase Line 368):
```python
timeout=30 # Increase to 60 for slower models
```
---
### Email Pre-screening Too Aggressive
**Symptom:** Important emails not appearing in briefing
**Fix:** Check pre-screening patterns in `should_process_email()` (Lines 297-334)
**Add more date patterns:**
```python
date_patterns = [
r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b',
r'\b\d{4}-\d{2}-\d{2}\b',
# Add custom patterns
r'next week',
r'by end of',
]
```
**Add more action keywords:**
```python
action_keywords = [
'due', 'deadline', 'task', 'todo',
# Add custom keywords
'please review',
'need your input',
'waiting on',
]
```
---
### HTML Not Rendering in Glance
**Symptom:** Blank iframe or "403 Forbidden"
**Fix:**
1. Verify Apache serving file:
```bash
curl https://www.netgrimoire.com/briefing.html
```
2. Check file permissions:
```bash
ls -l /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html
# Should be: -rw-r--r-- 1 1964 1964
```
3. Verify Apache VirtualHost config:
```bash
cat /data/nfs/znas/Docker/web/apache/netgrimoire.conf
```
---
## Future Enhancements
### Planned Integrations
**NetGrimoire Stack Status:**
```python
def fetch_netgrimoire_status():
"""Fetch Uptime Kuma API for service health"""
# Query Uptime Kuma heartbeat API
# Return: {services_up: N, services_down: N, warnings: [...]}
```
**Vikunja Tasks:**
```python
def fetch_vikunja_tasks():
"""Fetch tasks from Vikunja API"""
# Query Vikunja API for open tasks
# Merge into TODAY/THIS WEEK/LATER zones
```
**Additional Google Calendars:**
- PNC Fish and More
- Personal calendar
- Shared family calendar
**Gremlin Mood States:**
- Time-based commentary variation
- Workload-based sass level
- Stack health integration
---
## File Locations
### Script Files
```
/home/gremlin/
├── scripts/
│ └── gremlin-briefing-final.py
├── logs/
│ └── briefing.log
└── .config/
└── gcloud/
└── netgrimoire-calendar-sa.json
```
### Data Files
```
/home/gremlin/
└── .gremlin-email-cache.json
/data/nfs/znas/Docker/web/pages/netgrimoire/public_html/
└── briefing.html
```
### Nextcloud Files
```
/Briefings/
└── work-briefing.md
```
---
## Related Documentation
- [Google Calendar API](https://developers.google.com/calendar/api/guides/overview)
- [Nextcloud WebDAV](https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html)
- [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md)
- [Glance Configuration](https://github.com/glanceapp/glance)
---
## Maintenance
### Weekly Tasks
- ✅ Review flagged keywords effectiveness
- ✅ Check email cache size
- ✅ Verify Nextcloud work briefing uploads
### Monthly Tasks
- ✅ Clear email cache
- ✅ Review Ollama parsing accuracy
- ✅ Update work email export process if needed
### Quarterly Tasks
- ✅ Review Google Calendar sharing permissions
- ✅ Rotate Nextcloud app password
- ✅ Update federal holidays API if needed
---
## Support
**Logs:**
```bash
# Script execution log
tail -f /home/gremlin/logs/briefing.log
# Glance container logs
docker service logs glance_glance --tail 50
```
**Manual debugging:**
```bash
# Run script with verbose output
python3 -u /home/gremlin/scripts/gremlin-briefing-final.py
# Test individual components
python3 -c "from google.oauth2 import service_account; print('Google auth OK')"
```
---
*Last updated: April 18, 2026*
*Maintained by: Phil Harris (graymutt)*