From 8e5b76103ac1515817d438774cd814b0f54e5cfd Mon Sep 17 00:00:00 2001 From: Administrator Date: Sun, 19 Apr 2026 04:27:24 +0000 Subject: [PATCH] docs: create Gremlin/Morning_Briefing --- Gremlin/Morning_Briefing.md | 639 ++++++++++++++++++++++++++++++++++++ 1 file changed, 639 insertions(+) create mode 100644 Gremlin/Morning_Briefing.md diff --git a/Gremlin/Morning_Briefing.md b/Gremlin/Morning_Briefing.md new file mode 100644 index 0000000..b7e69e4 --- /dev/null +++ b/Gremlin/Morning_Briefing.md @@ -0,0 +1,639 @@ +--- +title: Consolidated Morning Briefing +description: +published: true +date: 2026-04-19T04:27:14.348Z +tags: +editor: markdown +dateCreated: 2026-04-19T04:27:14.348Z +--- + +# Gremlin's Morning Briefing System + +**Complete automated daily briefing integrating calendars, work taskings, and personal email** + +--- + +## Overview + +The Gremlin Morning Briefing is an automated system that generates a unified HTML dashboard displaying: +- **SIL Calendar** (Google Calendar synced to Nextcloud) +- **Work Email Taskings** (exported from Outlook as Markdown) +- **Personal Email Tasks** (parsed from MailCow IMAP via Ollama LLM) +- **Federal Holidays** +- **Future:** NetGrimoire stack status, Vikunja tasks + +The briefing organizes everything into three priority zones: +- 🔥 **TODAY** - Overdue tasks, today's events, high-priority items +- 📆 **THIS WEEK** - Next 7 days planning horizon +- 📬 **LATER** - Days 8-14, backlog items + +--- + +## Architecture + +### Data Sources +1. **Google Calendar (SIL)** → Service Account API +2. **Nextcloud** → Work briefing MD file via WebDAV +3. **MailCow IMAP** → Personal email scanning +4. **Ollama (llama3.2:3b)** → Email parsing for tasks/due dates +5. **Federal Holidays API** → date.nager.at + +### Processing Pipeline +``` +Google Calendar ──┐ +Work Email MD ────┼──> Python Script ──> HTML Briefing ──> Apache ──> Glance iframe +MailCow IMAP ─────┤ ↓ +Federal Holidays ─┘ Nextcloud CalDAV + (SIL + Taskings calendars) +``` + +### Key Optimizations +- **Email Pre-screening:** Regex filters reduce Ollama calls from 285 → ~30 +- **Caching:** Processed email IDs prevent re-processing +- **Zone-based bucketing:** Smart categorization by urgency and date + +--- + +## Installation + +### Prerequisites + +**System Requirements:** +- Python 3.9+ +- Docker Swarm (for Glance) +- Apache web server +- Nextcloud instance +- MailCow mail server +- Ollama with `llama3.2:3b` model + +**Python Dependencies:** +```bash +pip3 install --break-system-packages \ + google-auth google-auth-oauthlib google-auth-httplib2 \ + google-api-python-client \ + caldav icalendar pytz requests +``` + +--- + +### 1. Google Calendar Setup + +**Create Service Account:** +1. Go to [Google Cloud Console](https://console.cloud.google.com) +2. Create project: `NetGrimoire` +3. Enable **Google Calendar API** +4. Create Service Account: `n8n-calendar-reader@netgrimoire.iam.gserviceaccount.com` +5. Generate JSON key → save to `/home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json` +6. Set permissions: `chmod 600 /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json` + +**Share Calendar:** +1. Open SIL calendar in Google Calendar +2. Settings → Share with specific people +3. Add service account email (read-only) + +--- + +### 2. Nextcloud Setup + +**Create App Password:** +1. Login to Nextcloud: `https://cloud.netgrimoire.com` +2. Settings → Security → Devices & sessions +3. Create new app password: `Gremlin Briefing Script` +4. Save password for script configuration + +**Create Folders:** +``` +/Briefings/ + └── work-briefing.md +``` + +**Upload Work Briefing:** +- Export daily work tasking email as Markdown +- Upload to `/Briefings/work-briefing.md` (overwrite daily) + +--- + +### 3. Script Installation + +**Deploy Script:** +```bash +# Create directory structure +mkdir -p /home/gremlin/scripts +mkdir -p /home/gremlin/.config/gcloud +mkdir -p /home/gremlin/logs + +# Copy script +cp gremlin-briefing-final.py /home/gremlin/scripts/ +chmod 600 /home/gremlin/scripts/gremlin-briefing-final.py +chmod +x /home/gremlin/scripts/gremlin-briefing-final.py + +# Edit configuration +nano /home/gremlin/scripts/gremlin-briefing-final.py +``` + +**Configuration (lines 24-48):** +```python +# Nextcloud +NEXTCLOUD_PASSWORD = 'YOUR_APP_PASSWORD_HERE' + +# MailCow IMAP +MAILCOW_PASSWORD = 'YOUR_MAILCOW_PASSWORD_HERE' + +# Email Keywords (flagged for attention) +EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy'] + +# VIP Senders (always surface in TODAY) +EMAIL_VIP_SENDERS = ['cindy'] +``` + +**Test Run:** +```bash +/home/gremlin/scripts/gremlin-briefing-final.py +``` + +Expected output: +``` +============================================================ +Gremlin Briefing Generator - Optimized +============================================================ +Fetching Google Calendar events... + Found 15 SIL events +Fetching work briefing from Nextcloud... + Found work briefing (12453 chars) + Parsed 18 work events, 24 action items +Connecting to MailCow IMAP... + Found 285 emails in last 14 days + Fetched 8 new emails +Pre-screening 8 emails... + ✓ 3 emails need processing (filtered 5 junk) +Processing 3 emails with Ollama... + Parsing: Need to get Harvey a bath on april 19th... + ✓ Successfully parsed 3 emails +Syncing SIL calendar to Nextcloud... + ✓ SIL sync complete +Syncing email tasks to Nextcloud... + ✓ Added 2 email tasks to calendar +Fetching federal holidays... + Found 11 federal holidays +Organizing items into zones... +Generating HTML briefing... +Writing to /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html... +============================================================ +✓ SUCCESS! + TODAY: 3 events, 2 tasks, 1 flagged, 0 VIP + THIS WEEK: 5 days, 1 tasks + LATER: 2 days, 0 backlog +============================================================ +``` + +--- + +### 4. Cron Job Setup + +**Daily Execution (6:00 AM):** +```bash +crontab -e +``` + +Add: +```cron +0 6 * * * /usr/bin/python3 /home/gremlin/scripts/gremlin-briefing-final.py >> /home/gremlin/logs/briefing.log 2>&1 +``` + +**Manual Trigger:** +```bash +# Run now +/home/gremlin/scripts/gremlin-briefing-final.py + +# Clear email cache (force re-process all emails) +rm /home/gremlin/.gremlin-email-cache.json +``` + +--- + +### 5. Glance Integration + +**Update Glance Configuration:** +```bash +nano /data/nfs/znas/Docker/glance/glance.yml +``` + +**Add iframe widget:** +```yaml +- type: iframe + title: "☠️ Gremlin's Morning Briefing" + source: "https://www.netgrimoire.com/briefing.html" + height: 1200 +``` + +**Restart Glance:** +```bash +docker service update --force glance_glance +``` + +**Access:** +- Direct: `https://www.netgrimoire.com/briefing.html` +- Via Glance: `https://home.netgrimoire.com` + +--- + +## Configuration Guide + +### Adding Email Keywords + +Email keywords flag specific topics for your attention. Emails matching these keywords appear in the **🚨 Flagged Emails** section of TODAY. + +**Edit Script:** +```bash +nano /home/gremlin/scripts/gremlin-briefing-final.py +``` + +**Modify Line 45:** +```python +EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy'] +``` + +**Add keywords** (lowercase, no spaces): +```python +EMAIL_FLAG_KEYWORDS = [ + 'mxroute', # Hosting provider mentions + 'travel', # Travel-related emails + 'dts', # Defense Travel System + 'cititravel', # Travel booking system + '.mil', # Military emails + 'cindy', # From/mentions Cindy + 'vikunja', # Task management mentions + 'backup', # Backup-related alerts + 'downtime', # Service outage mentions +] +``` + +**Keywords are case-insensitive** and match anywhere in subject or first 500 chars of body. + +--- + +### Adding VIP Senders + +VIP senders always appear in TODAY's **💕 VIP** section, regardless of content. + +**Edit Line 46:** +```python +EMAIL_VIP_SENDERS = ['cindy'] +``` + +**Add multiple VIPs:** +```python +EMAIL_VIP_SENDERS = [ + 'cindy', # Wife + 'boss@company', # Boss's email + 'urgent@', # Any address containing "urgent" +] +``` + +**Matching is case-insensitive substring match** on the `From:` field. + +--- + +### Adjusting Email Search Window + +**Default:** 14 days (last 2 weeks) + +**Edit Line 44:** +```python +EMAIL_SEARCH_DAYS = 14 +``` + +**Change to 7 days:** +```python +EMAIL_SEARCH_DAYS = 7 +``` + +**Warning:** Larger windows = more emails to process. Pre-screening mitigates this, but 30+ days may be slow. + +--- + +### Customizing Ollama Model + +**Default:** `llama3.2:3b` (fast, good quality for email parsing) + +**Edit Line 49:** +```python +OLLAMA_MODEL = 'llama3.2:3b' +``` + +**Alternative models:** +```python +OLLAMA_MODEL = 'qwen2.5-coder:7b' # Better date extraction, slower +OLLAMA_MODEL = 'llama3.1:8b' # More accurate, much slower +``` + +**Test model availability:** +```bash +curl http://docker4:11434/api/tags +``` + +--- + +## Email Cache Management + +The script tracks processed emails in `/home/gremlin/.gremlin-email-cache.json` to avoid re-processing. + +**View cache:** +```bash +cat /home/gremlin/.gremlin-email-cache.json +``` + +**Clear cache (force re-process all emails):** +```bash +rm /home/gremlin/.gremlin-email-cache.json +``` + +**Cache grows indefinitely.** Periodically clear to prevent bloat: +```bash +# Monthly cache reset (add to crontab) +0 0 1 * * rm /home/gremlin/.gremlin-email-cache.json +``` + +--- + +## Work Email Export Process + +**Daily Workflow:** + +1. **Export from Outlook:** + - Select "Daily Executive Briefing" email + - Save as `.md` file + +2. **Upload to Nextcloud:** + - Navigate to `/Briefings/` + - Upload/replace `work-briefing.md` + +3. **Script automatically fetches** on next run + +**Expected MD Format:** +```markdown +## Daily Executive Briefing: April 18 - April 24, 2026 + +### Next Week's Schedule + +| Date | Time | Subject | Location | +|:---|:---|:---|:---| +| Mon, Apr 20 | 08:30 - 09:30 | Meeting Name | Conference Room | + +### Prioritized Action Items + +| Priority | From | Subject | Action / Deadline | +|:---|:---|:---|:---| +| **High** | Boss Name | Task Subject | Action required: Details here | +| Normal | Colleague | Task Subject | Action required: Details here | +``` + +**Script parses:** +- Schedule table → merges into calendar view +- Action items table → displays in **📧 Work Taskings** +- Filters out "Canceled:" meetings + +--- + +## Troubleshooting + +### Script Fails: Google Calendar Auth Error + +**Error:** +``` +Error fetching Google Calendar: invalid_grant +``` + +**Fix:** +1. Verify service account JSON file exists: + ```bash + ls -l /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json + ``` +2. Check calendar sharing in Google Calendar settings +3. Regenerate service account key if needed + +--- + +### Script Fails: Nextcloud Connection Error + +**Error:** +``` +Error fetching work briefing: HTTP 401 +``` + +**Fix:** +1. Verify Nextcloud app password +2. Test WebDAV manually: + ```bash + curl -u graymutt:YOUR_PASSWORD \ + https://cloud.netgrimoire.com/remote.php/dav/files/graymutt/Briefings/work-briefing.md + ``` +3. Regenerate app password if needed + +--- + +### Script Fails: MailCow IMAP Error + +**Error:** +``` +Error fetching emails: Login failed +``` + +**Fix:** +1. Verify MailCow credentials +2. Test IMAP manually: + ```bash + telnet 192.168.5.16 143 + # Type: a001 LOGIN phil@pncharris.com YOUR_PASSWORD + ``` +3. Check MailCow IMAP is enabled for user + +--- + +### Ollama Parsing Returns Null + +**Symptom:** Emails found, but `Successfully parsed 0 emails` + +**Causes:** +1. Ollama not running: + ```bash + curl http://docker4:11434/api/tags + ``` +2. Model not pulled: + ```bash + docker exec -it ollama ollama pull llama3.2:3b + ``` +3. Timeout issues (increase Line 368): + ```python + timeout=30 # Increase to 60 for slower models + ``` + +--- + +### Email Pre-screening Too Aggressive + +**Symptom:** Important emails not appearing in briefing + +**Fix:** Check pre-screening patterns in `should_process_email()` (Lines 297-334) + +**Add more date patterns:** +```python +date_patterns = [ + r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b', + r'\b\d{4}-\d{2}-\d{2}\b', + # Add custom patterns + r'next week', + r'by end of', +] +``` + +**Add more action keywords:** +```python +action_keywords = [ + 'due', 'deadline', 'task', 'todo', + # Add custom keywords + 'please review', + 'need your input', + 'waiting on', +] +``` + +--- + +### HTML Not Rendering in Glance + +**Symptom:** Blank iframe or "403 Forbidden" + +**Fix:** +1. Verify Apache serving file: + ```bash + curl https://www.netgrimoire.com/briefing.html + ``` +2. Check file permissions: + ```bash + ls -l /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html + # Should be: -rw-r--r-- 1 1964 1964 + ``` +3. Verify Apache VirtualHost config: + ```bash + cat /data/nfs/znas/Docker/web/apache/netgrimoire.conf + ``` + +--- + +## Future Enhancements + +### Planned Integrations + +**NetGrimoire Stack Status:** +```python +def fetch_netgrimoire_status(): + """Fetch Uptime Kuma API for service health""" + # Query Uptime Kuma heartbeat API + # Return: {services_up: N, services_down: N, warnings: [...]} +``` + +**Vikunja Tasks:** +```python +def fetch_vikunja_tasks(): + """Fetch tasks from Vikunja API""" + # Query Vikunja API for open tasks + # Merge into TODAY/THIS WEEK/LATER zones +``` + +**Additional Google Calendars:** +- PNC Fish and More +- Personal calendar +- Shared family calendar + +**Gremlin Mood States:** +- Time-based commentary variation +- Workload-based sass level +- Stack health integration + +--- + +## File Locations + +### Script Files +``` +/home/gremlin/ +├── scripts/ +│ └── gremlin-briefing-final.py +├── logs/ +│ └── briefing.log +└── .config/ + └── gcloud/ + └── netgrimoire-calendar-sa.json +``` + +### Data Files +``` +/home/gremlin/ +└── .gremlin-email-cache.json + +/data/nfs/znas/Docker/web/pages/netgrimoire/public_html/ +└── briefing.html +``` + +### Nextcloud Files +``` +/Briefings/ +└── work-briefing.md +``` + +--- + +## Related Documentation + +- [Google Calendar API](https://developers.google.com/calendar/api/guides/overview) +- [Nextcloud WebDAV](https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html) +- [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md) +- [Glance Configuration](https://github.com/glanceapp/glance) + +--- + +## Maintenance + +### Weekly Tasks +- ✅ Review flagged keywords effectiveness +- ✅ Check email cache size +- ✅ Verify Nextcloud work briefing uploads + +### Monthly Tasks +- ✅ Clear email cache +- ✅ Review Ollama parsing accuracy +- ✅ Update work email export process if needed + +### Quarterly Tasks +- ✅ Review Google Calendar sharing permissions +- ✅ Rotate Nextcloud app password +- ✅ Update federal holidays API if needed + +--- + +## Support + +**Logs:** +```bash +# Script execution log +tail -f /home/gremlin/logs/briefing.log + +# Glance container logs +docker service logs glance_glance --tail 50 +``` + +**Manual debugging:** +```bash +# Run script with verbose output +python3 -u /home/gremlin/scripts/gremlin-briefing-final.py + +# Test individual components +python3 -c "from google.oauth2 import service_account; print('Google auth OK')" +``` + +--- + +*Last updated: April 18, 2026* +*Maintained by: Phil Harris (graymutt)* \ No newline at end of file