diff --git a/Gremlin/Morning_Briefing.md b/Gremlin/Morning_Briefing.md deleted file mode 100644 index b7e69e4..0000000 --- a/Gremlin/Morning_Briefing.md +++ /dev/null @@ -1,639 +0,0 @@ ---- -title: Consolidated Morning Briefing -description: -published: true -date: 2026-04-19T04:27:14.348Z -tags: -editor: markdown -dateCreated: 2026-04-19T04:27:14.348Z ---- - -# Gremlin's Morning Briefing System - -**Complete automated daily briefing integrating calendars, work taskings, and personal email** - ---- - -## Overview - -The Gremlin Morning Briefing is an automated system that generates a unified HTML dashboard displaying: -- **SIL Calendar** (Google Calendar synced to Nextcloud) -- **Work Email Taskings** (exported from Outlook as Markdown) -- **Personal Email Tasks** (parsed from MailCow IMAP via Ollama LLM) -- **Federal Holidays** -- **Future:** NetGrimoire stack status, Vikunja tasks - -The briefing organizes everything into three priority zones: -- 🔥 **TODAY** - Overdue tasks, today's events, high-priority items -- 📆 **THIS WEEK** - Next 7 days planning horizon -- 📬 **LATER** - Days 8-14, backlog items - ---- - -## Architecture - -### Data Sources -1. **Google Calendar (SIL)** → Service Account API -2. **Nextcloud** → Work briefing MD file via WebDAV -3. **MailCow IMAP** → Personal email scanning -4. **Ollama (llama3.2:3b)** → Email parsing for tasks/due dates -5. **Federal Holidays API** → date.nager.at - -### Processing Pipeline -``` -Google Calendar ──┐ -Work Email MD ────┼──> Python Script ──> HTML Briefing ──> Apache ──> Glance iframe -MailCow IMAP ─────┤ ↓ -Federal Holidays ─┘ Nextcloud CalDAV - (SIL + Taskings calendars) -``` - -### Key Optimizations -- **Email Pre-screening:** Regex filters reduce Ollama calls from 285 → ~30 -- **Caching:** Processed email IDs prevent re-processing -- **Zone-based bucketing:** Smart categorization by urgency and date - ---- - -## Installation - -### Prerequisites - -**System Requirements:** -- Python 3.9+ -- Docker Swarm (for Glance) -- Apache web server -- Nextcloud instance -- MailCow mail server -- Ollama with `llama3.2:3b` model - -**Python Dependencies:** -```bash -pip3 install --break-system-packages \ - google-auth google-auth-oauthlib google-auth-httplib2 \ - google-api-python-client \ - caldav icalendar pytz requests -``` - ---- - -### 1. Google Calendar Setup - -**Create Service Account:** -1. Go to [Google Cloud Console](https://console.cloud.google.com) -2. Create project: `NetGrimoire` -3. Enable **Google Calendar API** -4. Create Service Account: `n8n-calendar-reader@netgrimoire.iam.gserviceaccount.com` -5. Generate JSON key → save to `/home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json` -6. Set permissions: `chmod 600 /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json` - -**Share Calendar:** -1. Open SIL calendar in Google Calendar -2. Settings → Share with specific people -3. Add service account email (read-only) - ---- - -### 2. Nextcloud Setup - -**Create App Password:** -1. Login to Nextcloud: `https://cloud.netgrimoire.com` -2. Settings → Security → Devices & sessions -3. Create new app password: `Gremlin Briefing Script` -4. Save password for script configuration - -**Create Folders:** -``` -/Briefings/ - └── work-briefing.md -``` - -**Upload Work Briefing:** -- Export daily work tasking email as Markdown -- Upload to `/Briefings/work-briefing.md` (overwrite daily) - ---- - -### 3. Script Installation - -**Deploy Script:** -```bash -# Create directory structure -mkdir -p /home/gremlin/scripts -mkdir -p /home/gremlin/.config/gcloud -mkdir -p /home/gremlin/logs - -# Copy script -cp gremlin-briefing-final.py /home/gremlin/scripts/ -chmod 600 /home/gremlin/scripts/gremlin-briefing-final.py -chmod +x /home/gremlin/scripts/gremlin-briefing-final.py - -# Edit configuration -nano /home/gremlin/scripts/gremlin-briefing-final.py -``` - -**Configuration (lines 24-48):** -```python -# Nextcloud -NEXTCLOUD_PASSWORD = 'YOUR_APP_PASSWORD_HERE' - -# MailCow IMAP -MAILCOW_PASSWORD = 'YOUR_MAILCOW_PASSWORD_HERE' - -# Email Keywords (flagged for attention) -EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy'] - -# VIP Senders (always surface in TODAY) -EMAIL_VIP_SENDERS = ['cindy'] -``` - -**Test Run:** -```bash -/home/gremlin/scripts/gremlin-briefing-final.py -``` - -Expected output: -``` -============================================================ -Gremlin Briefing Generator - Optimized -============================================================ -Fetching Google Calendar events... - Found 15 SIL events -Fetching work briefing from Nextcloud... - Found work briefing (12453 chars) - Parsed 18 work events, 24 action items -Connecting to MailCow IMAP... - Found 285 emails in last 14 days - Fetched 8 new emails -Pre-screening 8 emails... - ✓ 3 emails need processing (filtered 5 junk) -Processing 3 emails with Ollama... - Parsing: Need to get Harvey a bath on april 19th... - ✓ Successfully parsed 3 emails -Syncing SIL calendar to Nextcloud... - ✓ SIL sync complete -Syncing email tasks to Nextcloud... - ✓ Added 2 email tasks to calendar -Fetching federal holidays... - Found 11 federal holidays -Organizing items into zones... -Generating HTML briefing... -Writing to /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html... -============================================================ -✓ SUCCESS! - TODAY: 3 events, 2 tasks, 1 flagged, 0 VIP - THIS WEEK: 5 days, 1 tasks - LATER: 2 days, 0 backlog -============================================================ -``` - ---- - -### 4. Cron Job Setup - -**Daily Execution (6:00 AM):** -```bash -crontab -e -``` - -Add: -```cron -0 6 * * * /usr/bin/python3 /home/gremlin/scripts/gremlin-briefing-final.py >> /home/gremlin/logs/briefing.log 2>&1 -``` - -**Manual Trigger:** -```bash -# Run now -/home/gremlin/scripts/gremlin-briefing-final.py - -# Clear email cache (force re-process all emails) -rm /home/gremlin/.gremlin-email-cache.json -``` - ---- - -### 5. Glance Integration - -**Update Glance Configuration:** -```bash -nano /data/nfs/znas/Docker/glance/glance.yml -``` - -**Add iframe widget:** -```yaml -- type: iframe - title: "☠️ Gremlin's Morning Briefing" - source: "https://www.netgrimoire.com/briefing.html" - height: 1200 -``` - -**Restart Glance:** -```bash -docker service update --force glance_glance -``` - -**Access:** -- Direct: `https://www.netgrimoire.com/briefing.html` -- Via Glance: `https://home.netgrimoire.com` - ---- - -## Configuration Guide - -### Adding Email Keywords - -Email keywords flag specific topics for your attention. Emails matching these keywords appear in the **🚨 Flagged Emails** section of TODAY. - -**Edit Script:** -```bash -nano /home/gremlin/scripts/gremlin-briefing-final.py -``` - -**Modify Line 45:** -```python -EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy'] -``` - -**Add keywords** (lowercase, no spaces): -```python -EMAIL_FLAG_KEYWORDS = [ - 'mxroute', # Hosting provider mentions - 'travel', # Travel-related emails - 'dts', # Defense Travel System - 'cititravel', # Travel booking system - '.mil', # Military emails - 'cindy', # From/mentions Cindy - 'vikunja', # Task management mentions - 'backup', # Backup-related alerts - 'downtime', # Service outage mentions -] -``` - -**Keywords are case-insensitive** and match anywhere in subject or first 500 chars of body. - ---- - -### Adding VIP Senders - -VIP senders always appear in TODAY's **💕 VIP** section, regardless of content. - -**Edit Line 46:** -```python -EMAIL_VIP_SENDERS = ['cindy'] -``` - -**Add multiple VIPs:** -```python -EMAIL_VIP_SENDERS = [ - 'cindy', # Wife - 'boss@company', # Boss's email - 'urgent@', # Any address containing "urgent" -] -``` - -**Matching is case-insensitive substring match** on the `From:` field. - ---- - -### Adjusting Email Search Window - -**Default:** 14 days (last 2 weeks) - -**Edit Line 44:** -```python -EMAIL_SEARCH_DAYS = 14 -``` - -**Change to 7 days:** -```python -EMAIL_SEARCH_DAYS = 7 -``` - -**Warning:** Larger windows = more emails to process. Pre-screening mitigates this, but 30+ days may be slow. - ---- - -### Customizing Ollama Model - -**Default:** `llama3.2:3b` (fast, good quality for email parsing) - -**Edit Line 49:** -```python -OLLAMA_MODEL = 'llama3.2:3b' -``` - -**Alternative models:** -```python -OLLAMA_MODEL = 'qwen2.5-coder:7b' # Better date extraction, slower -OLLAMA_MODEL = 'llama3.1:8b' # More accurate, much slower -``` - -**Test model availability:** -```bash -curl http://docker4:11434/api/tags -``` - ---- - -## Email Cache Management - -The script tracks processed emails in `/home/gremlin/.gremlin-email-cache.json` to avoid re-processing. - -**View cache:** -```bash -cat /home/gremlin/.gremlin-email-cache.json -``` - -**Clear cache (force re-process all emails):** -```bash -rm /home/gremlin/.gremlin-email-cache.json -``` - -**Cache grows indefinitely.** Periodically clear to prevent bloat: -```bash -# Monthly cache reset (add to crontab) -0 0 1 * * rm /home/gremlin/.gremlin-email-cache.json -``` - ---- - -## Work Email Export Process - -**Daily Workflow:** - -1. **Export from Outlook:** - - Select "Daily Executive Briefing" email - - Save as `.md` file - -2. **Upload to Nextcloud:** - - Navigate to `/Briefings/` - - Upload/replace `work-briefing.md` - -3. **Script automatically fetches** on next run - -**Expected MD Format:** -```markdown -## Daily Executive Briefing: April 18 - April 24, 2026 - -### Next Week's Schedule - -| Date | Time | Subject | Location | -|:---|:---|:---|:---| -| Mon, Apr 20 | 08:30 - 09:30 | Meeting Name | Conference Room | - -### Prioritized Action Items - -| Priority | From | Subject | Action / Deadline | -|:---|:---|:---|:---| -| **High** | Boss Name | Task Subject | Action required: Details here | -| Normal | Colleague | Task Subject | Action required: Details here | -``` - -**Script parses:** -- Schedule table → merges into calendar view -- Action items table → displays in **📧 Work Taskings** -- Filters out "Canceled:" meetings - ---- - -## Troubleshooting - -### Script Fails: Google Calendar Auth Error - -**Error:** -``` -Error fetching Google Calendar: invalid_grant -``` - -**Fix:** -1. Verify service account JSON file exists: - ```bash - ls -l /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json - ``` -2. Check calendar sharing in Google Calendar settings -3. Regenerate service account key if needed - ---- - -### Script Fails: Nextcloud Connection Error - -**Error:** -``` -Error fetching work briefing: HTTP 401 -``` - -**Fix:** -1. Verify Nextcloud app password -2. Test WebDAV manually: - ```bash - curl -u graymutt:YOUR_PASSWORD \ - https://cloud.netgrimoire.com/remote.php/dav/files/graymutt/Briefings/work-briefing.md - ``` -3. Regenerate app password if needed - ---- - -### Script Fails: MailCow IMAP Error - -**Error:** -``` -Error fetching emails: Login failed -``` - -**Fix:** -1. Verify MailCow credentials -2. Test IMAP manually: - ```bash - telnet 192.168.5.16 143 - # Type: a001 LOGIN phil@pncharris.com YOUR_PASSWORD - ``` -3. Check MailCow IMAP is enabled for user - ---- - -### Ollama Parsing Returns Null - -**Symptom:** Emails found, but `Successfully parsed 0 emails` - -**Causes:** -1. Ollama not running: - ```bash - curl http://docker4:11434/api/tags - ``` -2. Model not pulled: - ```bash - docker exec -it ollama ollama pull llama3.2:3b - ``` -3. Timeout issues (increase Line 368): - ```python - timeout=30 # Increase to 60 for slower models - ``` - ---- - -### Email Pre-screening Too Aggressive - -**Symptom:** Important emails not appearing in briefing - -**Fix:** Check pre-screening patterns in `should_process_email()` (Lines 297-334) - -**Add more date patterns:** -```python -date_patterns = [ - r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b', - r'\b\d{4}-\d{2}-\d{2}\b', - # Add custom patterns - r'next week', - r'by end of', -] -``` - -**Add more action keywords:** -```python -action_keywords = [ - 'due', 'deadline', 'task', 'todo', - # Add custom keywords - 'please review', - 'need your input', - 'waiting on', -] -``` - ---- - -### HTML Not Rendering in Glance - -**Symptom:** Blank iframe or "403 Forbidden" - -**Fix:** -1. Verify Apache serving file: - ```bash - curl https://www.netgrimoire.com/briefing.html - ``` -2. Check file permissions: - ```bash - ls -l /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html - # Should be: -rw-r--r-- 1 1964 1964 - ``` -3. Verify Apache VirtualHost config: - ```bash - cat /data/nfs/znas/Docker/web/apache/netgrimoire.conf - ``` - ---- - -## Future Enhancements - -### Planned Integrations - -**NetGrimoire Stack Status:** -```python -def fetch_netgrimoire_status(): - """Fetch Uptime Kuma API for service health""" - # Query Uptime Kuma heartbeat API - # Return: {services_up: N, services_down: N, warnings: [...]} -``` - -**Vikunja Tasks:** -```python -def fetch_vikunja_tasks(): - """Fetch tasks from Vikunja API""" - # Query Vikunja API for open tasks - # Merge into TODAY/THIS WEEK/LATER zones -``` - -**Additional Google Calendars:** -- PNC Fish and More -- Personal calendar -- Shared family calendar - -**Gremlin Mood States:** -- Time-based commentary variation -- Workload-based sass level -- Stack health integration - ---- - -## File Locations - -### Script Files -``` -/home/gremlin/ -├── scripts/ -│ └── gremlin-briefing-final.py -├── logs/ -│ └── briefing.log -└── .config/ - └── gcloud/ - └── netgrimoire-calendar-sa.json -``` - -### Data Files -``` -/home/gremlin/ -└── .gremlin-email-cache.json - -/data/nfs/znas/Docker/web/pages/netgrimoire/public_html/ -└── briefing.html -``` - -### Nextcloud Files -``` -/Briefings/ -└── work-briefing.md -``` - ---- - -## Related Documentation - -- [Google Calendar API](https://developers.google.com/calendar/api/guides/overview) -- [Nextcloud WebDAV](https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html) -- [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md) -- [Glance Configuration](https://github.com/glanceapp/glance) - ---- - -## Maintenance - -### Weekly Tasks -- ✅ Review flagged keywords effectiveness -- ✅ Check email cache size -- ✅ Verify Nextcloud work briefing uploads - -### Monthly Tasks -- ✅ Clear email cache -- ✅ Review Ollama parsing accuracy -- ✅ Update work email export process if needed - -### Quarterly Tasks -- ✅ Review Google Calendar sharing permissions -- ✅ Rotate Nextcloud app password -- ✅ Update federal holidays API if needed - ---- - -## Support - -**Logs:** -```bash -# Script execution log -tail -f /home/gremlin/logs/briefing.log - -# Glance container logs -docker service logs glance_glance --tail 50 -``` - -**Manual debugging:** -```bash -# Run script with verbose output -python3 -u /home/gremlin/scripts/gremlin-briefing-final.py - -# Test individual components -python3 -c "from google.oauth2 import service_account; print('Google auth OK')" -``` - ---- - -*Last updated: April 18, 2026* -*Maintained by: Phil Harris (graymutt)* \ No newline at end of file diff --git a/home.md b/home.md index 0ace481..2f0cee7 100644 --- a/home.md +++ b/home.md @@ -2,10 +2,10 @@ title: The Grimoire description: Personal knowledge base — graymutt published: true -date: 2026-04-19T04:25:33.085Z -tags: +date: 2026-04-12T00:00:00.000Z +tags: editor: markdown -dateCreated: 2026-01-21T13:19:48.685Z +dateCreated: 2026-04-12T00:00:00.000Z --- @@ -73,8 +73,7 @@ wiki/ │ ├── Gremlin-Grimoire/ ← AI stack │ ├── Shadow-Grimoire/ ← acquisition │ ├── Green-Grimoire/ ← adult media -│ ├── Pocket-Grimoire/ ← portable lab -| └── Gremlin/ - Local AI and Auutomation +│ └── Pocket-Grimoire/ ← portable lab └── Periphery/ ← non-homelab ├── PNC-Harris/ ← family ├── PNC-Fish/ ← business