Administrator 8e5b76103a docs: create Gremlin/Morning_Briefing

2026-04-19 04:27:24 +00:00

14 KiB

Raw Blame History

title	description	published	date	tags	editor	dateCreated
Consolidated Morning Briefing		true	2026-04-19T04:27:14.348Z		markdown	2026-04-19T04:27:14.348Z

Gremlin's Morning Briefing System

Complete automated daily briefing integrating calendars, work taskings, and personal email

Overview

The Gremlin Morning Briefing is an automated system that generates a unified HTML dashboard displaying:

SIL Calendar (Google Calendar synced to Nextcloud)
Work Email Taskings (exported from Outlook as Markdown)
Personal Email Tasks (parsed from MailCow IMAP via Ollama LLM)
Federal Holidays
Future: NetGrimoire stack status, Vikunja tasks

The briefing organizes everything into three priority zones:

🔥 TODAY - Overdue tasks, today's events, high-priority items
📆 THIS WEEK - Next 7 days planning horizon
📬 LATER - Days 8-14, backlog items

Architecture

Data Sources

Google Calendar (SIL) → Service Account API
Nextcloud → Work briefing MD file via WebDAV
MailCow IMAP → Personal email scanning
Ollama (llama3.2:3b) → Email parsing for tasks/due dates
Federal Holidays API → date.nager.at

Processing Pipeline

Google Calendar ──┐
Work Email MD ────┼──> Python Script ──> HTML Briefing ──> Apache ──> Glance iframe
MailCow IMAP ─────┤         ↓
Federal Holidays ─┘    Nextcloud CalDAV
                       (SIL + Taskings calendars)

Key Optimizations

Email Pre-screening: Regex filters reduce Ollama calls from 285 → ~30
Caching: Processed email IDs prevent re-processing
Zone-based bucketing: Smart categorization by urgency and date

Installation

Prerequisites

System Requirements:

Python 3.9+
Docker Swarm (for Glance)
Apache web server
Nextcloud instance
MailCow mail server
Ollama with llama3.2:3b model

Python Dependencies:

pip3 install --break-system-packages \
  google-auth google-auth-oauthlib google-auth-httplib2 \
  google-api-python-client \
  caldav icalendar pytz requests

1. Google Calendar Setup

Create Service Account:

Go to Google Cloud Console
Create project: NetGrimoire
Enable Google Calendar API
Create Service Account: n8n-calendar-reader@netgrimoire.iam.gserviceaccount.com
Generate JSON key → save to /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json
Set permissions: chmod 600 /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json

Share Calendar:

Open SIL calendar in Google Calendar
Settings → Share with specific people
Add service account email (read-only)

2. Nextcloud Setup

Create App Password:

Login to Nextcloud: https://cloud.netgrimoire.com
Settings → Security → Devices & sessions
Create new app password: Gremlin Briefing Script
Save password for script configuration

Create Folders:

/Briefings/
  └── work-briefing.md

Upload Work Briefing:

Export daily work tasking email as Markdown
Upload to /Briefings/work-briefing.md (overwrite daily)

3. Script Installation

Deploy Script:

# Create directory structure
mkdir -p /home/gremlin/scripts
mkdir -p /home/gremlin/.config/gcloud
mkdir -p /home/gremlin/logs

# Copy script
cp gremlin-briefing-final.py /home/gremlin/scripts/
chmod 600 /home/gremlin/scripts/gremlin-briefing-final.py
chmod +x /home/gremlin/scripts/gremlin-briefing-final.py

# Edit configuration
nano /home/gremlin/scripts/gremlin-briefing-final.py

Configuration (lines 24-48):

# Nextcloud
NEXTCLOUD_PASSWORD = 'YOUR_APP_PASSWORD_HERE'

# MailCow IMAP
MAILCOW_PASSWORD = 'YOUR_MAILCOW_PASSWORD_HERE'

# Email Keywords (flagged for attention)
EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy']

# VIP Senders (always surface in TODAY)
EMAIL_VIP_SENDERS = ['cindy']

Test Run:

/home/gremlin/scripts/gremlin-briefing-final.py

Expected output:

============================================================
Gremlin Briefing Generator - Optimized
============================================================
Fetching Google Calendar events...
  Found 15 SIL events
Fetching work briefing from Nextcloud...
  Found work briefing (12453 chars)
  Parsed 18 work events, 24 action items
Connecting to MailCow IMAP...
  Found 285 emails in last 14 days
  Fetched 8 new emails
Pre-screening 8 emails...
  ✓ 3 emails need processing (filtered 5 junk)
Processing 3 emails with Ollama...
  Parsing: Need to get Harvey a bath on april 19th...
  ✓ Successfully parsed 3 emails
Syncing SIL calendar to Nextcloud...
  ✓ SIL sync complete
Syncing email tasks to Nextcloud...
  ✓ Added 2 email tasks to calendar
Fetching federal holidays...
  Found 11 federal holidays
Organizing items into zones...
Generating HTML briefing...
Writing to /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html...
============================================================
✓ SUCCESS!
  TODAY: 3 events, 2 tasks, 1 flagged, 0 VIP
  THIS WEEK: 5 days, 1 tasks
  LATER: 2 days, 0 backlog
============================================================

4. Cron Job Setup

Daily Execution (6:00 AM):

crontab -e

Add:

0 6 * * * /usr/bin/python3 /home/gremlin/scripts/gremlin-briefing-final.py >> /home/gremlin/logs/briefing.log 2>&1

Manual Trigger:

# Run now
/home/gremlin/scripts/gremlin-briefing-final.py

# Clear email cache (force re-process all emails)
rm /home/gremlin/.gremlin-email-cache.json

5. Glance Integration

Update Glance Configuration:

nano /data/nfs/znas/Docker/glance/glance.yml

Add iframe widget:

- type: iframe
  title: "☠️ Gremlin's Morning Briefing"
  source: "https://www.netgrimoire.com/briefing.html"
  height: 1200

Restart Glance:

docker service update --force glance_glance

Access:

Direct: https://www.netgrimoire.com/briefing.html
Via Glance: https://home.netgrimoire.com

Configuration Guide

Adding Email Keywords

Email keywords flag specific topics for your attention. Emails matching these keywords appear in the 🚨 Flagged Emails section of TODAY.

Edit Script:

nano /home/gremlin/scripts/gremlin-briefing-final.py

Modify Line 45:

EMAIL_FLAG_KEYWORDS = ['mxroute', 'travel', 'dts', 'cititravel', '.mil', 'cindy']

Add keywords (lowercase, no spaces):

EMAIL_FLAG_KEYWORDS = [
    'mxroute',      # Hosting provider mentions
    'travel',       # Travel-related emails
    'dts',          # Defense Travel System
    'cititravel',   # Travel booking system
    '.mil',         # Military emails
    'cindy',        # From/mentions Cindy
    'vikunja',      # Task management mentions
    'backup',       # Backup-related alerts
    'downtime',     # Service outage mentions
]

Keywords are case-insensitive and match anywhere in subject or first 500 chars of body.

Adding VIP Senders

VIP senders always appear in TODAY's 💕 VIP section, regardless of content.

Edit Line 46:

EMAIL_VIP_SENDERS = ['cindy']

Add multiple VIPs:

EMAIL_VIP_SENDERS = [
    'cindy',           # Wife
    'boss@company',    # Boss's email
    'urgent@',         # Any address containing "urgent"
]

Matching is case-insensitive substring match on the From: field.

Adjusting Email Search Window

Default: 14 days (last 2 weeks)

Edit Line 44:

EMAIL_SEARCH_DAYS = 14

Change to 7 days:

EMAIL_SEARCH_DAYS = 7

Warning: Larger windows = more emails to process. Pre-screening mitigates this, but 30+ days may be slow.

Customizing Ollama Model

Default: llama3.2:3b (fast, good quality for email parsing)

Edit Line 49:

OLLAMA_MODEL = 'llama3.2:3b'

Alternative models:

OLLAMA_MODEL = 'qwen2.5-coder:7b'  # Better date extraction, slower
OLLAMA_MODEL = 'llama3.1:8b'       # More accurate, much slower

Test model availability:

curl http://docker4:11434/api/tags

Email Cache Management

The script tracks processed emails in /home/gremlin/.gremlin-email-cache.json to avoid re-processing.

View cache:

cat /home/gremlin/.gremlin-email-cache.json

Clear cache (force re-process all emails):

rm /home/gremlin/.gremlin-email-cache.json

Cache grows indefinitely. Periodically clear to prevent bloat:

# Monthly cache reset (add to crontab)
0 0 1 * * rm /home/gremlin/.gremlin-email-cache.json

Work Email Export Process

Daily Workflow:

Export from Outlook:
- Select "Daily Executive Briefing" email
- Save as .md file
Upload to Nextcloud:
- Navigate to /Briefings/
- Upload/replace work-briefing.md
Script automatically fetches on next run

Expected MD Format:

## Daily Executive Briefing: April 18 - April 24, 2026

### Next Week's Schedule

| Date | Time | Subject | Location |
|:---|:---|:---|:---|
| Mon, Apr 20 | 08:30 - 09:30 | Meeting Name | Conference Room |

### Prioritized Action Items

| Priority | From | Subject | Action / Deadline |
|:---|:---|:---|:---|
| **High** | Boss Name | Task Subject | Action required: Details here |
| Normal | Colleague | Task Subject | Action required: Details here |

Script parses:

Schedule table → merges into calendar view
Action items table → displays in 📧 Work Taskings
Filters out "Canceled:" meetings

Troubleshooting

Script Fails: Google Calendar Auth Error

Error:

Error fetching Google Calendar: invalid_grant

Fix:

Verify service account JSON file exists:

ls -l /home/gremlin/.config/gcloud/netgrimoire-calendar-sa.json

Check calendar sharing in Google Calendar settings
Regenerate service account key if needed

Script Fails: Nextcloud Connection Error

Error:

Error fetching work briefing: HTTP 401

Fix:

Verify Nextcloud app password

Test WebDAV manually:

curl -u graymutt:YOUR_PASSWORD \
  https://cloud.netgrimoire.com/remote.php/dav/files/graymutt/Briefings/work-briefing.md

Regenerate app password if needed

Script Fails: MailCow IMAP Error

Error:

Error fetching emails: Login failed

Fix:

Verify MailCow credentials

Test IMAP manually:

telnet 192.168.5.16 143
# Type: a001 LOGIN phil@pncharris.com YOUR_PASSWORD

Check MailCow IMAP is enabled for user

Ollama Parsing Returns Null

Symptom: Emails found, but Successfully parsed 0 emails

Causes:

Ollama not running:
```
curl http://docker4:11434/api/tags
```

Model not pulled:

docker exec -it ollama ollama pull llama3.2:3b

Timeout issues (increase Line 368):

timeout=30  # Increase to 60 for slower models

Email Pre-screening Too Aggressive

Symptom: Important emails not appearing in briefing

Fix: Check pre-screening patterns in should_process_email() (Lines 297-334)

Add more date patterns:

date_patterns = [
    r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b',
    r'\b\d{4}-\d{2}-\d{2}\b',
    # Add custom patterns
    r'next week',
    r'by end of',
]

Add more action keywords:

action_keywords = [
    'due', 'deadline', 'task', 'todo',
    # Add custom keywords
    'please review',
    'need your input',
    'waiting on',
]

HTML Not Rendering in Glance

Symptom: Blank iframe or "403 Forbidden"

Fix:

Verify Apache serving file:

curl https://www.netgrimoire.com/briefing.html

Check file permissions:

ls -l /data/nfs/znas/Docker/web/pages/netgrimoire/public_html/briefing.html
# Should be: -rw-r--r-- 1 1964 1964

Verify Apache VirtualHost config:

cat /data/nfs/znas/Docker/web/apache/netgrimoire.conf

Future Enhancements

Planned Integrations

NetGrimoire Stack Status:

def fetch_netgrimoire_status():
    """Fetch Uptime Kuma API for service health"""
    # Query Uptime Kuma heartbeat API
    # Return: {services_up: N, services_down: N, warnings: [...]}

Vikunja Tasks:

def fetch_vikunja_tasks():
    """Fetch tasks from Vikunja API"""
    # Query Vikunja API for open tasks
    # Merge into TODAY/THIS WEEK/LATER zones

Additional Google Calendars:

PNC Fish and More
Personal calendar
Shared family calendar

Gremlin Mood States:

Time-based commentary variation
Workload-based sass level
Stack health integration

File Locations

Script Files

/home/gremlin/
├── scripts/
│   └── gremlin-briefing-final.py
├── logs/
│   └── briefing.log
└── .config/
    └── gcloud/
        └── netgrimoire-calendar-sa.json

Data Files

/home/gremlin/
└── .gremlin-email-cache.json

/data/nfs/znas/Docker/web/pages/netgrimoire/public_html/
└── briefing.html

Nextcloud Files

/Briefings/
└── work-briefing.md

Maintenance

Weekly Tasks

✅ Review flagged keywords effectiveness
✅ Check email cache size
✅ Verify Nextcloud work briefing uploads

Monthly Tasks

✅ Clear email cache
✅ Review Ollama parsing accuracy
✅ Update work email export process if needed

Quarterly Tasks

✅ Review Google Calendar sharing permissions
✅ Rotate Nextcloud app password
✅ Update federal holidays API if needed

Support

Logs:

# Script execution log
tail -f /home/gremlin/logs/briefing.log

# Glance container logs
docker service logs glance_glance --tail 50

Manual debugging:

# Run script with verbose output
python3 -u /home/gremlin/scripts/gremlin-briefing-final.py

# Test individual components
python3 -c "from google.oauth2 import service_account; print('Google auth OK')"

Last updated: April 18, 2026 Maintained by: Phil Harris (graymutt)

14 KiB Raw Blame History

Gremlin's Morning Briefing System

Overview

Architecture

Data Sources

Processing Pipeline

Key Optimizations

Installation

Prerequisites

1. Google Calendar Setup

2. Nextcloud Setup

3. Script Installation

4. Cron Job Setup

5. Glance Integration

Configuration Guide

Adding Email Keywords

Adding VIP Senders

Adjusting Email Search Window

Customizing Ollama Model

Email Cache Management

Work Email Export Process

Troubleshooting

Script Fails: Google Calendar Auth Error

Script Fails: Nextcloud Connection Error

Script Fails: MailCow IMAP Error

Ollama Parsing Returns Null

Email Pre-screening Too Aggressive

HTML Not Rendering in Glance

Future Enhancements

Planned Integrations

File Locations

Script Files

Data Files

Nextcloud Files

Related Documentation

Maintenance

Weekly Tasks

Monthly Tasks

Quarterly Tasks

Support

14 KiB

Raw Blame History