No description
  • Python 98.2%
  • Dockerfile 0.7%
  • Mako 0.7%
  • Shell 0.4%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
matthew 85a536e49d
All checks were successful
Build and Publish / build_and_publish (push) Successful in 1m15s
Merge pull request 'feat: add truth social scraping' (#1) from Terry/obba:feat/truth-social into main
Reviewed-on: #1
2026-09-06 20:22:13 +01:00
.forgejo/workflows Add CI 2026-08-22 21:32:52 +01:00
alembic Merge schedulebot 2026-08-23 00:38:50 +01:00
maubot_plugin actually upload videos 2026-08-22 23:24:45 +01:00
scripts add stuff 2026-08-22 21:28:43 +01:00
src/obba feat: add truth social scraping 2026-09-06 20:15:40 +01:00
.env.example feat: add truth social scraping 2026-09-06 20:15:40 +01:00
.gitattributes add stuff 2026-08-22 21:28:43 +01:00
.gitignore add stuff 2026-08-22 21:28:43 +01:00
alembic.ini add stuff 2026-08-22 21:28:43 +01:00
docker-compose.yml feat: add truth social scraping 2026-09-06 20:15:40 +01:00
Dockerfile add stuff 2026-08-22 21:28:43 +01:00
pyproject.toml feat: add truth social scraping 2026-09-06 20:15:40 +01:00
README.md feat: add truth social scraping 2026-09-06 20:15:40 +01:00
requirements.txt feat: add truth social scraping 2026-09-06 20:15:40 +01:00

One Big Beautiful API (OBBA)

Archive and search Truth Social, the White House X proxy, and daily presidential schedule. Optionally bridge new posts and schedule starts to Matrix via maubot, and keep image binaries locally.

Features

  • Polls Truth Social first for Trump, then we gapfill from the White House proxy
  • Archives the daily White House schedule (upstream is current-day only; history comes from OBBA)
  • Stores decoded post fields + full raw_json in PostgreSQL
  • Downloads images only (skips videos), max 20 MiB per file
  • Modular Matrix posting through maubot (trump for Truth, whitehouse for X, schedule for event starts)
  • Persists Matrix event id on each posted item
  • Search API: latest N, time ranges, author, text query; schedule today / current / search

Media longevity notes

Measured against live CDN headers (2026-08-22):

Source Host Cache-Control
Truth images static-assets-1.truthsocial.com max-age≈10y, immutable
X images/video pbs.twimg.com / video.twimg.com max-age=7d

OBBA still mirrors images from both so the archive does not depend on upstream retention. Videos are stored as URL metadata only (download_status=skipped_video).

Quick start

cp .env.example .env
docker compose up --build

API: http://localhost:8001
Docs: http://localhost:8001/docs

Local (without Docker app)

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
# start Postgres (compose db only), then:
export DATABASE_URL=postgresql+asyncpg://obba:obba@localhost:5432/obba
alembic upgrade head
uvicorn obba.main:app --reload

Backfill extra pages

obba-backfill both --pages 5
# or: python -m obba.cli both --pages 5

Truth backfill pages Truth Social via max_id. Polls also gap-fill from the White House Truth proxy. Matrix Truth bodies start with T before the URL; each id is sent at most once.

API examples

curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/posts/latest?limit=10'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/posts?source=truth&since=2026-08-01T00:00:00Z'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/posts?q=ballroom&limit=20'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/schedule/today'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/schedule/current'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/schedule/events?q=meeting&since=2026-08-01'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/schedule/2026-08-22'
curl -H "X-API-Token: $API_TOKEN" 'http://localhost:8001/stats'

Schedule notes

  • Upstream /schedule only returns the current Eastern day; OBBA stores each snapshot so /schedule/events can search past days after they have been polled.
  • live means the event is marked as livestreamed by the White House.
  • An event is running when start_at <= now and either before end_at, or within 1 hour after start if there is no end time.
  • Matrix schedule posts fire when an event’s start time is reached (same cold-start seed behavior as schedulebot: already-started events are not re-announced).

Matrix (maubot)

OBBA does not speak Matrix directly. It POSTs to a small maubot plugin shipped in maubot_plugin/.

1. Build and upload the plugin

./scripts/build-maubot-plugin.sh
# Upload dist/obba-poster.mbp in the maubot management UI

2. Create plugin instances

Instance Primary user Purpose
e.g. obba-trump your Trump bot MXID Truth Social posts
e.g. obba-whitehouse your White House bot MXID X / WhiteHouse posts
e.g. obba-schedule your schedule bot MXID Presidential schedule starts

For each instance, set config:

room_id: "!yourRoom:example.com"
auth_token: change-me
path: /post

Copy each instance’s webhook URL (typically
https://<maubot>/_matrix/maubot/plugin/<instance-id>/post).

3. Configure OBBA

MATRIX_ENABLED=true
MATRIX_CHANNEL_ID=!yourRoom:example.com
MAUBOT_AUTH_TOKEN=change-me
MAUBOT_TRUMP_WEBHOOK_URL=https://maubot.example/_matrix/maubot/plugin/obba-trump/post
MAUBOT_WHITEHOUSE_WEBHOOK_URL=https://maubot.example/_matrix/maubot/plugin/obba-whitehouse/post
MAUBOT_SCHEDULE_WEBHOOK_URL=https://maubot.example/_matrix/maubot/plugin/obba-schedule/post

Other X authors are archived but not Matrix-posted unless listed in MATRIX_X_USERNAMES.

Webhook body (sent by OBBA): JSON with body, optional room_id, images (base64 of archived photos), and videos (remote URLs). Maubot fetches videos and uploads them to Matrix; OBBA does not store video binaries permanently. Schedule announces are text-only. Response must include event_id.

Config

Variable Default Meaning
DATABASE_URL compose value Async SQLAlchemy URL
POLL_INTERVAL_MINUTES 10 Poll cadence
MEDIA_ROOT ./media Image storage
MEDIA_MAX_BYTES 20971520 20 MiB cap
MATRIX_ENABLED false Enable maubot posting
MAUBOT_*_WEBHOOK_URL empty Per-bot plugin webhook
MAUBOT_AUTH_TOKEN empty Bearer token for webhooks
WH_API_BASE WH v1 base White House API root (X, schedule, Truth gap-fill)
TRUTH_ACCOUNT_ID 107780257626128497 Truth Social account id to archive (@realDonaldTrump)
API_TOKEN empty If set, require X-API-Token on all routes except /health

Auth

When API_TOKEN is set:

curl -H "X-API-Token: change-me" 'http://localhost:8001/posts/latest?limit=5'

/health stays open for probes. Empty API_TOKEN disables auth.

CI

Forgejo Actions (.forgejo/workflows/build.yaml) mirrors truthbot: build on push/PR/release, publish git.shronk.net/projectslop/obba from main only. Requires secret CR_TOKEN.

GitLab CI (.gitlab-ci.yml) provides the same build/publish behavior (CR_TOKEN, optional CR_USER).

License

See project license if added; inspired by ProjectSlop/truthbot.