Retention for finished messages and completed cards, plus overview pages #191

Merged

robbertbos merged 1 commit from retention-and-overview-pages into main 2026-07-26 20:41:46 +00:00

Conversation 0 Commits 1 Files changed 77 +4174 -265

robbertbos commented 2026-07-25 21:38:38 +00:00

Owner

Copy link

Waggle kept everything forever. This adds automatic cleanup, under one rule:

Waggle never deletes what you added yourself or have not finished.

What is cleaned up, and what never is

Two per-user settings, each off / 30 / 90 / 180 / 365 days, defaulting to 90 for new accounts.

Cleaned up	Kept, always
Sent, cancelled and failed outgoing messages, with their attachments	Drafts
Completed and read cards	Cards carrying a note

A completed card is also spared while it still has a live outgoing message attached, or while it still owes Mattermost an unsave (cleanup_status pending/running/failed). That last guard matters: deleting such a card would leave the message flagged "saved" in Mattermost with nothing left in Waggle to un-flag it.

What disappears is Waggle's own copy. The Mattermost messages stay where they are. Only a message that was never sent is genuinely lost, and the settings copy says so.

Accounts predating the migration start switched off. Otherwise the first nightly sweep would have deleted years of archive before anyone had seen the setting. Those users get a one-time banner offering to switch it on at 90 days.

Three pages for what is kept

Reachable from the menu, rendered with the app's own card components rather than a bespoke list:

• Concepten (/concept-berichten) - a draft lived in no list at all before this, because both scheduled-post endpoints filter drafts out. A half-written reply was unreachable without opening the exact card it hung on.
• Uitgaande post (/uitgaande-post) - everything outgoing in three sections, failures first. Outgoing messages previously lived as sections inside the card lists, and sent messages were visible nowhere.
• Notities (/notities) - the cards that are never cleaned up.

Two defects that were not in the issue

Both found during review, both closed, each pinned by a test that fails against the bug:

1. Cancelled and failed rows shielded their attachments from the 7-day orphan sweep indefinitely. That sweep built its referenced set from select(OutgoingPost.files) with no state filter, so a dead row kept its attachment alive. Nothing cleaned up on cancel or on a failed send; only the manual "Gelezen, verberg" button did.
2. Deleting a completed card took its drafts with it through the FK cascade, including their attachment bytes, because the ref query had no state filter either. Reachable in practice: you can write a reply on a card you completed months ago.

Attachment bytes are released only after the deleting transaction commits, and only through _delete_orphan_refs, because a ref can legitimately be held by a live draft - editing a scheduled post cancels the row and re-hangs its files on the new draft.

Also here

The terms are re-readable on /voorwaarden, with the date they were accepted, from a single shared component so the retention sentence cannot drift between the two places. /welkom drops its connect-Mattermost step: a server is linked from Instellingen, and a new user without one gets a banner pointing there.

Accepted trade-offs

• A user can switch cleanup off entirely, so an operator cannot treat storage limitation as a system guarantee. Stated plainly in docs/deployment.md rather than left implicit.
• Existing accounts get no storage limitation until they act. The alternative was an archive deleted on deploy night.
• No warning before a card is removed. The user ticked the card off themselves, set the period themselves, and everything they added is spared regardless.

Issue #146

Closed with a different solution than it asked for. The issue wanted a maximum age on draft attachments; drafts are kept instead, and answered with a findable list. docs/deployment.md is updated accordingly.

Tested

Backend 1610 tests at 100% branch coverage, on SQLite and Postgres. Frontend 570 unit tests, 63 e2e. The migration's data step - the line that decides whether a deploy destroys data - has its own tests, because alembic/versions/ sits outside the coverage gate.

Separately filed: #205, our universal CSS reset strips the padding from NLDD components that style their host, and MinBZK/storybook#160 as the upstream hardening suggestion.

Waggle kept everything forever. This adds automatic cleanup, under one rule: > **Waggle never deletes what you added yourself or have not finished.** ## What is cleaned up, and what never is Two per-user settings, each off / 30 / 90 / 180 / 365 days, defaulting to 90 for new accounts. | Cleaned up | Kept, always | | --- | --- | | Sent, cancelled and failed outgoing messages, with their attachments | Drafts | | Completed and read cards | Cards carrying a note | A completed card is also spared while it still has a live outgoing message attached, or while it still owes Mattermost an unsave (`cleanup_status` pending/running/failed). That last guard matters: deleting such a card would leave the message flagged "saved" in Mattermost with nothing left in Waggle to un-flag it. What disappears is Waggle's own copy. The Mattermost messages stay where they are. Only a message that was never sent is genuinely lost, and the settings copy says so. **Accounts predating the migration start switched off.** Otherwise the first nightly sweep would have deleted years of archive before anyone had seen the setting. Those users get a one-time banner offering to switch it on at 90 days. ## Three pages for what is kept Reachable from the menu, rendered with the app's own card components rather than a bespoke list: - **Concepten** (`/concept-berichten`) - a draft lived in no list at all before this, because both scheduled-post endpoints filter drafts out. A half-written reply was unreachable without opening the exact card it hung on. - **Uitgaande post** (`/uitgaande-post`) - everything outgoing in three sections, failures first. Outgoing messages previously lived as sections inside the card lists, and sent messages were visible nowhere. - **Notities** (`/notities`) - the cards that are never cleaned up. ## Two defects that were not in the issue Both found during review, both closed, each pinned by a test that fails against the bug: 1. **Cancelled and failed rows shielded their attachments from the 7-day orphan sweep indefinitely.** That sweep built its `referenced` set from `select(OutgoingPost.files)` with no state filter, so a dead row kept its attachment alive. Nothing cleaned up on cancel or on a failed send; only the manual "Gelezen, verberg" button did. 2. **Deleting a completed card took its drafts with it** through the FK cascade, including their attachment bytes, because the ref query had no state filter either. Reachable in practice: you can write a reply on a card you completed months ago. Attachment bytes are released only after the deleting transaction commits, and only through `_delete_orphan_refs`, because a ref can legitimately be held by a live draft - editing a scheduled post cancels the row and re-hangs its files on the new draft. ## Also here The terms are re-readable on `/voorwaarden`, with the date they were accepted, from a single shared component so the retention sentence cannot drift between the two places. `/welkom` drops its connect-Mattermost step: a server is linked from Instellingen, and a new user without one gets a banner pointing there. ## Accepted trade-offs - A user can switch cleanup off entirely, so an operator cannot treat storage limitation as a system guarantee. Stated plainly in `docs/deployment.md` rather than left implicit. - Existing accounts get no storage limitation until they act. The alternative was an archive deleted on deploy night. - No warning before a card is removed. The user ticked the card off themselves, set the period themselves, and everything they added is spared regardless. ## Issue #146 Closed with a different solution than it asked for. The issue wanted a maximum age on draft attachments; drafts are kept instead, and answered with a findable list. `docs/deployment.md` is updated accordingly. ## Tested Backend 1610 tests at 100% branch coverage, on SQLite and Postgres. Frontend 570 unit tests, 63 e2e. The migration's data step - the line that decides whether a deploy destroys data - has its own tests, because `alembic/versions/` sits outside the coverage gate. Separately filed: [#205](https://code.overheid.nl/robbertbos/waggle/issues/205), our universal CSS reset strips the padding from NLDD components that style their host, and [MinBZK/storybook#160](https://github.com/MinBZK/storybook/issues/160) as the upstream hardening suggestion.

robbertbos force-pushed retention-and-overview-pages from 7892758a64 to 2aaad960a8 2026-07-25 21:50:01 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from 2aaad960a8 to f950032cfb 2026-07-26 06:58:46 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from 1839e06718 to b84b34b176 2026-07-26 10:08:32 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from a97ee477fc to b326d83f75 2026-07-26 19:18:09 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from b326d83f75 to 9b94083420 2026-07-26 20:11:10 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from 9b94083420 to 20e73883b1 2026-07-26 20:17:43 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from 20e73883b1 to 54bfffd2ca 2026-07-26 20:24:54 +00:00 Compare

robbertbos force-pushed retention-and-overview-pages from 54bfffd2ca to c4f334409b 2026-07-26 20:28:08 +00:00 Compare

robbertbos merged commit c4f334409b into main 2026-07-26 20:41:46 +00:00

robbertbos deleted branch retention-and-overview-pages 2026-07-26 20:41:46 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

nldd ds

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

nldd ds

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

robbertbos/waggle!191

No description provided.