Closes: #521 Closes: #705 Closes: #714

---

• Note the addition of yaml package (reco by e18e https://e18e.dev/guide/replacement-guides/js-yaml.html)
• working with pnpm-workspace.yaml from [email protected] (7 months old). If not fallback to package.json config like before this PR.

Closes: #521 Closes: #705 Closes: #714

---

• Note the addition of yaml package (reco by e18e https://e18e.dev/guide/replacement-guides/js-yaml.html)
• working with pnpm-workspace.yaml from [email protected] (7 months old). If not fallback to package.json config like before this PR.
• Targeting the "new tests branch" to see a smaller diff. #717 would need to be merged first.

pnpm-workspace.yaml

Closes: #521 Closes: #705 Closes: #714

tests

Proposal to change how tests are organized (to speedup tests again ^^)

• variant + kind = flavor AddonTestCase
• setupTest prepare the test mono repo with all flavors and only one install is needed
• remove ps-tree dep

On my machine (windows & wsl), add-ons part is going 30% faster!

Summary

Replaces the spoofable browser-supplied Content-Type allowlist in apps/processor with Google Magika–verified detection on every upload, locks down the serve path so it cannot be coerced into emitting a script-active type, and migrates the processor base image off Alpine so @tensorflow/tfjs-node can actually load in production. This is the processor half of a two-worktree change; a follow-up magika-web-uploads worktree will land the matching changes in apps/web once this merges. No apps/web/** files are touched here.

Why

The current multer.fileFilter in apps/processor/src/api/upload.ts is a prefix allowlist (image/, application/pdf, text/, application/vnd) keyed on whatever the browser claimed. That's trivially bypassable: a .py file arrives as text/plain and gets treated as a generic blob; a renamed Mach-O/ELF/PE binary arrives as image/png and is happily accepted; an HTML payload masquerading as text/plain rides through to the serve path and is later echoed back with whatever the DB row says.

PageSpace is pivoting toward cloud-IDE use cases where source-code uploads are first-class. We need an authoritative content-type signal that's independent of the browser. Google's Magika gives us ~5 ms CPU classification across 200+ labels, including code languages, with a vendored ~3 MB tfjs model.

What landed (commit-by-commit)

feat(processor): add magika-based content-type detector — Tasks A + B

• Pins magika@^1.0.0 in apps/processor/package.json.
• Vendors the standard_v3_3 model (3.1 MB) under apps/processor/assets/magika/standard_v3_3/ so detection runs without network access in dev, CI, and prod.
• New apps/processor/src/services/content-detector.ts:
  • Singleton MagikaNode instance lazily initialised via a memoised Promise<Magika | null> so concurrent first calls share one load.
  • detectContentType(filePath) → DetectedContentType { label, mimeType, group, score, source: 'magika' | 'fallback' }.
  • 250 ms Promise.race timeout ceiling.
  • All error paths return a frozen FALLBACK_DETECTION (application/octet-stream) and log via processorLogger using tempPath only — never originalName (preserves the PII-scrubbing posture from dbc52496).
• New apps/processor/src/services/__tests__/content-detector.test.ts (4 tests, all green): real-fixture classification (png/python/pdf), missing-file → fallback, sequential and concurrent-cold-start singleton reuse via vi.spyOn(MagikaNode, 'create').
• Whitelists @tensorflow/tfjs-node in the root package.json#pnpm.onlyBuiltDependencies so the native tfjs binding builds on fresh pnpm install (pnpm 10 blocks build scripts by default). This is one of the two files touched outside apps/processor/** (the other is the lockfile) — without it, fresh installs leave Magika unable to load and the feature silently degrades.
• Adds a magika/node paths mapping in apps/processor/tsconfig.json so the legacy moduleResolution: "node" setting can find Magika's subpath types.

feat(processor): enforce magika-verified types on upload — Task C

• Drops the allowedTypes prefix check from multer.fileFilter; keeps the empty-filename guard.
• Both /single and /multiple handlers now call detectContentType(tempPath) after the temp file is written and validated against TEMP_UPLOADS_DIR. If the verified label is in DENIED_LABELS = { pebin, elf, macho, dex, html, svg, xhtml }:
  • The temp file is fs.unlinked immediately (same warn-and-continue cleanup pattern used elsewhere).
  • /single returns HTTP 415 { error: 'Unsupported file type', detectedLabel }.
  • /multiple records the rejection in the per-file results[] and removes the path from tempFilePaths so the batch cleanup loop doesn't double-unlink.
• Verified mimeType and the new detectedLabel field are threaded through queueProcessingJobs into IngestFileJobData (extended in apps/processor/src/types/index.ts). queue-manager.ts dispatch is unchanged — it still keys on mimeType.startsWith('image/') / needsTextExtraction(mimeType), but now receives a trustworthy value.
• upload.test.ts: a hoisted mockDetectContentType defaults to image/jpeg so all 28 pre-existing tests pass unchanged. 4 new tests cover 415 rejection + temp cleanup for macho, verified text/x-python flowing into the ingest job, fallback application/octet-stream accepted, and a mixed-batch /multiple where one file is rejected and a sibling is accepted.

feat(processor): harden serve-path content-type handling — Task D

• Targets only the /original route in apps/processor/src/api/serve.ts. The /preset branch is intentionally left alone — those are processor-generated variants whose Content-Type derives from a known suffix.
• Content-Type is sourced strictly from fileRecord.mimeType (with pageRecord.mimeType as a per-page fallback). Whitespace-only stored values are normalised away. There is no caller-controlled fallback path.
• A new forceAttachment = isDangerousMimeType(contentType) || !hasStoredMime flag forces Content-Disposition: attachment plus the strict default-src 'none'; style-src 'unsafe-inline'; img-src data:; sandbox; CSP whenever the stored mime is missing OR matches DANGEROUS_MIME_TYPES (reused from utils/security.ts: text/html, application/xhtml+xml, image/svg+xml, application/xml, text/xml).
• X-Content-Type-Options: nosniff is set on every response (was already set; preserved).
• 5 new tests in serve.test.ts: safe-mime inline, script-active force-download, missing-mime → octet-stream + attachment, whitespace-only mime treated as missing, query-param override ignored.

build(processor): switch base image to node:22 bookworm-slim

• @tensorflow/tfjs-node has no Alpine prebuilds and Alpine isn't an officially supported tfjs platform — the binding requires glibc's ld-linux-x86-64.so.2 (tfjs#1425, tfjs#6556). As long as processor uploads run through Magika, glibc is a hard requirement, so the prod image had to migrate off Alpine.
• Migrated to node:22.17.0-bookworm-slim in both the build and production stages.
• Sharp 0.33+ ships self-contained glibc prebuilds that bundle libvips, so the entire cairo-dev jpeg-dev pango-dev giflib-dev pixman-dev pangomm-dev libjpeg-turbo-dev freetype-dev chain (plus the matching runtime libs in the prod stage) was deleted. Tesseract.js is pure JS + WASM with zero system deps.
• Build stage now installs only ca-certificates curl python3 make g++ pkg-config (the last three are kept so node-pre-gyp can compile from source if a future native dep ships without a glibc prebuild). Production stage installs only ca-certificates.
• Verified end-to-end with docker build --platform linux/amd64 (matches the ubuntu-latest CI runner used by .github/workflows/docker-images.yml) followed by an in-container smoke test:
  • require('sharp') → v0.33.5 loads
  • require('tesseract.js').recognize is a function
  • MagikaNode.create() loads the vendored model from /app/apps/processor/assets/magika/standard_v3_3/
  • libtensorflow announces AVX2 FMA at startup (proves the native tfjs_binding.node actually opened the underlying .so)
  • Magika correctly classifies a Python source as python and an HTML payload as html (which is in DENIED_LABELS, so the upload route would now reject it with HTTP 415)
• Final amd64 image: ~587 MB, smaller than the previous Alpine prod image once you account for the deleted dev-libs chain.

Files touched

File	Notes
apps/processor/Dockerfile	Alpine → bookworm-slim, deleted sharp dev-libs
apps/processor/package.json	+ magika@^1.0.0
apps/processor/src/services/content-detector.ts	new
apps/processor/src/services/__tests__/content-detector.test.ts	new
apps/processor/assets/magika/standard_v3_3/{model.json,config.min.json,group1-shard1of1.bin}	new (3.1 MB vendored model)
apps/processor/src/api/upload.ts	denylist + detector wiring + extended queueProcessingJobs
apps/processor/src/types/index.ts	+ detectedLabel?: string on IngestFileJobData
apps/processor/src/api/serve.ts	strict mime sourcing + forceAttachment
apps/processor/src/api/__tests__/upload.test.ts	mock detector + 4 new tests
apps/processor/src/api/__tests__/serve.test.ts	5 new lockdown tests
apps/processor/tsconfig.json	magika/node paths mapping
package.json (root)	pnpm.onlyBuiltDependencies: ["@tensorflow/tfjs-node"]
pnpm-lock.yaml	updated by pnpm install

Acceptance checklist

• pnpm --filter processor test — 801 / 801 passing (all 36 test files)
• pnpm --filter processor build — clean
• pnpm typecheck (turbo, repo-wide) — clean across all 12 workspaces
• pnpm lint (turbo, repo-wide) — clean
• docker build --platform linux/amd64 -f apps/processor/Dockerfile . — succeeds
• In-container smoke test: sharp + tesseract.js + tfjs-node native binding + Magika classification all working
• No any types introduced
• No originalName ever passed into processorLogger from the new detector code
• No edits to apps/web/**, packages/**, apps/realtime/**, etc.
• Detector singleton reuse, fallback path, denylist rejection, /single + /multiple temp cleanup on rejection, serve-path Content-Type sourcing, serve-path script-active force-download — all covered by tests

Known follow-up (out of scope here)

• The magika-web-uploads worktree will wire the verified mimeType and detectedLabel from IngestFileJobData into pages.mimeType / files.mimeType on the apps/web upload path, and use Magika's code-language labels to route .py, .rs, .ts, etc. to PageType.CODE.

🤖 Generated with Claude Code