Batch Optimization Pipeline — INP/Quality/Throughput 2025

To “optimize a lot of images safely,” you need a pipeline—not one‑off tools. This article details design and ops patterns that preserve INP/UX while delivering predictable quality.

Architecture overview

• Front: drag‑&‑drop, progress, cancel
• Workers: queue‑based processing with controlled parallelism
• Storage: fingerprinted naming for safe replacement

Internal links: Bulk compression tool, Advanced converter

UI that doesn’t hurt INP

• Make uploads cancellable with fetch() + AbortController
• Move heavy work to Web Workers/queues; keep the main thread light
• Show factual counts/time instead of shaky “ETA”

Quality presets

• Thumbnails: mid→high quality WebP/AVIF, prioritize resize
• Photos: prefer AVIF; generate JPEG/WebP for compatibility
• UI/logos: lossless PNG/WebP with palette reduction

Transform matrix (example)

• Input: PNG/JPEG/HEIC/TIFF → Output: AVIF/WebP/JPEG (bit‑depth/ICC per use)
• Sizes: 320/640/960/1200/1600px preset (aligned with sizes)
• Color: explicit P3→sRGB; embed sRGB ICC

Automated tests (pseudo)

// Visual difference guard
expect(butteraugli(original, output)).toBeLessThan(1.2)

Damage checks / structured data

// Verify metadata and content integrity
expect(hasICC(output)).toBe(true)
expect(readingTime(articleMdx)).toBeGreaterThanOrEqual(1)

Pre‑publish checklist

• [ ] LCP candidate images match sizes
• [ ] Alt/attribution integrated with content
• [ ] Fingerprints enable safe cache invalidation

Monitoring and rollback

• Tag jobs with IDs; on failure, fall back to originals
• Visualize CDN logs by format/size; track hit rates
• On degraded vitals (LCP/INP), auto‑rollback to safe presets

Queue design and backpressure

• Priority queues: thumbs/hero/low‑priority
• Backpressure: throttle intake when arrivals exceed capacity; enqueue retries
• Observability: chart queue length, wait time, failure rate

type Job = { id: string; kind: 'thumb'|'hero'|'bulk'; src: string }

Concurrency and resumability

• Derive parallelism from CPU/memory; adjust by resolution
• Chunk processing: commit in 100‑item batches; retry failed chunks only
• Resumability: idempotent via hash fingerprints; resume mid‑stream safely

const pool = new WorkerPool({ size: Math.max(2, os.cpus().length - 1) })

Storage (S3/GCS)

• Origins immutable under original/; public variants under public/
• Minimal metadata (e.g., x-amz-meta-icc: srgb)
• Lifecycle: move old variants by access frequency

Retry/idempotency/expiry

• Retry with exponential backoff + jitter; cap attempts
• Idempotency via src + params hash (store an idempotency-key per request)
• Expiry: GC old variants by key

Cost controls

• Pre‑generate hot sizes; raise cache hits
• Quality/width presets reduce failed attempts
• Under load, lower AVIF effort; re‑generate later for quality

Observability

• Log: jobId, src, fmt, w, q, duration, bytes, error
• KPIs: p50/p95/p99 time, success rate, retry rate, OOM rate
• Alerts: thresholds for p95 and failure rate

Worker (pseudo)

import sharp from 'sharp'

export async function process(job: Job) {
  const { src } = job
  const buf = await fetchBuffer(src)
  return sharp(buf)
    .withMetadata({ icc: 'sRGB.icc' })
    .resize({ width: 1200, withoutEnlargement: true })
    .toFormat('avif', { quality: 60, effort: 4 })
    .toBuffer()
}

Troubleshooting

• OOM: reduce parallelism; stream/tile huge images
• Quality uneven: introduce per‑purpose presets
• Pipeline stalls: prioritize hero queues; delay low priority

FAQ

Q. Should I provision many width steps?

A. A finite set matched to sizes is enough. Over‑granularity fragments cache.

Q. Is higher AVIF effort always better?

A. Practically, 3–6 is the best cost/perf trade‑off; too high hurts throughput.

Summary

Control parallelism, standardize quality presets, and include verification/monitoring/rollback. Start small and iterate with measurements.