Start
Introduction
Almaari's Add Clothes flow had gradually accumulated several separate responsibilities: image background removal, manual AI analysis for metadata extraction, credit accounting, garment persistence, and background metadata enrichment.
The system worked, but the workflow increasingly depended on the user advancing each step manually, while failures across processing services were difficult to recover from cleanly.
I began redesigning the ingestion pipeline around one durable workflow with explicit processing state, safe retries, and automatic garment analysis. The redesign is still being productionized; proposed components and design targets are identified below.
Overview
What I Changed
The central change is to treat image processing, analysis, and enrichment as one durable workflow centered on the Clothes record. Uploading an image becomes one asynchronous task with safer retries and idempotency.
Before
After
The flow
The Current Flow
The current upload experience asks the user to prepare the image, request analysis, and review the result as separate steps.
Step 1 · 1 of 4
Prepare the garment image
The user selects an image of their garment.
- 1The frontend resizes and compresses the image before sending it to the server as a Base64-encoded payload.
Current system
The Problem
The current pipeline has user-facing friction, but the deeper issue is that no single layer clearly owns the garment's progress through ingestion.
User-driven orchestration
AI analysis currently requires another explicit user action. The user is responsible for advancing a workflow that the system can already infer from the upload itself.
Billing coupled to ingestion
A standard garment analysis can involve a credit check, credit deduction, the AI request, and a refund when that request fails. That accounting logic makes a basic product workflow harder to retry safely.
Overlapping retry mechanisms
Reliability is spread across frontend submission guards, idempotency keys, MongoDB jobs, setImmediate processing, startup reclaim, and internal worker endpoints. Each mechanism solves a real problem, but together they obscure which layer owns recovery.
Partial failures
Background removal and AI analysis can succeed or fail independently. A successful crop should not need to run again because analysis failed later, but the current workflow does not represent that progress as one explicit checkpointed sequence.
Fragmented workflow ownership
Upload, crop, analysis, persistence, enrichment, and review are not represented by one durable state machine. The redesign moves that responsibility into persisted Clothes state.
Scope
Design Goals & Non-Goals
The redesign has three priorities:
- Cleaner architecture. Represent ingestion as one explicit workflow with understandable state transitions.
- Better reliability. Use durable task delivery and checkpointed processing so retries resume from completed work.
- Lower user-perceived latency. Start analysis automatically after upload instead of waiting for another user action.
Non-goals
- Rewrite every service.
- Replace MongoDB.
- Move every workload to GCP.
- Introduce Kafka or Pub/Sub.
- Optimize for millions of uploads.
The goal is to improve the reliability boundary without turning Almaari into a distributed system larger than its current scale requires.
Interactive system map
Explore the Architecture
The system map shows the proposed architecture, current implementation, successful path, and failure and retry paths.
The proposed workflow makes persisted Clothes state—not a browser or process—the orchestration boundary.
Workflow ownership
Current vs Proposed
The main change is where workflow ownership lives. Today, the frontend and several background mechanisms collectively move a garment through the system. In the redesign, the Clothes record becomes the durable source of truth and a worker advances it through explicit stages.
The redesign moves workflow ownership from the frontend and process lifecycle into persisted backend state.
Persisted orchestration
The Clothes Record Becomes the Workflow
A separate GarmentDraft model would add another lifecycle to create, reconcile, and eventually delete. Because the S3 flow already creates an owned Clothes shell, I decided to use the Clothes record itself as the workflow entity.
type ClothesStatus =
| "PROCESSING"
| "READY_FOR_REVIEW"
| "FAILED"
| "REPROCESSING"
| "ACTIVE";
type ProcessingStage =
| "UPLOADED"
| "CROPPING"
| "CROPPED"
| "ANALYZING"
| "READY_FOR_REVIEW"
| "FAILED";Proposed sequence
Request Lifecycle
The user-facing request ends early. Persisted state and durable task delivery carry the remaining processing forward without tying it to the API process lifetime.
1. Create and upload
The user selects an image. Express creates a pending Clothes record and returns a presigned S3 URL. The browser uploads the image directly to S3, then confirms completion with the API.
2. Queue processing
After confirming the upload, Express enqueues PROCESS_CLOTHING:{clothesId}. The ingestion worker receives the task and loads the persisted Clothes state before doing any work.
3. Process and checkpoint
The worker calls the crop service and saves processingStage = CROPPED when it succeeds. It then calls the vision service, persists the returned metadata, and advances the Clothes record to READY_FOR_REVIEW.
4. Review and finalize
The frontend refetches the Clothes record and unlocks editing. The user reviews the metadata, makes any changes, and submits. Finalization advances the garment to ACTIVE.
At-least-once delivery
Retries Are Expected, Not Exceptional
Cloud Tasks provides at-least-once delivery. The worker cannot assume that a task runs only once; duplicate delivery is normal control flow, not an unusual edge case.
Each workflow uses PROCESS_CLOTHING:{clothesId} as its stable identity. Before doing work, the worker reads the Clothes record and decides what remains from the persisted processing stage.
if (clothes.processingStage === "CROPPED") {
await analyzeGarment(clothes);
}Request-level idempotency
Protect repeated HTTP create attempts with an Idempotency-Key.
Workflow-level idempotency
Use clothesId + processingStage to determine which work remains.
Finalization idempotency
If the status is already ACTIVE, return the existing Clothes object instead of finalizing it again.
Degraded paths
Failure Recovery
The workflow preserves successful work and keeps a user path available when a dependency remains unavailable.
Crop service outage
A transient crop failure is retried by Cloud Tasks. If retries are exhausted, the garment moves to REPROCESSING and the user can continue with the original image. A delayed recrop can run later without blocking review.
AI service outage
Once cropping succeeds, the CROPPED checkpoint is retained. An AI retry skips cropping and resumes analysis directly. If analysis continues to fail, the user can enter metadata manually.
Browser refresh
The frontend refetches Clothes state and restores the current step. It does not create another workflow or restart completed processing.
Duplicate task delivery
The worker reads the persisted stage and skips completed work. Stage transitions and finalization are designed to be idempotent.
Delayed image reprocessing
If cropping continues to fail, the user should still be able to proceed with the original image. A later crop result may replace a pending image, but it must not silently replace an image the user has already approved.
if (clothes.status !== "ACTIVE") {
// processed candidate may replace pending image
}A future interface could offer the improved image as an explicit replacement.
Architecture decision records
Key Architecture Decisions
These decisions keep the reliability boundary explicit without adding infrastructure that Almaari's current scale does not need.
ADR-01
Google Cloud Tasks
Decision: Use Cloud Tasks for durable async delivery.
Why: Managed delivery and retries fit the current scale.
Trade-off: At-least-once delivery still requires idempotency.
ADR-02
Dedicated Cloud Run Worker
Decision: Separate ingestion from the public Express service.
Why: Independent scaling, resource isolation, and a cleaner failure boundary.
Trade-off: Adds one deployable service.
ADR-03
Reuse Clothes
Decision: Do not introduce a separate GarmentDraft.
Why: The S3 flow already creates an owned Clothes shell.
Trade-off: Clothes carries temporary processing state.
ADR-04
Single Checkpointed Task
Decision: PROCESS_CLOTHING handles crop and AI.
Why: Current scale does not justify a task per stage.
Trade-off: The worker owns stage orchestration.
ADR-05
Keep Railway
Decision: Keep crop and AI services on Railway for now.
Why: Migration does not directly solve the reliability boundary.
Trade-off: Revisit if latency, reliability, observability, or overhead becomes measurable.
ADR-06
Automatic Free Analysis
Decision: Standard garment analysis no longer consumes credits.
Why: Removes billing logic from normal ingestion and simplifies retries.
Trade-off: Each upload now has a model inference cost.
ADR-07
Sequential Crop → Analyze
Decision: Crop first, then analyze.
Why: AI receives a cleaner image and the pipeline stays easy to reason about.
Trade-off: Potentially higher latency than parallel analysis.
Deliberate constraints
Rejected & Deferred Alternatives
MongoDB as the primary queue
It reuses an existing datastore and resembles Almaari's current job handling.
Why I deferred it: Polling, delivery semantics, and abandoned-job reclaim would remain application-owned.
Trade-off: It avoids another provider but creates more queue infrastructure to maintain.
Processing inside Express
It avoids another service and is straightforward to deploy.
Why I deferred it: Long-running work would remain coupled to public request capacity and the API process lifecycle.
Trade-off: There are fewer deployables, but a weaker failure and scaling boundary.
One task per processing stage
Separate crop and analysis tasks would provide granular isolation.
Why I deferred it: The current upload volume does not justify the extra task definitions and orchestration.
Trade-off: Retries become more granular at the cost of a more complex workflow.
Move every service to GCP immediately
One provider could simplify networking and observability.
Why I deferred it: A migration does not directly solve workflow ownership or retry behavior.
Trade-off: Operations may become simpler later, but the near-term migration cost is high.
Keep manual AI analysis
It avoids inference for uploads the user may abandon.
Why I deferred it: It preserves a second user action, billing coupling, and a delayed processing start.
Trade-off: Model usage is lower, but user effort and perceived latency remain higher.
Introduce a separate GarmentDraft
A draft entity could isolate temporary ingestion state.
Why I deferred it: The existing S3 flow already creates an owned Clothes shell.
Trade-off: The conceptual boundary is cleaner, but another lifecycle must be reconciled and deleted.
Right-sized infrastructure
Cost & Scale
Almaari currently has more than 50 users and sees roughly 20–30 garment uploads per day. At that scale, the priority is operational clarity and failure recovery rather than maximum throughput.
Cloud Tasks volume is small, the worker can scale down while idle, and there is little reason to migrate functioning Railway services solely for architectural neatness. AI inference is more likely to dominate variable cost than task delivery.
Evidence
Metrics
Current scale
- More than 50 users.
- Roughly 20–30 garment uploads per day.
Design targets
- Under five seconds for normal garment processing.
- Safe duplicate task delivery.
- Partial retry from persisted checkpoints.
- Manual fallback after AI failure.
- No credit dependency for standard garment analysis.
What I plan to measure after rollout
- P50 and P95 processing latency.
- Crop and AI latency.
- Queue delay and retry rate.
- Crop and AI failure rates.
Implementation
Productionization Status
The redesign remains in progress. The lists below distinguish production behavior from planned work.
Already in place
- S3 production image path
- Clothes shell creation
- Railway crop service
- Railway vision service
In progress
- New processing state machine
- Cloud Tasks integration
- Automatic free analysis UX
Still to validate or implement
- Dedicated Cloud Run ingestion worker
- Checkpointed retry
- Latency benchmarking