# Build a Knittable.ai-style AI Knitting Pattern Generator with Cloudflare Workers AI

This guide walks through building a working clone of the core knittable.ai flow: describe a garment in text, or upload a photo of one, and get back a structured knitting pattern (row-by-row instructions, materials, sizing) — all running on Cloudflare Workers AI, no external AI API needed.

---

## 1. What you're building

**Two input paths, one output shape:**

- **Text → Pattern**: user describes a garment ("cropped cable-knit cardigan, worsted weight, top-down") → LLM generates structured pattern JSON.
- **Photo → Pattern**: user uploads a photo → a vision model describes the construction/stitches/shaping → that description is fed into the same text pipeline.

**Stack:**

| Layer | Tool |
|---|---|
| Hosting | Cloudflare Workers (+ Static Assets for the frontend) |
| AI inference | Workers AI (`env.AI` binding) |
| Image storage | R2 |
| User data / credits / saved patterns | D1 (SQLite) |
| PDF export (Studio/Pro tier) | `@react-pdf/renderer` or a Worker-side HTML→PDF step |
| Auth | Clerk, Auth.js, or Cloudflare Access — pick one; not covered in depth here |
| Payments | Stripe (for the paid tiers) |

You do not need a separate backend server — everything above runs inside Cloudflare Workers.

---

## 2. Project setup

```bash
npm create cloudflare@latest knittable-clone -- --type=hello-world --ts
cd knittable-clone
```

Add the Workers AI binding to `wrangler.jsonc`:

```jsonc
{
  "name": "knittable-clone",
  "main": "src/index.ts",
  "compatibility_date": "2026-08-01",
  "ai": {
    "binding": "AI"
  },
  "d1_databases": [
    { "binding": "DB", "database_name": "knittable-db", "database_id": "<your-id>" }
  ],
  "r2_buckets": [
    { "binding": "PHOTOS", "bucket_name": "knittable-photos" }
  ]
}
```

Create the resources:

```bash
npx wrangler d1 create knittable-db
npx wrangler r2 bucket create knittable-photos
```

No API key is needed for Workers AI — the binding authenticates automatically when deployed (and via your local Cloudflare login in `wrangler dev`).

---

## 3. Pick your models

From the [Workers AI model catalog](https://developers.cloudflare.com/workers-ai/models/), the relevant ones:

| Purpose | Model | Why |
|---|---|---|
| Pattern generation (text) | `@cf/openai/gpt-oss-120b` | Strong reasoning, function calling, good at structured/JSON output |
| Faster/cheaper alternative | `@cf/meta/llama-3.3-70b-instruct-fp8-fast` | Good quality, lower latency, function calling |
| Photo analysis (vision) | `@cf/meta/llama-3.2-11b-vision-instruct` | Purpose-built for image captioning/reasoning about visual content |
| Alternative vision model | `@cf/meta/llama-4-scout-17b-16e-instruct` | Natively multimodal, strong image+text understanding, function calling |

Start with `gpt-oss-120b` for text and `llama-3.2-11b-vision-instruct` for the photo step — swap later if you want different cost/latency tradeoffs.

---

## 4. Define the pattern schema

Decide on one JSON shape both pipelines (text and photo) will output into. This is the contract your frontend renders against.

```ts
// src/types.ts
export interface KnittingPattern {
  title: string;
  skillLevel: "beginner" | "easy" | "intermediate" | "advanced";
  garmentType: string;
  construction: string; // e.g. "top-down raglan, in the round"
  yarn: {
    weight: string;      // e.g. "worsted"
    yardage: number;
    suggestedFiber: string;
  };
  needles: string[];     // e.g. ["US 8 circular 24in", "US 8 dpns"]
  gauge: string;         // e.g. "18 sts x 24 rows = 4in in stockinette"
  materials: string[];
  sizes: string[];       // e.g. ["S", "M", "L"]
  sections: {
    name: string;         // e.g. "Yoke", "Body", "Sleeves"
    instructions: string[]; // row-by-row, one string per row/step
  }[];
  stitchChart?: {
    symbolKey: Record<string, string>;
    rows: string[][]; // grid of symbol keys
  };
  notes: string[];
}
```

---

## 5. Build the text → pattern endpoint

```ts
// src/index.ts
import type { KnittingPattern } from "./types";

const SYSTEM_PROMPT = `You are an expert knitwear technical designer. Given a garment
description, output ONLY valid JSON matching this TypeScript interface — no markdown
fences, no commentary:

interface KnittingPattern {
  title: string;
  skillLevel: "beginner" | "easy" | "intermediate" | "advanced";
  garmentType: string;
  construction: string;
  yarn: { weight: string; yardage: number; suggestedFiber: string };
  needles: string[];
  gauge: string;
  materials: string[];
  sizes: string[];
  sections: { name: string; instructions: string[] }[];
  notes: string[];
}

Rules:
- Row-by-row instructions must be complete and knittable, using standard abbreviations (k, p, k2tog, ssk, yo, etc).
- Gauge-check sizing math internally before writing stitch counts; keep counts consistent across increases/decreases.
- Include a materials list with realistic yardage for the stated size range.
- Default to sizes XS-3XL unless the user specifies otherwise.`;

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);

    if (url.pathname === "/api/generate-from-text" && request.method === "POST") {
      const { prompt, sizes } = await request.json<{ prompt: string; sizes?: string[] }>();

      const userPrompt = `Garment description: ${prompt}
Sizes requested: ${sizes?.join(", ") ?? "XS-3XL (default)"}`;

      const response = await env.AI.run("@cf/openai/gpt-oss-120b", {
        messages: [
          { role: "system", content: SYSTEM_PROMPT },
          { role: "user", content: userPrompt },
        ],
        // keep responses deterministic-ish and long enough for full instructions
        max_tokens: 4000,
      });

      const pattern = extractJson(response.response as string);
      return Response.json(pattern);
    }

    return new Response("Not found", { status: 404 });
  },
} satisfies ExportedHandler<Env>;

// Models sometimes wrap JSON in prose or fences despite instructions — defend against that.
function extractJson(text: string): KnittingPattern {
  const match = text.match(/\{[\s\S]*\}/);
  if (!match) throw new Error("Model did not return JSON");
  return JSON.parse(match[0]);
}
```

---

## 6. Build the photo → pattern endpoint

Two-stage: vision model describes the garment → same JSON-generation prompt consumes that description.

```ts
if (url.pathname === "/api/generate-from-photo" && request.method === "POST") {
  const formData = await request.formData();
  const file = formData.get("photo") as File;
  const imageBytes = new Uint8Array(await file.arrayBuffer());

  // Store the original for the user's history / re-generation later
  const key = `uploads/${crypto.randomUUID()}.jpg`;
  await env.PHOTOS.put(key, imageBytes);

  // Step 1: vision model reverse-engineers the construction
  const visionResponse = await env.AI.run("@cf/meta/llama-3.2-11b-vision-instruct", {
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: `Analyze this knitted garment photo. Describe in detail: garment type,
stitch pattern(s) visible (cables, colorwork, lace, texture), construction style
(top-down/bottom-up, seamed/seamless, in-the-round/flat), yarn weight estimate,
approximate gauge, and any distinctive shaping (raglan, set-in sleeve, yoke, etc).
Be specific enough that a knitwear designer could recreate it.`,
          },
          { type: "image_url", image_url: { url: `data:image/jpeg;base64,${btoa(String.fromCharCode(...imageBytes))}` } },
        ],
      },
    ],
  });

  const garmentDescription = visionResponse.response as string;

  // Step 2: feed that description into the same pattern-generation prompt as text flow
  const patternResponse = await env.AI.run("@cf/openai/gpt-oss-120b", {
    messages: [
      { role: "system", content: SYSTEM_PROMPT },
      { role: "user", content: `Reverse-engineer a full pattern from this garment analysis:\n${garmentDescription}` },
    ],
    max_tokens: 4000,
  });

  const pattern = extractJson(patternResponse.response as string);
  return Response.json({ pattern, sourcePhotoKey: key });
}
```

> Note: check the current input format for the vision model you pick on its [model page](https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct/) — Workers AI vision models sometimes expect `image` as a raw byte array in the request body instead of (or in addition to) the OpenAI-style `image_url` content block. Confirm against the docs before shipping.

---

## 7. Stitch chart generation (optional, harder)

knittable.ai renders an actual chart with knitting symbols. Two realistic approaches:

1. **LLM-generated symbol grid**: ask the text model to output a grid of stitch symbols (as in the `stitchChart` field above) and render it yourself as an SVG/HTML grid on the frontend. This works well for cables/lace/colorwork since it's really a structured-data problem, not an image-generation problem.
2. **Skip AI image generation for charts.** Text-to-image models (e.g. `@cf/black-forest-labs/flux-1-schnell` on Workers AI) are not reliable for precise technical diagrrams like stitch charts — don't use them for this part. Render the chart deterministically from the symbol grid the LLM produces.

---

## 8. Credits, tiers, and gating

Mirror the pricing model with a simple D1 schema:

```sql
CREATE TABLE users (
  id TEXT PRIMARY KEY,
  tier TEXT DEFAULT 'hobbyist',      -- hobbyist | studio | pro
  credits_remaining INTEGER DEFAULT 5,
  credits_reset_at TEXT
);

CREATE TABLE patterns (
  id TEXT PRIMARY KEY,
  user_id TEXT,
  prompt TEXT,
  pattern_json TEXT,
  source_photo_key TEXT,
  created_at TEXT DEFAULT CURRENT_TIMESTAMP
);
```

In the Worker, before calling `env.AI.run(...)`:

```ts
const user = await env.DB.prepare("SELECT * FROM users WHERE id = ?").bind(userId).first();
if (!user || user.credits_remaining <= 0) {
  return Response.json({ error: "Out of credits" }, { status: 402 });
}
// ... run generation ...
await env.DB.prepare("UPDATE users SET credits_remaining = credits_remaining - 1 WHERE id = ?").bind(userId).run();
```

Gate photo-to-pattern and custom measurements behind `tier !== 'hobbyist'` checks, same pattern.

---

## 9. The "Tweak" refinement panel

knittable.ai lets users iteratively refine a generated pattern. Implement this by re-sending the previous pattern JSON plus the user's change request:

```ts
if (url.pathname === "/api/tweak" && request.method === "POST") {
  const { pattern, instruction } = await request.json<{ pattern: KnittingPattern; instruction: string }>();

  const response = await env.AI.run("@cf/openai/gpt-oss-120b", {
    messages: [
      { role: "system", content: SYSTEM_PROMPT },
      { role: "user", content: `Here is an existing pattern:\n${JSON.stringify(pattern)}\n\nApply this change: "${instruction}"\nReturn the FULL updated pattern JSON, not a diff.` },
    ],
    max_tokens: 4000,
  });

  return Response.json(extractJson(response.response as string));
}
```

Charge 1 credit per tweak, same as the credit-check pattern above.

---

## 10. Frontend

Keep it simple: a static page (React, or plain HTML) with:
- A textarea + "Generate" button, or a file input for photo upload
- A results view that renders the `KnittingPattern` JSON into readable sections (materials list, gauge, row-by-row per section, stitch chart grid)
- A "Tweak" input box under the result

Serve it via Workers Static Assets by adding to `wrangler.jsonc`:

```jsonc
"assets": { "directory": "./public", "binding": "ASSETS" }
```

and letting the Worker fall through to `env.ASSETS.fetch(request)` for non-`/api/*` routes.

---

## 11. Deploy

```bash
npx wrangler d1 execute knittable-db --file=./schema.sql
npx wrangler deploy
```

Workers AI usage is billed per-neuron (Cloudflare's inference unit) — check current [Workers AI pricing](https://developers.cloudflare.com/workers-ai/platform/pricing/) since it affects your per-credit cost and margin on paid tiers.

---

## 12. What's *not* covered here

- **PDF export** — use a library like `@react-pdf/renderer` in a Worker, or render the pattern to HTML and convert with a headless-browser service (Workers AI itself doesn't do PDF generation).
- **Auth & Stripe billing** — standard integrations, unrelated to Workers AI specifically.
- **Prompt-injection / abuse protection** — since users can freely type into the prompt, add basic input validation and consider Cloudflare's `@cf/meta/llama-guard-3-8b` model as a content-safety check on inputs before they hit your main model.

---

## Summary of the core loop

```
User input (text or photo)
        │
        ▼
[photo only] @cf/meta/llama-3.2-11b-vision-instruct  → garment description
        │
        ▼
@cf/openai/gpt-oss-120b  (system prompt enforces JSON schema)
        │
        ▼
KnittingPattern JSON  →  render on frontend  →  store in D1  →  optional PDF export
```

That's the entire AI core of a knittable.ai-style app — everything else (auth, billing, UI polish) is standard web app work layered around these two Workers AI calls.