Deep dive
Improving recommendation quality
The first controlled live Duo run passed six of seven automated checks, and I still rejected the recommendation. That gap now defines the quality work.
Controlled Duo live result
Rejected run- Provider calls
- 1
- Checks passed
- 6/7
- My verdict
- Rejected
Selected film
The Grand Budapest Hotel stayed inside the fixed candidate set and satisfied the shared genre and runtime constraints.
Automated strengths
The response matched the schema, named Alex and Sam, represented signals from both profiles, and gave a sufficiently detailed explanation.
Failed bridge policy
The selected film was the deliberately weaker comparison candidate rather than one of the three pre-declared bridges.
Why I rejected it
The explanation made the overlap sound convincing, but the film still leaned too far toward Alex’s taste and did not offer enough kinetic energy for Sam.
Interpretation. One controlled run is not a general quality measurement. It is direct evidence of a narrower problem: the system can satisfy formal response checks while using a persuasive explanation to overstate a weak taste overlap.
The catalog became a cache, not a universe
The first catalog was large enough to demonstrate vector search and far too small to behave like a movie product.
The course started with twelve embedded films. I later added roughly three hundred titles by hand and included Soviet films so my parents could try the product with something closer to their viewing history. The expansion helped, but it exposed the real limit: a fixed local catalog can only recommend what its author anticipated.
PopChoice therefore moved toward hybrid retrieval. A request can use local PostgreSQL and pgvector candidates, then widen the pool through TMDB search or discovery when local quality is weak. Promising external candidates are enriched before ranking and can be queued for catalog seeding. The local database becomes a cache, identity layer, and reusable index—not a substitute for TMDB's breadth.
This also creates an operational quality problem. External records may duplicate an existing film, lack a release year, or fail embedding generation. Those cases are skipped or sent to review rather than accepted just because TMDB returned them. Backoffice catalog health and TMDB review surfaces make that work inspectable.
How the candidate universe changed
Course seed
The original loop used twelve films—enough to demonstrate embeddings, but too small for meaningful discovery.
Curated expansion
I added roughly three hundred titles, including Soviet films intended to make family testing more relevant.
Hybrid retrieval
Local pgvector search can be expanded with bounded TMDB search and discovery when the cached candidate set is too weak.
Reusable local index
Strong TMDB candidates can be enriched and queued for seeding, while duplicate or incomplete identities are rejected.
Why it matters. A broader source does not remove the need for local data. It changes local storage from the whole universe into the controlled layer where identity, embeddings, metadata, memory, and repeated retrieval can be made consistent.
Constraints come before generation
Recommendation quality improves first by removing known bad answers, not by asking the model to be more creative.
Quiz answers describe tonight: the mood, desired energy, era, discovery appetite, and hard avoids. Signed-in accounts add durable Movie Memory. The current memory model is deliberately conservative: watched, not-interested, and recently recommended titles leave the pool; wrong-mood feedback lowers a candidate; and an exact liked title adds a small boost.
The model receives the resulting candidate set and must pick from it. If its answer falls outside the set, deterministic code selects the strongest valid candidate and generates a safe explanation. This is a reliability boundary as much as an AI technique: retrieval owns possibility, while the model owns comparative judgment and language.
Memory is not mature personalization yet. It primarily reduces identical and obviously unsuitable recommendations from one request to the next. The longer-term direction is a single taste-signal model that can represent quiz answers, Movie Memory, result feedback, and a future swipe experience without giving each surface a separate ranking language.
Signals before ranking
Tonight signals
Mood, energy, era, discovery appetite, hard avoids, and optional reference films describe the immediate context.
Memory exclusions
Watched, not-interested, and recently recommended films leave the pool before the model ranks it.
Memory adjustments
Wrong-mood feedback downranks a candidate, while an exact liked film contributes a small positive signal.
Feedback loop
Result actions such as already watched, too obvious, more like this, and try another can shape later requests.
Why it matters. Known negatives are cheaper and more reliable than inferred taste. Removing repeats and explicit dealbreakers creates a stronger candidate set before the most variable part of the system begins.
Feedback-memory repeat avoidance
Deterministic fixture- Candidates
- 6
- Memory exclusions
- 3
- Returned picks
- 3
Tonight signal
A solo viewer asks for a gentle, heartwarming mystery after several recent misses.
Remembered boundaries
A watched romcom, a wrong-mood epic, and a rejected gore title must not become the answer.
Bounded result
Cozy Mystery leads; Ensemble Comedy and Family Adventure remain as valid alternatives.
Narrow claim
The fixture proves filtering and response contracts, not production model quality or user satisfaction.
Interpretation. This case makes one boundary inspectable: three known bad matches leave a six-film fixture before the response is scored. It remains a development regression test, not evidence that a live model will choose the same real film or that a viewer will enjoy it.
Evaluation matches the kind of risk
One evaluation mode cannot answer contract, data, environment, and live-provider questions at the same cost.
The cheapest layer runs deterministic fixtures with mocked model output. During the v0.2.0 evidence pass, the suite passed all 32 scenarios with 100 points per case. Twenty-seven cases cover the audience, recommendation depth, and candidate-source matrix. Five focused taste-control cases cover no-horror, not-obvious, no-long-runtime, no-gore, and feedback-memory repeat avoidance.
That result is useful precisely because its claim is narrow. It checks response shape, candidate validity, forbidden content, repeat avoidance, explanation requirements, audience representation, and source policy without paying for or depending on a live provider. It cannot tell me whether a person loves the selected film.
Higher layers answer different questions. A seeded database run checks whether fixtures and metadata exist in real catalog retrieval. Environment jobs persist source distribution and failures through Backoffice. A guarded live-provider run is manual because it spends credits and introduces OpenAI and TMDB variability. Separating these layers keeps the fast gate stable without pretending it is the final quality verdict.
Four evaluation layers
Deterministic fixtures
Mocked provider output makes candidate boundaries, safety, memory, explanation, audience, and source-policy checks cheap enough for every pull request.
Seeded catalog retrieval
A real database with controlled model output checks migrations, candidate availability, search, and catalog metadata quality.
Environment evaluation
Backoffice-triggered BullMQ jobs inspect the configured environment and persist summaries and per-case failures.
Guarded live-provider run
A manual action requires cost acknowledgement because OpenAI and TMDB behavior is variable and can spend provider credits.
Why it matters. Each layer becomes slower, more variable, and closer to production. The deterministic layer protects every pull request; provider-sensitive evidence is reserved for intentional validation where its cost and uncertainty are visible.
Deterministic contract baseline
Verified run- Scenarios
- 32
- Passed
- 32
- Score per case
- 100/100
Candidate validity
The lead pick and alternatives must come from the fixture candidate set rather than from an invented model title.
Hard constraints
Dedicated taste-control cases check horror, gore, long runtime, obvious picks, and other explicit exclusions.
Repeat avoidance
Watched, rejected, recently recommended, and fixture-forbidden titles must not return as the answer.
Explanation contract
The response must represent the audience and contain the scenario-specific terms needed to explain the choice.
Interpretation. This run verifies the scoring and response contracts with mocked provider output. It does not prove that a live model will always make a subjectively good choice; it proves that recommendation changes cannot silently cross the candidate boundary or ignore the fixtures' explicit constraints.
Evidence and operating contracts
- Current product behavior (opens in new tab)
The maintained contract for quiz modes, recommendation lifecycle, Movie Memory, feedback, catalog behavior, and group boundaries.
- Recommendation roadmap (opens in new tab)
The staged direction from a small embedded catalog toward reusable taste signals, TMDB-first discovery, and stronger evaluation.
- Evaluation fixtures (opens in new tab)
The source scenarios for audience, depth, candidate source, hard avoids, memory, and explanation checks.
- Backoffice eval operations (opens in new tab)
Persisted mock, real-data, environment, and guarded live-provider runs operated through BullMQ and Backoffice.
- Controlled Duo protocol (opens in new tab)
The fixed inputs, four-film candidate set, machine checks, one-call live mode, and separate human-review contract.
Why it matters. The behavior, scenarios, and operating modes live beside the code. That makes the quality claim reviewable and lets future changes update the same contract instead of relying on a one-off case-study description.
The next quality proof is deliberately harder
The first controlled live response passed most formal checks and still produced a compromise I would not recommend.
A deterministic Normal Match run proves the delivery path: two profiles reach one persisted result, and the same URL renders after reload. The harder protocol isolates ranking instead. Alex references Amélie; Sam references Mad Max: Fury Road; both rule out horror and films longer than 125 minutes. Four fixed candidates go directly to the production ranking function, with one provider call and no retrieval or catalog drift.
The one-call live run selected The Grand Budapest Hotel. The response stayed inside the fixed set, respected the hard constraints, named Alex and Sam, represented terms from both profiles, and gave a detailed explanation. It passed six of seven automated checks. The failed check mattered most: the film was the deliberately weaker comparison candidate, not one of the three pre-declared bridges.
I rejected it. The explanation made the overlap sound resolved, but the film still leaned too far toward Alex’s whimsical preference and offered too little of the kinetic experience Sam asked for. The next iteration must make bridge fit an explicit ranking constraint rather than a hint the explanation can talk around.
The fixed candidate set
Run Lola Run · 81 min
A strong bridge through romantic urgency and playful structure for Alex, plus kinetic pacing and bold visual style for Sam.
Scott Pilgrim vs. the World · 112 min
A strong bridge through playful romance and comic charm, paired with kinetic action and highly stylized editing.
Baby Driver · 113 min
A strong bridge through romantic escape and human connection, carried by kinetic driving and rhythmic visual direction.
The Grand Budapest Hotel · 100 min
A bounded comparison candidate: strong whimsical and visual fit, but deliberately weaker energetic overlap.
Why it matters. Fixing the inputs before the provider runs prevents retrieval quality, catalog drift, or repeated attempts from deciding the outcome. The test asks one narrow question: can the ranking layer choose and explain a credible bridge?