This week’s roundup pulls together three practitioner write-ups from Japan’s Zenn community. Each tackles a different layer of building real AI products — parsing reliability, agent UX, and business strategy — but they share a common thread: the gap between what looks fine in a demo and what survives contact with production.
Where LLMs lose track of your spreadsheet columns
Anyone who has fed a sales table into an LLM has probably wondered how far you can strip down the formatting before the model starts hallucinating numbers into the wrong column. One developer ran a structured experiment on exactly this question, testing three different LLMs against a sales table where column labels were progressively degraded — first replaced with symbols (with a legend given once at the top), then removed entirely.
The symbol-substitution approach held up reasonably well at first. But once the table grew past roughly five days’ worth of rows (about 15 cells), one of the three models being tested started mixing up values, even though it could still recite the legend correctly when asked. That’s the most useful finding here: a model correctly repeating back a mapping table is not the same skill as correctly applying that mapping across a growing dataset. Testing “does the model understand the key” in isolation will pass, while the actual aggregation task silently drifts.
Removing the header row altogether was worse — two of the three models eventually confused the actual order of columns, not just individual cell values. And critically, this degradation wasn’t driven by adding more columns; it got worse simply by extending the number of rows (the length of the series), even with the same number of columns throughout.
The practical takeaway for anyone building spreadsheet- or table-ingestion features on top of an LLM: don’t assume a technique that passes a short smoke test will hold at realistic data volumes. If your column labels are anything other than plain, repeated-per-row text, you need to specifically test long-series behavior, not just correctness on a handful of rows. Compressing headers to save tokens is a classic optimization that trades correctness for context-window savings, and this kind of length-dependent failure is exactly the kind of thing that won’t show up until someone pastes in a full month of data.
Teaching a resident agent when to shut up
The second piece is about a different failure mode entirely: agents that are technically correct but socially exhausting. If you’re building a chat-resident AI agent that watches channels, email, calendars, and meeting notes continuously, the natural engineering instinct is additive — every time you can detect something new, you wire up a notification for it. The author makes the case that this instinct is precisely what kills adoption. An agent that reports every minor task update, every small detection, quickly becomes noise that users learn to ignore or mute entirely, which defeats the point of having it resident in the first place.
The reframe is useful: treat “when not to speak” as a first-class design surface, not an afterthought bolted on after users complain. That means explicitly deciding thresholds for what’s worth interrupting a human for, batching or suppressing low-value updates, and possibly building confidence or importance scoring into whatever detection pipeline feeds the notification layer — rather than treating every detectable event as automatically notification-worthy.
This is a pattern that shows up across chatbot and agent design generally: the feature that ships fastest (“notify on X”) is rarely the feature that keeps users engaged long-term. If you’re building anything that runs continuously and has autonomy over when it speaks, it’s worth explicitly designing the silence budget before you design the next detection capability.
Is your AI product actually defensible?
The third piece speaks to a very different but equally practical anxiety: if you’re an indie developer or small team monetizing an AI-powered product, how do you know whether the next foundation model update will simply absorb your feature?
The author’s framing is a single diagnostic question: are you selling AI itself, or are you using AI to solve a pre-existing market problem faster and cheaper? Products in the first category — thin wrappers whose entire value proposition is “AI does X for you,” where X is a capability that plausibly becomes a native model feature — are structurally exposed. Products in the second category are ones where the underlying market pain existed before LLMs and will keep existing regardless of model capability; AI here is a means of production, not the product itself.
This isn’t a novel insight in isolation, but the value of the piece is turning it into a concrete self-test rather than a vague worry. If you can’t clearly articulate which of the two buckets your product falls into, that’s itself diagnostic — a product with a real moat should have an obvious answer to “what happens to us if OpenAI/Anthropic/Google ships this natively next quarter?” If the honest answer is “we’d have nothing left,” you’re selling AI capability, not solving a market problem, and you’re on borrowed time until the next model release.
The common thread
All three pieces are really about the same underlying discipline: stress-testing assumptions past the point where a demo looks convincing. Whether it’s a parsing technique that works at 5 rows but breaks at 20, a notification feature that feels helpful in isolation but becomes unbearable at scale, or a product whose value proposition quietly depends on a capability gap that model vendors are actively closing — the failure mode is always the same. Things that work in the small, controlled case you tested don’t automatically survive the dimension you didn’t stress: length, frequency, or time. If you’re shipping anything LLM-adjacent right now, it’s worth asking which of those three axes your own product hasn’t actually been tested against yet.