PICK YOUR SUPPORT STYLE
MONTHLY SUPPORT
Reader
$5/mo
Contributor
$15/mo
Architect
$50/mo
Recurring subscriptions auto-bill monthly via Stripe Checkout. Cancel anytime from the receipt email.
What was talked about: Opus 5.5 appears to be a substantial replacement for earlier model versions, with stronger day-to-day usability and especially impressive artistic output. Its visual taste, coding performance, and overall quality make it a credible reason to return to Anthropic models after spending more time with the GPT-6 family.
Takeaway: Opus 5.5 is a strong general-purpose release that may justify changing a primary model workflow.
Links shown: Claude: Introducing Opus 5.5
What was talked about: Early Opus 5.5 use has produced occasional unexpected refusals during ordinary work, raising concerns about unattended agent runs. GPT-6 has already completed continuous tasks lasting about 24 to 25 hours, while equivalent long-duration evidence for Opus 5.5 remains limited.
Takeaway: Test Opus 5.5 carefully before relying on it for unattended tasks where a mid-run refusal would be costly.
Links shown: Scaling01: Intelligence versus cost chart
What was talked about: Sonnet 5.5 arrived shortly after Opus 5.5, following a pattern in which capabilities from a large model move into smaller models. Comparisons with GPT-6, Sonnet 5.6, Luna, and Terra suggest that nominally similar models can occupy different practical tiers for intuition, instruction understanding, cost, and agent work.
Takeaway: Choose models by observed capability and workload fit instead of assuming that similar names or generations represent the same performance tier.
Links shown: Scaling01: Intelligence versus cost chart, Claude: Introducing Sonnet 5.5
What was talked about: Anthropic models were described as less disruptive than GPT models when working with unfinished products, preview interfaces, and complex agent hierarchies. They could implement direct product language and complete sensitive-looking technical operations without adding excessive warnings or blocking routine work, although experiences with refusals varied.
Takeaway: Safety behavior can materially affect implementation speed, so evaluate models with realistic product and agent tasks.
Links shown: Claude: Introducing Sonnet 5.5
What was talked about: The current acceptability of using consumer subscriptions through SDKs or server-side autonomous agents remains unclear. Anthropic announced restrictions around Claude command-line and SDK usage, then appeared to retreat from some changes, while subscription terms still commonly frame access as intended for humans rather than autonomous systems. OpenAI subscriptions were viewed as more tolerant in practice.
Takeaway: Do not treat technical access through an SDK as proof that continuous server-side agent use is permitted; review current service terms before deployment.
Links shown: Claude: Introducing Sonnet 5.5, Theo: Dev Day announcement overview, Subscription and API-cost analysis post
What was talked about: High-tier subscriptions can return far more inference value than their monthly price when used heavily, sometimes replacing thousands of dollars in API consumption. Provider descriptions of usage allowances and API-equivalent value can be difficult to interpret, especially when limits are expressed across different time periods.
Takeaway: Compare subscriptions with API pricing using actual workload volume, rate limits, and permitted usage rather than marketing language alone.
Links shown: Subscription and API-cost analysis post, Theo: Dev Day announcement overview, OpenAI: GPT-6.1 Sol
What was talked about: GPT-6 did not show a major improvement over GPT-5.6 for coding and autonomous component workflows in limited testing. Existing work therefore remained on GPT-5.6, although GPT-6.1 was considered a likely next model to evaluate.
Takeaway: A newer model is not automatically worth a workflow migration; test GPT-6.1 against established tasks before switching.
Links shown: OpenAI: GPT-6 Sol and Luna, Andon Labs: Model benchmark results
What was talked about: Grok-related products, Muse, Dots, and OpenClaw reflect a broader push toward personal consumer agents. Several products appear to build on existing agent frameworks, including OpenClaw and Codex-based foundations, rather than creating entirely new orchestration systems.
Takeaway: Consumer agent products increasingly compete through packaging and user experience while sharing similar underlying agent infrastructure.
Links shown: Andon Labs: Model benchmark results, SpaceXAI: Grok 4.7 announcement, Theo: Grok 4.7 review, Theo: Dev Day announcement overview
What was talked about: Muse was seen as a useful agent product, but its standalone model produced weak results for code review and did not stand out against alternatives such as DeepSeek Flash 401 and Luna Max. The product experience appears stronger than the underlying model when evaluated independently.
Takeaway: Evaluate an agent product separately from its base model because orchestration can provide more value than raw model quality.
Links shown: Andon Labs: Model benchmark results, Theo: Grok 4.7 review, CompleteSkeptic: Agent memory post, Gregor Žunić: Browser-use benchmark post
What was talked about: A general-purpose model can accept varied inputs and convert them into classifications, scores, or other quantified decisions. Its results were comparable to Gemini Flash Lite in an application test, with lower cost and similar speed, although the improvement was not large enough to require immediate support for another provider.
Takeaway: General-purpose classifiers can be practical low-cost substitutes, but integration work is only worthwhile when the operational benefit is clear.
Links shown: Charly Poly: Jev as a quantifier, Ohans Emmanuel: Jev coding-agent use case, Kieran Klaassen: Jev embedding alternative
What was talked about: Fast decision models can replace LLMs for routine tool selection and smart conditional logic. General-purpose models are especially useful during early product development because requirements and environments change too quickly to maintain a fixed fine-tuning dataset, while specialized training becomes more attractive after usage reaches substantial scale.
Takeaway: Start with a flexible general model, then consider specialization after the task, data, and scale become stable.
Links shown: Kieran Klaassen: Jev embedding alternative, Josh Kuechly: Jev zero-shot classification, Shreesh: Jev limitations, Decisions API announcement
What was talked about: Jev processes a task in one pass rather than performing an autoregressive pass for every generated token, which suggests lower latency and efficient batch execution. Open-source alternatives include approaches based on Gemma Diffusion and compact encoder models, though the best option depends on the target task and benchmark.
Takeaway: Single-pass decision models can deliver major speed advantages, and open alternatives should be compared on workload-specific benchmarks.
Links shown: Decisions API announcement, ModernBERT Large Zeroshot v2.0
What was talked about: Jev’s parameter count is unclear, but single-pass output can remain inexpensive even if the model is relatively large. Pricing was clarified as about four cents per million operations, and diffusion-style execution may gain additional efficiency in batches because it avoids repeatedly loading the model for token-by-token generation.
Takeaway: Architecture and execution pattern can matter more than parameter count when estimating the real cost of a decision model.
Links shown: ModernBERT Large Zeroshot v2.0, OpenRouter: GPT-6 Luna, OpenRouter: Qwen3.8 27B
What was talked about: NerfBench aims to measure whether hosted models become less capable after release. Current evidence is sparse, and subjective impressions can reinforce expectations whenever a poor response appears, making it difficult to separate real degradation from normal output variation.
Takeaway: Use repeatable benchmarks and enough data before concluding that a provider has reduced a model’s capability.
Links shown: Pangram: AI direction for Gmail, BridgeBench Nerf Bench
0:18 Alrighty, let’s get started here. Welcome back to AI Tools Club, everybody.
0:24 I hope you all enjoyed the week off that we had.
0:27 Sorry, I was gone in the woods last week.
0:31 Minimal contact with any models or Twitter or anything like that.
0:37 So this week will be a little bit different because there was a bunch of releases both next week in this week.
0:47 And I think we have enough time to even skip over sort of like the release notes from them.
0:51 So this will definitely be a bit more beginning of sort of what you’ve experienced with these different notes.
1:02 So yeah, let’s get into it. We’ll start with the first thing that was released last week, which was this point.
1:10 So yeah, this seems sort of basically from what I’ve been reading is sort of fable 1.1 replacement.
1:23 Sort of like usage compared to fable and the others much more usable data data.
1:34 One of the big things I’ve been seeing from people is its artistic capabilities.
1:39 I think I covered this, I forget which one recently, but like some of the good artistic capabilities.
1:44 This one definitely has the best. And it’s definitely dominating the negative because of it.
1:50 It was a bit of what it was. We made a bunch of cool demo videos.
1:55 Which I think is also a combination.
1:56 It seems that we’ve had a lot more interesting video creation tools coming up as well, making it easier for these models to go and create these sorts of videos.
2:04 But it seems that Opus Point has a lot of taste in this department.
2:11 Yeah. But this point obviously looks better across the board.
2:16 I’ve heard a few people, nothing too concrete saying that this is good, and it seems to be something about the GPT changes, but this actually looks like it may be a reason to go and switch
2:30 to opening and using this model. For the majority of the time.
2:40 But this looks like the first model that might actually convince me to go and switch back to the internet models.
2:46 So yeah, how have you guys been experiencing this model?
2:50 Has it been worth it using it versus previous sort of versions of Opus?
2:54 I know, like, Opus 5 was not a very good model.
2:57 Is this a better model to use day to day?
2:59 And how’s it compared to the OpenANA models?
3:03 Well, I’m barely using Fable anymore.
3:06 With Opus 5.5 around. It’s really, it’s really good.
3:12 Like, I know I can also use Fable. I suppose it would use up my week tuner, but 5.5 has just rocked.
3:21 And it’s only been a few days because I was at my daughter’s house over the weekend, so it wasn’t working much.
3:29 But man, it’s solid. This is a good release.
3:35 And I was just coming off the plan to start reducing my anthropic spend because I was having such a good time with GPT-6 family.
3:47 And then this came out. I was like, oh man, this is crazy.
3:53 Whiplash. And
4:23 to Nikolai’s point, he says Opus 5.5 has a high level of refusals.
4:31 I’ve only had two refusals in the last few months, and they were both 5.5.
4:38 And in either case, was I trying to hack the White House or anything?
4:52 My I started considering switching back from context to quad, but I tried the feedback from people that sometimes refuses to do stuff.
5:02 So it’s not as good for one-running tasks when you can just like in the middle of the task refuse to do something because things don’t have my
5:16 own experience with it. I guess recently I have not tried the longest running tasks on Opus 5.5.
5:28 Stall 6 ran a solid 24, 25 hours for me a couple of weeks ago, which was the longest ever I’ve seen.
5:39 And it was just kept cracking away.
5:42 Have not run open to that extent, so I can’t compare that.
5:48 Also, Quad released SunNet 5.5 yesterday or today.
5:53 So probably like they are basically releasing large models and then distilling into all small models consequently.
7:24 I’m used to both of them actually and I can like just from the vibe check some it is at least better in a lot of the like intuition kind of knowing what I mean kind of stuff.
7:41 In comparison the sixth school they also released the sixth.1 school which was released like probably two hours ago which I don’t think I will try yet but the sixth school like I was finding I was I kept like trying to squeeze
7:53 out every last token of my subscriptions and Luna was like way too dumb was like okay but so 5.6 and solve 6 seem like they’re like different tier models and I thought they’re both sole like the solo 5.6 is more like a luna
8:06 or more of like a terra model and it’s like notice would be kind of dumber in that case because also get like caps as inexpensive and fun it feels uh at least like on the vibes of like um also
8:21 the anatomic models the nice thing about them is they’re not as like safety used as the gbt models they don’t like uh like a hand um you know we do have like a what was it like a like a very basic landing page for an
8:36 and like unfinished product that’s like a reference to put in an app and you can see what else like put everywhere like stuff like this is preview this is like an unfinished product something something and they were like
8:48 really careful about like not accidentally like releasing something like reviewing something that they’re not supposed to and these one had and some of us like really fine just doing the same thing and saying
9:03 like I mean sign up here and like not actually be a pain in the ass and stuff attached to the whole thing so from that kind of point of view I haven’t seen enough any refusals
9:17 yet so far I feel like my use case agents are running in like some deep hierarchies and they like they they willingly know correct torrented stuff and they willingly
9:32 downloaded obliterated models like I don’t know I haven’t had anything that they have refused to I haven’t had them do the no the the son at five I’ve agreed to do an ability model already so um
9:47 what yeah yeah yeah my question is what is uh do we know what the overall like current and traffic policy on running the
10:01 subscription for agents like open club on like server side or something like that previously open again was much friendlier for this kind of use cases and entropic was against it but what was the current entropic they
10:19 had a thing in june like opposed june 15th they’re ending the official support for like club dash dashb and like sdk being a separate pool and then the like the d when it was supposed to end the backtrack
10:34 because i think they were losing graphic and they were like trying to not give people another reason to cancel the subscription so they haven’t come back to that so for now you can use cloud dashb and like sdk and that’s
10:46 what i’m doing like my honest is on cloud sdk so that definitely helps but i do know what they also been like back in the other button server side server
11:02 side for the thing i finish product subscriptions i pull them for my uh permanes on open class and open again is fine with that is
11:17 gonna pop it also a logo yeah i don’t know how there were some workarounds but i think they stopped breaking down and then or something i’m definitely using
11:31 like as i’m using a public like quad as d case which is just like less likely to catch that but it’s running like 24 seconds sometimes it’s like you know pretty clearly like if they were to do something
11:45 like behavioral patterns they would totally like change at some point it’s probably gonna be like another once once you know plus five point five is really good and people are happy with them we can send that and
12:10 they have the map compute maybe it seems super safe to use the SDK that’s I mean they they created that for you to use they they seemed freakier
12:26 about different uses of the API key through the Mac subscription yeah but when they were supposed to when they were like announcing restrictions to that they were including the
12:40 SDK and something that’s not allowed to be like coded prohibited like almost every autonomous subscription like subscriptions to use by humans only not by agents
12:55 like not like autonomous yeah but then they backed yeah but
13:26 if but if I’m like in between if I’m at startup and I’m running hundred open clothes but not five okay
13:42 okay okay yeah yeah but I yeah I had no problem with OpenAI at all so that
16:05 400 per week or per month month yeah that would be a monthly cost that was the poorest worded
16:19 description that I’ve read in a long time like are you just trying to confuse me but they’re trying to say you get the same work done as this or you get you know slightly more work but less API
16:34 dollars or less API bragging rights no by API prices I get thousands of dollars out of the subscriptions and I think they’re saying yeah it’s gonna be less than
17:13 a week
21:13 I I I tried a little bit uh six but not six point one um and I didn’t see the big difference between five point six and six uh for like my use cases like budding and
21:27 some autonomous uh components stuff like that so I I still like use five point six for most of my workflows but I probably will switch to six point one so now yeah
24:37 Andrew, what do you think about Grok models?
24:40 Like we are not usually using them, but people are like talking about them.
27:22 Yeah, but I think like, yeah, actually, I also have a grog bot.
27:26 It’s kind of similar to all this trend with Muse Instinct and Dots.
27:32 So they’re trying to build a personal agent based on Groc.
27:51 Yeah, dots. Yeah. Yeah. And I just Google and dots actually are not, yeah, it’s not OpenClaw, I think it’s Codex-based.
28:01 I’m wondering why they’re not using OpenClaw, though they bought OpenClaw half a year ago.
28:10 Yep, everybody wants to enter this consumer agent area.
28:27 Oh, already Muse is open cloud. Oh, wow.
28:40 Interesting. OpenCoh. Interesting. Yeah.
28:50 Yeah, the meta people did confirm that it’s pretty much just OpenClaw, if not slightly different.
29:06 Has anybody used the Muse model for anything?
29:13 Model? No. I think it’s not as good.
29:17 It’s like a regular model, but nothing.
29:20 I tried to plug it in for code review, got nothing out of it, dropped it.
29:25 Right now I’m using Deep See Flash 401 and Luna Max as my two models.
30:06 Let’s talk about the model, I think.
30:10 Yeah, yeah, but the bot is really good.
30:12 Yeah, it’s a good product. But the model itself is not as good.
30:19 It’s a model to use separately from Muse.
31:41 Wait, is it multimodal? About the screen, is it multimodal?
31:47 How do people? I see, okay. Okay. Yeah.
31:56 Yeah, I tried to use it for like I just played with it on Sunday, uh, a few Sundays ago.
32:02 Yeah, it’s like a good quantifier. But it’s like generic, so you can input anything, so you can like ask any question and it will try to quantify it for you.
32:15 So it’s kinda good general purpose and it’s convenient basically, but other than that, yeah, not sure.
32:26 I’ve tried to plug that into as an alternative in my app versus given I flash flight, and the results are not different, but the cost is lower, and the speed is about the same.
32:41 So it’s not bad. It’s as good as Gemini Flashlight, which I think is a really good value model, low latency model.
32:50 But it’s not like it’s going to steal the niche or make me want to support the model just for this use.
33:19 I think also people are using it for some routine like tool use, so deciding which tool to use for which action, something like that.
33:30 So it’s some basic it’s smart if if else or some that like you usually used
33:44 used to use LOM but now you can do the same actions like faster and cheaper.
35:58 I don’t agree. So like yeah if you do care about scale or cost you can train your own model.
36:04 But usually for most of applications you don’t want to train models so you have the same as LLMs.
36:13 Make LLMs work for any task. You don’t need to train a fine-tune LLM for most of tasks to be done well.
36:23 I think general and like general purpose specifiers will be quite useful because they will just work out of the box and build whatever I can for first million
36:37 users like later on yeah you may want to train everything yourself.
37:05 Yeah, it also yeah but also like when you’re building something
37:19 you don’t always understand what like what you are building so you don’t have a specific like fixed data set to find you into because you maybe you’re like building this and then you’re building that and
37:33 then like you’re changing it and like computer use maybe like you’re building it for Mac and then new version of Mac will be released and this model is too fucking tuned for some specific screenshots
37:48 of over something over a specific HTML standard if it’s browser use I don’t know so you sometimes want more general models so it it doesn’t break
38:03 when you change what you are building or the world is changing around us.
38:18 Yeah, Brendan is asking what about OpenAI decision APIs?
38:23 Yeah. One
39:04 of the things I found remarkable or interesting about Jev is that it’s a single pass.
39:09 So for any given task, it’s only one run-through.
39:13 It’s not one run-through per token.
39:15 So an LOM doing something is inevitably more than one slice through.
39:23 Some indication to me that it’s going to be fast.
39:44 Andrew, I’m having trouble hearing you.
39:46 I don’t know if it’s me or you. Yeah, yeah, I’m hearing fine.
39:52 Yeah, I had no trouble. So question is probably you.
40:22 So is there an open source general jev equivalent that is like an actual general or is it just like like is there a good starting point to finding from?
40:38 People have already put together the best open source ones.
40:41 Oh great. Of course there’s a bench already.
40:44 Yeah people were liking Gemma Diffusion some and a bunch of others but it probably depends on your use case.
41:40 It’s nothing new but probably they’ve got better because LMs got better and understanding of input tokens got better.
41:48 So you can build smaller models that are working better nowadays than four or five years ago by using BERT.
42:29 How big is GF? Like do we have an estimate?
42:37 Is it like so
42:55 we don’t even like know it’s like hundreds of billions or like dozens of billions or trillions interesting.
43:01 It’s cheapened. Yeah. Okay. Yeah. But it’s just a single pass, so it’s like it’s not generating
43:16 like lots of tokens. So output, it’s generating output in a single pass.
43:20 So it’s kind of cheap even if it’s big.
43:24 Because it’s someone who’s
43:42 a good idea wouldn’t
43:59 double check like impractical. Oh sorry, but real quick just because it feels like compared to like let’s say like QEN 207B pricing or like it’s still it’s
44:13 under a cents per million still but the and draft is like a cents per million.
44:18 So it’s just like it has to be like a solid order of magnitude like single magnetic order less like I thought that was kind of the biggest thing that it’s like so cheap.
44:28 It’s still really really cheap even for like pre-filled numbers.
44:32 Because you still get a mistake charges for pre-fill only.
44:36 And like yeah can they like is the pricing different for the future models in that case for some reason?
44:45 Or because it feels like if you just take the Quen27B and you fine-tune it to be a classifier, you won’t get that pricing, right?
44:52 So it has to be something more interesting that they’re doing there.
45:39 Okay, yeah, I just double-checked. I thought it was 40 like cents per billion, because that’s how they advertised it.
45:46 But actually, yeah, it’s 4 cents per million.
45:48 It’s 40. Because they put it as like $40 per billion.
45:52 And so, yeah. You’re right. It’s probably just something like that size.
46:05 I had heard in one of the interviews with Diego about Jeff, no, it was some podcast recently, that diffusion models are more efficient to run in batch than LLMs
46:19 because you don’t have to load and load again to run on the GPU.
46:26 The one pass through is more efficient.
46:30 I don’t have special knowledge here, so I don’t really know if that’s a true statement.
47:55 So Bridgepin promises to have something called Nerfbench, which I’d like the idea of hopefully an objective way of saying yes to nerfing or no to nerfing
48:10 for models. Right now, you can see there’s two, very few data points, four data points.
48:16 But I’d like to see this because I don’t know if I believe people saying everybody’s intuition on whether the models have gotten stupider or not.
48:29 But I’ve had the same intuition, so I’m not picking on other people.
48:35 But I thought that Opus V got smarter.
48:39 I thought it was bad at first, and then I started to depend on it.
49:51 I think it’s like superstition. So they think that people believe in something and they experience different answers from models, but when they experience bad answers, it reinforces the superstition.
51:06 When are you coming back? Okay,
51:28 great.
Subscribe to get the latest AI news in your inbox every week!
