PICK YOUR SUPPORT STYLE
MONTHLY SUPPORT
Reader
$5/mo
Contributor
$15/mo
Architect
$50/mo
Recurring subscriptions auto-bill monthly via Stripe Checkout. Cancel anytime from the receipt email.
What was talked about: Bridgewater and Thinking Machines fine-tuned Qwen 3.5 397B to recognize financially important information and outperform frontier models on a narrow domain task. Easier training services, mature open models, production data, and rising API bills are making fine-tuning relevant to more organizations after years of prompting, RAG, and context engineering being the better first options.
Takeaway: Fine-tuning is becoming worthwhile for well-defined, data-rich use cases, but teams should still exhaust simpler optimization methods first.
Links shown: Mira Murati on Bridgewater fine-tuning, Learning to Replicate Expert Judgment in Financial Tasks
What was talked about: Fine-tuned open models can cost substantially less to serve than frontier APIs while matching or exceeding them within a narrow specialty. The remaining open-versus-closed performance gap is already small on many saturated tasks, and domain-specific training data can give an open model capabilities unavailable through prompting or retrieval alone.
Takeaway: A smaller specialized model can deliver frontier-level results at a fraction of the inference cost when the task is narrow and measurable.
Links shown: Learning to Replicate Expert Judgment in Financial Tasks
What was talked about: Specializing a model can degrade its performance on unrelated tasks through catastrophic forgetting. On-policy distillation can help preserve general capabilities, while LoRA adapters allow one shared base model to support several interchangeable domain-specific specializations without maintaining a separate full model for each use case.
Takeaway: Use adapters or capability-preserving training methods when specialization must coexist with general-purpose performance.
Links shown: Learning to Replicate Expert Judgment in Financial Tasks, On-Policy Distillation
What was talked about: Narrow domain models require custom benchmarks, success criteria, and reward environments because public benchmarks rarely measure the behavior that matters to a specific product. Reinforcement learning environments reduce labeling work by requiring questions and reward criteria rather than a precise reference answer for every example, while production evals can later become the foundation of a training environment.
Takeaway: Build evaluations from the start because the same measurement system can support product monitoring, prompt iteration, and eventual fine-tuning.
Links shown: Learning to Replicate Expert Judgment in Financial Tasks
What was talked about: A homegrown evaluation suite for extracting family relationships, events, people, and locations uses expected structured outputs and metrics such as false positives and false negatives. Claude helped develop the harness and refine prompts, while CI rules keep test examples from leaking into prompts and creating misleading gains through memorization. Automated prompt optimization with GEPA produced little improvement over an already mature prompt.
Takeaway: A tailored test suite is essential for reliable extraction work, and evaluation examples must remain isolated from the optimization process.
Links shown: Learning to Replicate Expert Judgment in Financial Tasks, GEPA
What was talked about: Generated family-history prose is evaluated across dimensions including missing facts, invented details, correctness, and readability. Gemini 3.1 Pro produces the prose while Claude Opus acts as a judge, but the initial tests are too easy and produce consistently high scores. Subjective quality ultimately requires feedback from real users because language models often approve the same stylistic tendencies they generate.
Takeaway: LLM judges are useful for early qualitative checks, but difficult examples and human ratings are needed to measure taste and writing quality credibly.
Links shown: GEPA
What was talked about: General evaluation frameworks often provide too little useful abstraction for highly application-specific needs, making a small custom harness easier to build and refine with a coding model. Specialized ecosystems such as RAG evaluation can still benefit from tools like Ragas, but no single framework yet appears to cover the full range of functional, subjective, and optimization-oriented evaluations.
Takeaway: Choose an established framework when it closely matches the task; otherwise, a focused custom harness may be simpler and more adaptable.
Links shown: GEPA
What was talked about: A token-profiling tool drills into individual agent artifacts and tool calls instead of only showing aggregate spending dashboards. It estimates usage with an OpenAI-compatible tokenizer, reveals expensive test commands and runaway loops, and can expose repeated MCP call sequences that could be bundled into more efficient tools. Its output could eventually feed back into agent steering for automatic context and tool-use improvements.
Takeaway: Measure token consumption at the tool-call level to find concrete opportunities for shorter outputs, cleaner context, and fewer redundant operations.
Links shown: Token Profiler
What was talked about: Agent-generated code can be reviewed through file summaries, function signatures, module responsibilities, and interface contracts instead of reading every line of every diff. Strong module boundaries let an agent work inside one component without understanding unrelated internals, while failures remain isolated and easier to inspect.
Takeaway: Architectural boundaries and explicit contracts make agent output easier to review than exhaustive line-by-line inspection.
Links shown: Dax on reviewing agent changes, OpenCode, Token Profiler
What was talked about: Large groups of coding agents can generate excessive code, coordinate poorly, and damage a codebase when their responsibilities overlap. A more controlled design assigns each agent a specific module, service, or domain and uses a coordinator to route cross-boundary work. Microservices and hexagonal architecture provide possible boundaries, with ports and adapters keeping business logic independent from models, databases, and external APIs.
Takeaway: Multi-agent coding is safest when every agent owns a clearly isolated domain and integration happens through explicit interfaces.
Links shown: Token Profiler, Dax on reviewing agent changes
What was talked about: Coding agents can periodically audit a repository against rules in AGENTS.md, while CI checks and linters deterministically enforce requirements such as avoiding inappropriate regular expressions. Separate review agents can focus on code quality, business logic, or security instead of attempting one broad review. Fresh audits for code smells, god objects, and duplication can catch structural drift that implementation agents miss.
Takeaway: Convert important architectural rules into automated checks and use narrowly scoped review passes for concerns that require model judgment.
Links shown: Dax on reviewing agent changes, Cloudflare Template, skill-deslop
What was talked about: Some code duplication can improve locality, clarity, and context efficiency now that generating and updating code is inexpensive. Keeping logic close to the module that uses it reduces repository navigation and avoids distant abstractions, although large duplicate implementations remain risky when they are expected to evolve together. A proposed benchmark would compare architectures by agent token use and ease of navigation rather than correctness alone.
Takeaway: Prefer clear, local, and explicit code over abstraction for its own sake, while centralizing complex logic that truly needs one source of truth.
Links shown: skill-deslop, Write Boring Code
What was talked about: A new Tencent model with roughly 300 billion parameters appears competitive with GLM 5.1 despite being significantly smaller than GLM 5.2. Early impressions suggest Tencent may be becoming a more consequential participant in the Chinese open-model ecosystem, even if this release is not yet a major frontier event.
Takeaway: Tencent is worth watching as another Chinese lab approaching frontier-class performance with comparatively efficient models.
Links shown: Tencent Hunyuan HY3 model post, LLM-powered Tom Riddle diary
What was talked about: A physical project recreates Tom Riddle’s diary from Harry Potter using handwriting recognition, an LLM, and an e-ink display. Written input is captured with optical character recognition, processed by the model, and answered directly on the page as the original writing fades away.
Takeaway: Combining language models with tactile interfaces such as e-ink and typewriters can produce playful, compelling AI experiences beyond conventional screens.
Links shown: LLM-powered Tom Riddle diary
10:52 It looks good. I didn’t hear you say anything.
10:54 Okay, yeah, I had it on mute for a second there.
10:57 I just turned it off. So we hopefully should be hearing me now on the live stream.
11:06 We’ll see if that happens. Yes, I can hear you fine on YouTube.
11:12 Okay, great. Okay, so then with that, we’ll get started.
11:18 Welcome all to Tools Club for another week.
11:22 As per usual, toss anything you want to see in the chat that you want to discuss or any comments.
11:28 I’m on my laptop today, so I will not be able to as easily go and see the chat in Discord and stuff.
11:34 So feel free to just unmute and just chat in here will probably be the easiest way to go and do that.
11:42 So yeah, getting started with this week, I wanted to actually talk about fine-tuning models.
11:49 So Bridgewater, these guys are a big hedge fund, one of the most successful hedge funds out there.
11:56 They obviously use a bunch of AI and machine learning stuff under the hood behind the scenes.
12:01 And so recently, LMs, though, in the financial space have not been used too much.
12:06 Usually they’re not able to do anything or provide any insights that these companies weren’t already getting from their data sources that they already had.
12:15 But Bridgewater, in combination with Thinking Machines, Mira Marati, former OpenAI co-founder, and their startup, they were able to go and fine-tune a model that goes
12:29 and basically they tried to imbue financial taste into a model on what is important and what is not.
12:36 And so they were able to go and fine-tune, I believe this is like Quen 3.5 300B or something like that on this task.
12:44 And by doing so, I think they have a good chart here.
12:47 Yeah. They were able to make it much cheaper and much better than any of the Frontier models that are out there right now for this sort of thing.
12:56 And so this is actually individually, this is cool, but not really noteworthy.
13:02 What I wanted to discuss and sort of see what the community thinks right now is around fine-tuning models.
13:09 For a long time now, basically the last probably three years, I’ve told people that I’m working with, like consulting with, or just like advising on the products that they’re building that we should not be fine-tuning models
13:22 right now. You should go and be doing literally anything else.
13:25 The last thing you should be doing is going and trying to fine-tune an LLM because it’s very expensive and historically been very difficult.
13:32 And that you have all these other levers of like rag and prompt engineering and optimization and context engineering.
13:42 Those have all been more powerful and more easily accessible levers for you to be able to go and use to go and improve the performance of these models.
13:52 What we’re now seeing, though, is that as the field has developed, people are coming up with very concrete use cases for their models and sort of like are able to very well define and get a lot of data around these use cases
14:06 that they already have that’s already labeled from production.
14:09 And now we’re seeing with a wide variety of companies, we’ve talked about this a few times before now, but they are getting huge bills from LLM providers, you know, as their engineers and products are just burning through
14:24 tens of thousands, if not hundreds or millions of dollars worth of token every month.
14:30 And so they’re now actually looking at fine-tuning these models.
14:34 This is also spurred on by products like Thinking Machines Tinker and Prime Intellect Lab, which allow fine-tuning these models to be much more easily done, where they abstract away all of the sort of infrastructure
14:49 setup. And you just need to basically give it the data and say, here’s the way I want to feed the data through the model and how I want to score it to give the model reward.
14:58 And then they handle everything else behind the scenes for you.
15:01 So it’s becoming much more practical to actually go and fine-tune these models.
15:05 I still think that you should obviously utilize these lever, the sort of initial easier levers more.
15:12 This is still more work, obviously, than just optimizing prompts or putting in a half-decent rag system.
15:19 But this is now not becoming something I only recommend the sort of 1% of use cases.
15:24 And I think this is now right now, probably about 10% of use cases, could leverage fine-tuning.
15:29 And I think in six to 12 months, we could be seeing a lot more fine-tuning coming around as these services become more mature, and the base models, like
15:43 the open source models, are good enough to actually be worthy of fine-tuning.
15:48 So yeah, I wanted to see sort of like what the community had to say and what you guys had to say in terms of fine-tuning.
15:53 I know a few people here I’ve discussed fine tuning with before, but I think this is in the next six to twelve months going to become a much bigger deal.
16:01 So yeah. Like I know, Brandon, you were.
16:08 Oh yeah, no, Quentin, you can go. So yeah, Andrew, you were saying that this could be a cost-saving measure.
16:14 Is that because you would be fine-tuning an open source, a lower-cost model to just go spiky on whatever you wanted?
16:22 Exactly. So like you are fine-tuning the model.
16:26 And yeah, it’s a smaller model. Doing inference on a model yourself will always be cheaper than using the frontier models.
16:36 Even like, yeah, if you’re using a third-party provider, like if you’re using thinking machines, they will host your model that you fine-tune for you.
16:42 They charge $5 or $6 per million output tokens, whereas GPT 5.5 is $30 per million output tokens.
16:49 So that’s why you see large cost savings.
16:51 These open source models are just cheaper to run.
16:54 And they’re much more price competitive because everyone can run them.
16:59 All right. And the idea is that fine-tuning can bring it up to what you expect from the leading edge state-of-the-art models in some particular area for your use case.
17:11 Yeah, the gap between the open source and closed source models has been shrinking, especially on a lot of these easier tasks, because the frontier models, they’ve saturated these problems for a while in terms of what they
17:24 can feasibly get out of it with their training data.
17:26 And the open source models, they’ve just been playing catch up to sort of like that high water ceiling that’s been reached.
17:31 And so the gap is already relatively small.
17:33 And then when you go and add in your own data that you can fine-tune with that is outside of the scope of these base models, like stuff they haven’t seen or thought about before, essentially, you’re able to give the open
17:44 source models a boost to actually outperform the frontier models.
17:49 I still think as of right now, this is only easily applicable in smaller or more narrow domains where things are more well specified or are very out of distribution for frontier models.
18:03 But I think this will become viable.
18:05 Because right now, for this instance, they went and fine-tuned, I think it’s like Quen300B.
18:12 Quen 3.5, 397B is the model exactly.
18:17 But in the future, you’re probably going to be able to get away with fine-tuning the 30 billion parameter model, not the 300 billion parameter model, which will be about 10x cheaper than this.
18:27 So yeah, I think this will only go down in price because we’re starting to run into these problems where you need domain-specific data to be able to do well in them, that you can’t just necessarily get with only rag or only
18:39 prompting. So we’ll start seeing these use cases open up more.
18:45 Andrew, just a quick question. If I remember correctly, one argument against using fine-tune models was that in the general big enough models, which can be more
18:59 generalist, their performance will drop in unrelated tasks.
19:06 So if you find in a model, maybe it performs slightly better on your specific use cases, but then it will perform quite worse on other tasks.
19:17 Is that still the case? Yeah, that is still the case.
19:20 It’s actually interesting that you bring that up, because thinking machines, they’re the ones that came out with, I believe what’s this, connectionism somewhere over here.
19:29 I believe it’s on policy distillation.
19:32 But yeah, they talk about this catastrophic forgetting is what the phenomenon is called.
19:36 Yeah, the model, you find tuna on one domain, but all the other domains, like, they have their sort of intelligence sapped away so that the model can perform better in the new domain.
19:46 There are ways of going and getting this capability back using on-policy distillation, which is a very good read.
19:53 I think this is also becoming a very strong lever for people to use during fine-tuning.
20:00 But yeah, the reason they don’t talk about it in this blog post from Thinking Machines that they did with Bridgewater is they are training this model for this specific narrow task.
20:10 And so they don’t care about the general reasoning capabilities of the model.
20:14 Interestingly, the way that thinking machines and also Prime Intellect Lab, the way they train their models is using LoRas, which if you don’t know, LoRas are little like small weight adapters you would add onto the model.
20:25 So you still have the same big, large base model you load in, and then you add in these little adapters that fine-tune the model or apply the fine-tune to that model.
20:35 So you still have access to the original base model for general questions, but you can add in these LoRas as needed for specific use cases.
20:42 So that way you can only run, or you only have to run one big model, then with a bunch of LoRas, fine-tune four specific use cases.
20:49 And that way you have four specialized models and a general model that is being run.
20:57 Andrew, and just to go off of that, don’t you think that a lot of the problem with fine-tuning models is actually just defining your loss function itself?
21:08 Because in this case, for example, how do you even use a public benchmark to assess the model capability in a very narrow domain?
21:20 You essentially have to create your own, right?
21:23 Oh, yeah, correct. Absolutely. These sorts of problems, you have to have your own evals that are measuring what you care about.
21:31 You will not be able to go and like Bridgewater, they would not be able to figure out how well these benchmarks do or how well these models do for their use case if they didn’t go and build out their own benchmark.
21:40 So yes, defining your own benchmark, defining what success looks like, and then building an environment where the model can get rewarded for sort of doing things correctly in that environment.
21:50 That’s what it all is about right now.
21:53 Which is, it sounds like a lot, but it’s better than it was before.
21:56 Before is a lot of reading through data.
21:59 I mean, it still is a good bit. But RL environments have made it.
22:04 So you only have to really define the question and then the reward criteria.
22:07 You don’t have to define what precisely the correct answer is for every single question.
22:13 But yeah, no, that’s yeah, that’s still that is the thing that you are doing, is defining what the metrics are that define success for the model.
22:30 And so that’s why, like, for when I’ve been working with companies on rolling out sort of like AI-powered products, one of the things that I emphasize heavily that you sort of need to build out of the gate are evaluations.
22:44 I think if you’re building an enterprise or sort of like customer-facing AI product and you have no way of evaluating it, either before you actually go and release the system to the users, or while the system is actually
22:57 running, you have no way of measuring it at either point, then you’re just sort of flying blind.
23:02 Like any software engineer, if you deployed a regular software system like that, would think you’re crazy, you know, where you’re not actually capturing any user metrics, things are succeeding, if things are failing.
23:13 It’s not very smart to go and do that.
23:14 So I think this is sort of like a part of the standard operating procedure when you’re using AI and building AI products is this eval.
23:22 And now you’re able to go and take those evals and convert them into a training environment fairly easily to go and be able to go and train a model to do so.
23:35 Yeah, that brings up a third question of evals and email frameworks.
23:40 That’s it. Do you want to discuss that at all, Brandon?
23:42 I know you’ve been diving into that a bunch lately.
23:48 Yeah, I’d be curious to see if anyone else has any insights.
23:53 Interesting space. I don’t know if there are that many good frameworks out there yet.
23:57 We’ll get a lot of pretty questions.
24:01 Has anyone here built their own AI evals and have any sort of opinion on that experience at all?
24:10 I have a ton of evals that are about extracting family relationships and history events into a structured data structure.
24:23 And so I’ve got basically tons of either short sentences or longer sets of sentences with expected output.
24:31 And if I didn’t do that, I would be dead.
24:34 I could never, ever make any progress without this.
24:41 I started off with something super simple, and Claude has helped me make it, so I can barely understand all of the metrics that are being applied for false positives, false negatives.
24:55 It goes super deep, and it’s gotten my extraction working really well.
25:02 Nice. One of the nice things is, yeah, the Frontier models are fairly decent at least sort of like defining some metrics for you to be able to go and sort of like measure how well something is doing for your problem.
25:14 They’re not always necessarily correct or the best, but they definitely, if you don’t know where to start, you can definitely, oh yeah, they’re very, very good, very strong to leverage for that sort of use case.
25:26 Yeah, my topic, I think, is definitely in, like, way, way in the distribution.
25:32 This is not obscure stuff. If you say my mother, the model knows what mother is and what that implies.
25:42 So it’s pretty straightforward. It’s just the vagaries of language syntax when you’re talking about a bunch of family members.
25:52 How do you assign how do you extract from complex sentences?
25:59 That’s really where a lot of my work has gone.
26:04 Are you using these evals to help build out the harness and the prompts essentially that you are using?
26:11 Yes. Okay. Yeah. That’s why I’ve got some CI rules so that at least the exact tests aren’t leaking into the prompt.
26:23 That happened to me early, and then I realized why some of my numbers were getting so good because it was just memorized.
26:32 Yeah, that’s one of the things is, yeah, reward hacking is very much something the LMs are not afraid to do and you have to be aware of.
26:40 I do know, are you doing this yourself or automatically?
26:44 So I know there’s prompt optimizers like JEPA.
26:47 I couldn’t get anything out of JEPA.
26:51 So this is all me and Claude. And basically, I have some ideas.
26:57 Well, sometimes I throw some white papers at Claude to say if it’s useful.
27:03 I think we made some good progress.
27:06 Here I am talking about my imaginary friends.
27:08 But yeah, it’s been all homegrown. But it’s so important to this project, I couldn’t do it without it.
27:22 That’s also surrounding JEPA. That’s sort of the sentiment that I’ve heard, is that sometimes it works, but people mention it, it’s like, oh, it’ll just automatically prompt optimize anything for you, and it’ll just make
27:34 it better out of the box. But that’s not the case.
27:36 It sometimes works. A lot of the time, it doesn’t really do much at all.
27:40 That’s right. And that was the promise that I heard.
27:42 It’s like, oh, that’s cool. But I had already had Plod working on the prompt optimization for months at that time.
27:50 So it was not really something simple-minded that I threw at it.
27:54 It was already well-refined before JEPA had a chance.
27:59 So I didn’t give it any easy things to work on, considering that.
28:05 Yeah. Yeah, that’s super cool. So my latest part of the evals is, because I want family histories to
28:19 be extracted from the facts and the stories that you tell the model, is how do you judge the prose that is created from the facts that you enter?
28:32 Is it like good prose? Is it easy to read?
28:35 Is it correct? Does it add any crap and so forth?
28:39 And so that’s the latest level of eval to make sure that the draft prose that the app is creating is reasonable.
28:48 And that is much less mature than what I mentioned earlier, which was about extracting facts, people and relationships, locations, and so forth.
28:59 And how are you going about that? Because I know that is sort of like you’re trying to measure taste.
29:05 How good is the model writing? Is it slop or not?
29:07 What have you found works for measuring that sort of thing?
29:10 So I think a lot of people are interested in that.
29:13 So let me see. I could be more prepared next time, but there are five dimensions.
29:19 And the one dimension where I needed to use a different model was basically the hallucination dimension.
29:28 So one dimension is, are you missing things?
29:32 One, did you add things that didn’t exist?
29:35 And then I forget some of the others.
29:38 But I have, I brought in Opus to, because Gemini is my main model, 3.1 Pro generates the pros, and I have Opus checking to make sure that it’s
29:53 in good shape. But it came in sort of pre-saturated.
29:57 Like, I’m having a hard time getting bad pros with these tests.
30:05 So I’m not sure if I’m not trying hard enough.
30:07 I’m still working on it. But it came in in most cases good scores.
30:13 Like, oh, I need to try harder because it’s no good if everything is an A.
30:18 It’s yeah, that’s one of the things when I’ve been building out sort of evals with LLMs is they tend to produce relatively easy evals.
30:27 Like a lot of them that I’ve gone and made are usually like 80 or 90% accuracy, just like out of the box for the eval that you get with no tuning of anything.
30:36 See, I find that they really struggle with those sort of like really deep and nuanced evaluations to really suss out where the issues are with the model.
30:46 Yeah. And the next phase for the pros evals really is going to have to wait until I have a bunch of users so that I can have people who aren’t me looking in this saying this is not crap or this
31:01 is crap or something about it. Because now, you know, I think all my stuff is good.
31:08 And Gemini is happy with it, but that’s not really the point.
31:12 It’s got to be good for humans. And then final question, and then we’ll move on here.
31:19 But it’s how, are you using LLM as a judge for evaluating the pros or using some other method?
31:26 Okay. No, it’s absolutely LLM as a judge.
31:30 Okay. For the extraction, it can be very crisp because they’re specific facts, relationships, and events.
31:41 But here, it’s LM as a judge that is much more subjective.
31:48 It’s yeah. Okay, cool. It’s yeah. I know, yeah, LLMs, as a judge for like pros and like the issue is that they are the slop creators, and they initially think that the slop sounds good.
32:01 And so I find it interesting. Yeah.
32:05 Trying to measure sort of like quality and how you define quality becomes very specific for the judges.
32:14 So yeah, that’s super cool. Actually, quick question.
32:20 Are there any frameworks that exist for evails, or is it literally just like you’re just saying, oh, like have LLM look like a second LLM check?
32:28 Or have you just created the evals from scratch?
32:33 In my case, this has all been wired together for my particular app.
32:38 It’s all homegrown. I looked at some of the frameworks, and it didn’t seem to be a good match.
32:47 No criticism of it, but I just had Claud get it up and then refine it so it’s exactly what I need.
32:56 That’s also the general sort of like sentiment in my use, whenever I’ve built out Envalus as well, I’ve sort of started from scratch on my own rather than use an existing harness or anything like that.
33:08 I find they offer sort of like a little amount of abstraction that’s useful.
33:14 And just being able to sort of like vibe code and just sort of like set up the infrastructure myself and have it all custom exactly the way I want it to is just easier to go and do myself than using a framework.
33:25 As I know, Brainerd Schneider, you’ve been looking at eval frameworks, right?
33:30 And from your comment earlier, I assume none of them are that great.
33:37 It really depends what you’re trying to do, right?
33:39 If you’re trying to do Raggedy Dell, Ragas, or DPDAL, or keywords in that space, if you’re already plugged into a different ecosystem, then probably using that ecosystem makes a lot of sense.
33:53 But I haven’t found one that’s kind of in the final exam.
33:57 A lot of them have similar shapes, right?
34:00 So show across the board. They’re also triplets and then all sorts of different decisions you’re making, right, on how you optimize the emails because not just functionality.
34:14 It’s like a lot of complexity and nuance.
34:17 While I have found a really good format for happy style wikis, the open knowledge format from Google.
34:24 I haven’t found a unified one for details yet.
34:28 Cool. Good to know. So yeah, moving on to other projects from the community.
34:38 Brandon, I know last week you talked about wanting to present what you had been working on.
34:43 Not to put you on the spot here, but are you still interested in doing that today?
34:48 I’d be happy to. I’m actually walking in the rain right now, so I can’t really present it.
34:53 Okay. Yeah. But I mean, just yeah, like again, just I made a small post about it.
34:59 The idea here was just that I didn’t really see any tools online.
35:03 I know there’s like a lot of people literally trying to do the same thing where it’s like, oh, let me just show this pretty dashboard with how fast you’re burning tokens per week.
35:16 And I was just like, honestly, I mostly care about the artifact drill down.
35:20 And specifically saying, oh, like which individual tool calls are actually burning all my tokens?
35:27 Is there an NPM test for both that is burning all my tokens?
35:32 And so I just built a tool for that.
35:35 Nice, very cool. I’m currently digging through.
35:38 Was it in general that you posted it or somewhere else so I can get the link for people?
35:42 AI tools. AI tools. Yeah. Cool. Yeah.
35:50 See, I found this was interesting. Yeah.
35:52 Oh, sorry. Now you go. Yeah, I was just saying, like, it is still very rough.
35:59 And at the end of the day, it’s still an estimation.
36:05 But the fact that using the tokenizer that OpenAI claims to use for their GP models, it should still be pretty close just because it’s
36:19 based on the proportions. So yeah, if this is useful for anyone, if you’re curious to see what’s happening, why a certain session burn a crap ton of tokens, or honestly, just be
36:33 more aware of your context hygiene, because I’m sure when you run loops, it easily just ends up becoming tens of millions to hundreds of millions of tokens per session.
36:43 I think this is a good kind of a good tool, or a potentially helpful tool to just see what’s going on.
36:49 And one of my thoughts that I would be interested in using this for is identifying when certain sequences of, let’s say, MCP servers, where it’s like, oh, the model usually always calls these three tools from this MCP
37:04 server in a row. And can I then just go and bundle those three into one tool call that the model has to make instead of three?
37:10 And stuff like that. And what efficiencies can we find to sort of optimize the tool calling landscape that the model sees?
37:20 Yeah, and yeah, and I mean, to that point, the other thing that I was thinking about, I actually take a step back from working on that for now, but if people find it useful, my immediate next step was also
37:34 thinking, how can we make it so that the outputs are digestible to just put back into a loop, right?
37:43 So then let’s say you have some sort of hook that triggers this, and then you can see, oh, agent can improve their next session.
37:51 Say, okay, we’ll reduce, we’ll change the steering a bit to just, I don’t know, like call like certain tool calls less.
38:03 Or a big one for me was I realized, and I feel like this is kind of like a simpler case, but for the most part, test calls don’t really need to burn more than just one line tokens.
38:19 Just because all you need to do is just see, oh, like which tests are failing, and then look at those individual tests.
38:25 So yeah, kind of just hopefully helpful.
38:31 And honestly, it’s just hopefully useful at this point.
38:35 You know, Brendan, it seems like this would be a good capability to have in the frameworks like Cloud Code or Codex to self-introspect because they’re
38:49 running in various environments. Well, it’s maybe not in their interest because they sell tokens, but it would be very cool to have it be smart and maybe have some skills that just figure it
39:03 out, like, look, this is definitely not paying off.
39:07 We just dropped a million tokens for very little benefit.
39:11 Yeah, absolutely. I mean, honestly, the whole thing that spawned this is this fact that Codex doesn’t show any information on individual toll falls.
39:21 How many tokens they burn. But yeah, honestly, I think probably the bigger issue is just that there’s not really any specific outspoken community interest probably towards the specific tool calls.
39:33 Because I think as far as most people are concerned, they’re just like, oh, I just want my codec subscription to burn less tokens and for me to not run out on a $20 plan.
39:46 But yeah, I think as time goes on, I feel like, especially as GPT 5.6 comes out, I’m sure there’s probably going to be a subset of people who are just very, very interested in just diving into
40:01 how can I get the most out of more expensive models so that I don’t have to I can have the best of both worlds, right?
40:08 Like burning less tokens, but also not having to switch to a lower quality model and deal with lower quality output.
40:22 And yeah, I think, yeah, this is super cool.
40:24 I think there’s also sort of like a field outside of like codex and cloud code of just people who are building harnesses in general.
40:31 Even if it’s not this cool, I think these are the ideas and like the statistics essentially you need to be looking at when you’re building a harness and like getting insights into it, like talking about sort of like evaluating
40:42 models. This is one of the axes that you should be evaluating models on.
40:47 It’s sort of token usage and like optimal tool calling, that sort of thing.
40:52 So yeah, I think this is useful and something that people should be thinking about across the board to help minimize their token usage and maximize what they’re getting out of these models.
41:03 Yeah, super cool. Thank you for sharing that, Brandon.
41:06 I mean, thank you for bringing it up, Andrew.
41:10 Okay. Moving on now, still in the sort of like evals world.
41:18 I saw this this week, which is Dax.
41:20 This is the guy. He’s sort of, I believe, the lead of Open Code, which is like open source clawed code, essentially, if you haven’t heard of it before.
41:30 Meant to work. They have a really cool Go coding plan.
41:34 Wow, this page still doesn’t work. It’s the CSS on this page has been messed up for a while, but it’s called Open Code Go, and it’s like $10 a month, and you get access to pretty much all of the latest and greatest open source
41:46 LLMs with really generous rate limits and also very transparent rate limits where you can see, oh, you get 500 requests every six hours or whatever it is.
41:57 But anyway, he was talking about what he’s doing right now for code review.
42:02 And I thought this was interesting.
42:04 I wanted to see what the community right now is doing for that.
42:08 So he’s saying that he has moved on from just reading entire diffs that the agent outputs and instead have it go and sort of like summarize, here is what I did in each file.
42:22 And then based on those summaries of what it did, he’s able to suss out whether or not it did sort of what he wanted it to do in the way he wanted it to do it or not.
42:32 So yeah, he says, yeah, like just the general files that it’s operating within, and then also the function signatures of what’s getting passed in and out.
42:39 That’s what matters, but the actual implementation matters less.
42:44 So yeah, I thought that was interesting.
42:45 And I wanted to hear what people are doing right now for code review, if anything at all.
42:50 Like, is everyone just vibecoding nowadays and just not caring about it at all, what the code looks like?
42:56 Or do you have something else that you do?
42:59 Yeah. Oh, actually, I have some thoughts here.
43:05 So yeah, honestly, I feel like I’m still kind of behind in some ways where it’s like, I see other people talk about spinning up seven sub-agents to all code.
43:16 I’m like, crap, I’m still looking at one and just stressing out over, oh, crap, like, is this one agent even doing what I’m asking it to do?
43:26 And so just the general way I’ve been approaching it is I’ve kind of gotten into the habit of, and you can actually kind of see this in the token profiler project.
43:35 But I actually, so I’ve more focused on kind of like specific modules where I just make sure very, like very, very, like, very, very clean separation of responsibility.
43:47 And then I have just documentation on each contract that the module is supposed to follow.
43:52 So that the idea here is any given task that an agent executes, it should not ever go understand the internals of another module.
44:04 And it just assumes that that contract is correct.
44:07 And so the general way I evaluate whether the code’s doing what I want it to is whether the code still just respects that module boundary, like the contract.
44:17 And in the worst case scenario, if something goes wrong, then it’s isolated to one specific module.
44:27 Nice. Yeah, I feel like this aligns very much with sort of Dax’s take here, where he says, yeah, like files and functions and their signatures are what matter.
44:35 And yeah, maintaining that contract between all of them.
44:38 I know this is like the general structure that I have for a lot of my templates, which is where I start from, is sort of trying to imbue the structure and sort of like contracts similar to this into them and then let the
44:49 agent sort of like run wild within that.
44:50 And as long as it still adheres to those rules, I don’t care too much about what the code looks like under the hood.
44:58 So this is cool. I’m also of the same mind as you, Brandon.
45:04 I am not someone that spins up a bunch of agents usually.
45:08 I’m usually one agent doing something, and then I go and spot check it, make sure everything looks good, iterate on that, and then move on, as I don’t have it building massive features all at once.
45:20 But I do remember Jordan and I were actually talking about this in the car on the way to the retreat, how he and Arhennis, for Zero Claw, they had some Chinese bite dance engineers
45:35 who just had seven agents or seven teen agents, some crazy number of agents constantly just working on loops and saying, oh, build this CI pipeline.
45:49 And then it ended up just mangling their code base so badly that they had to just completely just revert back 100 commits.
45:59 Yeah, I know with those multi-agent systems, they end up being, in my mind, a lot of token slop.
46:04 Where I would trust multi-agent systems if they wrote the minimal amount of code by working together.
46:10 But I feel like with multi-agents, they end up writing the maximum amount of code to achieve a solution.
46:15 And like you said, it’s mangled all over the place.
46:17 They don’t communicate well between each other.
46:19 Which is why I need to be the lead more.
46:22 To that point, honestly, what I was thinking was literally a few days ago, I was actually thinking, if you’re kind of of the same mindset, Andrew, of this idea of very, very clear separation of boundaries and responsibilities,
46:37 I was actually thinking about the idea, I feel like an ideal multi-agent system would be you delegate to separate sub-agents, basically each sub-agent has its own domain of responsibility, right?
46:50 And then you would have a larger agent that just says, okay, this is the task.
46:55 Never deviate from certain boundaries.
46:57 Sub-agent. But each sub-agent, any time they need to share, they communicate with each other.
47:03 I feel like that might be a cool potential avenue to explore.
47:07 The idea of agents with module-specific boundaries.
47:12 So it’s each agent manages its own domain.
47:15 And I’ve heard about this in the context of everything should be a microservice now because of it, where you try and chop up your application to a bunch of small pieces, and then you have one agent that’s responsible for
47:27 that microservice. So then when you give a feature request, your coordinator agent goes and talks to all these different microservice agents that go and get, like, they own, I don’t know, like the user’s table and all the
47:38 auth logic. They go and talk to that guy, and then they talk to, I don’t know, like the dashboard agent who goes and builds out the dashboards and everything like that.
47:47 Well, as someone who hails from literally the creator of the microservice architecture, I have been thoroughly indoctrinated.
47:58 As a microservice hater, it’s interesting.
48:02 It does sound like a pretty clean paradigm.
48:05 I think we talked about the previous two weeks, actually, about multi-agents and how if you want to do it well, they need to have clear boundaries on when you switch from one to the other.
48:13 Microservices like that do give you pretty clear boundaries on when you should switch from one to another.
48:20 So, yeah, it does help in that regard for that sort of thing.
48:24 So I think it could be viable. Actually, I guess one last thing I would say on this, something that I came across while I was doing this that honestly, I generally put in my honestly all my projects now, anyone
48:38 that has any level of complexity is the idea of the hexagonal architecture or the ports and adapters architecture is another name for it.
48:48 And I’m a big fan of that just because it’s kind of just like the whole microservice architecture, except the idea is you separate the business logic, and then you kind of isolate everything to its own domains.
49:01 But then whenever you have it plugged into other things, for example, you want to have a different model, or you want to store something into a database, or you want to make an API call, that all is abstracted to the point
49:12 where all you need to do is build an adapter class that hooks up to that.
49:16 But for the most part, the business logic itself shouldn’t need to change.
49:20 And so I’ve been kind of putting that as enforcement in all my projects so that it’s very easy to upgrade models or very easy to just say, oh, I decided I want to use switch
49:36 from using DynamoDB to SQLite or something like that.
49:41 So something else that other people listening on the call could consider.
49:48 One of the things I’ve also noticed for all these different architectures is it doesn’t seem like most of them will work with LLMs in one way or another.
49:59 The main thing is just having one and having some rules for the agent to go and follow.
50:03 So that way it’s able to sort of consistently apply them across the entire code base.
50:08 I think that’s sort of like the most important thing.
50:10 If you’re listening to this and you’re like, I don’t know what this is, I don’t know.
50:12 I’ve never dealt with microservices.
50:13 I don’t want to deal with microservices.
50:15 But you have your own idea of sort of like how to architect things.
50:18 I think you should go talk with your local LM and design your own architecture.
50:22 And I think across the board, it will help having that level of structure and keep your code bases at least a bit more sort of in bounds than if you just let it go completely wild without any framework or structure.
50:36 And you could always ask your coding agent to assess any divergence from that architecture.
50:44 Like, where have I gone away from that?
50:50 Yeah, exactly. It’s the kind of boring work that you don’t want to do.
50:53 So you have the coding agent do it.
50:57 That’s something I actually have in my templates.
51:00 Or rather, like my templates have all these rules defined in them.
51:03 And then I’ll periodically just go and tell the agent and be like, all right, go look at the rules in the agents.md file and see if they are still being correctly applied across this entire code base.
51:14 And if they’re not, go and fix it, please.
51:17 As I know, I think Artem has also sent me, there’s no good way to go and pull it up without leaking DMs, but as I know there’s a bunch of D slop sort of skills out there that people
51:31 use, which basically say the same thing.
51:34 I also think, I believe the Codex code review and maybe also the Claude Code code review now that you have on GitHub with those GitHub integrations, they actually go and read the either clod.md or agents.md and will
51:49 check to see if it adheres properly to those files.
51:55 Like the job that you want to make.
51:57 That’s a promise that they make, and I found that it’s the way it is.
52:05 It’s a language model. So somebody here, it might have been Brandon who suggested it.
52:12 I just put some rules into CI to catch things and that has been super effective.
52:24 In my case, Claude was so comfortable with regular expressions when I wanted everything to be up a level, like semantic understanding.
52:33 Don’t put in some English language regular expression.
52:37 As soon as somebody starts talking Spanish, it breaks.
52:41 So now I just have something in CI saying, ah, that’s a regular expression.
52:46 If there’s no little escape clause next to it, it’s flagged.
52:52 Yeah, agents are super capable of writing in these rules and adding them to your CI or linters or whatever you run before the pre-checks.
53:02 And yeah, they’re very capable of going and sort of codifying all of the rules that you have for it.
53:07 And so that way you can do a static deterministic analysis on whether or not things work or not.
53:12 And so I actually have some of those that run as well.
53:14 Where I tell the agent, like, okay, once you’re done implementing something, go run these tests, go run the linter, and see what it catches, and then go and fix based on its recommendations, that sort of thing.
53:33 Also, actually, yeah, what I do, honestly, very, very simple.
53:36 I just say, are there any god classes or god objects?
53:40 And that usually catches most of. And specifically, I say, are there any code smells?
53:47 And then a lot of times it can catch stuff that way too.
53:52 And yeah, I feel like, yeah, just giving a fresh agent, just say, alright, go and find all the issues of this code.
53:58 I know one of the big things is my front-end code, I don’t have too much opinions on.
54:03 Front-end is front-end AI is good enough at it.
54:06 But a lot of times there’ll be just a ton of duplicated stuff in there.
54:08 So I’ll literally just spawn an agent and say, go find everything that’s duplicated and condense it all back down.
54:14 Because a lot of times I’ll already have built all the helper classes, but some agent won’t read the right file and we end up with copies of the same thing.
54:23 So yeah, a bunch of simple stuff like that.
54:28 And also going back to the GitHub code review stuff, the reason I think they say it tries to adhere to the agents.empting and textbook adhesion.
54:41 But it doesn’t always is because it’s doing a bunch of other stuff.
54:44 I think having your code review agents be either very strictly on code quality or having a separate one that is meant for actual implementation level, is this correct business logic?
54:57 Separating those out is good and keeping them more narrowly focused on one very particular thing that it’s trying to fix will help it catch it more often than trying to make one sort of like code review agent that does everything.
55:09 We’re just checking security, content, correctness, all that sort of stuff.
55:16 I’m kind of curious, Andrew. I’m curious your thoughts on the idea of.
55:20 I feel like, at least in my I’m actually more open to code duplication moving forward just because I feel like I would re I feel like it makes it sometimes it makes code more clear right like it like
55:36 as long as you you maintain separation responsibility I feel like as long as it’s kind of readable there there’s still benefits to like I guess like duplicating code right even if it’s a little bit uglier from a
55:50 programming perspective where to help with that oh yeah no for sure it’s definitely these LLMs like the reason they’re adding these duplicate files is because they’re sort of where they expect them to be you know so it’s
56:02 not necessarily bad a lot of the times to go and have duplicate files or duplicated logic yeah it’s I definitely for like a lot of smaller stuff it probably doesn’t matter all that much it is probably just a human like
56:16 you know like I have like once going back to front end I know like Claude and I think GPT as well they love making like is record helpers which is just like a one-line check and those get littered everywhere it’s very much
56:29 a human thing where like I don’t like those everywhere you know like I don’t think it affects the model in any meaningful way I think it does become an issue when you have sort of like large sort of complex pieces of code
56:41 that you want to go and modify that are supposed to do the same thing but have two separate implementations you know those are things that should be the same like file so but yeah like some level of duplication isn’t actually
56:52 that bad or harmful to the models I don’t think yeah actually honestly like one contrarian opinion that I have when I compare with some of my colleagues in software engineering is that now that writing
57:06 code is just so cheap sometimes I prefer to have more duplicates and things that are not like centralized because when I want to change something to make sure that the changes are more focused and then you know if I just
57:20 want to implement the same logic everywhere I just ask the model like hey just do this everywhere and I don’t have to like I don’t reuse as much code as I used to when I had to write all the code by myself by hand but
57:34 now you know I can control better what changes when I want to make a change to a function I don’t have to like keep looking everywhere that I use that function make sure that everything works you know I don’t know it’s just
57:47 a behavior change that is kind of funny but it goes against the norm but it’s just something that I find myself doing more and more often and something like that also will help with the context management of the model as
58:00 well because the model will just have all the code it needs is in this one file already even it’s duplicate even if it’s duplicated elsewhere it doesn’t need to know that it exists elsewhere you know if you have like one
58:09 central source it has to go dig through your code base and go and find sort of like where that definition is but if it’s all like a one file or like one sort of like module it’s much easier for it to go and find and dig through
58:19 all of that instead of having to find sort of like the super meta object you know that exists in some abstraction file elsewhere I think actually the kind of the concept we’re converging here like there’s
58:34 actually like I think the term is boring and explicit where it’s just like you you sometimes make code just like more dense you add more kind of unnecessary explanations and classes but
58:48 it just makes it very very easy for both an AI and a person go through the code base understand what’s happening even if that means like just duplicating logic as
59:03 yeah like I said I don’t think there’s really any wrong ways of doing it it’ll actually be interesting somebody should make a benchmark on that of having all these different sort of like multi-agent with microservices or
59:14 hub and spoke or yeah boring and explicit code and stuff like that and see sort of like how it changes how well the model performs and like how many tokens it uses and see if one is sort of like objectively better than another
59:27 I think that’d be a really cool coding benchmark to see where you’re not measuring accuracy per se but rather sort of like token use and like ease of navigating the code base for the model that’d be super interesting that
59:37 would be pretty funny like imagine just like you’re like okay build Google make no mistakes and then just use the different benchmarks it’s yeah basically cool
59:54 any closing thoughts on that otherwise we will wrap things up with a few small topics that I have to highlight seems not so I just wanted to call out that model
1:00:08 released this week Chinese model from Tencent one of the big players over there in China I actually have no idea what they do but they released a model this is around the original like Mr.
1:00:22 was it not Mr Mini Max M2 size around like 300 billion parameters so for reference this is about a little under half the size like 1.5x smaller
1:00:36 than GLM 5.2 and it seems to be around GLM 5.1 levels I’ve been hearing murmurings of like decent things about this model like Tencent hasn’t really produced any notable models previously but this model seems to be fairly
1:00:50 good for its size so I probably won’t cover this directly in the AI news this week I don’t think it is big enough yet but it’s just something to note that it looks like 10cent is sort of like becoming a more reasonable voice
1:01:03 or sort of like a bigger voice in the Chinese AI space and I expect probably like their next model or something will be interesting to go and use and look at so keep an eye out for them in the future let’s say if
1:01:18 anything happens you’ll of course hear it from me but yeah wanted to highlight that there continues to be more labs in China popping up that are producing close to frontier at least for China models and then finally
1:01:32 I thought this was a cool project this week so many of you if you’re Harry Potter’s fan you’ll know Tom Riddle’s diary where you like write stuff and then it gets like answered on the page somebody went and made that with
1:01:42 an e-ink display and an LLM so it will go and do sort of the optical character recognition reads what you wrote and then it will go and write back onto the page the answer for it so I thought this was a pretty
1:01:56 cool little demo here that they have where I think they’ve yeah it fades away and then there is the the answer to it so I thought this was a fun little project for some inspiration for anyone I know like the physical hacks
1:02:11 are always really cool to go and see integrating I know like AI with typewriters some people have done stuff like that so yeah thought this was also cool that’s cool so yeah that’s everything
1:02:25 from me this week looks like we have three minutes any closing thoughts or comments before we wrap things up here well the the Harry Potter power
1:02:39 diary makes me think that you should have a voice that says I’m Mr.
1:02:43 Misix you know good morning it’s yeah the Mises box yeah whenever whenever you you give it a request it says can’t say that’d be a fun hack all
1:02:59 the guidance from that Harry Potter book was don’t trust anything or you can’t see where its brain is of course that’s our whole life now it’s yeah exactly so
1:03:14 yeah it seems like that is all for this week thank you all for attending AI Tools Club be sure to go check out the YouTube channel if you haven’t already the live streams get uploaded there automatically it’s I also try and
1:03:28 make sure that there’s a summary of this whole meeting that gets released on Vector Lab hopefully within like an hour or two after the meeting so that way you can go and get the summary if you missed certain portions of it
1:03:38 today thank you all for coming see you all next week
Subscribe to get the latest AI news in your inbox every week!
