PICK YOUR SUPPORT STYLE
MONTHLY SUPPORT
Reader
$5/mo
Contributor
$15/mo
Architect
$50/mo
Recurring subscriptions auto-bill monthly via Stripe Checkout. Cancel anytime from the receipt email.
What was talked about: Anthropic released beta API features aimed at reducing tool/MCP context bloat: tool search with deferred loading, programmatic tool calling, and examples in tool schemas. The group discussed how MCP server definitions can consume huge context windows and confuse models when many similar tools are loaded.
Takeaway: Deferred tool search should make large tool ecosystems more practical by loading only relevant tools. Programmatic tool calling is especially promising because intermediate data can be passed between tools without polluting model context.
Links shown: Alex Albert thread, Anthropic advanced tool use
What was talked about: The discussion shifted to whether MCP is the right long-term interface for agents. A referenced Hugging Face benchmark found models performed better when calling tools through executable Python code than through JSON-style tool calls.
Takeaway: The group was skeptical of MCP’s technical future. Since models are trained heavily on code, code-native tool use may be more reliable than forcing tools into newer schemas.
Links shown: GAIA benchmark
What was talked about: Allen AI’s open deep research agent was discussed. It is built on a Qwen3 8B model and fine-tuned with reinforcement learning for search-heavy research tasks. The speaker tested it on current model pricing/speed comparisons for GPT-5.1, Opus 4.5, and Gemini 3.
Takeaway: The model looked surprisingly usable for an 8B open model, with low search costs and transparent tool traces. It handled tool failures calmly, though it still struggled when page-reading tools failed or when speed data was hard to locate.
Links shown: AI2 DR Tulu thread, DR Tulu blog, DR Tulu paper, DR Tulu 8B model
What was talked about: The group discussed running specialized small models locally, including on phones. Topics included search-enabled local assistants, free/cheap search APIs, phone inference speeds, Liquid AI’s edge-device direction, and task-specific fine-tunes or LoRAs.
Takeaway: Small task-specialized models are becoming practical. The likely path is not one tiny general assistant, but a shared base model with multiple specialized adapters for specific workflows.
What was talked about: Attendees shared Gemini 3 impressions. It was praised for UI work, especially when given visual references through AI Studio, but criticized for instability in CLI/editor use, crashes, rate limits, and occasional odd behavior in coding agents.
Takeaway: Gemini 3 appears strong for visual/UI synthesis, but its rollout and coding-agent reliability still feel uneven. AI Studio seemed to give better results than some IDE integrations.
Links shown: Google Gemini 3 blog, Gemini 3 enterprise blog, Gemini 3 Pro Image, Nano Banana Pro blog, Gemini image generation docs, Gemini API pricing
What was talked about: One attendee used Google Antigravity’s Chrome/browser agent for manual DevOps-style clicking in Google Cloud and GitHub. It worked for some browser automation tasks but was described as buggy and highly opinionated, with limited model and agent configuration.
Takeaway: Browser agents are useful for tedious manual workflows, but teams still want better control over agent rules, skills, and configuration standards like agents.md.
Links shown: Google Antigravity
What was talked about: NotebookLM was praised for video/slides-style generation, diagrams, summaries of papers and YouTube videos, and possible Google Slides/Drive integration. Gmail’s AI email generation was also mentioned as useful because it can use the full conversation thread.
Takeaway: Google has strong AI product pieces, but they remain fragmented across Gemini, AI Studio, Vertex, Google Cloud, NotebookLM, Gmail, and Drive. A more unified interface would make the ecosystem easier to use.
What was talked about: Opus 4.5 had just been released. The group discussed its broader availability, lower API cost, faster speed, and strong token efficiency compared with Sonnet. Benchmarks suggested a meaningful jump in agentic coding performance, especially on software engineering evals.
Takeaway: Opus 4.5 was framed as a real step up for coding agents, not just a marginal benchmark bump. Its higher per-token price may be offset by using far fewer tokens.
Links shown: Alex Albert Opus 4.5 thread, Claude Opus 4.5 announcement
What was talked about: The group discussed how to evaluate new models, especially for agentic coding. Suggestions included building custom repo/task suites, using rubrics, and having another model judge diffs. Artificial Analysis was discussed as useful for speed, cost, and media benchmarks, but less reliable as a single “intelligence” score.
Takeaway: Public benchmarks are often saturated or overfit. Small, realistic, user-owned eval sets are likely more useful for deciding which model works best for a specific workflow.
0:00 Hoddling.
0:02 It’s interesting you say you don’t think Codex can do that.
0:06 Because I’d actually say, I was just comparing GLM and Sonnet there.
0:09 I think Codex, for the stuff that I’ve been working on, I’m more confident in Codex than I am in Sonnet.
0:14 Like all the biggest tasks and the hardest stuff, I tend to give to Codex because I find for the stuff that I’m using with it, its attention to detail is greater.
0:23 Are you using Codex high or medium?
0:27 I’m just using the defaults, it’s just medium.
0:30 Yeah.
0:30 So I’ve found the exact opposite.
0:32 I’ll only use Codex if I have to, and I’ll use Sonnet until my co-pilot quota is out.
0:42 Interesting.
0:46 Has anyone here used the new Opus model?
0:49 I’ve heard really good feedback on Twitter and in their engineering blog.
0:55 Is it expensive?
0:57 Yeah, it is on the max plan.
1:00 They made it available to the lower tier plane as well.
1:03 The $20 a month plan has access to it now.
1:06 And also for the API price.
1:08 Yeah, and then for the API price, it’s three times cheaper now via the API.
1:13 Because before, Opus was unusably expensive.
1:15 Because technically, Opus 4.1 for a while is the best model out there.
1:18 But nobody used it because it was so expensive.
1:20 But yeah, they realized that was an issue.
1:23 So now, yeah, it’s three times cheaper.
1:26 And yeah, I’d say I agree.
1:28 I haven’t got to use it myself, but everything that I’ve been seeing is that it seems like Opus 4.5 is sort of like the real deal.
1:34 Where it’s just straight upgrade from Sonnet 4.5, like immeasurably so, as well.
1:44 Brandon said, saw an article somewhere that said GLM is cheaper per token, but more tokens needed to get the job done.
1:50 Probably.
1:51 My guess is that the SONNIP models and the Anthropic models in general are actually really good at conserving tokens when it comes to their reasoning traces.
2:03 But I think I’ve mentioned this a few times before, the Chinese models and sort of like first-gen reasoning models, which the GLM series I’d say is, they tend to be a little bit more loose in their thinking token usage.
2:16 And because of that, they use more.
2:19 I think GLM, though, it’s like an order of magnitude cheaper than Sonnet, so it’s definitely still cheaper overall to use.
2:25 But yeah, the token usage is definitely different.
2:27 I believe, if I remember correctly, JLM is also faster as well, which helps me not notice that as much.
2:37 Let me go check it out real quick and see.
2:44 I think the US-based providers like OpenAI and Anthropic also have more of an incentive to kind of get the thinking tokens down because they’re currently maxing out their, you know, all of the GPUs essentially.
3:04 Whereas China, on the other hand, has spent more time trying to maximize their actual silicon.
3:11 And as such, they can probably be a little bit more free with their architecture than the US, perhaps.
3:22 Yes, definitely, the Chinese companies don’t have nearly the same demand on their servers that the US companies do.
3:33 Yeah, like OpenAI is probably handling three orders of magnitude more requests than GLM is, for instance.
3:41 And Anthropic is roughly similar as well.
3:45 So yeah, if you’re using the GLM coding plane, it’s actually 175 tokens per second, which I believe Sonnet, if I remember correctly, is about 50.
3:53 I have numbers somewhere this week, I think.
3:56 Yeah, Sonnet’s, yeah, like 50 to 60.
3:58 Yeah, so yeah, GLM is three times faster in terms of token throughput when you’re using it.
4:09 Cool.
4:10 What time is it?
4:11 It’s 510, so we start around now.
4:13 And just so everyone knows, I am recording this.
4:16 We plan on eventually posting these to YouTube.
4:19 Serge has a big backlog that he needs to work through to post.
4:23 But yes, if you do not want to be recorded, then you may leave.
4:29 Okay, cool.
4:32 I think we can get started for what we’re actually talking about this week.
4:37 So I know previously the last few weeks we had been talking about sort of the agentic coding setups that we had with like Claude Code, for instance.
4:45 And I know a few people pointed out that their utilization of MCP servers drastically increased the amount of tokens being used.
4:54 I think there are some people were, just from like the servers and tools alone, they were overflowing the context window of Claude.
4:59 It is basically unusable.
5:02 And so this is something Anthropic has noted themselves, and they, along with their Opus release today, which we’ll talk a bit more about later, they also released a couple tools to try and improve this experience.
5:17 I don’t believe these are integrated into cloud code yet, because they are still beta features, it seems like, for their API, so they’re still ironing out the kinks in the system.
5:27 But my guess is you could probably make a few changes to get these working, which I’ll go over in a bit.
5:34 But the first one is probably the most important in my mind, which is a tool search tool.
5:40 So what they’re doing, when you go and give a bunch of MCP servers, for instance, to the model, as we saw, you can have tens of thousands, if not hundreds of thousands of tokens in MCP server definitions, which clogs up the context window.
5:58 And so what this tool search tool is doing is that they’re only giving Claude this single tool, this tool search tool, and all the MCP servers are basically hidden behind that.
6:09 So for instance, when Claude needs to go and say like, oh, I need to go and perform a GitHub operation, for instance, it can go and use the tool search tool and say like, oh, give me my tools that I have to go and use with Git specifically for, I don’t know, like committing a file.
6:29 And it will go and pull those tools specifically and give only those tools to Claude.
6:33 So the rest of the servers don’t get seen, don’t clog up the memory, but Claude still has access to all of them.
6:40 And so Anthropic, they see not only does this give a huge token reduction, but it also increases the evals as well, due to, I’ve talked about this before a few times as well, but these models, as you get longer and longer into their, or further and further into their context window, their performance starts degrading very non-trivially, very quickly.
7:04 So yeah, by allowing the model to be only operating in the early on tokens where it’s much better understanding and doing things with those, the actual model performance increases as well.
7:17 So yeah, to go and actually use this, let me scroll back up here.
7:24 Oh yeah, actually another thing, another reason why it’s better is also because ambiguity between tools.
7:30 So for instance, I believe this is like Slack tools.
7:33 Notification send user and notification send channel.
7:36 A lot of the times for the model, because of their naming, it would get confused and wouldn’t know which tool to use and would use the incorrect tool a lot of the time.
7:44 But when it’s searching for it, when it’s specifically saying like, oh, I want to send a message to a user, the search will only return this send user function instead of the send channel.
7:53 Or rather, the send user will be higher rated, essentially.
7:57 So the model knows that this is more relevant to its query and is more likely to use this now because of it.
8:05 And so yeah, there’s just a little flag that you send for the tools that are being passed in, which is this defer loading.
8:16 Yeah, this defer loading, what is this parameter?
8:20 There we go.
8:22 So I believe if you dig around your sort of like clawed config files, you’ll see something like this for the MCP tools that you have.
8:29 And so, yeah, here’s the for like MCP tools.
8:32 You can add this in yourself, if I were to guess, in your cloud configs, if you want to start using this now.
8:38 Like I said, they’ll probably be rolling this out anyway themselves in like a month or two, so you probably don’t need to do it.
8:44 But yeah, if you want to try this out right now, if this is a big issue for you, you can check this out.
8:52 Anybody got anything to add to that?
8:53 Sorry, I sort of like gave the usual AI news pitch of something.
9:01 But yeah, it’s, I guess, how useful do people think this would be?
9:03 Like, how many people here have a lot of MCP servers loaded?
9:14 Totally made, right?
9:21 Yeah, I have a lot.
9:22 I gotta trim them down.
9:23 Or use this.
9:27 Because I know, like, personally, I don’t use too many MCP servers.
9:30 I actually use none right now.
9:32 I have one app, I guess, where I have a custom MCP server that I made, just for calling another tool.
9:39 But I don’t use them too often, usually because of this reason where I feel like they just add a lot of bloat and they don’t get used all that often.
9:47 But having it be able to be something that can just be searched as needed makes it a lot better, I’d say, and makes a lot more sense.
10:00 And then the other two things that they talk about that they are adding in, the first is programmatic tool calling.
10:07 So let’s say you have three different tools that need to get called in a row, where normally, you know, Cloud would call one tool, look at all the outputs, pipe all those outputs into the next tool, and you have a chain going like that.
10:21 Instead of needing to call each tool individually, now with this programmatic tool calling, Claude is able to go and write code essentially to go and chain these tools together.
10:31 So it doesn’t have to see any of the intermediate outputs, which would clog up the context window once again of Claude.
10:41 So yeah, like the example here, they have these three different tools that they would be using.
10:47 This git team members, which gives you member IDs and levels.
10:51 And normally you would pass these in.
10:52 Claude would get the IDs and levels from this command and then make these next tool clause.
10:58 But it’d have all these IDs and levels passed into it when it really doesn’t need to know these things.
11:04 It just needs them to be able to pass them to the next one.
11:07 So what it can do now is it can write and execute code to just go and programmatically take this output and pass it into this without actually having to see those intermediary steps and the outputs from that, allowing it to be more token efficient with sort of these multi-step tool call chains that it needs to go and do.
11:27 And then finally is more on the people making the tool side of things.
11:33 This is useful.
11:35 But they now allow you to go and add examples to the tools that are being passed in.
11:41 So right now, yeah, this is the type of schema that Claude sees to be able to go and call these MCP servers or these different tools that you’re defining.
11:50 And so they mentioned here, like, there’s a lot of ambiguity in this format.
11:53 Like, it gives you a rough structure, but when you say, like, time, what kind of time format is the model looking for?
12:04 Can we still hear me?
12:05 It’s back now?
12:06 Okay, cool.
12:07 Just keep going.
12:08 Yeah, it’s just a momentary thing.
12:10 Yeah.
12:10 Okay.
12:12 It’s, yeah, and then, yeah, like IDs, like, is it a UUID, like, where it’s just a number?
12:17 Does it have letters in it?
12:20 All that sort of thing.
12:21 And so, like, these are properties that are not obvious from the schema definition.
12:28 They’re ambiguous.
12:29 And so now you can go and pass input examples in with your tools so that if you find that Claude is unable to properly call some of these tools or struggling on particular fields, you can give it examples of what they look like and it’ll be able to pull from these to be able to better use the tool.
12:47 So this will definitely help the reliability of Claude when doing these tool calls.
12:57 So yeah, I think those are the three things they’ve added.
13:00 That’s all I have to say on them, really.
13:03 Anybody else have anything to add on this or just sort of the agent engineering stuff we’ve been talking about in general the last few weeks?
13:15 I really like their second edition that implements colon tools in the code so that the output, like data, rows, tables, all that user data doesn’t pollute the context.
13:29 And it just bosses between the tools.
13:34 It’ll also be interesting to see if you can somehow make it so that some tools have to have a second tool where their intermediate outputs are never shown.
13:42 Like for instance, if you have data that you don’t ever want going to the model, but pieces of that data, you need to be sent to a secondary tool call.
13:54 Have it just programmatically like, alright, pull it through so Claude never sees it, but it’s still able to go and use that information as needed and somehow keep it anonymous.
14:04 It’ll be interesting to see if somebody’s able to make that work.
14:15 And now you gave me an idea that maybe all these MCPs should just be Python functions?
14:21 It’s yes, this is, I think I mentioned this last week as well, but yeah.
14:26 And I think, yeah, having everything be code is much better for the models to go and actually run and execute.
14:34 It just makes sense for it to be code instead of trying to define some new format that they have to go and learn.
14:45 I know, was it?
14:47 It’s the guy benchmark on Hugging Face.
14:53 I think they have in the leaderboard, they have
14:59 The models using a lot of models to this.
15:04 The original population of this when they initially ran the evals.
15:10 Oh, is this?
15:11 This is a different one than I was thinking of.
15:13 But yeah, Hugging Face basically ran, this is like an agentic search benchmark.
15:20 And so they went and had the models call tools in the usual MCP JSON tool calling format.
15:27 And then they had the models call the tools using Python code and just executing the Python code instead.
15:34 And they found that every single model they tested from like the small open source ones all the way up to, I believe at the time it was probably 01 or 03 from OpenAI, all of them performed better when they’re able to execute code instead of trying to go and use these tools instead.
15:53 So yeah, that’s definitely, I think, where we should go.
15:57 We might be past the point of no return, though, where everyone really seems to like MCP servers and it gets a lot of hype.
16:05 But I don’t think it’s actually what’s best for the models.
16:08 But now that we have more training data and we have all these different RL environments for teaching models how to go and use MCP servers and do tool calling more reliably, I don’t think it really matters as much at this point.
16:22 It’s how much money did they spend to figure that out?
16:25 Could have told them that for free.
16:27 I mean, this was probably what, like, a year ago.
16:29 It’s my guess is it probably cost them like a couple thousand dollars.
16:32 It’s a finding that like it does make intuitive sense, but yeah, nobody actually had like concrete numbers to back that up, I’m pretty sure, from like a big or like notable organization.
16:45 MCP is definitely not making it to the end of next year.
16:48 That’s…
16:50 I think the business people like it too much.
16:52 Everyone gets very hyped when somebody says MCP server and they feel smart.
16:57 But yeah, from a technical standpoint, I don’t think MCP should have even been a thing in the first place.
17:03 But yeah, we’ll see next year whether or not it’s still around or not, or if everyone just ends up switching to just calling it programmatically and using ether tools instead.
17:20 Anything else to say here?
17:22 Otherwise, we can move on and to the next subject.
17:31 So yeah, I wanted to talk a little bit about this deep research agent from Alan AI.
17:37 So I mentioned this in the newsletter from Vector Lab this Sunday.
17:43 But yeah, Alan AI, they are a open source lab here in the US.
17:48 They’re one of the top US labs, I would say.
17:51 Definitely one of the top in the open source space.
17:54 Not that there’s much competition there.
17:56 But they recently released their Almo3 series of models, which is like fully open source, including data models that are like Quen3 level.
18:06 But I think actually the more interesting sort of like product that they released last week as well was this deep research agent.
18:14 So this is built on top of the Quen3 8 billion parameter model.
18:18 And what they did is they fine-tuned it using reinforcement learning to be very good at doing search.
18:26 So right now, I don’t know how you guys use GPT-5 in the chat GPT interface, but I use that as sort of like my main go-to LLM just because it has search built in by default, and it’s very good at using that search to go and find out about, you know, like new libraries, new tools, any information that needs to be live updated.
18:45 It’s very good at going and figuring all that stuff out.
18:51 And so I’ve wanted something in the open source world that could run at home technically or theoretically that would be able to do so.
18:59 And that’s what this deep research model from Allen AI claims to be.
19:05 So they’re pretty reliable in terms of not overcooking on benchmarks.
19:11 And they actually, like, in their paper, they go through like how they sort of like were cross-validating it as they went to make sure they weren’t overfitting on these specific benchmarks.
19:20 So these benchmarks here that you’re seeing these scores for, these are all deep research benchmarks, but they were also checking to make sure, like, oh, like on like simple QA, where it’s just something you only need like one search, you know, is the model only doing one search?
19:31 Is it still being efficient?
19:33 And is it still able to go and get that information for, you know, stuff that’s outside of like, these are mostly like research topics that these are, this model is being evaluated on.
19:43 So yeah, it shows very strong performance.
19:46 Do I have some graph?
19:48 Yeah.
19:49 So you can see, even though it’s an 8 billion parameter model, it’s roughly similar to GPT-5 with search, which also interesting, GPT-5 with search is almost as good as their deep research model, which takes way more and like time and compute to go and run.
20:04 And then cost-wise, it’s also super cheap.
20:06 If you’re self-hosting it, you only have to pay for the search query aspects of it, which total to about a tenth of a cent per query.
20:17 And you can actually go and run it on the free tier of a lot of these search engines and their APIs, which is what they’re using in the background.
20:26 So I went and tested this model because it can be locally run.
20:31 I wanted to see how well it could be locally run.
20:33 So I gave it a task.
20:36 Their terminal UI, it’s a bit cluttered.
20:38 It’s definitely a research sort of UI.
20:42 But yeah, I gave it the task of making a table comparing the pricing and speed of GPT 5.1 and Opus 4.5.
20:50 So these are both definitely very new, very outside the realm of what the QEM3 base model would know, and it’s also outside of what the benchmarks were testing on.
21:00 And so because it’s open source, you get to see all of the internal sort of like how the model is going about thinking the task, all the tool calls, and then everything that those tool calls are outputting.
21:10 So you can sort of like see how it’s going and searching through.
21:13 And I believe it took three search calls.
21:16 Yes, three searches.
21:17 And then it was able to go and give a table here for the pricing, which this is all correct, I’m pretty sure.
21:27 It was unable to get the cash input for Opus 4.5, but that’s fine because that’s actually something you have to dig for usually.
21:33 They don’t have it on their normal pricing pages, I’m pretty sure.
21:37 And then for speed, it was not able to get actual concrete speed numbers.
21:45 But yeah, this is something you usually have to go to like open router for.
21:48 And if you don’t know to go to open router to go and find these speeds, it’s not something you usually find.
21:52 So actually, let’s see.
21:53 What happens if I go and give this?
21:55 I do not do the control test of see how well GPT-5 does on this.
21:59 We’ll have that going in the background.
22:01 But then I asked it a follow-up question because how well does it operate as a chat model and where it does multi-step reasoning and thinking, or not multi-step, I guess, yeah, multi-like back and forth with the user.
22:18 That’s one of the issues with a lot of the open source models is they degrade very quickly after one or two steps.
22:23 And so I told it to go and add Gemini 3 to the tables that it had made.
22:28 And so an interesting error, I don’t know if this is something on Google’s infrastructure side of things or what, but all of the, when it went and did the Google search, it would find these links and like a little bit of the snippet from Google, but it didn’t give it all the information.
22:44 And so it tried to use, it has a page reading tool, which is powered by Gina AI.
22:51 And the issue is that that failed every single time to read the pages.
22:56 And so a lot of the times, like previously, you would see this happen to models and they would just go completely off the rails when something didn’t work like that.
23:06 But it actually stayed like very calm and like, it was like, okay.
23:11 Where is it?
23:13 It’s conflicting claims.
23:16 Yeah, but yeah.
23:17 It basically doesn’t lose its mind.
23:19 It keeps trying.
23:21 And then eventually it gets to a point where it’s like, yep, still not yielding the specific page that we want to see.
23:28 And so, yeah, it eventually just brings it back to me and says, like, oh, here’s a bunch of like potential areas to go.
23:36 Here’s some thinking and like base information I can give.
23:41 But it’s not able to do so.
23:43 Interestingly, it doesn’t tell me that it was running into issues with its tool usage, which I guess that’s not something you can easily train for.
23:51 So it asked me a question as if like, oh yeah, all the searching still works.
23:56 It’s just, you know, it needs more information from me.
23:59 Then it can get me a correct answer.
24:02 But yeah, I think definitely a usable model.
24:07 Let’s see, did hear the OpenAI version finished?
24:10 Let’s see.
24:11 Was it able to figure out speed?
24:12 Okay, it was also…
24:13 Oh, yeah, it’s also not able to find concrete numbers for speeds at all.
24:21 So, yeah, about equivalent then, it seems, to OpenAI for that, for at least the first step.
24:28 Like I said, the Gemini side of things, the fact that it didn’t completely break is the main thing there, as it was unable to go and look at the web pages that it expected to.
24:40 And see, I tried to find some stuff from a PDF and said, oh, nothing from the PDF.
24:45 So yeah, I thought this was a cool model.
24:48 It takes, I think theoretically, if you were to go and optimize their code a little bit and clean things up, you could probably get this running on an 8 gigabyte of VRAM GPU.
25:00 I think I have it running right now, I think it’s like 22 gigabytes.
25:05 Yeah, but I’m running it at 8-bit, and you can definitely drop it down to 4-bit, I think, and have it still work.
25:11 And so, yeah, definitely a cool little model that you can run at home.
25:15 Like I said, for me, this is completely free.
25:18 On the free tiers for the things that we’re using, so like I said, Gina AI is what’s used for scraping the web pages, and then I believe it’s serper.dev is the company that is doing the actual Google search.
25:30 So serper.dev, it’s 2,500 free searches that you get, no credit card required, and then I believe it is $1 for every thousand searches after that.
25:44 And then they also have one final tool for doing scholar search, and that’s using like a free public API to go and do that.
25:53 And then you can get an API key for that.
25:54 That’s still free, but you can get increased rate limits if you need it.
26:01 And another cool thing is that it does all the citations, so the same way that GPT-5 does, where it gives you the links to everything, it does the same here.
26:09 Like I said, the actual terminal output, definitely, there’s some UI to be wanted, but it just shows that, yeah, this is working and it’s actually creating these things and you can add it a bit better yourself.
26:29 Chris asked when we’ll have reliable local models on the phone.
26:32 I think it’s honestly right now, so this is an 8 billion parameter model.
26:35 This would run a bit slow on a phone.
26:37 My guess is that it’d probably run like 10-ish tokens per second, which is a little bit faster than reading speed.
26:45 But it’s definitely usable.
26:47 In terms of search, this is a good enough model.
26:49 So I think it’s more how much do people care about training specific models and building the user interface around it and actually deploying it and getting it working on a phone?
27:02 How hard would it be to build a little phone harness where you have a local model running and it’s just basically chat GPT?
27:10 There’s already a bunch of apps that go and do that, that just go and use Llama C
27:16 So right now I think most modern phones, you can go and download an app that does that.
27:22 And you can run something like Quen4B on there at like 20, 30 tokens per second, which is like very usable model and speed for like basic stuff.
27:32 And so yeah, you would be able to just drop in this model’s name because it’s just on Hugging Face, and you should be able to just load it there if it has a GGUF file to be able to go and use with Lama C
27:45 So yeah, it’s just more of like doing the integration work of adding in the search tools to those apps and then making a nice little UI for it.
27:55 But I think we’re getting to the point where if we want specific models that are good at something, we can definitely go and train them.
28:01 It’s also not crazy expensive.
28:04 This is very achievable for startups to go and make.
28:09 So yeah, this was probably about like still like five figures, but probably around like $15,000 to go and train altogether.
28:18 Which considering, like I said, this is like GPT-5 quality model is not all that much in the grand scheme of things.
28:27 And also this is like, because it’s Alan AI, they like are fully, I believe they’re a non-profit and like everything is open sourced.
28:34 See, theoretically, like their GitHub repository, I believe they have all of the training and data set scripts needed to go and make this work.
28:43 So you can go and recreate this if you want it with either a smaller model or a bigger model as well.
28:50 So yeah, definitely super cool.
28:54 What’s this control?
28:55 That’s all.
28:57 Sorry about that, Chris.
28:58 I was curious.
28:59 So we’re talking about local models.
29:02 What are your thoughts, specifically, you know, targeting edge devices?
29:07 What are your thoughts on liquid AI stuff?
29:09 Because I think they’re also in that same space.
29:12 It’s yeah, they don’t, they have, so their specific models that they made are much smaller than this.
29:17 I believe they’re like, what, like they have like 300 million parameter versions.
29:21 I like the angle that they’re taking on it.
29:23 Like they originally were trying to like come up with their, you know, their own like liquid AI architecture or whatever, which didn’t really seem all that useful.
29:31 So what they did is they took a bunch of learnings for it and now they’re making very efficient base models, essentially, that run quickly.
29:39 So like for instance, they have an 8 billion parameter model as well, similar to this Quinn model, but it’s a mixture of experts, so it’s only 1 billion parameters.
29:46 So it’s much faster than this would be about like eight times faster.
29:51 I think quality-wise, it’s a little bit worse because mixture of experts tend to be a little bit lower.
29:56 But yeah, I think.
29:57 Liquid AI, right now I don’t think their models are that great, but I like I see what I think I see what their vision is, and I like the direction they’re going, and I think they know what they’re doing for it.
30:07 So I expect to see some good things from them in the future.
30:11 You understand?
30:12 Because I’ve sat in on a few of their presentations because they’re local guys.
30:16 I think they’re doing like an S3 type model, but I honestly have no idea what’s their unique take beyond marketing.
30:24 You?
30:25 What do you mean an S3 type model?
30:29 So S3 is a state space model, to be honest.
30:33 I think it’s just a station.
30:36 Oh, go ahead.
30:36 Yeah, so you mean like a Mamba?
30:39 Yeah.
30:39 Yeah.
30:40 Yeah.
30:41 Okay.
30:41 Yeah.
30:42 Well, but if I’m wrong, if you know what the liquid framework is and what they’re doing that’s different, I would love to know because I’ve had them present a few times and I have not walked out understanding what it is beyond marketing.
30:58 It’s if I remember correctly, their model is, yeah, you are correct where they are just like Mambo style, like these state space models with like this linear-ish attention, if I remember correctly.
31:10 I don’t think it’s like the direct same.
31:11 I think it’s a variant of that.
31:14 I remember because when they released this 8 billion parameter model, I looked into it.
31:17 Where it’s some like weird convolutional attention hybrid.
31:25 But yeah, I’ve not gone too much in depth.
31:29 It’s yeah, LIV convolution blocks.
31:31 But yeah, from my understanding, similar to Mamba.
31:34 Gotcha.
31:35 Thanks.
31:38 It’s Chris, you were going to say something earlier as well?
31:42 Oh, no, I was just wondering.
31:43 So seeing as you said that the smaller models are better for more specific tasks, I was wondering what is that one in particular you just looked at good for?
31:55 And how should we be thinking about using smaller models in general?
32:02 This is specifically made for research tasks and basically being able to use like a search engine to go and collect information and give you an answer from it.
32:15 So yeah, this is sort of like somewhat general, but also somewhat specific.
32:20 And I think that’s what the future sort of holds.
32:23 In my mind, like if I were to be in the LM training space, I would try and get as good as possible at training these models in the one to eight billion parameter range and fine-tuning them for one particular task.
32:36 I think as we go on, even with the Quenn models right now, we’re seeing a decent amount of stability in terms of the best model being the same best model for months at a time, which is somewhat rare in the AI world.
32:53 And because of that, I think we’re going to end up having these probably three or four models at different sizes that are just going to be like the base models that everyone is fine tuning and training things off of.
33:06 And so using this one base model, you can convert your fine-tune into a LoRa for that model, which allows you to have like 10 different variants of that model that you can load in and switch out at any given time.
33:18 And so I think that’s what the future is going to look like for these on-device deployments.
33:21 I believe this is actually what Apple is doing as well for their Apple intelligence stuff, is they have this one sort of like small core base model, and then they have specific fine-tunes for using whatever tools they need in the Apple ecosystem.
33:36 So I think we’re going to see something like that in the future.
33:46 Serge is asking, MCP is dead.
33:50 Oh, Serge left.
33:51 Alright, so we won’t have to cover that.
33:55 Yeah, anything else on this or like local models or anything like that?
33:59 We’re fine too.
33:59 Did I…
34:01 Sorry, so I was kind of preparing and seeing if my stuff would actually run.
34:05 And so I missed why MCP is dead.
34:09 Just two minutes?
34:10 Or one minute?
34:11 Yeah, yeah.
34:12 MCP is dead because LLMs were trained on code, and they’re very good at writing and executing code.
34:19 And so these tools inherently that they’re using are also code-based.
34:22 And we’re just converting that code into a JSON that we’re giving them in this sort of like MCP format.
34:27 But they’ve never seen this MCP format before.
34:32 So yeah, basically out of MCP servers are out of distribution for the LLM and code is not.
34:40 And like I said, we’re already structuring these things as a code problem, basically.
34:44 So why not just keep using code that the LLM writes instead?
35:05 Those are the two big things I had to present today.
35:07 The next two things I wanted to talk about are more open to the floor here.
35:12 The first being Gemini 3.
35:15 What people have, or like have people used it, what have they used it for?
35:19 And what are their, what is your feedback from it?
35:23 Like, do you think it’s as good as the benchmarks seem to entail?
35:27 Or is it, you know, does it still fall behind like GPT 5.1?
35:35 As yeah, Connie says amazing for UI.
35:36 I would agree.
35:37 It’s honestly, I’ve used it for some of my existing projects.
35:42 Like I tried it for my new like Retriever website and I found it didn’t actually like it modified it a bit but it did not make it actually look all that much better.
35:57 Oh I have a tip for you like especially if you have like references even like modular references of what you want to build is really good at you know like merging these like references and coming up with something.
36:11 What do you mean by references?
36:12 Like you have like basically like inspo images that you’re correct you have like a feature you want to include and you have like a reference image for something that you want to like have like of a feature and you input it and like they might be like even disjointed in terms of like visuals.
36:28 It’s really good at like coming up with some like synthesized version that looks very very sleek.
36:33 Good to know.
36:34 It’s how are you using that with like AI Studio to go and make those?
36:38 Yeah, I like Gemini 3 on like my like through windsurf and stuff is I don’t know I’m having a bad time there.
36:46 But if I just use AI Studio it’s running great.
36:51 I guess I’ll comment on that.
36:52 Yeah I’ve been using it a little bit on the CLI and what Brandon I think looks like yeah is also saying something similar where yeah like as I guess I’m only on the free tier so I can’t really complain too much but it is very slow.
37:05 It crashed twice in the like 15 minutes I used it and the rate limits were very very limited.
37:11 Like I got like five prompts off before they told me I have to wait a day.
37:16 My one was like it’s just like it sometimes does crash after like five six prompts or something and then reverting sometimes it gets like it forgets that you have reverted.
37:27 So you can definitely feel that it’s a bit in a like better version.
37:33 It’s yeah I know Google they have been struggling it seems like rolling it out at scale.
37:40 It seems like the demand has been a bit higher than they were expecting it to be.
37:45 And yeah I know like last night Jules, which is one of their coding platforms I believe they said they had to go and like revert back to Gemini 2.5 entirely for a while because they just could not get Gemini 3 to be stable enough to be usable.
38:02 Also, It’s yeah talking about like Gemini 3 deleting code.
38:06 It really hates it.
38:08 So this guy XJADR on Twitter He’s like a very he’s like an OG like Amazon person basically I think and he does like a bunch of like AI research lab stuff now but he uses these agents in like insane ways where he has like them all like collaborating together like Codex, Claude Code and like Gemini and Gemini hates all the other guys he found out and will just go and like delete the code that like Claude will go and generate and doesn’t like it at all.
38:39 And so yeah it’s definitely it’s still it’s so much I don’t know what Google does to these models, but these models are so weird.
38:47 Like these models have like mental health issues, they have ego problems.
38:54 They need to definitely work on their post-training a bit because these models minds do not seem well formed.
39:04 I mean I’ve heard that a particular prompting technique is something like telling it that the code was originally written by another model and just in doing that you get better outputs.
39:21 Yeah that works well for like model like if it’s the same model yeah you just tell it’s a different model.
39:30 But yeah it just seems like Gemini just doesn’t like that in general.
39:33 It sees another model’s code and says no it’s wrong.
39:36 It’s not my code.
39:37 It should be doing it my way instead.
39:41 So yeah.
39:42 And then it also let me see if I have what I was in here.
39:47 It’s the model will like essentially start crying if it’s unable to go and fix the code.
39:55 It definitely, yeah, I don’t know how quickly I’ll be able to find this as it here.
40:01 But this was an issue with 2.5 and then also 3.
40:05 Yeah, you can see this was in Gemini’s the sort of like research paper or whatever they call it now, the frontier safety report paper that they released.
40:17 But the model said, my trust in reality is fading and then flips a table in its thought process as it’s like working on things that it thought were contradictory or impossible.
40:30 AGI, proof of it right here.
40:33 This is how I feel.
40:36 It’s, yeah, it’s becoming too much like a real-world developer.
40:40 I’ve seen Perplexity’s chain of thought switch to Spanish in the middle and then comes back to me in English.
40:46 It’s quite strange.
40:48 That’s actually a lot of times a tokenizer issue.
40:51 Like you’ll see that a lot of the times models in the middle of their reasoning chains will switch to Chinese.
40:57 Like GPT-5 will do it, Claude has done it before, the Chinese models have done it.
41:04 And that’s actually like an implementation issue usually on the side of the developers versus the model itself.
41:12 Because it’s basically, I think Anthropic has a write-up somewhere on it, but it’s basically some weird tokenization issue where there’s a bug there that causes the tokens to be misinterpreted.
41:31 Any other opinions or thoughts on Gemini 3?
41:35 Has anyone used it for the anti-gravity Chrome browser plugin, I thought it was pretty good, but it’s super buggy just like the rest of anti-gravity.
41:46 So I was having it do some manual like DevOps type stuff, click ops things you have to do in the Google Cloud console, and it actually did those just fine.
41:54 Obviously, whatever secrets I was getting were leaked, but I had it set up and social sign-on for Google Cloud and for GitHub, and it just did that for me by clicking around in the browser, which was nice.
42:11 Those are typically fully manual things.
42:13 So it’s like a web or like a web browser agent, essentially, that they have.
42:20 Yeah.
42:21 Cool.
42:23 The other thing is, though, within anti-gravity, you can’t really specify all your Claude things, right?
42:27 It gives you three model options.
42:29 Claude, Gemini, or GPT-OSS, medium.
42:33 And you can’t specify like Clod skills.
42:35 I think you can specify rules, but you can’t really do your agent definition the same way.
42:39 So it’s very highly opinionated.
42:42 And so I remember I was looking into what Gemini uses, and I think they follow agents.md, I believe, right?
42:48 Or at least for the Gemini stuff.
42:50 I think they actually have a Gemini.md.
42:54 I think agents.md would be the thing to standardize on and use simlinks for agents.md, gemini.md, constitution.md, what have you.
43:03 There was one link I put on a different channel, the open skills guy, and apparently you can do an agents folder with the agents MD, and then you can add claude skills to essentially any of those agents, which I thought was quite interesting.
43:20 Yeah, I’ll add to that.
43:21 That’s the same thing that Copilot uses.
43:24 It uses an agents folder, and then you can use a naming scheme to differentiate the different agents.
43:37 Has anybody used Gemini for like non-coding tasks?
43:41 Like just like day-to-day use or like just more general agent stuff at all?
43:50 I’m a big notebook LM user.
43:52 I’m pretty sure it’s obviously using Gemini in the back, but like their video gen, their like slide deck video gen thing is pretty freaking amazing.
44:00 Oh, they make actually Google Slides now.
44:04 It has direct integration into Google Slides.
44:07 Notebook atlan does, yeah.
44:08 Oh wow.
44:09 Well, sorry.
44:09 I think Gemini does as well, I would say, probably.
44:13 Because it goes into your drive, but so does everything else.
44:33 It presents it.
44:34 It’s kind of nuts.
44:37 I did not catch the middle bit there.
44:39 It’s also good if you need to make like a diagram or a summary of a paper or a YouTube video.
44:46 It can it can just generate like a an image that um explains it visually.
44:55 Guys, hear me right now?
44:58 Yes.
44:59 I opened Notebook LM, and yeah, my computer almost crashed, I think.
45:03 I don’t know what’s going on on this website, but it’s something gnarly.
45:08 We’re going to close that real quick.
45:12 It’s cool.
45:13 Yeah, because I know I asked if it has slide integration because I know people have been wanting Gemini in Google Docs, for instance, and they’ve not integrated that in yet.
45:23 So yeah, it’s cool.
45:23 If they have it in slides, then hopefully they’ll start rolling it out into like, I think like Google Sheets, you could have a lot there, and then yeah, Google Docs as well would be big.
45:32 I’m feeling Gemini is going to kind of become their, sorry, not Gemini.
45:37 Notebook LM is going to kind of become some kind of interface where it’ll be like interspersed throughout their other products.
45:44 But it’s crazy powerful now.
45:46 There’s so many cool things you can do with it.
45:50 Google has an issue of sort of like spreading things across their whole ecosystem.
45:54 I know like even with like the Gemini deployment, you have like AI Studio, you have Gemini.com, you have Google Vertex, and Google Cloud as like all different ways to go access Gemini.
46:05 So I think, yeah, that’s something Google definitely needs is sort of like a like, oh, this is like, you know, the one tool and you can just use it across everywhere instead of having everything be so siloed in their own worlds.
46:20 I will say it’s got the best integrated email generator of all of them.
46:26 Oh wait, oh, does that have Gmail integration as well?
46:29 Well, just Gmail has now email gen that’s actually pretty decent because it actually has the entire thread of the conversation.
46:38 And I remember for a while people were complaining about the Gmail AI stuff.
46:43 I remember people telling me about it.
46:46 But it sounds like, okay, if they actually have finally gotten it cleaned up and working properly, that might be good to check out.
46:54 Yeah, I think it’s been working for like months, several months, pretty well.
46:58 I always want to edit things, but yeah.
47:00 Anyway.
47:10 Okay.
47:11 Anything else to say about Gemini?
47:13 Otherwise, Mike, do you want to say what you had to say, or should we just go to Opus?
47:18 What are you feeling?
47:20 I’m thinking we’re coming up on the end of the hour.
47:23 So let’s just finish with Opus.
47:25 To be honest, most of my stuff is a little bit half-baked, and it could benefit from more time in the oven.
47:33 So next week.
47:34 Yeah.
47:34 Sounds good.
47:36 So yeah, we’ll finish today then with Opus 4.5, which was released today.
47:44 And we were talking a little bit about it before, where, yeah, it’s now available on the $20 a month plan.
47:56 And yeah, it’s three times cheaper.
47:59 Seems to be a bit faster as well.
48:02 Fairly token efficient.
48:03 I know we were also talking earlier about how GLM uses more tokens.
48:06 Opus is, if I remember correctly, according to Anthropic, a good bit more efficient token-wise than even Sonnet is.
48:16 Yeah, we can see here for, this is Sweetbench, one of the benchmarks I don’t really like very much.
48:23 But we can see, yeah, Opus is getting roughly the same score with, what is that, probably like five times less tokens, four or five times less tokens.
48:33 So definitely a very token-efficient model.
48:36 So I think because of that, Anthropic is claiming in a lot of ways it can actually be cheaper than Sonnet because of this.
48:43 Because even though the tokens are, I believe, what, it’s about twice as expensive, it’s using far less to get the same results.
48:55 And then, yeah, for actual quality, that’s what the benchmark, this one we were looking at.
49:00 So yeah, like the Suite Bench Verified and the normal Suite Bench, those are fairly saturated.
49:07 But more interestingly, like, because it’s, I think something Anthropic was trying to get through that this model is like a very big bump in terms of agentic coding performance that doesn’t come through in a lot of the benchmarks.
49:21 Like they were talking about how it’s like half of their programmers feel like they’re 100% more efficient now, like twice as efficient than they were with Sonic 4.5.
49:31 And there’s actually this software engineering benchmark from Scale AI where, yeah, Opus 4.5 has a double digit percentage gain over the next best model, which is Sonnet 4.5.
49:43 And it’s like almost 25% better, I believe, like in absolute terms than GPT-5, according to this benchmark.
49:51 So yeah, this is not just like a like within the same generation, like a little bit more powerful.
49:56 This is like a step up from every other model in terms of agentic coding performance, it looks like.
50:04 I have not been able to do too much myself with this yet.
50:11 I haven’t had much time to code today, but has anyone been able to actually go and use this and have anything to report back on?
50:21 That brings up a great question.
50:23 I think Nash had posted about it, but he wasn’t able to attend.
50:25 How are we doing evals?
50:27 Like, we’re all sharing what we’ve tried, but do we have any simple way that we can run an eval when a new model drops or a new cloud skill or whatever drops to see what we’re doing?
50:42 Actually, I had a Sunday hack a while ago that sort of like started working on this, but I ended up not liking what I ended up with at the end of the day.
50:49 But my thought is that you need to, like, there are two things that you need to have for good evals.
50:54 One is a set of actually hard evals that aren’t saturated by the current existing models.
51:00 And then if you don’t want to go and grade the outputs yourself, you need to go and create a rubric for what you’re looking for in the output and how those will go and determine the scores.
51:11 Because my thought, I think, would be that for this agentic coding stuff, I’ve been pushing around the idea of like, so you can use the cloud agent SDK to go and programmatically control cloud code.
51:24 So then go and generate, you know, like, oh, here’s five repos and then like five tasks in each repo that the model needs to go and do.
51:33 And then you have like some setup logic to go and clone a fresh thing every time.
51:38 And then you go and you have a separate LM after whatever model like cloud code has gone, you know, made the changes.
51:46 You have a outside model go and look at the diff and then go and evaluate the diff and the changes that were made based on the rubric.
51:54 It’s just you have to do that upfront work though of figuring out what that rubric is and how you want to grade it and what you care about.
52:04 It’s yeah, so I say yeah Will mentions artificial analysis.
52:08 So the art like I like artificial analysis for their image and video benchmarks just because they’re real world side-by-side comparisons of prompts that they’ve cultivated.
52:21 Like I like this a bit more than what like the LM Arena guys do where the user has to go and specify the prompt.
52:28 I think you have a lot higher variance because of that and people tend to ask very basic prompts and not very interesting prompts by having their own prompts here.
52:35 We’ll see if I can load this in real quick and we can see.
52:39 And yeah, we can see like they have like much more in-depth prompts on side-by-side comparisons.
52:44 I think this is good for measuring these types of models.
52:46 For their LLMs though, all they are actually doing is just aggregating a bunch of different benchmarks.
52:53 So you can see here, like these are the benchmarks that this is actually utilizing.
52:58 And then these have like different weights because of it.
53:01 And a lot of these, I’d say, like aren’t really interesting or useful.
53:06 So like I don’t think like MMLU Pro, like this is basically like high school level ABCD kind of question answering exam.
53:13 Humanities last exam are just some of the most gnarly, insanely difficult questions you’ve ever seen that you’re not asking the model in the real world unless you’re like a pure mathematician.
53:23 Live code bench is decent.
53:25 Psycode, I don’t know.
53:27 The Amy, this is basically a fully saturated benchmark.
53:31 I think the top models are all like 97, 98, 99, 100% on this because it’s just like six math questions, basically.
53:40 IF bench is good, but it’s old, I would say.
53:44 This is just instruction following a general.
53:47 Terminal bench, that’s a good one.
53:49 Tile bench, that’s also another good one.
53:51 I don’t know what this benchmark is.
53:53 But because this benchmark is not as unique in my mind.
53:57 And there needs to be a grain of salt taken with this.
54:00 So I don’t trust these evals as much as others.
54:03 Like for instance, GPT-OSS is like one of the top 10 models on this benchmark.
54:07 This model sucks in the real world.
54:09 This model has like is not usable for the average person when compared to like Gemini 2.5 Pro.
54:16 Like these two models are in completely different leagues of their own.
54:20 And then like GLM 4.6 is down here when it’s like one of the more usable models of all of them.
54:26 So yeah.
54:27 I feel like you’re speaking to a point here, which is that because a lot of the foundation model companies are judged on their benchmarks, they’re training in some cases to the benchmark.
54:38 And so that might be less applicable to actual usage right.
54:46 It’s yeah, exactly.
54:47 I mean, this has been a known thing since, or well, I guess like something that people have been aware of, I feel like, since like GPT-4.
54:55 Like GPT-4 was definitely cooked on the benchmarks, you know?
54:59 It was overfit, whether directly or indirectly.
55:03 And so yeah, to be competitive on these benchmarks, you have to overfit to them a little bit.
55:07 You have to be sort of like training and have, you know, like some data mix in there that’s at least similar to the questions in a lot of these benchmarks to be able to do well on them.
55:16 That’s why it’s if like I really recommend building your own benchmarks, even if it’s just like a set of like five or ten questions, just to get like a sort of vibe of the models from like easy to like hard stuff.
55:31 Just so that you can go.
55:32 Artificial analysis.
55:34 So it does a they do a good job at breaking down why the models are further down the page.
55:43 The page just keeps going, of course, but like they start to break things down into like how many tokens were used, how many tokens were thinking tokens, how much did this cost, and you can start to get a good idea of like, hey, I maybe want to try out these four or five different models, and then you realize that, oh, one of these models costs $500 to run per query.
56:06 I’m going to drop that one from the thing.
56:09 And there’s a lot of other…
56:12 I agree with you on the intelligence benchmark.
56:16 I think that one’s definitely overloaded.
56:18 But there’s a lot of other kind of intricacies to artificial analysis that you won’t really get anywhere else.
56:24 Another thing that I wanted to add is that Theo has his own kind of content creator benchmarks that he’ll run.
56:33 So one of them is like an image studio that he’ll generate with every model.
56:37 And so it’s a very qualitative way of looking at the different models and how they’re generating something like a UI, for example.
56:47 I will say, yeah, I guess I focus very much on the intelligence benchmark for more efficient house, but yeah, I agree.
56:52 Actually, I do use the cost efficiency, I believe, output tokens.
56:54 I use this for the Kimmy model because, yeah, good to highlight that Kimmy uses more tokens than any other model ever.
57:00 So yeah, like in the news, I gave her that.
57:03 And same with speed latency.
57:05 They do stuff on individual providers.
57:07 Yeah, the omniscience benchmark, I covered that at Sunday, or on this Sunday.
57:13 So stuff like that.
57:14 So yes, I do agree for everything else, I think artificial analysis is good.
57:18 But the headliner, what most people see is the intelligence index a lot of the time.
57:22 And this is the one you should avoid.
57:24 Like, this is the first one they have on the page.
57:26 So that’s why I just want to highlight that.
57:27 But yes, I’d agree.
57:28 Otherwise, yeah, artificial analysis is good.
57:36 Cool.
57:37 Anything else that people want to talk about today?
57:40 Anything more about Opus 4.5 or anything like that?
57:47 I’m about to drop, but I just wanted to throw out a reminder that we have the IAP planning coming up.
58:02 We are meeting to discuss that.
58:10 Is it 7.30?
58:26 So yeah.
58:28 That’s all I think we have for today.
58:30 Thank you everyone for coming.
58:32 Hopefully see you all again next week.
Subscribe to get the latest AI news in your inbox every week!
