PICK YOUR SUPPORT STYLE
MONTHLY SUPPORT
Reader
$5/mo
Contributor
$15/mo
Architect
$50/mo
Recurring subscriptions auto-bill monthly via Stripe Checkout. Cancel anytime from the receipt email.
What was talked about: GPT-6 arrives with exceptional vision, spatial reasoning, computer-use, and benchmark performance. Examples include creating Blender animations, designing custom building sets, and reaching 95% on iBench V3, far ahead of the previous results below 60%. Its image understanding now appears much closer to its text intelligence.
Takeaway: GPT-6 is especially valuable for visual understanding, spatial work, and computer-use tasks.
Links shown: OpenAI GPT-6 announcement, Vector Lab GPT-6 initial assessment, Adonis Singh EyeBench result, Adonis Singh score-versus-cost chart
What was talked about: GPT-6 often produces compact code-golf-style implementations that are difficult for humans and other models to audit. It can also stop after one step, request confirmation too often, and avoid standard editing tools in favor of opaque Python or TypeScript scripts. Despite these workflow problems, its planning, cybersecurity review, and bug-finding abilities are very strong.
Takeaway: Use GPT-6 for planning and review, but closely inspect its code and give explicit instructions for sustained autonomous work and auditable edits.
Links shown: Adonis Singh score-versus-cost chart, Sue Maru GPT-6 example, Tenobrus GPT-6 coding post, Mitsuhiko code example, Rhys Sullivan GPT-6 assessment, Gegam GPT-6 impressions
What was talked about: Practical coding experiences varied from the initial criticism. Sol and Astra completed multi-hour bug-fixing, front-end, prompt, and data-logic tasks successfully, while vision work with posters and SVG files was particularly impressive. Rate-limit experiences differed by workflow and subscription, with heavier users reporting substantially less available capacity than with earlier models.
Takeaway: Actual results depend heavily on the workload, reasoning level, and plan; reserve the most expensive model for tasks where its added intelligence matters.
Links shown: Gegam GPT-6 impressions, Tibo profile, Zry Miller model-use post
What was talked about: GPT Image 2.5 produced noticeably stronger logos and better alignment with project context than the previous image model. The release includes larger and smaller variants for API use and performs strongly across image generation, image editing, and multi-image editing.
Takeaway: GPT Image 2.5 is a strong general-purpose choice for context-aware image generation and editing.
Links shown: Zry Miller model-use post, OpenAI GPT Image 2.5 announcement, Arena GPT Image 2.5 rankings
What was talked about: High and extra-high reasoning modes often consume limits two or three times faster while providing only a small performance improvement. Medium reasoning produced comparable results for many Sol and Astra tasks with less verbosity, while cheaper models such as Luna could be used at higher settings without meaningful concern about limits.
Takeaway: Start with medium reasoning and increase it only for difficult planning, critical review, or tasks that fail at the lower setting.
Links shown: Tibo profile and media, OpenAI GPT-6 announcement, Gegam GPT-6 impressions
What was talked about: GPT-6 remains imperfect despite early claims that it approaches AGI. Known problems include sloppy code, unnecessary confirmation pauses, incorrect assumptions, and destructive Git behavior. OpenAI is reportedly aware of the post-training issues, making later GPT-6.1 or GPT-6.2 revisions more promising for enterprise adoption.
Takeaway: Treat GPT-6 as a powerful but immature model, keep version control recoverable, and require confirmation before destructive operations.
Links shown: Gegam GPT-6 impressions, Yann Dubois GPT-6 assessment, Doug Cook GPT-6 safety post
What was talked about: GPT-6 costs about twice as much as Sol, and its token efficiency does not consistently offset the higher price. Easier tasks already have limited room for further efficiency gains, so frontier models can waste capacity when a smaller model would work. Routing tasks to cheaper models or specialized sub-agents is becoming more practical.
Takeaway: Use GPT-6 selectively and route routine work to less expensive models to preserve limits and reduce cost.
Links shown: Doug Cook GPT-6 safety post, Vector Lab GPT-6 initial assessment
What was talked about: An analysis compared effective token costs across major AI coding subscriptions using blended input, output, and cached-token usage. Luna on OpenAI’s $200 plan appeared exceptionally inexpensive, while Sol and several Claude models also ranked reasonably well. Fable, Grok subscriptions, and some other plans delivered much less usage per dollar, although measurement noise and changing limits complicate exact comparisons.
Takeaway: Compare plans by usable work and blended token consumption, not advertised multipliers or API prices alone.
Links shown: Doug Cook GPT-6 safety post, Fei coding-subscription cost chart
What was talked about: Anthropic’s larger subscription tiers may not scale proportionally with price, making two smaller subscriptions potentially more economical than one larger plan. Five-hour windows, changing boosts, separate Claude Code accounting, and internal ratios for input, output, and cached tokens make the true allowance difficult to calculate. API price reductions also do not necessarily transfer directly to subscription usage.
Takeaway: Measure actual Claude Code usage before upgrading because the nominal plan multiplier may not reflect the capacity received.
Links shown: Fei coding-subscription cost chart, Vector Lab GPT-6 initial assessment, Artificial Analysis GPT-6 pricing post
What was talked about: The viral Ponytail skill uses roughly a thousand lines to promote disciplined software engineering, but a very short instruction reportedly achieved similar results with lower token and time costs. Long AI-generated skills often contain repeated rationale and unnecessary prose that enters the context on every use.
Takeaway: Benchmark a compressed version of each large skill and keep only the instructions needed to preserve behavior.
Links shown: Nate Berkopec on the Ponytail skill
What was talked about: PStack applies Cursor-inspired orchestration principles such as guarding the context window, routing work to specialized sub-agents, and loading only the skills relevant to each task. Its structured workflows can reduce irrelevant context and divide large jobs into smaller responsibilities, though some elements are Cursor-specific or expensive in token usage.
Takeaway: Protect context by giving each agent only the instructions and responsibilities required for its part of the workflow.
Links shown: Nate Berkopec on the Ponytail skill, Cursor PStack plugin, The Complete Guide to PStack, PStack Poteto mode skill
What was talked about: SpecKit emphasizes explicit planning, implementation, testing, documentation, and auditability, while PStack provides a broader orchestration layer that selects workflows and skills automatically. Structured decomposition becomes especially important when agent-generated pull requests are too large for complete manual review.
Takeaway: Use SpecKit when traceability is mandatory and orchestration patterns when repeatable delegation and automated quality checks are the priority.
Links shown: GitHub Spec Kit, Cursor PStack plugin, Nate Berkopec on the Ponytail skill
What was talked about: Large skills can be reduced by moving deterministic logic into scripts, placing templates in separate files, and removing duplicated rationale. Core instruction files require careful manual review because every extra token is repeatedly loaded and amplified. Claude Code documentation can lag behind the CLI, and Claude Code does not automatically load AGENTS.md unless it is connected to CLAUDE.md, such as with a symbolic link.
Takeaway: Keep always-loaded instructions short, verify behavior against the current CLI, and use the instruction filename that each harness reliably loads.
Links shown: Nate Berkopec on the Ponytail skill, AGENTS.md
What was talked about: OpenAI claimed an AI-assisted solution related to the Navier-Stokes Millennium Prize problem, alongside controversy about overlapping Anthropic research, access to research logs, and proper credit. OpenAI stated that its proof used a different approach. The reported effort used 10,000 sub-agents and about $6.5 million in API credits, showing both the promise and the high cost of frontier automated mathematics.
Takeaway: AI can produce significant mathematical results, but independent proof verification, transparent provenance, and clear attribution remain essential.
Links shown: Konsti Wohlwend Navier-Stokes timeline, Sam Altman response
What was talked about: Agent Skew tracks token usage and local context across agent sessions, offering a more developed alternative to small custom trackers. CodexBar and CC Usage provide related visibility into subscription consumption. These tools help identify expensive workflows as model capability makes context management and prompt quality increasingly important.
Takeaway: Track token and context usage across sessions so model selection and workflow changes can be based on measured cost.
Links shown: Sam Altman response, AgentsView, CodexBar
0:10 Uh all right, cool. Everything is looking good.
0:13 Let’s get started. Uh welcome back everyone to AI Tools Club.
0:17 Um yeah, this week we are going to be talking about GPT-6.
0:22 We’re also going to be talking about how OpenAI has allegedly solved the Navier-Stokes Millennium Prize problem.
0:29 Also, we’re going to be talking about some interesting research into how much value you’re actually getting out of the different subscription plans and comparing them all and see which one you get the most value out of.
0:41 And then finally, we will be discussing some stuff about skills and how much content really needs to be within them.
0:49 We’ll also be covering anything else that the community wants to talk about here as well.
0:52 So feel free to chime in to the discussion.
0:56 So yeah, getting started here with GPT-6, big headliner model that OpenAI released last week.
1:03 A lot of people are very impressed by this model, basically calling this potentially AGI and seemingly just destroying every benchmark that we throw at it.
1:15 OpenAI actually in their release post talked about how they had to go and work with a bunch of private benchmark makers and make them make their benchmark marks more difficult for them.
1:27 So yeah, the initial vibes, as you probably saw on social media, the model is extremely good at spatial awareness and also
1:42 being able to go and do computer use with stuff like Blender.
1:46 You probably saw a ton of the different Blender animations that it was going and making.
1:50 It’s also this, and from the AI News this week, I demonstrate, or I’ve showed the example of it going and designing custom, basically like Lego sets from scratch and then rendering them to make these cool videos for
2:04 different subjects. So yeah. But I think a lot of the initial hype was because it’s very good.
2:13 Like it’s very obviously the best vision model out there right now.
2:16 I think, yeah, like the iBench V3, this is the main vision benchmark that I sort of rely upon for like if the model can actually sort of like properly see things in the real world.
2:26 Previously, we can see next best model didn’t even get 60% on this benchmark.
2:31 But GPT-6 is now at 95%. So basically saturating this entire benchmark.
2:36 It’s just way better than any other model when it comes to vision, which is good to see.
2:40 For a long time now, I feel like the image capabilities of these models to be able to understand images and videos by proxy as well really struggled.
2:51 It was much further behind its text-based capabilities.
2:54 But it seems GPT-6 is finally bringing it up to that echelon so that they’re roughly equal in terms of how good and strong they are.
3:04 So yeah, but I think outside of these sort of browser use, computer use, image understanding tasks, the model actually seems to struggle a little bit.
3:15 So the main thing I’ve been seeing a lot of people talking about is how the model is, if we pull up the right one, yeah, it writes awful code.
3:28 So this is due to the fact that the model has been trained to use as few tokens as humanly possible.
3:36 And during the course of training, it learned to go and do this by writing basically code golf solutions, where the code is just very dense and like unreadable for a human being.
3:50 And also most other models, when they go and look at the code that GPT-6 goes and writes, they all go like, yo, what the fuck is this?
4:01 We don’t really know how to read this and understand.
4:03 It’s too dense. So this has been the main gripe that I’ve seen.
4:07 I’ve also seen, like for instance, this guy and also myself when I’ve been using it, is that the model doesn’t actually seem to be very eager to be agentic.
4:17 It will go and go one step for you, but it won’t do three or four steps.
4:23 It’ll very often come back to you and say like, oh, here’s what I found.
4:26 Do you want me to keep going? And so it’s a lot more like, it’s almost like the model has a stutter when you are going and using the model because you constantly have to go back in there and course correct it constantly.
4:39 It does though, however, in terms of intelligence, this is the best model.
4:43 So like for stuff like planning and orchestrating other models, it is very good.
4:48 It is by far the smartest model that we have out there right now.
4:51 And then this also flows in the other direction for reviewing outputs.
4:55 So like we saw with the initial OpenAI benchmarks that it is extremely good at sort of cybersecurity tasks and doing those properly.
5:04 And a lot of that is bug finding. So using this model as a bug finding model, it will be extremely strong as well.
5:13 And then of course, like I said, the vision capabilities of the model as well.
5:18 But yeah, the sort of like actual UX of using it in the Agentic applications themselves and in the harness, the model doesn’t like to do that as much.
5:29 It feels very much like the original GPT-5 when that first came out, which was a very smart model, but was actually a bit lackluster when it came to actually going and using a coding harness.
5:40 And this is something that Gemini 3 Pro as well, that model also struggled to exist in a coding harness.
5:46 It doesn’t struggle in the same way, or GPT-6 doesn’t struggle in the same way that those older models did, where it’s able to use them, but it seems to do things in its own way.
5:55 Like, for instance, it hates using the edit code tool in Codex, as I’ve been coding with it.
6:02 It loves writing to files using Python scripts or TypeScript scripts.
6:07 Instead of just editing the file directly, it wants to go and write its own code file for it, which is a bit annoying because it’s much harder to go and audit the code that it wrote, at least within the chat, because you
6:17 don’t get shown the diff or anything like that.
6:18 It’s hidden behind some large opaque script that gets run instead.
6:24 And so yeah, I think I’ll stop there in terms of my feedback for the model.
6:29 I want to hear, because this has been out for about, what, half a week now, what is the community’s thoughts on this?
6:37 I used SAL exclusively for fixing bugs yesterday.
6:43 And I was just looking at the code.
6:46 I don’t see any of that. Like, I saw it work on problems for a few hours without quibble.
6:53 And at one point it stopped, but that’s because it said, you know, you said I could spend $6 on tokens, and I’m almost in $6.
7:04 So, and yesterday it was Sol Medium.
7:08 Today I’ve been working at Sol Low based on something I saw on Twitter on X from Thibault, I think his name is.
7:20 Who said Sol Low is as good as this is all Astra activity.
7:29 I’m just confused. I’ve been using Astra Low today and Astra Medium yesterday.
7:34 No quibble at all. Like, this is all good.
7:38 I don’t have a way to compare it, but it just is handling all sorts of things, both front end and some modifying prompts and logic for extracted data.
7:52 Just crank it through it. So I don’t know.
7:59 How have you been finding the rate limits for the model?
8:01 Because I know I’ve heard a decent number of people saying they’re going through rate limits quicker, and like some much quicker than normal.
8:11 Yesterday was the first day I really tried to use Sull, and I did not notice it.
8:17 But I think the, so basically I had Fable create a big test harness and run through the tests for my app.
8:27 And I was the meat proxy that basically did the clicks that Fable couldn’t do.
8:32 And then I was throwing the identified issues at Astra.
8:38 And at that rate, I did not run into any limits.
8:45 But it was a little slower than usual, I think, because bugs were being identified over the course of several hours.
8:57 Interesting. So that’s one person for Astra.
9:02 Do we have other feedback from others here?
9:05 Yes, I can also share. So I use it like so, like before.
9:09 I don’t have problems with rate limits, but maybe in the beginning I got now a lot of free credits.
9:16 I got pre-resets again. And they resetted it also yesterday or something.
9:22 No problem there. I tested the vision stuff for like figures and to make nice posters.
9:28 And this is actually impressive. I really liked it.
9:30 And I said I use InSkip for SVG and it could handle this.
9:34 So this was crazy cool. For the dense code, I didn’t check it.
9:38 But I think some people argue this might be the future that you, like LLMs, they write assembly code or machine code directly and humans don’t read it anymore.
9:48 I don’t know, but I could imagine that we go this direction and humans will lost or are not needed anymore to understand code.
9:57 I think part of the code quality thing is, yeah, like, well, even if you assume the humans don’t have to understand it, like, how hard is it for another model to come and look at that code and still understand it?
10:06 You know, another version of Astra that doesn’t have the exact chain of thought that the original model had to be able to go and understand it.
10:14 I think that’s the main question. Because we see sort of like right now, like current AI slop code, it makes it very large and expansive code bases.
10:23 You have a lot of code duplication throughout everything.
10:25 And code bases become burdensome. It takes a long time and very expensive to add features.
10:31 And is this another sort of symptom of that?
10:33 Or is this some different paradigm, I guess?
10:38 Yeah. And I also don’t use loops that crazy or let it run.
10:43 I code mostly in the terminal. And then I never had rate limits until now.
10:49 With the 200 bucks subscription. That’s fine for me.
10:56 Yeah, I definitely noticed the rate limits.
10:59 And it was from like GPT 5.4 to 5.5, the rate limits noticeably decreased.
11:05 And then from 5.5 to Sol, they also noticeably decreased.
11:09 And now there’s another decrease. And it’s like, I feel, I still sometimes use GPT-5.4 as like an alternative to Luna because it’s almost as cheap as Luna.
11:20 Well, it’s still more expensive, but it doesn’t do some like very stupid things that Luna does.
11:26 But I do find myself just using Luna because it also feels like they generally lowered the amount of total limits because they like, what is it, like 10x to their user base or something over the last few months.
11:40 So I’m guessing they’re just like spreading the compute more thin.
11:44 Because it did feel like a few months ago, it was like really hard to hit the weekly limit.
11:49 And now I can do it in a day. Especially when they announced the reset, I will do it.
11:54 But I still kind of uncontrollably do it.
11:57 Like I’m in 30% off the weekly limit for today.
12:00 And now I’m going to get by with the 30% for the rest of the week.
12:05 It’s kind of silly. Yeah, so definitely the rate limits for…
12:10 So just, you know, makes me more selective for models and literally using Astra only for like visual things or like some really important planning where I do need like this extra bit.
12:22 Yeah. And yeah, I do know for like in terms of the like codex numbers, yeah, they talk about how like every three days they’re adding another million users to it.
12:31 I think a lot of that like they’re measuring it in a weird way where they have like the ChatGPT desktop app and they have like a little tab on there called codex.
12:39 And I think if anyone sends any message to codex there, they’re counting them as a new user, which is why we’ve seen new users explode.
12:46 I don’t think many of their users, or a lot of their growth anyway, is coming from CLI usage, like I assume most of us here probably are.
12:55 Yeah. Chris also mentions you got three.
12:57 It’s like they have an IPO plan. Yeah, exactly.
13:00 They’re pumping the numbers up for the investors, showing big growth.
13:04 That’s what they like to see. I was kind of assuming they mean the paid plan or something, or that people sign up for the $20 at least plan.
13:14 Oh, so they have, but I guess the plan is not for Codex specifically.
13:18 So it would be like someone sign up for the plan and click something in Codex and that counts as a Codex user.
13:23 And someone might have just had a 20 bucks plan and accidentally clicked it or something.
13:27 So that’s the. Okay. Interesting. Okay, well, then they just were, you know, they still probably have more codecs usage because it feels like that’s more mainstream now anyways, just from the vibes.
13:40 And it does feel like they decreased the amount of like compute per person.
13:45 Like it’s, I mean, in my memory, it was definitely really hard for me to like exhaust all the codecs usage.
13:52 And now it’s much easier. Also using, you know, before there wasn’t a model that could train it that fast, I guess, too.
14:02 I have a chart for after we talk about GPT-6, we can discuss more of that because somebody went and broke down how much usage we get for each model.
14:10 Chris says he got three weekly rate limit resets for free.
14:13 Yeah, everyone did. They had a thing where basically the rollout of GPT-6 got delayed.
14:20 They announced it last week, I believe Wednesday, but most users didn’t get it until Friday.
14:25 And so for each day that it was delayed, they gave out a free banked reset, as they call it.
14:30 So yeah, if you have a code of subscription, you now have two free rate limits that you can use whenever.
14:37 And then I think there’s also a third one from back in late September.
14:42 I forget for what, but they gave another bank reset for that as well.
14:47 Let’s see, Codex definitely I haven’t heard an announcement, but when I went to generate some images, it seems to be an improved model today in ChatGPT.
15:01 Yes, you are correct. We can bring back on that.
15:03 Yeah. Yeah. So I hate the logo that I have for my app, but I have not focused on getting something better and my few attempts have been lame.
15:14 And today I got 16 choices that most of them were really, like, really good.
15:21 And, you know, ChatGPT has the context.
15:23 There’s nothing hidden from it. But bam, like 16 logos, really good.
15:29 On topic, understanding the message.
15:32 Big difference from the previous image two or whatever the other, the previous image model, which was really good.
15:40 So noticeably better today. Yeah, this came out, it was officially announced two hours ago that they rolled it out.
15:48 And yeah, you’re right. Yeah, GPT Image2, the previous model, was already probably the best image generation model, at least for just sort of like day-to-day general image generation.
15:57 And yeah, this builds upon it, GPT Image 2.5.
16:01 There’s two models, I think a bigger and a smaller one.
16:04 I think if you have a ChatGPT subscription, you don’t really get to choose, but if you’re using it via the API, I think Flare, the smaller one, is a bit cheaper.
16:14 But yeah, they top the leaderboards for just image generation, image editing, and also multi-image editing.
16:20 So yeah, super strong image generation model there as well.
16:23 That’s bundled with your subscription.
16:28 Maybe also you guys, you use the, in Codex, you use the Max and Ultra?
16:32 Or you use always the XHI? I think the general sentiment, at least from me and what I understand, I think Quentin, I believe it was, also touched on this.
16:42 I was trying to find the post from Teebo.
16:44 But the general vibe for at least GPT 5.6, Sol, and also Astra is that you’re getting minimal amounts of performance from using extra high or high reasoning modes.
16:56 So like for instance, for me, I usually stick around medium.
16:59 For like GPT 5.6 and Astra, since I’ve been using it, it’s always been medium reasoning.
17:05 I think with 5.6, I would sometimes go up to high, but I never went up to those higher levels because you’re usually burning your rate limits two or three times faster and only getting 5% better performance.
17:17 So it just wasn’t worth it most of the time for me.
17:20 I don’t know if other people have other thoughts here on that.
17:25 I tend to use the higher levels when I have a rate reset because I know I can burn through it and then start over again.
17:34 But I hardly ever go, I don’t think I ever go above extra high for Sol, Terra, or Astra.
17:42 And I use Luna at max or whatever the high one is for Luna.
17:47 Because Luna’s so cheap, I don’t care.
17:54 I don’t think I’m going to be able to run, hit my limit using Luna with my, you know, the way I use it.
18:01 Sasha, you have something to say? Yeah, I mean, I think out of my experience, I actually started off just totally maxing it all the time.
18:10 And then it just, it became very verbose.
18:13 So then I don’t even know when I switched to media, but for soul medium, I was like, oh, wait a minute, this is just as good.
18:20 It’s a little bit less verbose and it’s, you know, it still works at that soul level.
18:25 So yeah, I actually, I’m kind of surprised by that because I, you know, the last like six months, I’ve always been just like, just maxing everything that isn’t a swarm, whatever.
18:36 But I mean, I think just to add on that, I mean, I think it is kind of a quandary now because some of these models, where they are now, it’s like, I think if you don’t understand what’s happening anymore, does
18:51 it really matter? Right. So it’s almost as if the premium is really more just can the model explain things in a way that you can keep up with the velocity and high fidelity of like
19:05 execution? Yeah, I think a lot of those questions come longer term.
19:13 So it’s much harder to tell now with the code quality of these models and such.
19:19 I mean, I don’t have a qualitative sense, but I just, yeah, something does feel improved between Sol and Astra, but I mean, I’m not really sure.
19:29 Like, it just feels, I think it’s more just kind of like, hey, like, I asked it to do something and it executed on it.
19:36 But to some degree, I also still have to be like, all right, what did you fuck up?
19:39 Like, let’s go back and check. Yeah, I think that’s the biggest takeaway, I think, for this model is that it’s good, but it’s still not perfect per se.
19:49 A lot of the initial sort of like day one hype I saw from people was that this is sort of a perfect model or it could do it all, but it still very much has gaps if you know where to look.
20:01 But I’ve said this for a long time, depending on the person, what your definition of AGI is changes.
20:07 You know, I think for a lot of people, GPT-4 was AGI for them.
20:11 So I think as we progressively make these models smarter and better and more reliable, we slowly encapsulate more and more people into that, like, this model is AGI crowd.
20:23 And, you know, more and more people, the model is good enough for them.
20:27 Is Nikolai here? Nikolai was claiming AGI, I think, after GBD3 or something.
20:36 Yep. I think AGI is here. So anyway, but these issues, my thought is that this model still needs
20:50 a decent amount of polish. The OpenAI team, this guy, Jan, he’s part of the post-training team at OpenAI.
20:57 They are aware of these things. Like he basically mentions, yeah, the sloppy coding and then also the confirmation issue I was talking about before and how OpenAI will be polishing that.
21:06 I think GPT-6 is good. I think for, like, I have a lot of people I talk to in enterprise settings.
21:12 I don’t think I’ll recommend GPT-6 to them.
21:14 But I also think GPT-6.1 and 6.2, those will be very strong models because I assume they’re going to clean up these small little things and just make it really polished.
21:24 And I think those will be extremely strong models to go and use.
21:29 So yeah, looking forward to that. I also just remembered actually Astra doing accidentally a Git hard reset on my Markdown knowledge repo
21:43 and wiped out a bunch of files and then spent a solid 15% of my weekly limit trying to resurrect them from elsewhere, from agent transcripts of what did they write in those files and putting them back or something.
21:57 And I don’t know how it made the mistake.
22:00 It was like, oh shit, it was the wrong Git repo and it was working in that one and only repo.
22:04 So it’s whatever horror stories people were telling about 5.6 about well, it wasn’t a home directory.
22:13 Thanks for that. But it’s still a bit naughty, even though it still asks for permissions for some silly things.
22:19 So it’s a bit imbalanced. Yeah. I think it’s one of those things, once again, going back to like it is meant to be very succinct in its reasoning.
22:29 I think there’s a balance to that because if it just sort of assumes things, like it sounds like it ears, it assumed it was in one Git repo, but it’s actually in a different one, but it assumes something because it’s trying
22:39 to use less tokens because of that it went in, yeah, removed a bunch of changes.
22:44 Which I guess that’s another thing we did not discuss is the pricing increases for this model.
22:48 It’s about two times more expensive than Seoul to go and use.
22:53 They claim that it about evens out in terms of efficiency, but I think we’re getting to the point now where these models are already so smart and so efficient for a lot of tasks, specifically the easier tasks, they’re
23:07 already as efficient as they possibly could be.
23:10 And there’s not much token or cost savings that you could have because of it.
23:14 So that’s why we see on artificial analysis, GPT-6, even though OpenAI, they claim that the efficiency and price increase about counteract each other, on the AA benchmark, they find that it’s actually more expensive
23:29 pretty much across the board. I think only astra low is a little bit cheaper than sole extra high and sold max.
23:36 And so yeah, that’s why I asked about the rate limits because I think this model will consume rate limits faster than Sol will.
23:44 And like I said, I think we’re getting to the point now where sort of like having model routing or using some quote-unquote dumber sub-agents is becoming much more viable because using these big smart agents is just not needed
23:56 anymore. It’s just more expensive. So I think we’ll be seeing more of that in the future as well.
24:04 But yeah, any closing thoughts here about GPT-6 before we go and move on to the next topic?
24:15 All right, moving on then. We were talking about subscription usage and costs, so this segues very well.
24:23 Somebody, I believe, yeah, Chinese, they went and aggregated what the effective cost per million output tokens is for all the different models across all the major different coding plants that you could have right
24:37 now. So yeah, they calculate it in terms of price per million tokens.
24:41 I believe this is a blended rate, factoring in input tokens, output tokens, and cached input tokens as well.
24:49 For those that don’t know, cached input tokens, those are the vast majority of tokens you use.
24:54 And they’re just sort of like previous message history and stuff like that and all the previous tool calls that just get re-ingested into the model after every single turn or message that you send to it.
25:04 But what they went and found is that the OpenAI plan, specifically Luna, is the best price per million output tokens of any model out there.
25:15 So if you’re on the $200 month subscription plan, I believe it is 83 cents per billion tokens that you use.
25:25 So yeah, because I give it in terms of billions, because millions, it’s was at 8,000 or 800ths of a cent.
25:33 So extremely cheap. And so I believe that means you get about 2.4 billion tokens per $200 on that plan.
25:42 So it is by far the most cost efficient.
25:46 And then, yeah, we also see, interestingly, that the Cloud models are actually fairly decent for Sonnet and Opus.
25:52 They’re both in the top 30. And same with GPT 5.6 Sol, I believe, as well.
25:59 It’s somewhere here. Yeah, 33rd. Where they’re both actually pretty price effective, like one cents per million tokens total.
26:07 And then looking on the other side of things of what’s really expensive, Fable is very expensive.
26:13 You get much less usage out of Fable than you do other models, both in terms of rate limits and also just raw value as well.
26:20 You can see here this scale on the right, unlike the left, the left is all linear scaled.
26:27 The right is all logarithmic scaled.
26:29 So you can see here the Fable usage, it’s about what, like 16 cents per million tokens.
26:35 So what is that? It’s like eight times higher than what we’re seeing at the top here with like GLM 5.3.
26:42 So yeah, if you want frontier level model quality, it seems the OpenAI subscription, you get much more work with Sol than you do with Fable on the Anthropic subscription.
26:52 So I thought that was very interesting.
26:53 And then we have all the different coding planes between as well.
26:56 Open CodeGo looks like a pretty good option for a lot of the cheaper Chinese models.
27:01 Seems to be the most cost-effective.
27:04 And yeah, the Grok, if you’re a SpaceX fan and you’re using any of their subscriptions, they pretty much exclusively are on the expensive end of things.
27:14 So yeah, hopefully this is useful. Yeah, interesting.
27:19 Fable is cheaper on a smaller MaxPlan than on a larger.
27:28 Yeah, interesting. So you have more fable if you have two $100 subscriptions and one $200 subscription.
27:39 I assume that with these measurements, I think he just aggregated across a bunch of people.
27:43 Because like I said, this seems to be a blend in terms of token cost between input, output, and cash tokens.
27:49 So if you’re using them in different ratios across different problems, there’s measurement noise.
27:53 So I would say like plus or minus like.
27:54 40% difference. So if it has a 40% measurement noise, the whole benchmark is kind of meaningless.
28:03 I mean, like, you could see orders of magnitude here, though.
28:06 Like, you could see that Luna is two orders of magnitude cheaper than OpenCodeGo.
28:10 So it’s like the general vibe. Because you assume everything is plus or minus 50%.
28:15 Because we’re talking orders of magnitude difference between all of these models.
28:18 But also, actually, I heard that there was a thing, and Anthropic is being sued for that, for their 20x subscription that was marketed as 20x.
28:28 It’s not actually 20x the like the 20 subscription.
28:32 The 20x only applies to the five-hour window and like the $200 subscription that’s twice the price of the $100 subscription is actually 1.7 times the usage.
28:44 So you’re actually better off, which actually matches this benchmark seems like, because it’s 1.0 for Fable on a, what is it, $100 one?
28:54 And 1.6. So it actually matches what I heard.
28:57 That is, like, it is more economical to have two $100 subscriptions, at least until they fix it, or until it’s, you know, they get good court or like whatever.
29:08 Because so. Makes sense. That seems to align.
29:16 Did you hear about that, Andrew, as well?
29:18 Yeah, it’s I saw. I mean, there’s people internally complaining.
29:20 It’s interesting that the Claude Code subscription plan seems to get way more flack on Twitter than the OpenAI plan.
29:27 I think part of that is due to the fact that OpenAI resets the rate limits all the time for basically no reason.
29:34 So, you know, it calms the crowds. Anthropic doesn’t do that nearly as much.
29:38 But yeah, they’ve been having issues with this since basically day one of their coding plan releasing.
29:43 Yeah, I remember people going and measuring it and saying, oh, it’s off for these different reasons, or you’re getting more or less tokens.
29:51 I think in terms of the lawsuit, I remember seeing that lawsuit initially come up.
29:54 I think that was a while ago. I think that was at least three or four months ago.
29:58 I don’t think the lawsuit from my reading of it had any legs or like there’s anything like provable or tangible.
30:05 I think Anthropic, they definitely fudged the limits a bunch.
30:07 Like interestingly, they say, yeah, after September 14th, these measurements were taken.
30:12 Because I think, yeah, depending on the time you take these, Anthropic is changing them constantly.
30:19 But yeah. It’s yeah, they’re definitely not as good.
30:25 Like OpenAI is probably still the better value.
30:29 Even if you like, ignoring the fact that GPT 5.6 is just a better model than Opus, you’re just getting way more usage.
30:38 So if you’re trying to force Fable, else I would be curious too, because this question has the Fable 5, and I’m guessing, because Fable 5.1, the input tokens are much cheaper, right?
30:49 So I would guess that would skew this by like a significant portion, right?
30:55 This graph and probably move Fable closer up.
30:58 I don’t think so. So from what I was seeing initially with Fable is that similar to GPT 5 or GPT-6, where did I come up with Fable?
31:11 I think you mentioned that it’s less efficient for token, but I think that benchmark you said is not like per task, but it’s just like tokens or like billions of or billions of tokens.
31:19 So that benchmark it would be more connected.
31:23 Yeah. So yeah, according to artificial analysis, at least through the API, the model is more token hungry.
31:31 So it costs more and it actually overshines the cost decreases on the input token pricing.
31:38 So yeah, I believe on the artificial analysis coding benchmark suite that they have, it was actually more expensive than the original Fable 5.
31:48 So yeah, I don’t think it’ll change too much.
31:52 It’s also, they have their own custom limits in cloud code, I remember that they announced.
31:57 Like that the 75% decrease doesn’t actually apply to cloud code usage because they already have their own built-in sort of like ratios for there where they price things differently.
32:08 So yeah. What is the 75% decrease? I don’t think I heard of that.
32:15 So the 75% decrease is only for API usage.
32:19 The input tokens, the way they get built in cloud code, they have their own internal ratios.
32:25 Like the ratio between input, output, and cached tokens in cloud code are all calculated differently, which also might be why people then go and get different limits or different costs for the subscription plans than what
32:38 seems to be publicly enough, because they don’t actually follow their own API pricing when they’re pricing those tokens out when it’s coming from cloud code.
32:47 Yeah, I guess whatever they can do to obscure the actual stuff you get in the subscription, that makes a smart move.
32:59 Also still in the benchmark you were showing, it wasn’t taken into account cost per task.
33:03 It was just like billions of tokens.
33:05 So Fable 5.1 should still get more billions of tokens per cent, right?
33:11 Even because they’re cheaper, supposedly.
33:14 Though they haven’t measured it here, right?
33:16 Oh, yeah. I’m guessing they would still pass it.
33:19 They would still pass it in the subscription.
33:20 I’m guessing it might not be 75%, but I’m guessing they just have slightly more efficient attention mechanism or something.
33:27 They would probably actually give more tokens to the end users of cloud code.
33:32 So this would be a bit better for 5.1, hopefully.
33:37 I mean, even if it’s four times better, it still doesn’t even get into the left side.
33:42 I don’t think. It’s still worse than all the GPT 5.6 sole models.
33:47 I guess these are as well. But yeah.
33:50 Cable’s expensive. That’s the takeaway.
33:55 They’re also increasing the limits since like in two weeks or something, right?
34:02 They’re decreasing the limits by 15% or something like that.
34:06 Yeah, they had been doing a boost basically in limits, I believe, up to 50% higher than normal.
34:13 And now they’re decreasing it by like 30%.
34:17 So you’re still ending up like net 20% increase from what it was like three months ago.
34:22 But it is a decrease relative to what it was like last month, if I remember correctly.
34:27 And they got a bunch of flack for that when they announced that because they worded it very poorly.
34:36 But yeah, any other thoughts here about the pricing?
34:39 They also interestingly have cursor in here as well if you’re wondering how much money you get out of your cursor pro subscription for any people that are still on that and all that sort of thing.
34:50 So yeah. Alrighty, moving on then to the next thing.
34:59 This was, I thought this was pretty interesting, where there was this new skill that was going around, somewhat viral, called Ponytail, which is meant to be sort of, you know, like senior software engineer vibes that tries
35:12 to do like the minimal amount of work to try and reduce a lot of sort of like the sloppy code that LLMs go and write.
35:19 And what this person on Twitter went and mentioned is that this skill is a thousand lines of code long and that basically a single, you know, what is this, like 10 word quote unquote skill is
35:33 able to go and do just as well as their, you know, thousand line of code or lines of, I guess, skill or lines of markdown one as well.
35:44 And that, yeah, the token savings are about the same amount of time and it actually ends up being slightly more cost and time effective.
35:51 So I thought this was very interesting because I know many people here, they will go and use their AI agents to go and write their own skills, which ends up making very verbose skills, very long.
36:03 But I think going and taking those verbose skills and then telling your agent like, oh, can you reduce this down to like a one or two liner will probably go and save you, you know, like thousands of tokens per each skill,
36:14 potentially. Because based on this, I’m pretty sure this was also, I doubt this person has a thousand lines worth of thoughts to write about, you know, this code quality style.
36:25 So yeah, this is an interesting, I think, auto-research task, where if you have an existing skill that you know that works, go and tell your agent, hey, go and minimize and make this skill as succinct as possible for me.
36:37 That still goes and does all the same core tenants as the big skill does.
36:41 And then, yeah, just having go and optimize and benchmark itself on its own skill.
36:45 I think that’s a very interesting use.
36:49 Okay, so I’ve got an 800-line skill that I use for releasing to production.
36:54 I’ll give that a try. Try to trim it down because I’ll be able to figure out if it gets worse.
37:00 Because basically everything has to work perfectly.
37:04 So my longest skill is a good test case for me.
37:09 And I have a few sort of test skills as well for different testing things where I think I probably have like 2,000 lines worth of skills where I’ll probably see if I can condense them down to like 20 instead and still get
37:20 the same performance out of them. Actually, I have some thoughts to contribute here.
37:28 Are you familiar with, have you encountered PSAC?
37:32 I have not. No. Oh, definitely. I think everyone here, I’ve been telling everyone.
37:39 So PSTAK is by Lauren Dan. So she’s a principal enterprise at Cursor and SpaceX.
37:46 And she’s like a baby on X. She has like 90K followers.
37:51 But she recently published both a tutorial and she’s been producing articles on how to use the kind of just what they’re doing inside Cursor.
38:03 And I think it’s a very good look at what the workloads, like option level workloads look like with skills.
38:10 And I actually think that you have most principles regarding token or context window.
38:19 And one of the principles I really, really like that they have is this idea of guard the context window.
38:24 Because there’s a principle that every token should be worth its cost.
38:29 Because there are a lot of ways that AI tends to say things remotely.
38:34 But you are incurring a cost every single token that goes into every single context window.
38:44 So I’m very much a big fan of spawning sub-agents when the context isn’t directly relevant to an agent session.
38:56 And kind of like having this whole entire composition workflow where it’s like agents, like sub-agents only get irrelevant skills at their relevant level of orchestration.
39:14 I think this is sort of like, this sounds like progressive disclosure kind of things.
39:20 Absolutely. Yes. Yeah. So is this sort of like tools around progressive disclosure is what they’re giving them?
39:29 Yeah. I mean, I think that that’s kind of the name of the game at this point in time.
39:34 I mean, very much, I will say with a caveat that I think this is in part very much cursor trying to kill their harness.
39:48 But I do think that a lot of the principles do make sense.
39:52 And I think it’s definitely worth doing.
39:58 Especially if you’re burning lots and lots of tokens, decomposing some of the skills and some of the principles and incorporating them into your own workforce.
40:10 So have you been using this brand? Like how have you sort of like experienced this?
40:14 Is it good? So I’ve actually been doing it just like piece by piece, manually taking certain skills.
40:22 Like I really like, for example, they have a bunch of skills related to investigation.
40:27 But what I really, really like about the workflow is actually through the docs section here that kind of explains each part of the workflow.
40:34 But the general idea here is like there’s this kind of entry level, like the entry layer, where it’s like the way that they have, like, her team does this in a cursor, it’s like slash potato mode.
40:46 And then every single command, like they say, like, has that skill, but now there’s like an orchestrator that routes the relevant, like, picks, like the relevant skill and then, like, the relevant set of skill, or like relevant
41:00 workflow, and then executes a bunch of silver into that workflow.
41:03 So then now, like, it’s kind of like, you don’t even need to, like, it’s not like you’re just like doing a raw codec or like a clawed session anymore.
41:12 It’s kind of just more like you have a session with like enhanced by like specific sets of skills.
41:20 And I mean, like, there are certain things where it does feel a little bit bad codey, but like, like, I think that this is kind of partially a publicity stunt, again, to like get people to use
41:34 cursor. But I do agree with a lot of the what they’re trying to do.
41:40 And so I’ve actually just been kind of like picking out and like there’s something like very cursor specific about the inputs that they have.
41:47 Like they’re like certain steps where it’s kind of like they have an arena skill where it’s kind of like four different models from four different providers, each doing a design, comparing, picking out people.
42:00 And that’s kind of a pain in the butt instead of incursor.
42:02 And I’m actually not the cheapest token-wise.
42:05 But I do like the idea of, like, I very much like an idea.
42:10 And like, I’ve been just working and putting it in practice, so I could probably give a better answer at the next Coast Club meeting.
42:18 Yeah, we’d love to hear more about that.
42:20 This seems like, in a way, very similar to like SpecKit, but in a more general way, where SpecKit sort of defines like planning stages and implementation and testing, all that sort of thing.
42:31 But this is sort of a general way of the agent goes and picks out those different pieces for you.
42:35 And you can also customize them more.
42:38 But StackKit, oh, it’s overboard. When I’ve used StuckKit, I’ve literally just stopped looking at what StepKit puts out.
42:47 It spawns like five different files.
42:58 I’m sorry, was that Quentin? Sorry.
43:00 I didn’t catch the alternate to PSTAC that you’re discussing.
43:04 What’s the name? SpecKit. SpecKit. Yeah, which we’ve discussed.
43:08 SpecKit is very different in that SpecKit is more so like if you’re trying to do Spectrum development and you want to just like, you know, building out a feature, you say, oh, help me plan out this feature.
43:19 VStack is more so kind of just like, you’re like, kind of just like, if from now on, you know, like, if you’re going to use quad code, right, like put it into this orchestration flow that now enhances every single one of
43:31 your and like adds more structure and process to every single thing that you say to your cloud session.
43:41 Yeah, it seems like spec kit is what I recommend to all of my enterprise clients where they need that sort of like structure and auditability of like what they’re doing.
43:49 You know, you can’t just vibe code features.
43:50 You need sort of like the paper trail of like, here’s what we were trying to build.
43:53 But it seems like yeah, this, what is it, PSkill or PStack is sort of the more meant for like vibe coders almost, you know, where you don’t have to think about it as much.
44:02 You don’t have to go and read everything.
44:04 It just has a bunch of stuff built in and customized by you to do that.
44:08 I really like that principle of it.
44:10 At least like working at Maritime, right?
44:12 Like one of the biggest things that I’m concerned about is the fact that I have certain PRs that are like 20,000 lines and I’m like, can I really try like there’s no way I can keep up with just manually looking at every single
44:25 line of code change. So like how do you how do you ensure that there’s like a certain level of quality each of the like what the what the biocoding like the agent actually did?
44:36 So it’s like kind of like if you can break up responsibilities into smaller sub-agents and make sure that each one does each task very, very well rather than just one agent a lot of it.
44:49 Yeah, that’s cool. Yeah, we’d love to hear more about it as you use it more, get more familiar with it.
44:54 And yeah, that’s super cool. Yeah, anything else that anybody else wants to discuss around skills or anything like that?
45:05 Fable just told me that it thinks it can get my 800 line skill down to 250 lines.
45:11 Beautiful. By moving some logic into scripts, which is smart, and moving some templates into other files, and then just getting rid of the blather.
45:22 Rationale, history, yada, yada. Stuff that clearly you don’t need.
45:27 But I will say, like, at least like my philosophy is I very much think that if there’s anything that you should manually review, like I think it should be the skills.
45:38 Because you have like agents who love to say things really, really redundantly.
45:42 And like I like, I think that if you’re ever composing skills, right, like no line, like no statement should ever be said about or and like I know that Glaud also likes to like or every agent likes to like kind of slop their
45:56 answers and say things like in a much more verbose way than necessary, which also tends to burn a token.
46:05 Like for example just by virtue of saying like you know like this does like x does y in order to double z right it could have just been x does y to z and then that that like means the same
46:20 exact things totally but like in the other case like you burn like three extra tokens when you multiply that like several hundred times over you start getting like 1,000 line ponytail skills that only need one line.
46:36 I’ll also note that the anthropic documentation for quad code and the other stuff lags behind what the CLI actually accepts.
46:47 I tried to make an agent that understood the documentation as well as the CLI to optimize my setup for things like skills and hooks and rules and agents and stuff like that.
47:01 And it ran into a lot of issues when it was trying to use the official documentation, whereas when it just asked itself what was going on, it was far better.
47:12 So I think that, you know, and that’s also how I learned about how skills in cloud code have a different set of permission availability than agents do and how to kind of guardrail
47:26 against that. So it’s just kind of a word of caution if you’re trying to develop this kind of stuff is it’s not well documented.
47:35 Even at trillion dollar companies, they don’t like updating the documentation.
47:42 But yeah, I think, yeah, I think yeah, Brandon, you were saying that, yeah, people should be manually auditing their skills.
47:49 I agree. I think actually specifically your agents.md and clawed.md files should be very clean and direct because that’s every single session starts with those.
47:58 So any issues or sort of verbosity or anything weird in there gets amplified throughout the rest of the session.
48:04 But I also know most people don’t want to sit down and read through thousands and thousands of lines of AI generated markdown files.
48:13 So having a somewhat automated way, just telling your AI to go and make it more succinct will at least get you to halfway there, get you started down the right path.
48:21 A quick clarification on that is that agents.md does not autoload.
48:26 And as much as the harness may kind of help you do it occasionally, I was testing this out this weekend.
48:35 And the only surefire way if you don’t want to use cloud.md is to sim is to simlink it essentially as a shortcut to agents.md.
48:49 So that way you’re not reliant on the harness or the model to figure it out.
48:55 It just feels invisible and it’s just a file system thing.
49:00 I believe, yeah, you’re talking about cloud code specifically, correct?
49:02 Because yeah, cloud code does not read agents.md.
49:06 Yeah, it will if it will if you do the symlink correct, but otherwise, yeah, no, it completely ignores it.
49:14 Yeah. It’s I believe, okay, yeah, Anthropic is weird about it, because yeah, agents.md is a protocol made by OpenAI, so they refuse to adopt it.
49:22 But I think most other harnesses, like pretty much every other harness, I believe, follows the agent.md spec.
49:29 I think there’s a list on their site for everyone that is using it.
49:34 But yeah, if you’re a cloud code user, yeah, exactly.
49:37 Just simlink it to the cloud.md or ask your agent to.
49:43 This is simlink. But yeah, that’s, yeah, to make sure it gets autoloaded all the time.
49:51 Yeah, basically it’s different from a lot of the suggestions that you’ll see are to go into your collada MD and type in like at agents.md but that relies on the harness slash model to actually properly
50:05 read it and do it. Whereas with a sim link, it can’t, when it goes to the memory address, it only finds those instructions.
50:16 Yeah, exactly. We have four minutes left.
50:22 I think I will quickly highlight the fact that OpenAI is claiming to solve a millennium problem, Navier-Stokes, which is a big open question in the fluid dynamics field of whether or not you can have runaway singularities,
50:36 essentially, in fluid dynamics. It appears that the answer is yes, you can.
50:40 The problem itself being solved actually is not the most interesting thing that’s happening right now, though.
50:45 It is the fact that Anthropic and OpenAI are currently getting into a bit of beef about who actually solved it.
50:52 So basically, Anthropic has this guy Levant, I think is how you pronounce he’s actually a professor here at Harvard, who he’s done a lot of the other sort of like popular, I think he’s the one that published the Jacobian
51:05 conjecture results about a month or two ago.
51:09 But he works at Anthropic or like with Anthropic, but he also uses the OpenAI models to go and do this math research that he’s been doing as well.
51:18 And so, yeah, internally, it is rumored that Anthropic had solved one of these sort of major problems or something like that.
51:25 And now there’s a bunch of drama around the fact that did OpenAI read Levent’s logs to go and see what he came up with and what the solution was or what problem he was working on, and then sort of like scooping that research
51:39 and then claiming it as their own. Because what happened is they solved a sort of tangentially related problem to Navier-Stokes, sort of like a stepping stone along the way.
51:50 But they did not actually solve the full Millennium problem like OpenAI did.
51:54 But there’s drama around that because basically they offered Tristan, one of the collaborators with Levent, to go and write a paper with them where they could be first offered.
52:08 But they’re like, Levent, because he’s associated with Anthropic, they’re like, you can’t be a part of this paper at all for this millennium problem being solved.
52:17 So yeah, there’s a lot of drama around accreditation.
52:21 It seems, according to OpenAI at least, that they sort of solve this proof in a different way than the Anthropic team did.
52:30 And so it seems like there isn’t too much here.
52:32 Like this drama might be a little bit made up or just sort of like they happen to be solving a related problem.
52:38 I know DeepMind as well was also working on trying to solve Navier Stokes as well.
52:42 It seemed like the most low-hanging fruit, I guess, of all the Millennium problems.
52:48 But yeah, so OpenAI is now posting a bunch of proof and screenshots of like, oh, here’s what we actually did here.
52:53 We didn’t access anything from Anthropic.
52:56 But yeah, some interesting drama. But cool that we now have an AI solution for a Millennium problem.
53:02 Like for those that don’t know, Millennium problems are the biggest problems in math in terms of popularity and sort of like notoriety.
53:09 They were, it’s, I believe, the seven hardest problems or like open-ended problems that a bunch of mathematicians, I believe back in 1900, when said like, these are the ones we want to have solved by the millennium.
53:22 One has been solved by a human back in the 90s.
53:25 And now we have the second one, which has been solved by AI today.
53:30 So that’s pretty cool to see. They also come with a $1 million reward.
53:35 Interestingly, though, someone is doing the back of the envelope math, because OpenAI, to solve this problem, they spawned 10,000 sub-agents of a unreleased model.
53:43 We’ll call it GPT 6.5, because it seems like a new pre-train from what they’re saying.
53:49 But yeah, they spent $6.5 million worth of API credits to go and solve this problem.
53:55 So they actually ended up not making any money off the price, which isn’t really the point.
53:59 But yeah, it is very expensive to do this frontier math research, even with these super powerful and smart models.
54:06 But we still get super cool results like this.
54:10 So yeah. And with that, we have reached six o’clock at the end here.
54:15 Any final closing thoughts, comments, or concerns for this?
54:20 Poking max at its finest. Yes. I wanted to bring up another tool that could be used for people.
54:28 So the creator for Pandas, I think, are a few weeks ago, or I came across like a couple weeks ago, something called Agent Skew, which is basically just an objectively better version of the token tracker that I made by
54:43 the guy who created the Pandas library.
54:45 So if you don’t want to use a crappy vibe-coded day token tracker, you can use something that’s a real deal.
54:53 Can you give a quick 30-second overview of what this tool has done, or what this tool is doing, and why you should use it?
55:01 Well, I mean, just because, like, again, I think that it’s very, very, like, I think that the product of AI tools nowadays are the people who understand very well how to properly manage their context.
55:13 And so I think that the global-purse tools that actually manage your token usage and your local context are always going to be really, really valuable.
55:22 And I think that we’re getting to the age where, like you mentioned, Andrew, the models themselves are more than good enough for 99% task.
55:33 And so we’re going to reach this point where people are expecting the models to be better to account for human error.
55:39 But at some point, it’s going to be because the prompting should have been better, not because the model wasn’t good enough.
55:53 It’s yeah, so yeah, this project, I believe, yeah, they’re tracking token usage and yeah, where it’s going and what it’s being used on across all your different sessions.
56:02 For those that don’t know. So yeah, Brandon, yeah, I believe, yeah, previously at Toolscliff had presented his version of it.
56:07 But yeah, like I said, this is the more full-fledged, yeah, over a thousand commits version of it.
56:13 So yeah, Will also mentions CodexBar, which I believe is a Macs app or a Mac app for tracking token usage as well.
56:23 I think it doesn’t give the same insights that you do from agents view would be my guess.
56:30 But yeah, I definitely recommend it.
56:33 If not for no other reason, it’s fun to see how much money you’ve spent.
56:36 Like I use a tool called CC usage and getting to see how much value I get out of my different subscription plans is super cool to see.
56:45 And it sort of incentivizes me to use them more.
56:49 But yeah, with that, we are a bit over time.
56:53 Thank you all for coming today, and hopefully see you all next week.
Subscribe to get the latest AI news in your inbox every week!
