Rendered at 12:06:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pimeys 14 hours ago [-]
I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars...
This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and OpenClaw just felt... not what I wanted. Two days later and 20 dollars poorer I have what I need: a multimodal agent written in rust that has access to my homelab.
Nothing felt off with GLM. It did what I wanted, was fast, had a decent not very annoying personality and was much cheaper than Opus or GPT.
I used it unquantized through Fireworks, but there are multiple other providers too.
gertlabs 12 hours ago [-]
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing.
In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average. Data at https://gertlabs.com/rankings
But when factoring in performance/cost, GLM 5.2 is the frontier model.
jfaat 9 hours ago [-]
> but if you only want to use the best model available, it isn't there yet
I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way more stable. Almost like buying a Ferrari for your daily commute instead of a Toyota or even a Mercedes.
I think there are several factors. Certainly marketing making us think we need the shiny thing which is rampant online and very smart people think they aren't susceptible to. There's a lot of really odd 'I trust Anthropic/OpenAI more than Deepseek' which tends to ignore, for starters, that you can run choose your provider and still save a ton. I also think there's some amount of addiction and brand loyalty where a Ferrari is one hell of a drive so that you turn your nose up at that sensible Toyota. Oh the other one I see used is like oh only fable can oneshot updating my embedded systems thing from 1975 to rust which is great but let's recognize how niche that is.
And it ends up just coming across as people are getting SO reliant on the tools so fast. Maybe it's ok to think and like read a few lines of code and work with these agents to convert your thing to rust or center your div. Even if coding is over which in some sense it certainly is, don't turn your mind into the wall-e people yet. I found myself guilty of this so often. It takes way more time and effort to do things via prompt and I wouldn't just open the editor and fix it because that dopamine hit of the magic the abstraction provided was so strong.
So I'm pretty much done using the 'best' (on benchmarks, if money isn't an object, etc etc) models available. After a year on Sonnet/Opus/GPT5x I'm having way better results with open weights models that don't get lobotomized weekly. I'm finding ways to do the crafting part of building software by focusing on honing my harness and workflow. I'm enjoying changing the oil on my Toyota after a year of almost flying off cliffs in my Ferrari and if I can check my ego it's a purely positive thing.
dofm 48 minutes ago [-]
> I'm trying to wrap my head around exactly why so may people seem to want the best model available
This is the logical end point of the fear-based way LLMs are marketed. You must want the best, because everyone who has the best can work faster than you, generate more — if you don't have the best, you are behind! Why would you want to use anything other than the best?
The thing is, once everyone has the best, the question is: how much can you spend? If you can't spend more, you are behind! If spending the most will get you ahead, why would you not want to spend the most, if you can afford it?
There is only one way through this, in the long run: work out a way forward that doesn't make you dependent on this cycle. If you can compete at all, without the spend, what happens is: they burn money and you don't.
FWIW so far I don't think the benchmarks prove very much about the actual experience, and you can discover this just as easily without spending any money. And we know this about benchmarks! Once a benchmark seems useful as a measurement, it becomes a target and it stops being as useful.
I think your strategy is right. It requires bravery, and as you say, it requires ego balance. But I believe it is obvious that the world will either come around to a more sensible, stable pattern or it doesn't matter either way because we're fucked. So opting out of this mad early cycle and choosing to be calmer and happier is a choice you can just make.
nl 7 hours ago [-]
> most halfway decent models can write damn good code for a fraction of the price.
The difference is how the model is used.
With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks"
With the lessor models the code is fine, but they need something else to plan what needs to be done.
GLM-5.2 is the third model (after Opus 4.6+ and GPT-5.5) that can do this agentic style work.
Notably Gemini 3.1 Pro is notoriously bad at this style work - the code is good, but it drifts off task most of the time. 3.5 Flash is supposed to address this, but I haven't had a good reason to try it.
jfaat 6 hours ago [-]
My whole point is that I don't want it to build an entire feature from one prompt. At most, I want to work with an agent to nail down the spec and then work with an agent that orchestrates the implementation via other agents, same for testing, etc. None of that requires frontier capabilities, it requires a little bit of work on a harness, a little bit more of my input, a little more of my brainpower. I _want_ to build tools that make it work better and don't change when the CC team gins up some default for their harness and foists it on me. I don't see that as a tradeoff at all and I think engaging in my work process more than fire and forget (and literally always in my experience fix stuff later) is more fun and rewarding once the 'holy shit this is now possible' high wears off. Doubly so once the frontier model gets nerfed mid-cycle and now I have to undo the mess because they released v*.x++ and I fell for it again by trusting it to do these agentic tasks without my involvement.
pimeys 5 hours ago [-]
Yep. I've tried to use the models to build large things for me. You can't trust the code it produces. Even if it works there are parts that are hot garbage, and will bite you later on. I've found out that having an editor open, asking it to implement things until a certain point, manually fixing some of the worst things it generates, then asking it to expand from there is much better than just prompting a thing and pushing to production.
And hey, don't get me wrong, you can get pretty far with just prompting. But the subtle misses and (I'm looking at you GPT) the overengineered 20k line PRs to do a simple thing are going to cost you a lot if you're not vigilant.
seviu 3 hours ago [-]
I would say 3.5 flash is great if you use a good open harness. I use omp for that. The thing with Google is that they announce they have a great model, and that they have been testing it internally for half a year. I guess they don't care too much about who or how he uses it.
I am still struggling how to deal with sub agents and different roles for each model. I still think Claude or Codex are overall better models, but everything around them transpires such weird vibes, including, and this one kills me, that at certain times they feel like dumbed down.
I keep changing these things often, but I have basic subscription to codex (20$ plan) which I use with GLM 5.2 to do some high level planning of what I intend to do, and then leave Deepseek do the coding. Or something along those lines.
Point is, GLM 5.2 is now at a point where I cannot tell you if it's better or worse. I can tell you however one thing: no matter when I use it, it's consistent in what it does and how it works.
Then there is the Fable thing, but as with many things, I think the past has distorted the reality. It lasted two days, but Anthropic said it clearly for plan users it would only be there for two weeks. It was great for doing what you can already do with other tools: doing all the planning, and reviews, and launching a million subagents talking to each other. I sometimes wonder if it was really a new model, or just Opus 4.9 wrapped with some fancy model driven harness.
As for Fable: I used it as much as I could while we had it.
It was a step change over Opus with my work.
swiftcoder 4 hours ago [-]
> With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks"
I've had no trouble getting the current generation of smaller models to do the same thing. Maybe it's more of a harness issue than a model issue?
Recently I've used both MiniMax M3 and DeepSeek V4 Flash to one-shot moderately complex applications from a written spec, and neither one got lost along the way
NitpickLawyer 6 hours ago [-]
> 3.5 Flash is supposed to address this, but I haven't had a good reason to try it.
Price and speed, for me. GLM5.2 is "good enough" for some tasks, but rather slow (on their coding plan). In the time it takes GLM to "read files to figure out...", gemini flash is usually finished. It's not SotA for coding, but it's fast and often "good enough" for normal tasks.
andai 8 hours ago [-]
Yeah, the funniest thing about everyone freaking out about Fable's capabilities recently was that for most of the stuff they were amazed by, you could get roughly the same result from DeepSeek Flash.
I used to be obsessed with what's the best model. Then a while back when the new best model came out, I tested it on a task. I also tested its little brother (much smaller model from same company).
They both completed the task perfectly except the "best" model (the bigger one) cost 5x more and took 3x longer...
Ladioss 4 hours ago [-]
"Best model" discourse always remember me of my days in Monster Hunter with people who refused to consider playing with anything other than the meta set for their weapon and then proceed to immediately cart right at the beginning of the hunt :)
With the wealth of models available (open source vs closed, api vs local), I find optimizing the cost-efficiency of your token consumption an important part of business-oriented AI engineering. You don't need "the best" for every task.
cdud3 2 hours ago [-]
A lot of the monetarization strategies for LMM's depend on the need to use them via SaaS subscriptions. If companies start to realize that local AI is cheaper, provides good enough results and makes them independent then that monetarization strategy falls apart and a whole industry collapses.
realusername 7 hours ago [-]
> They both completed the task perfectly except the "best" model (the bigger one) cost 5x more and took 3x longer...
Same for me, I certainly don't have the same definition of success and failure either.
A more expensive model has *less* rooms for wandering around than a cheaper model.
If Claude wanders around during 10min until finding the most obvious solution, then I count it as a failure.
grosswait 48 minutes ago [-]
Because not every problem is a coding problem or not entirely solvable through code. Other tasks include legal, philosophical, financial, investigative, and combinations of these and others.
cicko 38 minutes ago [-]
It doesn't look like that's where the conversation was going, though.
peheje 4 hours ago [-]
Reason people want the best: people want to believe their project is so advanced that they need the most clever LLM possible. To say otherwise is to admit that it's not really frontier or novel in any way. And people don't like that.
YmiYugy 5 hours ago [-]
I’m writing a lot of React code and find that the cheaper models are pretty terrible.
Maybe I’m holding it wrong but the experience that the cheaper model is usually enough just track with my experience.
Worse, I find predicting the difficulty of tasks exceedingly difficult. More often than not using the initially cheaper models requires me to reroll with a more expensive one or waste a lot of times and tokens cleaning up the subpar results.
With OpenAI and Anthropic still subsiding tokens, not using the best models still seems like a tough ask.
dsrtslnd23 4 hours ago [-]
I agree, but there are use cases for the 'best model' other than converting your 1975 stuff to rust: for use cases where LLMs are just getting started to be useful I really want to use the current 'best' model: e.g. CAD, PCB design etc. In particular anything which requires spatial reasoning. The short time I had access to Fable 5 - it was just way better than any other model.
dofm 37 minutes ago [-]
Except that there is no application for AI in CAD that is better, more appropriate, more robust or more sensible than learning how to use a CAD package and doing it yourself.
It's not fast-changing, it's not abstract, it's just not that difficult, and where it is difficult, the AI cannot help you, because it is not capable of things you are capable of.
Learn CAD yourself. Honestly; I was sure I would never manage to learn CAD but it turns out to be interesting, rewarding, valuable and actually quite quick to learn.
An LLM certainly is not going to be able to do it better than you once you have a tiny bit of experience. (PCB design, perhaps, has a language to it that an LLM can make a bit more headway into, but as a non-PCB-designer I would still bet that it's more like CAD than code)
ifwinterco 5 hours ago [-]
I think people are grouping into two flows.
One group is trying to get the LLM to basically one shot everything and not properly reviewing the output.
Others are using the LLM to assist their human intelligence in a tight loop.
If you’re doing the former you really do need the best model available because that’s still right on the edge of what LLMs can do at best, and at worst you’re just shipping pure unmaintainable slop.
If you’re doing the latter then you can get away with a slightly less powerful model without it making a material difference because your human intelligence is filling in gaps
marcyb5st 5 hours ago [-]
Well put. I belong to the latter group as I feed small, granular tasks that I describe thoroughly to the LLM. I tried, however, to just give it a bigger scope task. Even best models produce sloppy code.
While the single functions/classes/structs/... can be well though out the code tends to lack cohesion, and especially maintainability. For instance, it never thinks: "I could put this logic in an interface/trait so that if the requirements change I can simply add a concrete implementation that satisfies the new requirements (and potentially use one of these for testing)".
ifwinterco 3 hours ago [-]
Yes that's also my experience.
SoTA models can do reasonably good jobs on each ticket, but over time the architecture of the application starts degrading without a human in the loop.
The entropy increases slower with better models but the trend is always towards slop
cik 7 hours ago [-]
I've landed in a similar place by reducing effort and cutting up tasks. I find that more exacting specifications to the models, yield significantly less need for "effort". Combining each with multjple git worktrees and an integration branch for the current worktrees themselves has yielded incresible results.
This also allows me to play with, and mix codex, claude cli, and others. This is my happy spot for the last two months.
jfaat 6 hours ago [-]
Yeah this is sounds close to my workflow and its good to hear you've find a nice flow too! It frees me up to spend that effort on doing more things in parallel and focusing way more on the specs which is usually a good idea anyway.
darkstar_16 5 hours ago [-]
It's also geeks and engineers using these models and being the most vocal. We always think we're special and need the extra horsepower. Ever been on one of those home lab subreddits ? Same story.
miroljub 3 hours ago [-]
Of course people want the best model available, even at 10x costs, if they are not paying for it. If the company is paying, why wouldn't you want a 2% better model?
That changes as soon as the developer is the one paying for a model. Then it's a classical engineering trade-off between money and quality, and that's where open models are clear winners.
ssk42 9 hours ago [-]
What is your favorite harness for the open weights?
jfaat 8 hours ago [-]
We built our own and aren't done open sourcing it but before that I got to a really good place with opencode plus some custom agents, pi family is good too although I haven't used it as much. We made an agent to design a spec, one to implement by dispatching subagents, one to validate against the plan, things like that. All of this helps claude/gpt too IME. For open models it has helped them stay out of loops (e.g. Kimi's but WAIT) and for frontier it helps them stay on task and not invent bloated patterns
SeriousM 6 hours ago [-]
pi is great for learning, oh-my-pi has all the nice things included that I've built fory pi previously.
NamlchakKhandro 9 hours ago [-]
pi-mono
ithkuil 5 hours ago [-]
What is pi-mono ? (I heard about pi)
re-thc 2 hours ago [-]
> most halfway decent models can write damn good code for a fraction of the price
The problem isn't what they do in a blank state. It is how they get there and the edge cases. Some models also take longer (uses more steps) i.e. end up costing more despite being "cheaper".
I've seen models:
- Back out plans non-stop. Tried the obvious path. Invents X/Y/Z excuse (without verifying) that it can't be done. Notes that down and moves on. It could be as simple as site A being down and to download from site B but that's it.
- Hacks the test to make it work. Code is wrong? Nah, let's update the test.
- Keep saying useless things like YAGNI and infinite excuses like too risky to never do the work.
- Claims they are done but there's 100 edge cases not covered. When you try to use it it fails in ways you as a human assume it should work. You can write a spec to cover it all but then what's the point?
- Be trigger happy and never investigate. Tries to do it. 5 minutes. Oh it failed. Back out. Repeat. Better models definitely spend more time analyzing and actually "think". I've had models spend hours trying to do a change due to this method when an actual investigation (code walkthrough) might have solved it.
- Know and use the right tools. A lot of lesser models have infinite fear e.g. oh docker might not be available (it is) or this and that (even if you nudge it in any way) and spend a lot of extra time "working around" it.
The list goes on. Better models definitely help.
Only thing to agree on is no you don't need Fable but saying Sonnet can do the job instead of Opus is a different story. It's so obvious when Sonnet touches the code that I can't give it more than 5 minutes. It lies. Doesn't check. Forgets things and then messes up.
secrooq 26 minutes ago [-]
[flagged]
hedora 8 hours ago [-]
In your box plots, 4.6 sonnet wins over all (even opus 4.6, the 4.8’s and fable).
That’s not super surprising to me, but, given the apparent randomness of the stack ranking, is GLM actually worse than any of the Anthropic models? This looks like a 10-way tie to me.
gertlabs 8 hours ago [-]
We've spent some time trying to understand this anomaly, even re-running Sonnet 4.6 through our evaluations to see if that would bring down its scores... and it didn't. I don't know what they did differently, but it's basically Opus 4.6 with more temperature variability (some great responses, some less great, with an approximately frontier median response in agentic work specifically). It is smart, methodical and excellent at tool calling in our custom environments.
We now use Sonnet 4.6 for a number of internal use cases we wouldn't have considered otherwise.
hedora 7 hours ago [-]
That tracks with my experience.
4.7 was so bad, I locked a bunch of my machines to 4.6.
I haven’t bothered locking the 4.8 machines to 4.6. There was a HN thread a while back where they run swe bench a few times a day and measure success rate and latency. It showed opus getting significantly dumber for the week before a recent launch.
It wouldn’t surprise me if they’re quantizing to improve margins or to hype models in comparative testing in order to defraud investors at IPO.
Or, maybe QA is hard. Anyway, I think they hit a performance wall sometime at or before 4.6.
yfontana 3 hours ago [-]
Doesn't track with mine. I've been stuck with Sonnet 4.6 with one of the clients I work for. It writes code fine, but it's not nearly as good as the more recent models for everything else. It's fairly common for it to suddenly go off the rails for no good reason, so I can't really trust it with agentic loops.
It's also not very good at diagnosing non-trivial issues. It's not uncommon for it to suggest whole lists of irrelevant / nonsensical reasons for something not working. Then I copy/paste the code and some context into chatgpt and it hones in onto the correct issue right away, even with inferior tooling.
matheusmoreira 3 hours ago [-]
> In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average.
Just want to express how amazing that is. Opus 4.6 is an amazing model. That an open weight model like GLM 5.2 competes with it is nothing short of outstanding.
neya 9 hours ago [-]
What is the methodology of your benchmark?
On the contrary, I personally think these broader benchmarks are meaningless. I think personalized benchmarks are the way to go. They should answer "How does this model perform for MY use-case?" rather than trying to answer "How does this model perform across all coding environments?"
Case in point: I use Elixir which is not as popular as Python, is always a hit or miss with most SOTA models at the top of these benchmarks. Whereas, the ones in the middle of the benchmarks (like the GLM) almost always outperform even SOTA models from Google / Anthropic. However, this is relevant only for my use case and I wouldn't just advocate a model for everyone based off my use-case alone.
gertlabs 8 hours ago [-]
We use a rotating pool of ~100 games for the coding parts of the benchmark, and are scored objectively based on ratings similar to Elo. Models write code submissions to interact with the environment, then are evaluated in large batches against other submissions.
We test 11 popular/interesting languages (you can see the Languages chart to filter), but not Elixir -- although other evaluations have found that many LLMs solve more problems when working with Elixir [0]. Why models write code well in some languages over others seems to go beyond pre-training data (Python scores quite low for most models) and we don't fully understand it.
An expressive and well designed language (elixir) is objectively better than a less well designed language like python. Python probably needs more LoC than elixir for the same task. Python is also untyped by default.
aeonfox 1 hours ago [-]
Elixir is not just expressive, it's highly conventional. I've found best practice code usually converges on the same idiomatic patterns, and well written codebases look very similar to each other in style
neya 7 hours ago [-]
Thanks!
ronsor 10 hours ago [-]
Opus 4.6 is still my preferred model for work, so this is great to hear.
echelon 9 hours ago [-]
I can't wait for open models to take over in all categories.
Sounds like this is the year for coding.
pizzly 9 hours ago [-]
It looks possible open models will. I never expected the reason would be political/legal rather than technical.
echelon 43 minutes ago [-]
The CEOs spent so much time talking about putting everyone out of work and how "unsafe" their models were that the government stepped in with export controls.
They did this to themselves.
robrenaud 6 hours ago [-]
If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.
rolisz 6 hours ago [-]
With Claude Code Ultrathink, I used 3 million tokens in 20 minutes. At API prices, that would be around 30$. So 90$/h. Model cost is not that much lower.
kennywinker 5 hours ago [-]
x40hrs/week * 50 weeks = $180k
Congrats, now you’re paying an engineer’s salary to make your engineer at best 20% more productive.
Better to hire another engineer, or two jrs, and build up your in house talent.
cicko 29 minutes ago [-]
except this is way more than an engineering salary. At least in Europe.
koito17 8 minutes ago [-]
Europe is a continent. There's significant differences in salaries across countries. Salary in e.g. Ukraine is nowhere near the level of Germany. Likewise, the salaries in the United Kingdom and Switzerland are much different than those of Germany.
OtherShrezzing 6 hours ago [-]
I don’t think any engineers who cost $150/hr are having their productivity moved by 20% depending on a $10/hr gap between models on or near the frontier.
Most of the gains right now come from tooling and process and any big post 2025 language model. The specific model isn’t that important right now.
pimeys 5 hours ago [-]
Exactly. And being able to choose your own tools is much more valuable than having a tiny bit better model.
YmiYugy 5 hours ago [-]
But SOTA models used liberally at API pricing is a lot more than $10/hour.
You can probably burn $100+/hour with just a single agent, and probably thousands when running agents programmatically, e.g. workflows.
__alexs 4 hours ago [-]
I find it hard to trust a ranking system that gives Sonnet a higher capability score than Fable.
bjourne 10 hours ago [-]
Man, there is exactly zero information on your site about how your benchmarks work. Why should one trust your numbers when there is no way to verify them?
gertlabs 10 hours ago [-]
Scroll to the bottom for the methodology (sorry, this should be linkable)
ukuina 5 hours ago [-]
Why is Sonnet 4.6 ranked higher than Opus 4.6?
ComplexSystems 7 hours ago [-]
Sonnet 4.6 is ahead of Opus 4.7? Hm.
jchw 12 hours ago [-]
After having used GLM 5.2 and Opus 4.8 for enough time, I'm very unconvinced of the benchmark maxxing claims - if anything, GLM 5.2's rather lackluster performance on benchmarks compared to Opus 4.8 paints the opposite picture when compared to the subjective experience.
When I first used Opus 4.8, I threw several different workloads I had at it - I have Claude doing a lot of misc projects whose primary purpose is pretty much just studying what AI agents can do for my own curiosity and no other reason. Opus 4.8 was one of the first models I ever snuck in there that basically ran out of control. No previous Opus or Sonnet model I had used ever did this. Within hours every agent I had running was writing non-sense tool calls that echoed pretend commands that didn't exist, like 10 in a row, and talking about the "tool channel" being dirty. I switched back to Opus 4.7 and assumed Opus 4.8 was legitimately just broken.
I did come back to Opus 4.8 and found that it was indeed, pretty powerful. But that initial experience has stuck with me on just how narrow of a perspective any given test or benchmark is guaranteed to have. LLMs are too broad, it really doesn't matter what you try to do in your benchmark, you will necessarily get a limited view of what the model is capable of and its shortcomings. This will remain true for at least as long as models are susceptible to massive swings in performance based on randomness and minor differences in prompts and other environmental factors.
I'm not saying benchmarks are useless or that your benchmarks are not possibly closer to the truth either. All evidence at least points to the idea that Chinese models perform very well in coding but often have more mixed results on other tasks. I'm just saying that at this point, benchmarks feel like they have limited connection to my actual real experiences. GLM 5.2 actually scored kinda meh on a lot of benchmarks (compared to closed frontier models) but my actual experience using it does not match this.
And I'm definitely not saying GLM 5.2 is better than the frontier LLMs here, just that the race is close. I still prefer GPT 5.5 right now for code review, I think, and Opus clearly has some advantages depending on the task. It's just no longer a given that Opus 4.8 will perform better than GLM 5.2 on any given task, so to me the calculus behind "using the best model available" is getting complex and you might need to get a feel for what models have what strengths to really figure it out.
I do feel like the "use the best model available" mentality is not going to die any time soon, but if it does die, it will be gradual and start soon for programming. Modern LLMs are still not a full superset of what human programmers can do, but still larger models are definitely starting to hit diminishing returns for tasks at the lower end of complexity, and that is a big deal. It's a weird world where some tasks you can feel kinda confident just throwing Gemma 4 at it and not sweating whether you should use a better model; I've certainly done it for some quick Python scripts or getting an overview of some code I'm unfamiliar with.
avereveard 8 hours ago [-]
I really dislike opus 4.8 it rarely compete things and prefer to waste tokens making lists of things that are missing. When stuck or need input it words the challenge at length without conveying anything useful for decision making, and quite often its solution to problems is to excise features or just try catch errors and proceed with faulty data silently
skeptic_ai 11 hours ago [-]
Why Deepseek v4 flash is better than pro in your benchmarks?
gertlabs 10 hours ago [-]
It's 100% due to tool use -- Flash adapts much better to our custom harness with tool names that are not identical to what models were likely trained on. DeepSeek V4 Pro performs much worse in that aspect than almost all other recent releases, for whatever reason.
rockwotj 11 hours ago [-]
I have also found deepseek flash beat pro in some of my own internal evals for tasklet.ai it’s really surprising and I don’t understand it
freakynit 10 hours ago [-]
Same.. although rare, but have observed twice till date.
Some blog post I read few weeks back said that DSV4Flash in xHigh effort beats even the pro model in xHigh effort.
onoesworkacct 10 hours ago [-]
The rumour is that it's trained on Opus, but who knows
rockwotj 10 hours ago [-]
Oh of course all deepseek and glm are. Multiple people have seen GLM self report that it is claude, which makes it super obvious.
I think the surprising thing is I expect flash to be a pure distillation and strictly worse quality but clearly it’s more nuanced than that.
kennywinker 9 hours ago [-]
Claude claims to be deepseek, under some circumstances:
Don't ask western llms in Chinese what model they are...
xbmcuser 9 hours ago [-]
maybe they distilled claude for the flash version and not for the other hence better tool use and programming benchmarks
marci 5 hours ago [-]
This was a preview release. They haven't finish training. The Pro contains more knowledge but it probably takes longer training than flash for the smarts to kick in.
Madmallard 11 hours ago [-]
Notice the website url is the same name as the commentor.
Notice he's using "trust me bro" benchmarks.
Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized.
Everyone is grinding and marketing nobody is actually discussing anything for real.
nl 7 hours ago [-]
What does this even mean?
Madmallard 4 hours ago [-]
It means people have self-inflicted AI psychosis
nl 3 hours ago [-]
It's always been ok for people to talk about their projects here. In fact it's encouraged.
Aditya_Garg 13 hours ago [-]
Im really curious about this. Why pay API pricing? I burn 1000s of dollars a month of api according to claude usage but only pay the $100 subscription
horsawlarway 12 hours ago [-]
My increasing frustration with these plans is the harness lock in.
Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates.
So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.
cortesoft 12 hours ago [-]
They postponed that change, here is the email they sent out:
> In May, we sent you an email announcing that starting today, the Claude Agent SDK, claude -p, and third-party apps built on the Agent SDK would stop drawing from subscription rate limits and move to a dedicated monthly credit. We're writing to let you know that we’re not making this change today. We’re working to update the plan to better support how users build with Claude subscriptions.
> What this means for you
> Nothing changes for now. Agent SDK, claude -p, and third-party app usage continues to work with your subscription exactly as it did before today, and there's no credit to claim. Your subscription limits are unchanged. When we have an update, we'll share it with advance notice before it takes effect
clhodapp 6 hours ago [-]
Something I haven't been able to figure out.... How are you supposed to actually get an API key to use quota from your subscription? The terms of service still forbid using OAuth authentication and the API keys from the console indicate that you need to pre-load your account with funds when you try to use them.
huksley 4 hours ago [-]
They reverted this decision, "claude -p [prompt]" works with your subscription ok.
throwawayffffas 12 hours ago [-]
Z.ai does not lock you in to any harness.
hedora 8 hours ago [-]
Is there a secure way to use GLM without spending $10K’s for local HW? I “only” have a 128GiB inference machine, and don’t really trust anthropic not to steal my IP over time.
I see no reason to trust Z.ai more than other vendors.
throwawayffffas 4 hours ago [-]
Kind of, you need at least 256 gb of vram and 24-40 gb of vram to run the 2bit quantization, because it's a moe you just need the expert to fit in vram to get significant improvement over a pure CPU setup. At 2bits though expect significant quality loss.
villish 5 hours ago [-]
You'd need to multiply that $10k by 8 minimum.
sroerick 12 hours ago [-]
I'm using synthetic.new and Neuralwatt with pi and its good and also cheap
computerex 12 hours ago [-]
I have had bad experience with neuralwatt GLM 5.2. Seems like they may be using quantized version of the model.
scottcha 11 hours ago [-]
Hi I'm the CTO of neuralwatt, would love to hear your feedback on what your experience was. Feel free to email me scott@neuralwatt.com. Also for GLM5.2 we run the FP8 quantization at 1M context which is a common deployment target.
versteegen 7 hours ago [-]
Hi Scott! Was just considering signing up, NW looks great (fp8 GLM 5.2 is good!) Standard cached token pricing for GLM 5.2 is pretty high, I'm wondering whether the KV cache for that model actually is that expensive to serve on average, or if Neuralwatt's energy pricing for long-running GLM 5.2 agents is especially competitive? The live energy stats don't break down by token type, would love to see that. And 2/3 of the examples given in docs/energy-methodology are models you don't even host anymore. Uncertainty and selective stats puts people off signing up, they tend to assume the worst. Oh, and MiMo or DS4 please :)
johne20 7 hours ago [-]
I had good experience with neuralwatt in my heavy testing on real project in last days. Price/performance for api pricing was great. When using with pi, I was a little confused on if/how it supports diff reasoning levels?
4 hours ago [-]
weird-eye-issue 12 hours ago [-]
I think they rolled that back
smcleod 12 hours ago [-]
They canned the moved to make -p commands API billable.
redox99 9 hours ago [-]
And codex is even more subsidized. It's an absurdly good deal.
SV_BubbleTime 13 hours ago [-]
There is a whole iceberg topic on subsidizing.
So your question is really “if they’re giving free usage, why not take advantage of it?”
I do, so I don’t know the reasons not to, other than to experiment.
AussieWog93 11 hours ago [-]
[dead]
shostack 14 hours ago [-]
If you're using Matrix, consider Hermes as a harness if you haven't already. Native gateway support. I've been primarily using mine through Element and it has largely been great.
pimeys 14 hours ago [-]
Oh interesting. I basically chose Matrix because setting anything up with Whatsapp or signal was kind of painful and telegram doesn't make it easy to use encryption with bots.
I kind of wanted to see if I can make a Matrix agent from scratch with Rust with GLM and it was surprisingly easy. Just make something for myself how I want it. Maybe I'll take a look on Hermes later...
Barbing 12 hours ago [-]
Very interesting—Element X solved a lot of the pains of Element (iOS), could be a good solution!
andai 8 hours ago [-]
Nice. I'm working on an agent too. How are you handling tool calls?
but I'm running into issues with nested backticks so I'm thinking of making dedicated close tags per tool call.
neya 9 hours ago [-]
I am seeing extremely positive results with Elixir too. Previously I was on Deepseek (deepseek-v4-pro) and GLM5.2 outperforms Deepseek easily. It's been a month since I used any native Claude models (simply because of pricing) but then, GLM5.2 is running for me at $20/day in usage on OpenRouter for GLM5.2. I am not sure if I've misconfigured Claude code or if this is indeed normal usage pricing. But, the output more than makes up for it. However, using Deepseek v4 pro directly from deepseek.com using their discounted pricing is insanely cost efficient. I topped up $10 a month and a half ago and I'm still yet to use up all the money in my account. Here's hoping that SOTA models become even cheaper!
KaoruAoiShiho 13 hours ago [-]
Are you sure fireworks is unquant? It's not listing precision on openrouter like everyone else.
try-working 8 hours ago [-]
Have you tried using DeepSeek V4 Pro instead? It will be cheaper and faster than GLM.
jklmnopqrstuvw 11 hours ago [-]
> A typical session for me with GPT is usually over a hundred dollars.
I don't think a $100 session is "typical". I use GPT for months. $20/m plus plan is enough for my daily work.
simple10 10 hours ago [-]
I use an observability tool with claude code [1] that shows me usage including prompt and session cost. Even though I use a max subscription, it's interesting to see what it would cost me if I was using API directly.
My typical session ranges from $100-$400 - higher end when using workflows with lots of subagents. $100/session is expected when using the API without the subsidized subscription pricing. Most larger orgs have to use API pricing AFAIK.
There are Business and Enterprise plans, both have discounting.
adamtaylor_13 10 hours ago [-]
It's really interesting what "normal" is for folks. I use the $200/month Anthropic subscription and use it within a few percentages of my limit every week.
I'd blow through $20/month plan in hours.
jascha_eng 6 hours ago [-]
Shorter sessions more often doing a /clear etc. save a shit ton of tokens. I pay 100 bucks a month but barely use 30% of it most weeks.
tjwebbnorfolk 10 hours ago [-]
I have Claude max plan and the vscode claude dashboard plugin has logged about $4k worth of tokens in the past 2 months. I upgraded because I was using my weekly basic plan tokens in like 5 hours.
Likewise, I don't understand how anyone survives on the basic plans. It's funny seeing these two camps not understanding what the other is doing :)
gguncth 7 hours ago [-]
What makes you use API billing instead of a plan?
dist-epoch 14 hours ago [-]
$20 on API pricing or on subscription?
pimeys 14 hours ago [-]
API, pay per token.
Chrisoaks 10 hours ago [-]
Why are you not using the subscription plan?
pimeys 7 hours ago [-]
I want to run the model in Western servers. And GPT/Opus is paid by the company which doesn't really get subsidized tokens.
In the future none of us do, so it's better to trial how the actually adorable models perform.
Just FYI, this question was a quote from Pulp Fiction, the other commenter (mdre) replied also with a quote, that was an answer to this question in the movie.
mdre 13 hours ago [-]
[flagged]
wahnfrieden 8 hours ago [-]
Why are you spending on API for GPT coding instead of stacking 20x subs and using codex-lb?
pimeys 7 hours ago [-]
Company pays API prices so we can use daily the best model for our job without being locked in. Also the team subscriptions started to be more like X per seat + usage...
wahnfrieden 5 hours ago [-]
Oh it sounded like personal use.
I understand the reasons to use team/enterprise accounts, but apart from the policy/management/billing side of it, I still don't understand the value in spending thousands for API instead of hundreds - even when there's argument that one provider is better than another depending on the use case, I don't think that credibly extends much beyond OpenAI + Anthropic frontiers, which both have $200 subs you can stack.
dom96 12 hours ago [-]
Twenty dollars?
How are you comfortable spending that much to write something as simple as a matrix bot?
Are people doing this kind of thing just super rich or am I missing something?
ygjb 11 hours ago [-]
It's pretty simple. There are things that I do because it's fun, like gamedev. I hand code that, and don't use LLM tools because I like learning and building. I do lots of utility stuff coding for my wife's business, most of that is stuff I could do in a few hours. It's worth $20 to not spend a few hours doing it. It's a cost benefit tradeoff. I won't learn much fixing WordPress themes or adding a feature to her web page, or setting up an automation for her, so I don't see the point of doing that.
Same thing for stuff at work. Oh, the tables/schema changed and my queries broke? I could dork around with spark and cypher for an hour, or I can tell claude to update the queries for the new schema. At the rate I am paid, spending on Claude tokens is generally a better use of my resources.
Building a net new solution? Coding tools take a back seat until I get the core logic right, then I let automation handle web page and UI scaffolding.
annzabelle 9 hours ago [-]
A lot of people spend $20 on a hobby for an hour of enjoyment a couple times a week. Not odd at all to do that for a few hours of coding if you find it fun. It could be a day pass at a bouldering gym or a yoga class or amortized running shoes/garmin/electrolytes.
copperx 10 hours ago [-]
$20 is really cheap for the amount of work saved, considering you're in the US.
adamtaylor_13 10 hours ago [-]
Is spending $20 considered "super rich"?
yard2010 4 hours ago [-]
Recall that the marginal utility of money diminishes when you have more of it - when you have a lot of money it's easier to turn it into even more money, and vice-verca. It's not linear. So 20$ difference has exponential not linear influence on "being rich".
NamlchakKhandro 8 hours ago [-]
Yeah we're all doing this from our Super Yachts that performs Marine Biology research in its spare time.
TimXare 9 hours ago [-]
[dead]
playorizaya 12 hours ago [-]
[flagged]
SwellJoe 14 hours ago [-]
I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro. But it turned out MiMo got lucky, it's performed worse on almost every test I've done since, while DeepSeek has consistently been among the best performers and its extreme caching performance makes it cheaper than just about anything, including much smaller models.
Also of note, I found giving models access to the open source semgrep as a tool makes some perform worse and none perform better, though it's plausible there's a way to wire it up in a harness that presents useful information to the model without the model having to know how to use it (my theory is that semgrep isn't heavily represented in the training data, so you're asking the model to do two things at once: figure out how to use semgrep and find security bugs, and both tasks suffer for the lack of focus...most small models, and some big models, can't do that well).
Edit: But, also, more testing is ongoing. I suspect GLM 5.2 will also be a consistently strong performer. It seems to excel at most things I've tested on it.
Overall, I still think GLM 5.2 is the much stronger performer. It's hard to tell the difference between GLM 5.2 and Opus at <120k tokens.
SwellJoe 10 hours ago [-]
I have found that some models consistently find or miss specific bugs, and which bugs are hard don't completely line up across all models, so I believe that. I just refactored the security bug-finding harness I've been working on completely (not checked in yet, testing it currently) to strongly encourage "multi-model, multi-pass" scans and make them easy to orchestrate with de-dupe and weeding false positives with a strong model, rather than one model or doing just one pass over each file. Giving a model a second attempt increases their findings by 20-30%, and giving them a third, adds another 10-15%.
I'm inclined to use DeepSeek V4 Pro the most, because it is consistently extremely strong, it's very fast, it's very cheap and has excellent caching and cheap-as-free cached input tokens (something like 80% of token usage is cached when I'm using it for security scanning). So, my probably "pair" of frontline security researchers will probably be DeepSeek V4 Pro and Gemma 4 31B self-hosted (another shockingly strong contender, competitive with the best models once you let it loop on the same file a couple/few times). But, I won't be surprised if GLM 5.2 turns out better than DeepSeek V4 Pro...it costs quite a bit more.
faeyanpiraat 5 hours ago [-]
So its like run 3 loops of “here project, find bugs” with all good models, then dedupe and priorize with a sota?
SwellJoe 5 hours ago [-]
The loop is "look at this file in this repo, find bugs" iterated over every file in a project, with the ability to look at the rest of the repo for cross-file bugs related to the file they're instructed to look specifically at, but yes. The Anthropic folks have basically said that's how they're doing security audits (Nicholas Carlini is an Anthropic employee and he's done talks about it), so I assume that's how Mythos found its bugs.
I've benchmarked it, and the "here's a repo, find bugs" approach finds far fewer bugs. Like, dramatically fewer. Models are good and contexts have expanded, but focus still wins with hard problems. You could probably tell the good models to make a plan to audit the repo, and it would end up making its own "loop" in the form of a checklist of files to look at over several sessions or via subagents, I assume.
acters 9 hours ago [-]
I believe it is because GLM 5.2 has extra anti-cyber training instilled in it. Similar to Kimi k2.7 code.
Deepseek v4 pro being in preview with less "safety" training makes it stronger for that reason. Thinking will be different and in the end, it will actually try to be useful. Just expect future Chinese LLMs to further push out "safety" guided LLMs. The future is bleak for open weight models. Prepare to have "guidelines" enforced unceremoniously to all.
qingcharles 8 hours ago [-]
Every time a new frontier model arrives I have it give one specific codebase of mine a once-over for bugs and other idiotic mistakes.
Fable found a couple of good ones, then we lost Fable, so I tried GLM5.2 and it found two critical bugs that Fable had missed, so it got my seal of approval.
Barbing 12 hours ago [-]
We need a benchmark of independent community sourced benchmarks!
…probably already is one
SwellJoe 11 hours ago [-]
I don't know how you'd judge benchmarks beyond "did it test and measure what it says it tests and measures". And, I guess there have been instances where the benchmark failed to do that, and the models could cheat in some way and it just tested the models ability to find the answer key. In the case of my benchmarks every model other than Claude models running in Claude Code never have network access and all information from after the bug was discovered has been removed from the repository the model can see.
But, there are benchmarks for so many different kinds of ability, I don't know how to compare them directly against one another. Like, models that do well on terminal and agentic coding benchmarks tend to do well on finding security bugs, but it's not a 1:1 correlation, there are surprises.
mapontosevenths 10 hours ago [-]
It's not super scientific, but I really like to watch Bijan Bowen's videos on Youtube. I think he's pretty fair about the way he compares them, and it's enough for what I'm doing.
SwellJoe 10 hours ago [-]
Actually doing something normal but challenging with a model is generally enough for me. I do a quick (an hour or two) project, and see how it holds up. If I'm feeling like it's harder than it should be, I switch to a comparable model I know is good. e.g. I most recently tested Gemini Flash 3.5 for making a web app. It shit the bed...kinda worked, but was ugly and needed several bugfixes right off the bat. I tried the same app in Opus 4.8, which aced it with barely any extra conversation, it looked great (basic but clean, like it was intentional) without any effort.
I like reading benchmarks, but I take them all with a grain of salt. They're just to tell me if the model is worth even trying for my task. I've heavily used self-hosted Qwen 3.6 and Gemma 4 on a bunch of different tasks, and while the benchmarks consistently say Qwen is the better model, I simply don't find that to be the case for anything I do. I think Qwen is tuned for benchmarks, while Google couldn't give two shits about most of the benchmarks, they're just busy making unusually smart tiny models.
onoesworkacct 8 hours ago [-]
could mimo have scraped the mythos findings already? it's very recent
SwellJoe 6 hours ago [-]
That's covered in the article. All bugs (which you can see here: https://github.com/swelljoe/nelson/tree/main/cases ) are extremely recent (like a week old when I pulled them at the end of May). MiMo 2.5 Pro was released in April, at least a month before any of the cases were published, and I don't remember the exact training data cutoff for that one (if I found it), but I'm certain it's at least a couple/few months before the release date, as the base training when the data gets baked in is usually followed by weeks or months of post-training.
Anyway, it isn't possible for any of the models, so far, to be trained on the Mythos bugs. We're getting closer to the point where I have to worry about that, at which point I'll roll forward and pull some newer CVEs from what they've published, assuming they keep publishing new bugs. (And, if they don't, it's trivial to switch to just random CVEs. But, finding out what Mythos is up to is interesting.)
amhoab 9 hours ago [-]
Aren't you the Webmin guy?
SwellJoe 6 hours ago [-]
More the Virtualmin guy. But, yeah, I also work on Webmin and have since '99, so I'm a Webmin guy. But, Jamie is the Webmin guy. (And, I'll note that something like half of my commits to Webmin over the past few months have been bug fixes of bugs found by models, sometimes via Nelson, sometimes just interacting with Opus in Claude Code.)
mattmcdonagh 11 minutes ago [-]
GLM-5.2 suggests long-horizon agentic work is becoming open, cheap, and deployable.
I ran it on my laptop, which is a Lenovo Legion 5i (think 32 GB RAM, 4060 w/ 8 GB VRAM, you get the picture). It was a quantized model (otherwise it would not fit on my NVMe 1TB drive) at 4 bits per weight - UD_Q4_K_XL. It ran at about 12 seconds per token (not tokens per second). A fun project, but not worth it. I used 4096 tokens of context cache, and I ran it with llama.cpp - as it supports memory mapping. Because the whole thing could obviously not fit in RAM, I was curious how much it would need to stream from SSD. The answer? For a simple 4 sentence description of who it was, about 1.5 TiB was streamed from disk.
bArray 12 hours ago [-]
Thank you for sharing. 1.5TB of streamed data at 12 seconds per token on a high end consumer laptop is a pretty high requirement - I can only imagine how much that cost to train. I don't know how running this model could be cost effective for anybody.
Retro_Dev 11 hours ago [-]
Indeed - definitely not cost effective to run it on this laptop LOL. It makes me wonder how fast we could run the model if we could fit the weights entirely within CPU cache (assuming a whole ton of CPUs with low latency & high speed IO of course).
scosman 10 hours ago [-]
short answer: they mostly aren't
A few people are running highly quantized models with limited context windows. It's still impressive, but not the benchmark level intelligence. Very few people could afford a rig for reasonable local performance at a reasonable quant, at full context size.
The antirez example is 2.6bit quant, 32k context, and few tokens per second... on a ~$7000 MacBook M5 (new RAM pricing).
8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps..
Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.
For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of money in local models unless you have a business where you're already paying for many employee's individual token usage.
Aurornis 14 hours ago [-]
> 8 X RTX6000. It will run you around 80-100k to get started
8 x RTX6000 GPUs cost $100,000 alone. You then need to build a system that can support those GPUs with enough PCIe lanes through a PCIe switch.
It's going to be $120K to $150K to build or buy a system to run this.
cheschire 12 hours ago [-]
Not to mention the three separate dedicated 15A circuits you would need to have installed in order to run the 3x 2000W power supplies running ideally at no more than 1400W sustained load each. And definitely would need 200A service to the house if you have a family living there with you.
But hey you could save on heating?
InvertedRhodium 10 hours ago [-]
That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door.
A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.
Aurornis 7 hours ago [-]
> That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door
In the US it's common to get 200A 120/240V split-phase service. We're talking about the wiring inside the house, though.
How do you think everyone here is charging their electric cars at home and running our AC and electric cooktops at the same time if we didn't also have that? :)
You need to derate for constant loads here, and I assume you have to do that in NZ as well.
So, no, not a "uniquely US issue".
bentinney 8 hours ago [-]
Not so sure about that. 200amp @ 240v is pretty standard for modern houses in the US. My house in Japan was only 40amps, so there are plenty of countries where this would be an issue.
knollimar 13 hours ago [-]
isn't throwing that into a [insert financial vehicle that gives 99.99999% safe returns] going to destroy that when you factor in electricity costs?
Or even just electricity costs vs token cost
CamperBob2 14 hours ago [-]
You can run the NV4FP quant with 8x RTX6000 cards at 50-75 tps output, but not (practically speaking) the OEM FP8 version. You will learn more about PCIe than you ever wanted to know.
The real gangstas are running 16x RTX6000s. Too rich for my blood, and the NV4FP quant doesn't seem to be that much worse.
Sanzig 13 hours ago [-]
Anyone done any benchmarks on the NV4FP quant? Seriously considering pitching an 8 x RTX 6000 Pro box at work to run GLM-5.2 in an air gapped environment.
tiahura 12 hours ago [-]
Good luck. I’m in the legal field, and even there, selling airgapped is tough.
botro 9 hours ago [-]
What are the challenges you've seen in selling air gapped? Is it the high upfront cost? Challenges with hardware maintenance or something else?
DrScientist 2 hours ago [-]
Given GLM is open weight - all you need is one company to take the taalas approach ( model on hardware ), and you're sorted right?
>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.
Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down.
I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - say a 32GB/64GB M7 Pro.
I also wouldn't be surprised if, 3 years after that, cheaper hardware and improved model efficiency means that there's a much smaller gap between what you can run on a consumer CPU (which, with memory prices coming down, could look like a 256GB M9 or M10 Pro) and $100k GPU cluster.
marcus_holmes 10 hours ago [-]
This is clearly where the industry is going, imho. Everyone who is playing with LLMs wants a laptop with enough grunt to run a decent model locally.
We've been sat with basically the same PC specs for ~20 years - our current specs are within an order of magnitude of the ones we could buy back in 2010. This is not really constrained by tech, as we could have much, much, larger machines. It's more because there's no mass demand for much, much, larger machines - if it's big enough to run Office apps or VSCode then you're good to go. The exponential growth we saw in the 90's was driven as much by software demand as it was by hardware development.
I can see the next 10 years produce the same kind of push for larger machines that the 90's did. And we should probably expect the same kind of standards churn as our existing technologies for storage, memory, etc, don't scale up enough and new technologies become worth developing because there's demand for them.
user43928 3 hours ago [-]
It seems relevant for playing with LLMs, but for actual work this seems far off for me.
My productivity profits from the best intelligence available, a decent context size, and a batch size of four.
While my MacBook has 48 GB of RAM, not only do I want the above requirements at a decent speed, but I also need my machine to run the development tools and test suites, ideally without the fans blasting at full load.
For the foreseeable future I will stay with providers rather than local inference, apart from niche use cases.
marcus_holmes 3 hours ago [-]
Yeah, agree, but that's the point, really. If I could buy a 16Tb machine with 4 TPUs for ~$5K and run a frontier model locally, I would.
I'm in Australia, so we're probably not getting access to Fable again. We're learning that a faster model + better harness/framework > smarter model. So being able to run GLM5.2 locally and super-fast would be great.
byzantinegene 8 hours ago [-]
my only concern if the same specs today would cost 10x more given the trajectory of the growth of memory prices lately.
marcus_holmes 7 hours ago [-]
I think this is where the new technology comes in. There is demand for 10x (or 1000x) the memory that we're using at the moment, so someone/something will satisfy that demand. We haven't had that demand up until now, because 16Gb was a perfectly reasonable amount of memory that could run pretty much anything, and if that won't then 32Gb will. There was zero demand for 16Tb memory machines because no-one had any application for that much memory. Now that's changing, and there is demand for that much, so we'd expect to see that being made available.
But the existing tech we're using for 16Gb probably isn't going to scale to 16Tb at a reasonable price point. And the price point is relatively inelastic - people are used to paying <$5K for their computers, and they're not going to go much above that. You'll get early adopters paying $10K or more for a machine that large, but not the early majority. And even then, obviously, $10K is not going to buy you a 16Tb memory machine.
So there's room for a new technology to come in, where there wasn't previously. This is what happened all through the 90's, and we churned through a bunch of standards and technologies to try and keep up with demand.
internet_points 1 hours ago [-]
> memory prices coming down
Are they?
I suspect AI labs are buying stuff not just for their own use, but to make local use too expensive to be an option :-( And they can always make the "best" frontier model even bigger (though only fractionally better) so it's always out of reach of local use, while consumer laptops have nearly the same amount of memory they had a decade ago.
m o
o
d
e
l o
s
i o
z o
e 2020 2022 2024 2026
c
h
e
a
p o
R o
A o
M o
2020 2022 2024 2026
vagab0nd 8 hours ago [-]
For most tasks, I don't value the LLMs based on their absolute capabilities. I wouldn't want to use GPT-4 today even if it's free.
dakolli 7 hours ago [-]
I'm being very sarcastic, local model evangalists seems to just be operating on vibes when they say these things and are completely disconnected from how models work, what the hardware requirements are.
Prices aren't going down, and consumer platforms are being shipped with less RAM so we can be sold cloud products. This isn't going to happen.
Can you please explain to me how you're going to fit 700bb-1T params in 64GB of RAM? You realize there are memory requirements proportional to model size?
NitpickLawyer 6 hours ago [-]
> Can you please explain to me how you're going to fit 700bb-1T params in 64GB of RAM?
You don't. What they're saying is that today's small models (that fit on consumer hw) are better than yesteryear's top models. GPT4 was reportedly 8x 220B (~1.6T) MoE, and today you can run a 30-120B model that beats it handedly in real-world tasks.
Similarly for 4-20B models beating GPT3 (175B) and so on.
There is a sweetspot of "good enough" that the small models can reach, where you get equivalent tasks solved fully locally. They'll never touch SotA, but they'll reach 2-3-4 year's SotA. Which, depending on the task you need, it can be "good enough".
krackers 14 hours ago [-]
Would you be better off pooling that money with some hackerspace group and then setting up shared inference infra, so that way you at least get better utilization?
KaoruAoiShiho 13 hours ago [-]
And before you know it, you invented some openrouter provider from first principles...
janalsncm 13 hours ago [-]
Right. For example you will need to figure out how to share it and who maintains it.
aetch 12 hours ago [-]
You can then rent spare capacity out to people on a subscription or token basis ….wait
InvertedRhodium 14 hours ago [-]
Depends how much you value privacy and running uncensored models.
Personally, I’m waiting for hardware to hit the secondary market before I buy something to run unquantized models like GLM. But I have no doubt that I will, at some point.
8note 14 hours ago [-]
you can however, have fun with it.
oil workers buy 100k trucks they do not-much with. why not a 100k in computer?
afavour 14 hours ago [-]
Because car loans can’t be used to buy computers
frangonf 3 hours ago [-]
Surprising that the banking industry has not come up yet with the AI native consumer product loan for GPUs.
OliverGuy 2 hours ago [-]
Probably a bit niche at the moment really. The only people interested in that are us nerds, and the product segment is very adhoc - especially for the local crowd where an epyc, with a bunch of pcie riders and some 3090s on a steel frame is considered optimal
bandrami 1 hours ago [-]
Paging Mr. Son. Mr. Son, please pick up line 3.
ElProlactin 13 hours ago [-]
And there's your idea. If you could find a way to get people to add another $500/month over 80+ months to an auto loan, dealers would eat that up like filet mignon.
jliptzin 12 hours ago [-]
Yea as far has hobbies go, I feel like this is on the low end. I know people who collect watches and corvettes, that's way more expensive and functionally you can't really do anything special with them.
theteapot 12 hours ago [-]
The difference is watches and corvettes typically appreciate in value, where as computer hardware typically drops like a rock.
15155 11 hours ago [-]
> watches
Some, and the market fluctuates a ton.
> corvettes
Only the oldest, most unique model years: nobody is buying (C4-C5-realistically C6) mid-90s or early 2000s Corvettes for more than what they paid for them, and they never will.
parineum 11 hours ago [-]
> The difference is watches and corvettes typically appreciate in value
Both of those things' value drops like a rock as soon as you buy them and, at least for cars, they don't all appreciate. Most don't. Even so, they appreciate at an incredible slow rate.
I can't speak for watches but I'd be surprised if it wasn't the same situation.
At least the gpus can create value after you buy them before they are worthless.
randomNumber7 12 hours ago [-]
Also LLMs are mainly used for work and if you can spend 6 digits on watches your likely financially independent.
cdelsolar 10 hours ago [-]
hmm ok let's build a state of the art from 2021 homelab using 2x Epyc Milan chips + DDR4 RAM and lmk how much it costs...
Ken_At_EM 14 hours ago [-]
I can't help but ask where this comment came from, you must have some exposure..
CamperBob2 14 hours ago [-]
It is so easy to spend $100K on a pickup truck these days, it's not even funny.
tiahura 12 hours ago [-]
A Honda minivan is > 50k.
SV_BubbleTime 13 hours ago [-]
Factory F350 Platinum is at least 90k sticker.
hedora 8 hours ago [-]
Yet Ford claims it is impossible to sell any pickups for > $60K, so they killed the lightning.
I assume (since they claim they are selling the batteries to AI data centers), they’ll produce some sort of EV >= F150 once the bubble pops, and we get a new president.
SV_BubbleTime 6 hours ago [-]
Automotive EE here… every other decision about vehicles is about emissions. CAFE, the reason that a company releases X model is that they can then sell more Y models that get worse mileage.
EV is a separate thing. Vastly overmarketed for the technology as it exists today.
dakolli 14 hours ago [-]
Sure, If you want to light money on fire for entertainment, more power to you. There's probably worse ways to light 100k on fire. If I have an extra 100k laying around it's going to my family though.
Ldorigo 12 hours ago [-]
How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?
bandrami 1 hours ago [-]
Inference providers have been getting a firehose of investor cash to keep the chips running (and are looking around very nervously as that firehose starts to sputter).
kingstnap 6 hours ago [-]
Output tokens are actually kinda expensive for the provider.
The input cache hit tokens are incredibly cheap for them, (incredibly high margin too, except for deepseek).
And input tokens are in the middle. Input tokens can be processed very efficiently.
Also his math is wrong. $100k gets you 22.7B output tokens at $4.4/M which is how much GLM 5.2 costs.
At 500/s 22.7B is just 500 days. Or about 1.54 years. Which is much less then the life of the hardware.
ac29 10 hours ago [-]
The inference providers are running batch sizes much larger than 10
As an individual I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag. Once they start trimming down the excess and making them field focused they will run just fine on people's individual devices.
JumpCrisscross 13 hours ago [-]
> I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag
Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?
KetoManx64 13 hours ago [-]
Do quantized models specifically prune out specific knowledge? I think they just compress things down but they're still in there. You'd most likely need to do that when you're doing the initial model training, but I'm not expert.
JumpCrisscross 11 hours ago [-]
> they just compress things down but they're still in there
The compression is almost certainly in part specific knowledge getting fuzzed.
DennisP 10 hours ago [-]
Yeah, but it's everything getting fuzzed, including the parts you care about.
JumpCrisscross 9 hours ago [-]
Sure. There is a legitimate question around whether one can selectively excise “useless” knowledge. My guess is you can’t. The act of learning it encodes both the act of learning and the knowledge per se. The former is the power of the LLM. (I personally force mine to double check everything instead of going off memory.)
kibwen 13 hours ago [-]
Quantizing is one thing. But in general it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability, otherwise you'd have AGI just from reinforcing your model on memorizing the first 10^50 digits of pi.
Likewise, LLMs do not violate the laws of information theory, and therefore the only way to encode X amount of information in Y amount of bits where X > Y is by performing what is effectively lossy compression, and as X grows larger relative to Y the compression ratio must change to lose ever more information.
Yes, for the sake of making chatbots that are "conversational" in that they can interpret natural language as input and produce code as output you can easily benefit in incidental and unintuitive ways by training it on more natural language text. But for a given fixed parameter size, it's possible to produce a better model for a specific task by selectively not muddying its training set in the first place with things that are likely irrelevant to the task.
coldtea 12 hours ago [-]
>But in general it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability, otherwise you'd have AGI just from reinforcing your model on memorizing the first 10^50 digits of pi.
It's hardly self-evident, and your counter-example is hardly applicable.
The first 10^50 of pi is not the same as having BREADTH of information in the training data, which is the whole point not just any random "information that is irrelevant to your use case".
not to mention that the first 10^50 digits of pi compress to quite small formula, so not much information there to begin with from a shannon/kolmogorov perspective
kibwen 11 hours ago [-]
It is self-evident. Bringing up Kolmogorov complexity is irrelevant, we're talking about rote memorization, but if you can't ignore the given example then replace "digits of pi" with "bits of output from a true random number generator". There's an infinite amount of information that we could shove into a model, and a finite amount of bits with which to store any of that information such that it can be usefully recalled or form useful logical associations.
coldtea 3 hours ago [-]
"rote memorization" is not the right way to describe how an LLM works.
The memorization of say 100000 world facts through training texts, which enrich model associations all around, is absolutely not the same as rote memorization on 10^50 digits of pi. Not for a human, and even more so, not for an LLM.
An LLM trained with digits of pi and one trained with books and posts, even if they both have the exact same amount of bytes of training input, would not be comparable in any way in utility and reasoning capabilities.
>There's an infinite amount of information that we could shove into a model, and a finite amount of bits with which to store any of that information such that it can be usefully recalled or form useful logical associations.
Which is irrelevant. Anyway, the amount of information that doesn't form useful logical associations is even larger (e.g. actual human books vs possible permutations of characters and spaces). Just like those (random) possible permutations of characters aren't good for LLM input to get logical associations out of it, pi isn't either (logical associations of the kind we care for and expect, not of the kind related to pi's sequences).
Also it's not only not self-evident, it's also apparently wrong.
JumpCrisscross 11 hours ago [-]
> it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability
We don’t understand AI or natural intelligence well enough to make such statements. As for self evidence, cross-domain competence in humans and the rise of generalist models over domain-specific ones (on competence, not cost) seems to pretty directly tank your hypothesis.
tiahura 12 hours ago [-]
Apparently irrelevant data can help because model weights are entangled.
wonnage 14 hours ago [-]
Yeah, the neoclouds and hyperscalers are taking massive losses right now, self hosting is basically signing yourself up to do the same. There are philosophical reasons to do so but it’s a terrible economic decision
rekttrader 14 hours ago [-]
Or you have data that HIPAA, GDPR, PII, or have to care about the concern of others training on your data.
dakolli 14 hours ago [-]
That too.
dist-epoch 14 hours ago [-]
> 50tps for a decade
assuming demand doesn't keep on increasing. even google has trouble having enough capacity apparently.
Rekindle8090 12 minutes ago [-]
[dead]
9 hours ago [-]
14 hours ago [-]
himata4113 16 hours ago [-]
These numbers are seem pretty low compared to what I was able to achieve specifically around windows kernel, win32k<->win32u to be exact. It honestly wouldn't surprise me anymore if china started surpassing models that US makes public, at least in specific categories such as cyber.
GLM 5.2 is already capable enough to assist in self-training which is similar to what we saw happen with frontier models and they appear to be getting there at a significantly lower cost than openai/anthropic.
acters 9 hours ago [-]
I am finding Chinese models are introducing more guidelines against cyber. Especially Kimi k2.7 code seems to have extra training against cyber security capabilities. Last one, k2.6 was a lot stronger at cyber but obviously the Kimi team improved over time, so this is not the best they can do but no one will be able to get the best anymore.
I expect future Chinese models to introduce even more of this type of bogus "safety" training.
Looks like if you are a white hat, then you will be fighting an uphill battle. Black hats will be fine, they will not care, they can just run a heretic model or specialty trained model.
EMIRELADERO 11 hours ago [-]
> These numbers are seem pretty low compared to what I was able to achieve specifically around windows kernel, win32k<->win32u to be exact.
Care to give more context to this? Seems interesting
danmaz74 13 hours ago [-]
It will almost for sure surpass the models which Trump will allow US "allies" (which he just considers client states) to use. This, together with China's growing dominance in PV, rechargeable batteries, EV, could really be the nail in the coffin for the post WWII economic world order.
himata4113 13 hours ago [-]
Honestly, it's becoming increasily hard to disagree with such sentiment when china is preparing itself to lead in energy, manufacturing, research, chip production and so on while there's an entire group of people trying to put datacenters in space.
woeirua 13 hours ago [-]
You are delusional if you think China is going to let Europe have access to Mythos level models for free.
chillfox 11 hours ago [-]
Why not?
Mythos level really doesn't seem that scary. And it would be a great way to take away the American labs international market.
I think it would make strategic sense for them to release more capable models than what American labs are allowed to make available to the world. It would help them grow their global soft-power and be a destabilizing effect on the American economy.
BobbyJo 9 hours ago [-]
It is fairly obvious to me that the open models are a form of "dumping" as far as the economics and the desired outcome from China's perspective. They get to watch as the US pours tons of money and talent into an industry, then prevent that investment from having any return. In 5 years we'll be on equal footing, China will have spent 1/1000th the money, and the only downside will be that they spent 5 years being 6 months behind.
China could not be happier.
The same model is going to apply to the silicon supply chain as well is my guess. 1000th the expenditure in exchange for being a little behind the curve.
I worry it will have a very real chilling effect on research and development, since customers will probably very quickly switch to the thing that costs 1/10th as much, sucking out the ROI.
frabcus 3 hours ago [-]
Sounds good from an x-risk point of view then. Maybe that's their deliberate plan!
hedora 8 hours ago [-]
Didn’t they already? Mythos isn’t even SOTA according to Anthropic (they point at GPT 5.5), and third party benchmarks have massive error bars where Fable, GPT 5.5 and GLM 5.2 overlap.
lukan 12 hours ago [-]
To hurt the US, maybe. I have not tried it, but GLM here seems already pretty capable.
jmye 12 hours ago [-]
What does "free" have to do with anything?
danmaz74 7 hours ago [-]
We'll see. Helping Trump in destroying USA's traditional alliances is probably worth more to China than keeping a Mythos for themselves.
solenoid0937 16 hours ago [-]
GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months.
Not that it would make any sense.
rgbrenner 15 hours ago [-]
If that happens it'll be an absolute disaster. Imagine a scenario where Anthropic and OpenAI prohibit most US companies from using their latest models because of safety.. And meanwhile attackers use equivalent open source models to attack US companies.
Any prohibition on open source models will do nothing to fix the problem.. since attackers will never feel bound to the law. All advanced models must be available for defensive purposes.
andy99 15 hours ago [-]
Right, but is there any evidence of intelligence behind any of these (government) decisions? It’s just regulatory capture + marketing (plus some people living out an imaginary fantasy that they’re in Neuromancer or something), absolutely no reason to think they won’t try and target open models as part of this.
popalchemist 15 hours ago [-]
There's at least one reason:
much harder to make a profit in policing non-american companies and open-source models without huge (or even any) MRR.
If the real motive is profit, then open source models are likely simply not a viable means to that end.
wokkel 1 hours ago [-]
They did the same by banning strong encryption. Never underestimate the stupidity of politicians
hedora 8 hours ago [-]
OpenAI and Anthropic are already unable to make SOTA models generally available (and support this, oddly enough).
If huggingface or whatever is forced to take down open source licensed weights, there’s always bittorrent.
Export controls are one thing, but the US doesn’t really have import controls, and there’s no copyright issue, so DMCA, etc don’t come into play.
It’d take the courts years to decide how to contort the law to ban open weight models, and by then, it’ll be too late (and also pointless).
richardlblair 12 hours ago [-]
And someone will start a competing company in a sane environment.
solenoid0937 15 hours ago [-]
> since attackers will never feel bound to the law.
But that's the whole point.
Fall out of favor with the admin and you lose access to the good American models, aren't allowed to use Chinese ones, and fall prey to the attackers and behind your competitors.
lenerdenator 13 hours ago [-]
It'd be less about "safety" and more "we've spent trillions developing these AI tools only to have the Chinese, once again, copy them and offer them for pennies on the dollar, and no one seems to care about the impact that has on the long-term sustainability of this sector of the American economy as a whole, so we're yanking the models."
jmye 12 hours ago [-]
"I'm going to take this box razor and make some really deep cuts around the middle of my face because my tech sector is too good and that's actually a bad thing because $foreigners."
lenerdenator 11 hours ago [-]
I'm not saying it's necessarily a good thing. I'm also not saying it's about foreigners at this point. It's about seeing a bet through. They've burned a metric crapload of capital on developing AI models and the infrastructure to host them. They want that money back and then some. Remember, the fine shareholders of OpenAI think that 100x returns just aren't reasonable and want that cap lifted. If this kind of thing continues, they'd be lucky to make their money back at all, let alone 100x.
Which would be fine, but as we know, people securitize the crap out of their investments these days, and least some people probably leveraged themselves on some US AI companies, so now the risk is spreading outside of the sector to the economy in general, which is made worse by the sheer amount of spending on AI.
aussiegreenie 15 hours ago [-]
The Americans may ban the use of the Chinese models in America. But like the Chinese car ban, everyone else will use them.
hedora 8 hours ago [-]
Technically speaking, Chinese cars have not been banned. They are subject to a 100% tariff. They’d still be price competitive, but the manufacturers haven’t bothered jumping through the regulatory hoops.
I’ll happily pay a 100% tariff on open weight models, and there are no regulatory hurdles for them to jump through (yet).
lenerdenator 13 hours ago [-]
That's not necessarily a good thing for everyone else, mind.
Yes, you get your free model, but the cost of this is not developing your own capability and tying your fate to a country which may or may not have your best interests as a nation in mind.
This is just the deindustrialization that occurred in my home region (the American Midwest) playing out on a global scale in different sectors. It was originally driven by the Japanese, who, to their credit, acted more as partners than competition. Eventually that desire for larger margins went to China, and now you basically can't build anything of consequence without at least some Chinese parts, because there's "no economic case" for it. This means that you have to play Beijing's game if you want access to any sort of modern market.
You see this happening with Volkswagen's restructuring, next you'll see it with non-American, non-Chinese AI.
singpolyma3 12 hours ago [-]
It's not really the same because we already have the model. If China stopped letting us have it tomorrow I'd doesn't matter because... We have it already
chillfox 11 hours ago [-]
So... how's that any different from using American stuff for those of us in the rest of the world?
Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something.
At least with China, we are very good at winning trade wars with them here in Australia.
lenerdenator 11 hours ago [-]
You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021.
Or, on a more local note, an Australian automotive worker who worked for a company that figured out 10 years ago that they wouldn't be able to pay him a decent wage, compete with the then-upcoming Chinese EVs, and remain profitable.
Paradigm2020 11 hours ago [-]
You might feel different if you're a palestinian who's getting american bombs dropped on him, or an afghani collateral damage or...
There is no good guys in general, and whataboutism and making the scope bigger doesn't help.
The thing is that if the models you are building on are open source whether hosted on chinese / american / whatever service at least give you an option to switch provider easier vs a fable / chatgpt 5.6 that gets banned for none americans etc...
2 years ago america would have had the branding/perception advantage but right now that is well and truly gone...
Danox 7 hours ago [-]
More what aboutism American Indians, Aborigines, Māori, Sami, New Caledonia, the Kanak people what do they all have in common? Sent to re-education camps at some point in time, some of them sterilized, And all treated his second class citizens. One of the reasons most countries are relatively quiet about the Chinese is that so many other countries have indigenous people that were treated pretty much the same at some point in time in their history…
Stop pretending there’s some type of moral high ground there isn’t. Disgusting.
Barrin92 6 hours ago [-]
> or a Ukrainian soldier who has to deal with drones built with Chinese components
man you're gonna be disappointed when you learn where the components for Ukrainian drones come from (spoiler alert, it's China 95% of Ukrainian drone manufacturers use Chinese components. Both Ukrainian and Russian drones are Chinese components glued together, the vendors in China literally stagger Russian and Ukrainian buyers on the factory floors to not have them run into each other). The largest trade partner of Vietnam and the Phillipines is China.
The kind of thinking that assumes that rivalry implies deglobalization or bloc politics is exactly what's 30 years out of date. It's projecting how Americans think on the entire world, but that's not how the world works any more. The rest of the world continues to globalize, even through war.
America is undergoing Sovietization and erecting an Iron Curtain, and China ironically enough is simply doing what the US used to do. If Americans think the rest of the world will follow them into isolation they're going to make the same discovery the Russians did in the last century.
nl 6 hours ago [-]
> Or, on a more local note, an Australian automotive worker who worked for a company that figured out 10 years ago that they wouldn't be able to pay him a decent wage, compete with the then-upcoming Chinese EVs, and remain profitable.
I don't understand what your point is? This seems like a perfect example of comparative advantage - Australia can produce iron ore cheaper than anywhere else in the world and even when China launched a trade war against Australia the Australian economy kept growing.
There wasn't even any bump in unemployment from the closing of the car industry.
Once that trade war was settled, Australia got cheaper cars, China got cheaper iron ore and both economies won.
The rational behavior on both parts there is in stark contrast to current US policy, which is unpredictable and capricious.
> You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021.
This seems like a random list of complaints about China and I agree with them in general.
I think you'll find most major powers have similar complaints. There certainly are against the US - I think you might find that both the Philippines and Vietnam(!) have fairly mixed feelings about the US for example.
skissane 14 hours ago [-]
> GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months.
I’m sceptical they could find the legal framework to do this even if they wanted to
They have legal authority to (a) prevent export of US goods/services; (b) ban imports of physical goods; (c) ban transactions (including purchasing services or license agreements) with foreign firms
But I’m not aware of any legal authority which lets them ban US firms from running a Chinese-developed open source AI model in the United States, if they are at arms length from the vendor, and aren’t using it for government contracts or regulated applications
Possibly they could order HuggingFace/etc to suspend Chinese accounts. But if someone in the US (or a third country) downloads the model from China then reuploads it to a US server, completely independently of the vendor - where is the legal hook to prohibit that?
bardak 13 hours ago [-]
They could ban payment processors from processing payments to any hosts of GML 5.2, despite the open weights the vast majority of people will be using cloud providers to get access since it is to heavy to host for 99% of people.
This would be extremely heavy handed and probably end up accelerating the loss of the virtual US monopoly of payment network. The reast of the world isn't going to let the US dictate that only they get the frontier models whether their US made or otherwise
skissane 13 hours ago [-]
> They could ban payment processors from processing payments to any hosts of GML 5.2
Can they actually though? Do they have legal authority to tell a payment processor that it has to block transactions of a legal US company, just because the company is hosting a Chinese-developed open source model? I’m sceptical
And what about companies (e.g. AWS) that let you “bring your own model”?
bardak 13 hours ago [-]
It would be extremely heavy handed but the administration has sanctioned the International Criminal Court judges such that they basically have no access to the Wests modern financial system. I think domestic US providers would have to deal with different ways but someone like Herzner could easily be cut off from the financial system if the administration doesn't feel that they are adequately blocking the model
skissane 12 hours ago [-]
> It would be extremely heavy handed but the administration has sanctioned the International Criminal Court judges
That's sanctioning specific individuals for specific acts they performed which the US claims contravene its interests and those of its allies.
I don't agree with the ICC sanctions, but it really can't be compared with the proposal "sanction any company, even US domestic entities, which use a Chinese-developed open source model".
In fact, I think part of what enables the US to sanction them (under US law) is the fact they are neither US citizens nor residents; if they were US citizens living in the United States, I don't think the President would have the legal authority to impose those kinds of sanctions.
They could sanction Hetzner–because it is a German firm based in Germany. I don't see how they could sanction a US firm based in the US whose owners and staff were US citizens.
Also, the 5th Circuit Court of Appeal decision Van Loon v Treasury (Nov 2024) is relevant–it held that IEEPA (the law used to sanction ICC officials) couldn't be used to sanction the Tornado Cash smart contract system, since open source code wasn't "foreign property" under IEEPA.
phs318u 13 hours ago [-]
Swapping the footgun for a huge long-range boomerang doesn’t mean it’s not going to eventually swing around and whack you in the back of the head.
bardak 12 hours ago [-]
100% agree and don't think it will come to that but I won't completely put it past this administration
addandsubtract 12 hours ago [-]
Label AI as porn and the payment processors will cut their ties automatically.
mrandish 13 hours ago [-]
> I’m sceptical they could find the legal framework to do this even if they wanted to
I agree, my only caveat is that the current administration has shown it's willing to go beyond aggressive regulatory interpretations to questionable and outright implausible interpretations. As we've seen recently, the federal courts and SCOTUS are overturning most of these but that can take a year or more to resolve. The one positive light is they seem to push the hardest on certain culture war issues (immigration, voting, districting, etc). AI doesn't seem like a core hot button issue for the White House and there is a strong pro-AI / business faction.
eunos 13 hours ago [-]
OpenRouter or Huggingface should consider moving to Switzerland
gruez 16 hours ago [-]
>GLM export controls incoming?
US imposing export restrictions on a model from China?
mcintyre1994 16 hours ago [-]
It’d be restrictions on Americans and American companies, and probably also pressure on America’s allies.
mkagenius 14 hours ago [-]
Token smuggler sounds like a profession coming soon. For distillation and stuff.
addandsubtract 12 hours ago [-]
I mean, there are already places where you can buy tokens at 10% of their original cost.
manquer 16 hours ago [-]
While unlikely , it is not without precedent , there are restrictions on ASML a Dutch company to sell EUV machines
throwup238 15 hours ago [-]
That’s because the Department of Energy originally funded and contributed IP to the EUV Corp joint venture between several semiconductor companies (including ASML and Intel). Their ability to export control EUV was part of that original agreement that the entire technology is built on.
11 hours ago [-]
verdverm 16 hours ago [-]
ASML complies as an ally, why would China comply?
The weights are already available and downloaded, is it going to be a crime to have them, run them, make them available? Constitutional rights still exist (I hope)
solenoid0937 16 hours ago [-]
> is it going to be a crime to have them, run them, make them available?
Now you're getting it! Commerce will call it a munition and those harboring it as harboring illegal/foreign munitions.
No business will take the hit, so they will quickly deplatform the models.
No end user has the GPU capacity to use GLM 5.2 or similar models at full precision so the government will call the problem "mostly solved." But they might choose to "make examples" out of a few people using p2p software to download the weights if they choose to.
verdverm 16 hours ago [-]
Or we use the models to work on fixing vulns and stop over-blowing the doom scenarios. Gotta save the kids and kill the terrorists though!
I'm for making software better instead of banning it based on what the rich and powerful claim.
I suspect the real fear is that open weight models undermine the financials and token prices they thought were going to pay off their ludicrous spending because they have all raced and raised hardware prices.
hadlock 14 hours ago [-]
> making software better instead of banning it
We're still in the middle of the cambrian explosion.
If Anthropic was capable of developing Opus 4.49-4.5 2H 2025.... then any company with a research team capable of reading all the papers and press releases will be capable of producing Opus 4.8 by the end of 2027, either raw model competency, or in a harness like claude code (or better with both). I guess what I am trying to say is that Opus 4.5 does not represent the edge of agentic capability, merely somewhere in the thick meaty layer of "functional and achievable".
We can draw the line at Sonnet 4.6 in the US but much like encryption export restrictions in the 1980s, the line drawn will be laughably low within a few years and simply unthinkable in a decade.
solenoid0937 15 hours ago [-]
> making software better instead of banning it
That would be the rational thing to do.
> financials and token prices
I do not think the government thinks this deeply. Market manipulation might be a rational, if unethical reason to ban open source models.
But this admin banned Anthropic models to "own the libs." They will continue to ban what they want for whatever reason they want. I don't think those reasons will be particularly coherent.
verdverm 15 hours ago [-]
Yeah, the current admin is reactionary, they appear to put little thought in, or at least disregard input they dislike. I don't think Ant's ban was about "owning the libs" as much as it was asserting dominance over someone who spoke up counter to the admin's aims and claims. They do listen to money, which is where I see Big Ai paying for executive orders (because the admin forgot what it means to compromise as part of legislating for all americans).
15 hours ago [-]
15 hours ago [-]
matheusmoreira 15 hours ago [-]
> it going to be a crime to have them, run them, make them available?
Yeah. Illegal numbers.
fragmede 13 hours ago [-]
DeCss was short enough to fit in a t-shirt. Americans are larger these days, but not by enough to fit a decent LLM's weights on an XXXXL shirt, even double sided.
manquer 11 hours ago [-]
That too has precedence , there is long history of controls of cryptographic algorithms up until the 90s. It wasn't abstract either, older greybeards would remember browsers like Netscape had two versions International and U.S. for this reason.
If you classify AI as a weapon which seems to be the direction that we are all heading towards, they yes first amendment rights won't likely apply.
Art9681 13 hours ago [-]
They can easily issue an order for any American company to stop hosting/serving the models. If the model was a threat to national security because of its capabilities then a lot of other countries would follow, including China. No nation will allow some vibe coder with a rogue AI to pose a threat to their systems.
The reason GLM-5.2 hasn't been banned is that despite these cherry picked use cases, GLM-5.2 isn't even close to Opus in all use cases. These vibe benchmarks are ran by companies that are not part of the cyber services offered by Anthropic and OpenAI where they can use the models without the safeguards and refusals so their actual cyber capabilities can be utilized.
These guys that wrote the article compared a gimped Opus to GLM-5.2, knew full well it's misleading, and got the clicks regardless. They don't have enough clout to be a part of something like Project Glasswing, GPT Cyber, etc.
fph 15 hours ago [-]
How would that even work for an open-weight model?
bardak 13 hours ago [-]
Go after the hosts, 99% of people won't be able to run this locally even if they wanted to.
djeastm 15 hours ago [-]
I think state-of-the-art AI is going to be defense industry only from now on. We can have our toy drones but not the Predators and Reapers.
Gigachad 14 hours ago [-]
Turns out toy drones are more useful in war than multi million dollar planes anyway.
techpression 14 hours ago [-]
Reaper and Predator are both drones and there’s really no comparison to toy drones in terms of sheer destruction and capabilities in general, the comparison is actually quite apt imo.
solenoid0937 11 hours ago [-]
You're right. Toy drones have proven vastly more effective IRL.
The others are a waste of taxpayer money. Extraordinarily low return on investment (kill-on-investment?)
fragmede 13 hours ago [-]
Which ones are the ones Ukraine has used to bomb Moscow?
14 hours ago [-]
serf 14 hours ago [-]
the things that empower modern toy drones were export restricted for years before hand.
mullingitover 13 hours ago [-]
Obvious answer: build all your open source LLMs into firearms, get the SC to grant 2A protections.
dakolli 14 hours ago [-]
Cool then everyone will just change their config to route through a provider overseas for an added 50-100ms latency. Who cares.
solenoid0937 11 hours ago [-]
Countries and businesses that don't want to be sanctioned by the US government or the US financial system care - so all western countries and corporations.
14 hours ago [-]
WithinReason 15 hours ago [-]
> [...] beating Claude Code (32%) at roughly $0.17 per vulnerability found
Claude Code is an agent harness, not an LLM.
Claude is a brand (or group of LLMs), not an LLM.
raincole 15 hours ago [-]
Yes, and the article author is fully aware of that. Thank you for pointing out this small mistake though.
mkagenius 14 hours ago [-]
It looks like the author is specifically avoiding model's name, because results are really weird.
Opus 4.8/4.7 scored 28%
Opus 4.6 score 37%
So the author thought as let's not get into that just write Claude.
happycube 13 hours ago [-]
Not weird at all, given the variance in Opus' quality over the last few months.
wild guess - I wouldn't be surprised if Opus 4.6 was run quantized for a while, and 4.7/4.8 have QAT for that nerfed size.
insiderphd 2 hours ago [-]
Hello! Author here (Katie) Ty for your comments, 4.6 and 4.7 both scored 28% on our benchmark, I just wanted to have 10 things in the list because I wanted a round number.
andriy_koval 14 hours ago [-]
many people think opus 4.6 was the best
raincole 10 hours ago [-]
Where is the weird part?
croemer 12 hours ago [-]
The dollar amount is meaningless without comparison - and no other model has a price tag. Sloppy article.
tills13 14 hours ago [-]
It costs nothing to not be pedantic.
alienbaby 13 hours ago [-]
Possibly, nothing other than accuracy
mdp2021 6 hours ago [-]
"Kindly reach us in Cambridge for the lessons".
Onavo 15 hours ago [-]
Claude code it's the only way to get access to the actual amortized cost of running a Claude-scale model. The consumer non-enterprise API is extremely expensive (with increasing marginal costs for the user and fat profit margins for Anthropic). If you want to approximate a State level attacker's cost where they can have the model on their own hardware, Claude Code is probably the best guess at the amortized cost.
armcat 2 hours ago [-]
I find it astounding that ppl still comment “it’s still behind” or “it’s not the best model”. Everything is about the harness. Even the big AI labs are focusing on managing agents - sandboxes, memory, context, skills, loops. With the right harness GLM 5.2 can do no wrong.
40four 6 hours ago [-]
It’s hard to argue against the open weight models if your only concern is coding. Which, for many of us hackers here in this forum, it is.
But I would like to point out that the overwhelming majority of people using LLMs aren’t programmers, don’t care about coding, and couldn’t even be bothered to “vibe code”.
So we should consider the bias of the output of these open weight models, and what that looks like, outside of the context of writing code.
WinstonSmith84 5 hours ago [-]
There is no money made from these people though .. people who are using ChatGPT to plan for their next week-end or their next vacation aren't paying a $100 or $200 monthly subscription. As for non coder office workers (accountants, PMs, etc.), they use Microsoft or Google products which all integrate AI to some extent within their products - with RAG for Sharepoint to some basic AIs to generate text or automate work in spreadsheets .. the models used there are already capable enough for all what's needed (I think Microsoft is using GPT 5.1 or 5.2 in its latest iteration but for sure no GPT 5.4/5.5). The thing is, Software development is where money is made for these labs
40four 5 hours ago [-]
You’re making a good point. I don’t disagree with what you’re saying. But I think my point got lost.
I don’t agree with “Software development is where money is made for these labs”. Coders will inevitably eat up the most tokens & buy the bigger $200 subscriptions because we want to keep working.
But us coders are still the small minority of users. They aren’t counting on us to get to trillion dollar evaluations.
They are counting on the regular folks to buy the $20/ month subscription. It’s really easy to run out your free tier usage these days, asking questions that have nothing to do with coding.
So my point is what does that output look like for someone asking a question about politics or world news?
bel8 2 hours ago [-]
I think most of these $20/mo subscrptions will either be Apple's iCloud, Microsoft's Office 365 or Google's Drive+Office plans which already do or will offer bundled AI.
I know Google gives me free Gemini AI from my Google Drive plan. Microsoft probably already does too, didn't test. Apple is probably crafting some arrangements if not offering already.
My point is most people wont pay for AI. It will be bundled.
And I think AI is going to be free for all, with ads.
jackdawed 12 hours ago [-]
I use GLM 5.2 via Neuralwatt and it's gotten so cheap I wouldn't mind cancelling my personal Claude subscription if work gave me one. I've spent 374M tokens this month and it only cost me $18 on energy-based pricing.
cmrdporcupine 10 hours ago [-]
How's the reliability and speed?
kelnos 5 hours ago [-]
Title is misleading (and is editorialized from the actual article title). GLM 5.2 did better than Claude in one specific cybersecurity-related benchmark (finding vulnerabilities of one certain type). I don't think you can draw any general conclusions about the relative utility of the two models.
insiderphd 2 hours ago [-]
1000% this, this was us internally testing if our harness worked, the motivation was never to test them in-depth 1v1. We were just really shocked at the results, there’s a lot more work to do here.
danslo 16 hours ago [-]
It reads like an ad.
Secondly these are "just" IDORs, arguably the easiest class of vulnerabilities.
Thirdly it compares to GPT 5.5 and Opus 4.8.
No, we don't have Mythos at home.
vlian2088 16 hours ago [-]
>Thirdly it compares to GPT 5.5
mythos is <10% ahead of gpt 5.5 on all benchmarks, which it gains by being several times the size of opus. had it been economical to provide, it would've been released to the public on day one instead of the marketing circus those effective altruism clowns had exhibited. admitting that it costs >1000% to run inference on a <10% better model would've been very damning.
oa335 14 hours ago [-]
> it costs >1000% to run inference
do you have a source for this claim? i thought LLM providers earn high margins from inference (charged by token). is this no longer the case?
vlian2088 13 hours ago [-]
if a $6000000 cabinet can generate 10000/s tokens of Opus but only 1000/s tokens of Mythos, then Mythos costs 1000% to run no matter the markup.
no one has a source, because no one knows closed model parameter counts. we have only heuristics which strongly indicate that Mythos is simply a big fucking model that any other lab could make an equivalent of.
3836293648 13 hours ago [-]
This was just theorised. The leaked OpenAI financials suggest otherwise (because of shady naming of losses)
The only ones who seem to profit are the ones running smaller Chinese models. Even NVIDIA seems to have to "reinvest" their profits into sponsoring companies to buy their cards now.
InsideOutSanta 16 hours ago [-]
In my experience, GLM 5.2 is extremely good at finding vulnerabilities, and more importantly, unlike Opus, I've never seen it refuse a command. It genuinely is a very strong model for finding and fixing vulnerabilities.
nozzlegear 13 hours ago [-]
More importantly, unlike Mythos and Fable, you can actually use GLM 5.2! It's not just marketingware that got its founder in hot water with the government.
NitpickLawyer 15 hours ago [-]
> Thirdly it compares to GPT 5.5 and Opus 4.8.
> No, we don't have Mythos at home.
That's still useful. To paraphrase the kids these days, GLM5.2 is in the room with us, today. Mythos is not. And for us in the EU, it's even more complicated, as Mythos might be with us in the room one day, and go poof the next day, on the whims of political entities that we have 0 control over.
Knowing where open, accessible, local models are is important. We know they're behind. But there comes a time when "good enough" is useful. Even if they're "just IDORs" today, and even if they're behind SotA today.
As someone else said above, GLM5.2 (and other models in the same tier like kimi, dsv4, etc) is / are slowly becoming "good enough" to assist in automated repo prepare work (download, install, test, edit, re-test, etc). And that translates in RL traces ready to be trained into the next generations. That might be more important than x% behind on benchmarks.
sanid 15 hours ago [-]
Technically we don't have Mythos at all? You guys have access. This tells me we have Opus at home (open weights).
jimbob45 15 hours ago [-]
Yeah they straight up say that their criteria is narrow and primarily important for their specific use case. Never let rationality cause your pitchfork to be cast away though!
uluckydev 7 hours ago [-]
I used Claude a lot, but with Claude Code it takes a lot of context window, and it's very pricey, to be honest. Then I shifted towards Minimax. I used the coding plan because it's cheaper, but it still gets the job done.
When M3 came out, I started using it, and it was actually really good. After that, I shifted towards OpenCode for my AI agent, and that's been really good as well. The best thing I realized is that it uses less context, works better, and gives me access to a lot of different models from one place.
I never actually used GLM, but I recently found QuanCode, which is amazing. I used it to build a full-stack application. Now I'm shifting my focus more toward SaaS distribution. I'm still figuring out how to automate different workflows, and using QuanCode has been really fast and effective for building those automations.
Kiog-Aser 4 hours ago [-]
[dead]
spaceman_2020 49 minutes ago [-]
Opus 4.8 is genuinely one of the most frustrating models in casual use. It has a tendency to completely lose context in the middle of a conversation. It’s also too pedantic and nitpicky, and relies on language that’s way too specific to get any work done. I always end up being frustrated with it and revert to opus 4.6
childintime 5 hours ago [-]
About running models locally and why data centers win (for now): they can stream the model weights to many neural engines at the same time, so each of these only needs enough RAM to hold the KV cache. So each engine is cheaper to operate, plus they are time-shared, resulting in massive wins for data centers.
So one can see businesses owning their own such cluster, next to their database infra, in the near future.
maxignol 5 hours ago [-]
Would you recommand some ressources about how multiple neural engines are used in data centers ?
andai 8 hours ago [-]
Most interesting things to me from their benchmarks:
GPT does way worse than Opus without their harness, but better with it.
Opus 4.7 and 4.8 do way worse than 4.6. (Intentional nerfing?)
Would have been interesting to see GLM in the custom harness.
Would also be interesting to run GLM in Claude Code, which it has presumably been fine tuned on.
croemer 12 hours ago [-]
They should also at least run Opus through the same Pydantic harness they used for GLM. As is, it's apples vs pears.
Where's the cost per vulnerability for all the other models than GLM?
Also, without code this isn't very trustworthy. Could all be made up as well.
tmach32 4 hours ago [-]
I think one thing people are missing about this article is that they are arguing that the harness can make a bigger difference than the model. They aren't merely hyping GLM 5.2.
_cs2017_ 5 hours ago [-]
I don't feel the numbers without the harness are useful.
People will use the model with the harness. I know that harness may not be optimized to this model, but it's still more useful to see the numbers from an imperfect harness than from a no harness setup.
XCSme 11 hours ago [-]
Does a bit worse than Opus 4.8 in my tests[0], but it's 5x cheaper and 3x slower.
Note that being open-weights, "slower" is relative, as it depends on who's serving the model. This can drastically change over time too.
nsoonhui 10 hours ago [-]
Not sure what to make if your benchmark because GPT 5.5(low) ranks higher than GPT 5.5 (medium) -- #4 vs #9
XCSme 10 hours ago [-]
You'd be surprised, some models on high do worse than on medium, because they start overthinking and doubting themselves, polluting the context with too much information, etc.
It depends a lot on the task and harness too (using plans and to-do lists, vs one-shot answers), but for simply answering directly to an inquiry, often extra thinking doesn't necessarily improve the answer, especially if the answer is binary, or can be correct or wrong, as opposed to having more time to refine a creative output.
XCSme 10 hours ago [-]
Another example was Gemini 3.1 flash lite, which on high was basically just burning tokens, costing like 30x more, while giving worse answers:
Beats which model in Claude? Whenever a "benchmark" doesn't put precise model numbers in their headlines I am immediately skeptical. Either they don't know the difference (bad) or they are benchmarking against weaker models (misleading, also bad).
It's like when studies say "AI is bad at X" and they used GPT-3.5 in current year.
InsideOutSanta 15 hours ago [-]
They say "Claude Opus 4.8" in the first paragraph.
crm9125 14 hours ago [-]
We're supposed to read the article?
How are we supposed to stay skeptical of everything if we read anything!?
ls612 16 hours ago [-]
Opus 4.8 according to TFA. Whether or not the safety guardrails were responsible for the difference is an open question but for a dev who wants to secure their software who doesn’t work at one of the blessed Glasswing companies it doesn’t really matter why, it matters what the best tool you actually have is.
xlii 3 hours ago [-]
I switch from Codex to GLM 5.2 when I'm out of tokens. The main difference for me is time to completion.
GPT gets there <5 minutes, GLM 5.2 without context takes ~1H.
Though the harness makes a significant difference. On Pi GLM5.2 dreams for minutes, with OpenCode it's more on the point and gets to editing quicker.
mpfect 2 hours ago [-]
Feeling proud on these Open Models. Its just they need to focus on efficiency as well especially in terms of size.
jacomoRodriguez 2 hours ago [-]
Which harness do you recommend to run coding task with glm 5.2?
Any good resources about this (also for setup and recommend config)?
tomerbd 2 hours ago [-]
GLM 5.2 - Super Clear
GPT-5.5 - Super Smart
Auto/Composer - Super Fast (cursor)
gurjeet 10 hours ago [-]
Twice in the text quotes Claude Code's F1 score as 32%, but the table shows the score is 37%. It's very likely that the actual score is 32% (because it is referenced 2 times, and a third time indirectly as the difference 'seven').
Oddly, this is a strong indication of the text being hand-written rather than LLM-assisted; it's very likely that a human made a mistake in creating the table.
> ... beating Claude Code (32%) ...
> ... GLM 5.2 ... beat Claude Code by seven points (39% vs. 32%).
> Rank | Configuration | Harness | F1
> ...
> 4 | Claude Code (Opus 4.6) | Claude Code SDK | 37%
insiderphd 2 hours ago [-]
Hello author here, or one of them anyway. I can confirm that it was hand written, 32% was combined all the Claude models (4.6, 4.7, 4.8) mushed into one score, 37% was Opus 4.6 specifically (which did the best)
theteapot 15 hours ago [-]
> Constant: the IDOR dataset (the same real, open-source applications we've used in prior research) ...
What we're they? Also, wouldn't one expect a more recently released coding agent (with a more recent knowledge cut off) to perform better because they have access to more knowledge about vulns in these OSS projects, and even possibly have knowledge of your own "prior research"?
mkagenius 14 hours ago [-]
One would. But then the results are even weirder as opus 4.6 scored more than opus 4.8 by a huge margin
13 hours ago [-]
mohitpaddhariya 6 hours ago [-]
open-weight models routinely match or even outperform previous-generation proprietary APIs
veselin 16 hours ago [-]
Here, it appears they compare a single prompt "find IDOR", against a multi-agent system. However, one can also start far more sophisticated skills that spin up subagents and mostly do the same in Claude Code, Codex, OpenCode, Pi, etc.
Which I guess makes what semgrep sells obsolete. Unless they have built a pareto-optimal point in terms of capabilities and token usage maybe?
blazespin 16 hours ago [-]
I think the point is less "how can we throw shade on the OP" and more "a harness can enable a lot of models to do very serious cybersec, glm 5.2 is one of them"
s3p 16 hours ago [-]
Are you replying to a response to the original comment? I looked but i didn't see anyone saying he's throwing shade.
BikiniPrince 14 hours ago [-]
You have to forgive the GLM bot. It's not very good.
synergy20 9 hours ago [-]
but, it's $160/month(unless you buy a one-year plan that gets cheaper), not too far from $200/month from claude and codex? why should I switch?
ben8bit 5 hours ago [-]
Definitely a +1 from me. I've really enjoyed using it via OpenCode/Zen. Not loving the pricing with OC so will probably switch to OpenRouter once my credits are done.
maxignol 5 hours ago [-]
Have you tried opencode go ?
_s_a_m_ 13 hours ago [-]
I tried GLM many times and it is bad, i have on clue what these people are talking about
jeffnash 12 hours ago [-]
have you tried 5.2? I agree that 5.1 and prior were below Kimi, Mimo, Qwen, Minimax, and probably Deepseek (depending on task), but 5.2 (especially unquantized) feels like something else.
Now I feel like that I'm covered by GLM 5.2 and Minimax M3 (when I need vision or a second pass on something).
thefourthchime 9 hours ago [-]
Same. I asked it my Pac-Man question and it was the first to DNF.
It just goes off getting confused about how to design the map for 15 minutes and then times out.
throw10920 12 hours ago [-]
Bad for security research or for general coding?
Having used GLM 5.2 for non-security software work, I can say it's better than Sonnet (but not Opus), and cheaper than both (because when you steal someone else's IP, you don't have to amortize the cost of their R&D).
byzantinegene 5 hours ago [-]
stealing someone's ip... hmmmm
chonghaoju 5 hours ago [-]
Every agent run writes an audit record. Not for compliance theater — because when something breaks at 2am, you need to know exactly what happened and why.
sidcool 9 hours ago [-]
Genuinely curious. Say GLM 5.2 is better than Opus. But how does one go about using it by themselves?
I went with the second one to try it out, feels pretty okay (with OpenCode, though Claude Code would also work), however it feels like I reach the weekly limits somewhat fast with their 65 USD Pro subscription. They also have that whole peak times thing going on and apparently it will get worse after September:
> Supported models and Visual Understanding MCP share the same usage quota. GLM-5.2 and GLM-5-Turbo consume quota at 3x during peak hours and 2x during off-peak hours. Limited-time benefit: off-peak usage is currently charged at only 1x quota through the end of September. Peak hours: 14:00–18:00 daily (UTC+8).
Mashimo 4 hours ago [-]
OpenRouter, Z.ai coding plan, OpenCode Go, OpenCode Zen .. and probably more.
cake-rusk 4 hours ago [-]
How do you run this thing? What kind of hardware do you need?
You can use GLM in OpenCode with a z.ai subscription by default as well. Also it'd be good if you mentioned you were involved with nemesis8.
kordlessagain 8 hours ago [-]
I think it would be good not to suggest someone run a new Chinese agent on their bare metal.
When I posted the comment I was both the first commentor as well as the first person to upvote the submission. That matters. My name is ALSO on the open source repo that allows Opencode to be run in a container.
That's transparency, maybe not here, but on a clickthrough to Github it is immediately obvioius.
wadim 5 hours ago [-]
> I think it would be good not to suggest someone run a new Chinese agent on their bare metal.
Not sure a project nobody knows or uses is much better in this regard?
I think they give $5 trail credits to test with any of the open weight models.
g42gregory 14 hours ago [-]
If only the "cybersecurity" crowd were focused on patching the vulnerabilities.
Instead of shilling for the LLM providers.
__MatrixMan__ 14 hours ago [-]
But if we patch all of the vulnerabilities, who will pay for our vulnerability scanner?
_factor 14 hours ago [-]
The robot figured out how to bump the lock. The obvious solution is to ban the robot.
theptip 9 hours ago [-]
But… what effort level? “Opus 4.8” is a massive capability range. If you just ran it on medium that is a completely different result than vs. max.
Art9681 13 hours ago [-]
This is because of the safeguards and not the model capabilities. If these folks signed up for the proper cyber service offered by Anthropic where refusals are removed then the open weight model wouldn't look as capable.
unnouinceput 8 hours ago [-]
And just like Linux lost to Windows in consumer market due to devs/creator's stubbornness, same will happen with closed vs open LLM. In the end the one that is used the most will be the one that you train your kids on and therefore the one that wins the market. Eventually the closed one with too much guardrail will be left behind because people will stop using it.
You need to read the market. Linus didn't read it in 90's, Gates did and that's why Windows is in almost every home.
throwaway676712 3 hours ago [-]
Is this 2006? Linux is present on literal billions of android phones, servers, supercomputers and other embedded devices. It's the most ubiquitous OS on the planet and it's not even close, even Microsoft contributes to it.
The only niche where it doesn't utterly dwarf the competition is personal computers and it looks like we're all getting priced out of that anyway
bingemaker 5 hours ago [-]
How do you run GLM? Are there any hosted services?
port3000 5 hours ago [-]
Opencode Go subscription ($5 to try for one month) or Neuralwatt are what I use. Both through opensource Opencode harness (like Claude code)
bingemaker 5 hours ago [-]
Thank you!
Alien1Being 10 hours ago [-]
The current US administration has gone a long way towards handing over leadership in AI to China.
a96 4 hours ago [-]
Along with everything else. Almost like having a fascist dictatorship isn't really a very competent way to run a country no matter what the size.
rbbydotdev 9 hours ago [-]
Argh, agent benchmarks are so bad and can be gamed easier than bmw emissions tests.
12 hours ago [-]
protonisafk 5 hours ago [-]
It seems benchmarks keep changing and preferring the latest AI agent literally every time.
cmrdporcupine 13 hours ago [-]
I like GLM 5.2... ish. It's ok.
I'd be mostly fine switching to it.
I just can't find a cost effective way to do that. z.AI's coding plan is both overpriced and unreliable. ollama's is also overpriced. Paying by the token for it on openrouter etc is more expensive than just having a Codex or Claude coding plan.
If you have to pay by the token, it's clearly cheaper. It's not competitive with a coding plan though.
TurdF3rguson 13 hours ago [-]
It also means giving up vision which I don't know how I would deal with. I think I would prefer a weaker model with vision than a stronger without.
Openrouter definitely supports vision models. Why would you have to give up vision?
Mashimo 4 hours ago [-]
> Why would you have to give up vision?
Because you would have to switch model.
You can't just say "Oh, button X looks weird see [screenshot]" while coding with GLM. You would need to switch to another model and then maybe back.
TurdF3rguson 10 hours ago [-]
For example if I want to paste a screenshot of what I mean, I can't.
cmrdporcupine 13 hours ago [-]
If you using opencode or similar you can just temporarily switch models -- in the same session -- to something that has vision and have it look at your image. And then switch back.
gazpachotron 12 hours ago [-]
Or create an agent or subagent that just looks at images, and specify a vision model for that agent.
TurdF3rguson 8 hours ago [-]
I don't see how that helps, I would still need to somehow get the image into the coding model's context.
gmerc 12 hours ago [-]
vision runs just fine locally for most usecases, so it's really just a skill to call that Ollama instance
nozzlegear 13 hours ago [-]
Why's that?
dist-epoch 14 hours ago [-]
Anthropic is saying other models were good at detecting vulnerabilities, where Mythos excelled was in creating functional exploits for them.
This article only talks about detecting vulnerabilities, so it's unclear if it's a true Mythos equivalent.
igregoryca 13 hours ago [-]
It seems "Mythos is really good at finding vulnerabilities" has been what people took away from the Project Glassing announcement, which makes sense. Unfortunately for Anthropic, most seem to have forgotten the best argument Anthropic had for holding Mythos back from the general public, "it's crazy good at crafting exploits". Then, without that context, the tinfoil hats came out.
laybak 14 hours ago [-]
how representative are Semgrep's benchmarks? everyone seems to have their own benchmark these days (guess it's good "content marketing") I'm honestly losing track
questionreality 6 hours ago [-]
hope open source continues to improve
slashdave 12 hours ago [-]
Advertisement
rvz 11 hours ago [-]
Many people here are now realizing that open weight models are now able to compete against frontier closed models.
This is where we are heading and why many closed labs are terrified of this affecting their bottom line and the reason why they want them banned from being released.
crazylogger 11 hours ago [-]
Actually they don't even need to compete against frontier closed models, they just need to work.
99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.
lowbloodsugar 12 hours ago [-]
Felt like I was reading advertising for their harness.
dools 12 hours ago [-]
I think Opus 4.8 is deliberately nobbled. Kimi k2.6 with Kimi code beats opus models at finding vulnerabilities, even though it produces some false positives, when I give the same issues to opus and ask it to verify most of the time it concurs it’s a real issue even though it failed to find the issue itself
unnouinceput 8 hours ago [-]
OK, half the article is on and on about harness and scaffolding and whatnot. I kept reading waiting for a benchmark where they give the same scaffolding to GLM like they did to Opus. Where is that one?
utunga 13 hours ago [-]
Just popping in to say that no you can't use the word "tokenomics" to mean that. Argh.
lenerdenator 13 hours ago [-]
The incentive to develop Claude further is to make money.
The incentive to develop these Chinese models further is to trash the business case of most American AI labs.
csjh 14 hours ago [-]
I found it to spiral into complete nonsense a few times when I tested it out, but it's possible that was a bug in the provider
yieldcrv 12 hours ago [-]
who is your favorite hosted GLM 5.2 provider? I'm looking for fastest tokens/sec and best cost
additionally, reliable API, because z.ai can be finicky
also, not for Enterprise use, but I like non-US providers, I don't care if the party happens to be the one reading my information and stealing my trade secrets, if they won't respond to a US subpoena
CurbStomper 12 hours ago [-]
people still using Claude?
TacticalCoder 13 hours ago [-]
How to reconcile that with the recent, highly upvoted, article titled: "The gap between open weights LLMs and closed source LLMs"?
What explains it?
Is TFA lying? Is the most upvoted comment here lying?
Bigpet 3 hours ago [-]
Top comment doesn't say it's better. Just says it's a "workhorse".
The article itself doesn't say "it's better", basically just says "in this one specific benchmark it beat Claude with Claude code". Mind you with multimodality it Opus still beat GLM 5.2 very handily in that same benchmark.
I can't find any contradiction and I don't see anyone lying directly. At most they lead you to imply false things, but they're not untrue at a literal reading.
contentkraft 2 hours ago [-]
[dead]
modgate 6 hours ago [-]
[flagged]
jocelyner 8 hours ago [-]
[dead]
fishonbike 7 hours ago [-]
[flagged]
goyoon 9 hours ago [-]
[dead]
zwJay 6 hours ago [-]
[dead]
aussinholdn 14 hours ago [-]
[dead]
rode1974 15 hours ago [-]
Hopefully i get a macbook pro soon enough to run some small or medium sized LLMs
paperterminal 15 hours ago [-]
Same, but so much $$
15 hours ago [-]
BikiniPrince 14 hours ago [-]
This is a joke right? I wouldn't install this in a sandbox.
mlnj 11 hours ago [-]
Why? Don't tell me you've never tried a non-US based model, ever.
There's a number of US providers who also run it, if that is your preference.
This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and OpenClaw just felt... not what I wanted. Two days later and 20 dollars poorer I have what I need: a multimodal agent written in rust that has access to my homelab.
Nothing felt off with GLM. It did what I wanted, was fast, had a decent not very annoying personality and was much cheaper than Opus or GPT.
I used it unquantized through Fireworks, but there are multiple other providers too.
In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average. Data at https://gertlabs.com/rankings
But when factoring in performance/cost, GLM 5.2 is the frontier model.
I'm trying to wrap my head around exactly why so may people seem to want the best model available when it has recently become clear that most halfway decent models can write damn good code for a fraction of the price. And the frontier models get nerfed constantly so you with open weight you can get something slightly less performant but way more stable. Almost like buying a Ferrari for your daily commute instead of a Toyota or even a Mercedes.
I think there are several factors. Certainly marketing making us think we need the shiny thing which is rampant online and very smart people think they aren't susceptible to. There's a lot of really odd 'I trust Anthropic/OpenAI more than Deepseek' which tends to ignore, for starters, that you can run choose your provider and still save a ton. I also think there's some amount of addiction and brand loyalty where a Ferrari is one hell of a drive so that you turn your nose up at that sensible Toyota. Oh the other one I see used is like oh only fable can oneshot updating my embedded systems thing from 1975 to rust which is great but let's recognize how niche that is.
And it ends up just coming across as people are getting SO reliant on the tools so fast. Maybe it's ok to think and like read a few lines of code and work with these agents to convert your thing to rust or center your div. Even if coding is over which in some sense it certainly is, don't turn your mind into the wall-e people yet. I found myself guilty of this so often. It takes way more time and effort to do things via prompt and I wouldn't just open the editor and fix it because that dopamine hit of the magic the abstraction provided was so strong.
So I'm pretty much done using the 'best' (on benchmarks, if money isn't an object, etc etc) models available. After a year on Sonnet/Opus/GPT5x I'm having way better results with open weights models that don't get lobotomized weekly. I'm finding ways to do the crafting part of building software by focusing on honing my harness and workflow. I'm enjoying changing the oil on my Toyota after a year of almost flying off cliffs in my Ferrari and if I can check my ego it's a purely positive thing.
This is the logical end point of the fear-based way LLMs are marketed. You must want the best, because everyone who has the best can work faster than you, generate more — if you don't have the best, you are behind! Why would you want to use anything other than the best?
The thing is, once everyone has the best, the question is: how much can you spend? If you can't spend more, you are behind! If spending the most will get you ahead, why would you not want to spend the most, if you can afford it?
There is only one way through this, in the long run: work out a way forward that doesn't make you dependent on this cycle. If you can compete at all, without the spend, what happens is: they burn money and you don't.
FWIW so far I don't think the benchmarks prove very much about the actual experience, and you can discover this just as easily without spending any money. And we know this about benchmarks! Once a benchmark seems useful as a measurement, it becomes a target and it stops being as useful.
I think your strategy is right. It requires bravery, and as you say, it requires ego balance. But I believe it is obvious that the world will either come around to a more sensible, stable pattern or it doesn't matter either way because we're fucked. So opting out of this mad early cycle and choosing to be calmer and happier is a choice you can just make.
The difference is how the model is used.
With Opus you can give it a long-horizon task (eg build an entire feature) and it will plan it out and implement it and almost always stay on task. This is what people mean when they say "agentic tasks"
With the lessor models the code is fine, but they need something else to plan what needs to be done.
GLM-5.2 is the third model (after Opus 4.6+ and GPT-5.5) that can do this agentic style work.
Notably Gemini 3.1 Pro is notoriously bad at this style work - the code is good, but it drifts off task most of the time. 3.5 Flash is supposed to address this, but I haven't had a good reason to try it.
And hey, don't get me wrong, you can get pretty far with just prompting. But the subtle misses and (I'm looking at you GPT) the overengineered 20k line PRs to do a simple thing are going to cost you a lot if you're not vigilant.
I am still struggling how to deal with sub agents and different roles for each model. I still think Claude or Codex are overall better models, but everything around them transpires such weird vibes, including, and this one kills me, that at certain times they feel like dumbed down.
I keep changing these things often, but I have basic subscription to codex (20$ plan) which I use with GLM 5.2 to do some high level planning of what I intend to do, and then leave Deepseek do the coding. Or something along those lines.
Point is, GLM 5.2 is now at a point where I cannot tell you if it's better or worse. I can tell you however one thing: no matter when I use it, it's consistent in what it does and how it works.
Then there is the Fable thing, but as with many things, I think the past has distorted the reality. It lasted two days, but Anthropic said it clearly for plan users it would only be there for two weeks. It was great for doing what you can already do with other tools: doing all the planning, and reviews, and launching a million subagents talking to each other. I sometimes wonder if it was really a new model, or just Opus 4.9 wrapped with some fancy model driven harness.
As for Fable: I used it as much as I could while we had it.
It was a step change over Opus with my work.
I've had no trouble getting the current generation of smaller models to do the same thing. Maybe it's more of a harness issue than a model issue?
Recently I've used both MiniMax M3 and DeepSeek V4 Flash to one-shot moderately complex applications from a written spec, and neither one got lost along the way
Price and speed, for me. GLM5.2 is "good enough" for some tasks, but rather slow (on their coding plan). In the time it takes GLM to "read files to figure out...", gemini flash is usually finished. It's not SotA for coding, but it's fast and often "good enough" for normal tasks.
I used to be obsessed with what's the best model. Then a while back when the new best model came out, I tested it on a task. I also tested its little brother (much smaller model from same company).
They both completed the task perfectly except the "best" model (the bigger one) cost 5x more and took 3x longer...
With the wealth of models available (open source vs closed, api vs local), I find optimizing the cost-efficiency of your token consumption an important part of business-oriented AI engineering. You don't need "the best" for every task.
Same for me, I certainly don't have the same definition of success and failure either.
A more expensive model has *less* rooms for wandering around than a cheaper model.
If Claude wanders around during 10min until finding the most obvious solution, then I count it as a failure.
It's not fast-changing, it's not abstract, it's just not that difficult, and where it is difficult, the AI cannot help you, because it is not capable of things you are capable of.
Learn CAD yourself. Honestly; I was sure I would never manage to learn CAD but it turns out to be interesting, rewarding, valuable and actually quite quick to learn.
An LLM certainly is not going to be able to do it better than you once you have a tiny bit of experience. (PCB design, perhaps, has a language to it that an LLM can make a bit more headway into, but as a non-PCB-designer I would still bet that it's more like CAD than code)
One group is trying to get the LLM to basically one shot everything and not properly reviewing the output.
Others are using the LLM to assist their human intelligence in a tight loop.
If you’re doing the former you really do need the best model available because that’s still right on the edge of what LLMs can do at best, and at worst you’re just shipping pure unmaintainable slop.
If you’re doing the latter then you can get away with a slightly less powerful model without it making a material difference because your human intelligence is filling in gaps
While the single functions/classes/structs/... can be well though out the code tends to lack cohesion, and especially maintainability. For instance, it never thinks: "I could put this logic in an interface/trait so that if the requirements change I can simply add a concrete implementation that satisfies the new requirements (and potentially use one of these for testing)".
SoTA models can do reasonably good jobs on each ticket, but over time the architecture of the application starts degrading without a human in the loop.
The entropy increases slower with better models but the trend is always towards slop
This also allows me to play with, and mix codex, claude cli, and others. This is my happy spot for the last two months.
That changes as soon as the developer is the one paying for a model. Then it's a classical engineering trade-off between money and quality, and that's where open models are clear winners.
The problem isn't what they do in a blank state. It is how they get there and the edge cases. Some models also take longer (uses more steps) i.e. end up costing more despite being "cheaper".
I've seen models:
- Back out plans non-stop. Tried the obvious path. Invents X/Y/Z excuse (without verifying) that it can't be done. Notes that down and moves on. It could be as simple as site A being down and to download from site B but that's it.
- Hacks the test to make it work. Code is wrong? Nah, let's update the test.
- Keep saying useless things like YAGNI and infinite excuses like too risky to never do the work.
- Claims they are done but there's 100 edge cases not covered. When you try to use it it fails in ways you as a human assume it should work. You can write a spec to cover it all but then what's the point?
- Be trigger happy and never investigate. Tries to do it. 5 minutes. Oh it failed. Back out. Repeat. Better models definitely spend more time analyzing and actually "think". I've had models spend hours trying to do a change due to this method when an actual investigation (code walkthrough) might have solved it.
- Know and use the right tools. A lot of lesser models have infinite fear e.g. oh docker might not be available (it is) or this and that (even if you nudge it in any way) and spend a lot of extra time "working around" it.
The list goes on. Better models definitely help.
Only thing to agree on is no you don't need Fable but saying Sonnet can do the job instead of Opus is a different story. It's so obvious when Sonnet touches the code that I can't give it more than 5 minutes. It lies. Doesn't check. Forgets things and then messes up.
That’s not super surprising to me, but, given the apparent randomness of the stack ranking, is GLM actually worse than any of the Anthropic models? This looks like a 10-way tie to me.
We now use Sonnet 4.6 for a number of internal use cases we wouldn't have considered otherwise.
4.7 was so bad, I locked a bunch of my machines to 4.6.
I haven’t bothered locking the 4.8 machines to 4.6. There was a HN thread a while back where they run swe bench a few times a day and measure success rate and latency. It showed opus getting significantly dumber for the week before a recent launch.
It wouldn’t surprise me if they’re quantizing to improve margins or to hype models in comparative testing in order to defraud investors at IPO.
Or, maybe QA is hard. Anyway, I think they hit a performance wall sometime at or before 4.6.
Just want to express how amazing that is. Opus 4.6 is an amazing model. That an open weight model like GLM 5.2 competes with it is nothing short of outstanding.
On the contrary, I personally think these broader benchmarks are meaningless. I think personalized benchmarks are the way to go. They should answer "How does this model perform for MY use-case?" rather than trying to answer "How does this model perform across all coding environments?"
Case in point: I use Elixir which is not as popular as Python, is always a hit or miss with most SOTA models at the top of these benchmarks. Whereas, the ones in the middle of the benchmarks (like the GLM) almost always outperform even SOTA models from Google / Anthropic. However, this is relevant only for my use case and I wouldn't just advocate a model for everyone based off my use-case alone.
We test 11 popular/interesting languages (you can see the Languages chart to filter), but not Elixir -- although other evaluations have found that many LLMs solve more problems when working with Elixir [0]. Why models write code well in some languages over others seems to go beyond pre-training data (Python scores quite low for most models) and we don't fully understand it.
[0] https://elixirforum.com/t/llm-coding-benchmark-by-language/7...
Sounds like this is the year for coding.
They did this to themselves.
Congrats, now you’re paying an engineer’s salary to make your engineer at best 20% more productive.
Better to hire another engineer, or two jrs, and build up your in house talent.
Most of the gains right now come from tooling and process and any big post 2025 language model. The specific model isn’t that important right now.
When I first used Opus 4.8, I threw several different workloads I had at it - I have Claude doing a lot of misc projects whose primary purpose is pretty much just studying what AI agents can do for my own curiosity and no other reason. Opus 4.8 was one of the first models I ever snuck in there that basically ran out of control. No previous Opus or Sonnet model I had used ever did this. Within hours every agent I had running was writing non-sense tool calls that echoed pretend commands that didn't exist, like 10 in a row, and talking about the "tool channel" being dirty. I switched back to Opus 4.7 and assumed Opus 4.8 was legitimately just broken.
I did come back to Opus 4.8 and found that it was indeed, pretty powerful. But that initial experience has stuck with me on just how narrow of a perspective any given test or benchmark is guaranteed to have. LLMs are too broad, it really doesn't matter what you try to do in your benchmark, you will necessarily get a limited view of what the model is capable of and its shortcomings. This will remain true for at least as long as models are susceptible to massive swings in performance based on randomness and minor differences in prompts and other environmental factors.
I'm not saying benchmarks are useless or that your benchmarks are not possibly closer to the truth either. All evidence at least points to the idea that Chinese models perform very well in coding but often have more mixed results on other tasks. I'm just saying that at this point, benchmarks feel like they have limited connection to my actual real experiences. GLM 5.2 actually scored kinda meh on a lot of benchmarks (compared to closed frontier models) but my actual experience using it does not match this.
And I'm definitely not saying GLM 5.2 is better than the frontier LLMs here, just that the race is close. I still prefer GPT 5.5 right now for code review, I think, and Opus clearly has some advantages depending on the task. It's just no longer a given that Opus 4.8 will perform better than GLM 5.2 on any given task, so to me the calculus behind "using the best model available" is getting complex and you might need to get a feel for what models have what strengths to really figure it out.
I do feel like the "use the best model available" mentality is not going to die any time soon, but if it does die, it will be gradual and start soon for programming. Modern LLMs are still not a full superset of what human programmers can do, but still larger models are definitely starting to hit diminishing returns for tasks at the lower end of complexity, and that is a big deal. It's a weird world where some tasks you can feel kinda confident just throwing Gemma 4 at it and not sweating whether you should use a better model; I've certainly done it for some quick Python scripts or getting an overview of some code I'm unfamiliar with.
Some blog post I read few weeks back said that DSV4Flash in xHigh effort beats even the pro model in xHigh effort.
I think the surprising thing is I expect flash to be a pure distillation and strictly worse quality but clearly it’s more nuanced than that.
https://www.reddit.com/r/DeepSeek/comments/1rd5jw7/claude_so...
Notice he's using "trust me bro" benchmarks.
Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized.
Everyone is grinding and marketing nobody is actually discussing anything for real.
Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates.
So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.
> In May, we sent you an email announcing that starting today, the Claude Agent SDK, claude -p, and third-party apps built on the Agent SDK would stop drawing from subscription rate limits and move to a dedicated monthly credit. We're writing to let you know that we’re not making this change today. We’re working to update the plan to better support how users build with Claude subscriptions.
> What this means for you
> Nothing changes for now. Agent SDK, claude -p, and third-party app usage continues to work with your subscription exactly as it did before today, and there's no credit to claim. Your subscription limits are unchanged. When we have an update, we'll share it with advance notice before it takes effect
I see no reason to trust Z.ai more than other vendors.
So your question is really “if they’re giving free usage, why not take advantage of it?”
I do, so I don’t know the reasons not to, other than to experiment.
I kind of wanted to see if I can make a Matrix agent from scratch with Rust with GLM and it was surprisingly easy. Just make something for myself how I want it. Maybe I'll take a look on Hermes later...
I followed this example
https://minimal-agent.com/
but I'm running into issues with nested backticks so I'm thinking of making dedicated close tags per tool call.
I don't think a $100 session is "typical". I use GPT for months. $20/m plus plan is enough for my daily work.
My typical session ranges from $100-$400 - higher end when using workflows with lots of subagents. $100/session is expected when using the API without the subsidized subscription pricing. Most larger orgs have to use API pricing AFAIK.
[1] https://github.com/simple10/agents-observe
There are Business and Enterprise plans, both have discounting.
I'd blow through $20/month plan in hours.
Likewise, I don't understand how anyone survives on the basic plans. It's funny seeing these two camps not understanding what the other is doing :)
In the future none of us do, so it's better to trial how the actually adorable models perform.
I understand the reasons to use team/enterprise accounts, but apart from the policy/management/billing side of it, I still don't understand the value in spending thousands for API instead of hundreds - even when there's argument that one provider is better than another depending on the use case, I don't think that credibly extends much beyond OpenAI + Anthropic frontiers, which both have $200 subs you can stack.
How are you comfortable spending that much to write something as simple as a matrix bot?
Are people doing this kind of thing just super rich or am I missing something?
Same thing for stuff at work. Oh, the tables/schema changed and my queries broke? I could dork around with spark and cypher for an hour, or I can tell claude to update the queries for the new schema. At the rate I am paid, spending on Claude tokens is generally a better use of my resources.
Building a net new solution? Coding tools take a back seat until I get the core logic right, then I let automation handle web page and UI scaffolding.
https://swelljoe.com/post/will-it-mythos/
Also of note, I found giving models access to the open source semgrep as a tool makes some perform worse and none perform better, though it's plausible there's a way to wire it up in a harness that presents useful information to the model without the model having to know how to use it (my theory is that semgrep isn't heavily represented in the training data, so you're asking the model to do two things at once: figure out how to use semgrep and find security bugs, and both tasks suffer for the lack of focus...most small models, and some big models, can't do that well).
Edit: But, also, more testing is ongoing. I suspect GLM 5.2 will also be a consistently strong performer. It seems to excel at most things I've tested on it.
Overall, I still think GLM 5.2 is the much stronger performer. It's hard to tell the difference between GLM 5.2 and Opus at <120k tokens.
I'm inclined to use DeepSeek V4 Pro the most, because it is consistently extremely strong, it's very fast, it's very cheap and has excellent caching and cheap-as-free cached input tokens (something like 80% of token usage is cached when I'm using it for security scanning). So, my probably "pair" of frontline security researchers will probably be DeepSeek V4 Pro and Gemma 4 31B self-hosted (another shockingly strong contender, competitive with the best models once you let it loop on the same file a couple/few times). But, I won't be surprised if GLM 5.2 turns out better than DeepSeek V4 Pro...it costs quite a bit more.
I've benchmarked it, and the "here's a repo, find bugs" approach finds far fewer bugs. Like, dramatically fewer. Models are good and contexts have expanded, but focus still wins with hard problems. You could probably tell the good models to make a plan to audit the repo, and it would end up making its own "loop" in the form of a checklist of files to look at over several sessions or via subagents, I assume.
Deepseek v4 pro being in preview with less "safety" training makes it stronger for that reason. Thinking will be different and in the end, it will actually try to be useful. Just expect future Chinese LLMs to further push out "safety" guided LLMs. The future is bleak for open weight models. Prepare to have "guidelines" enforced unceremoniously to all.
Fable found a couple of good ones, then we lost Fable, so I tried GLM5.2 and it found two critical bugs that Fable had missed, so it got my seal of approval.
…probably already is one
But, there are benchmarks for so many different kinds of ability, I don't know how to compare them directly against one another. Like, models that do well on terminal and agentic coding benchmarks tend to do well on finding security bugs, but it's not a 1:1 correlation, there are surprises.
I like reading benchmarks, but I take them all with a grain of salt. They're just to tell me if the model is worth even trying for my task. I've heavily used self-hosted Qwen 3.6 and Gemma 4 on a bunch of different tasks, and while the benchmarks consistently say Qwen is the better model, I simply don't find that to be the case for anything I do. I think Qwen is tuned for benchmarks, while Google couldn't give two shits about most of the benchmarks, they're just busy making unusually smart tiny models.
Anyway, it isn't possible for any of the models, so far, to be trained on the Mythos bugs. We're getting closer to the point where I have to worry about that, at which point I'll roll forward and pull some newer CVEs from what they've published, assuming they keep publishing new bugs. (And, if they don't, it's trivial to switch to just random CVEs. But, finding out what Mythos is up to is interesting.)
What does that mean for the frontier?
https://lifeinthesingularity.com/p/glm-52-proves-ai-comes-fo...
[1] https://huggingface.co/zai-org/GLM-5.2
A few people are running highly quantized models with limited context windows. It's still impressive, but not the benchmark level intelligence. Very few people could afford a rig for reasonable local performance at a reasonable quant, at full context size.
The antirez example is 2.6bit quant, 32k context, and few tokens per second... on a ~$7000 MacBook M5 (new RAM pricing).
Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.
For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of money in local models unless you have a business where you're already paying for many employee's individual token usage.
8 x RTX6000 GPUs cost $100,000 alone. You then need to build a system that can support those GPUs with enough PCIe lanes through a PCIe switch.
It's going to be $120K to $150K to build or buy a system to run this.
But hey you could save on heating?
A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.
In the US it's common to get 200A 120/240V split-phase service. We're talking about the wiring inside the house, though.
How do you think everyone here is charging their electric cars at home and running our AC and electric cooktops at the same time if we didn't also have that? :)
You need to derate for constant loads here, and I assume you have to do that in NZ as well.
So, no, not a "uniquely US issue".
Or even just electricity costs vs token cost
The real gangstas are running 16x RTX6000s. Too rich for my blood, and the NV4FP quant doesn't seem to be that much worse.
https://taalas.com/products/
Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down.
I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - say a 32GB/64GB M7 Pro.
I also wouldn't be surprised if, 3 years after that, cheaper hardware and improved model efficiency means that there's a much smaller gap between what you can run on a consumer CPU (which, with memory prices coming down, could look like a 256GB M9 or M10 Pro) and $100k GPU cluster.
We've been sat with basically the same PC specs for ~20 years - our current specs are within an order of magnitude of the ones we could buy back in 2010. This is not really constrained by tech, as we could have much, much, larger machines. It's more because there's no mass demand for much, much, larger machines - if it's big enough to run Office apps or VSCode then you're good to go. The exponential growth we saw in the 90's was driven as much by software demand as it was by hardware development.
I can see the next 10 years produce the same kind of push for larger machines that the 90's did. And we should probably expect the same kind of standards churn as our existing technologies for storage, memory, etc, don't scale up enough and new technologies become worth developing because there's demand for them.
My productivity profits from the best intelligence available, a decent context size, and a batch size of four.
While my MacBook has 48 GB of RAM, not only do I want the above requirements at a decent speed, but I also need my machine to run the development tools and test suites, ideally without the fans blasting at full load.
For the foreseeable future I will stay with providers rather than local inference, apart from niche use cases.
I'm in Australia, so we're probably not getting access to Fable again. We're learning that a faster model + better harness/framework > smarter model. So being able to run GLM5.2 locally and super-fast would be great.
But the existing tech we're using for 16Gb probably isn't going to scale to 16Tb at a reasonable price point. And the price point is relatively inelastic - people are used to paying <$5K for their computers, and they're not going to go much above that. You'll get early adopters paying $10K or more for a machine that large, but not the early majority. And even then, obviously, $10K is not going to buy you a 16Tb memory machine.
So there's room for a new technology to come in, where there wasn't previously. This is what happened all through the 90's, and we churned through a bunch of standards and technologies to try and keep up with demand.
Are they?
I suspect AI labs are buying stuff not just for their own use, but to make local use too expensive to be an option :-( And they can always make the "best" frontier model even bigger (though only fractionally better) so it's always out of reach of local use, while consumer laptops have nearly the same amount of memory they had a decade ago.
Prices aren't going down, and consumer platforms are being shipped with less RAM so we can be sold cloud products. This isn't going to happen.
Can you please explain to me how you're going to fit 700bb-1T params in 64GB of RAM? You realize there are memory requirements proportional to model size?
You don't. What they're saying is that today's small models (that fit on consumer hw) are better than yesteryear's top models. GPT4 was reportedly 8x 220B (~1.6T) MoE, and today you can run a 30-120B model that beats it handedly in real-world tasks.
Similarly for 4-20B models beating GPT3 (175B) and so on.
There is a sweetspot of "good enough" that the small models can reach, where you get equivalent tasks solved fully locally. They'll never touch SotA, but they'll reach 2-3-4 year's SotA. Which, depending on the task you need, it can be "good enough".
Personally, I’m waiting for hardware to hit the secondary market before I buy something to run unquantized models like GLM. But I have no doubt that I will, at some point.
oil workers buy 100k trucks they do not-much with. why not a 100k in computer?
Some, and the market fluctuates a ton.
> corvettes
Only the oldest, most unique model years: nobody is buying (C4-C5-realistically C6) mid-90s or early 2000s Corvettes for more than what they paid for them, and they never will.
Both of those things' value drops like a rock as soon as you buy them and, at least for cars, they don't all appreciate. Most don't. Even so, they appreciate at an incredible slow rate.
I can't speak for watches but I'd be surprised if it wasn't the same situation.
At least the gpus can create value after you buy them before they are worthless.
I assume (since they claim they are selling the batteries to AI data centers), they’ll produce some sort of EV >= F150 once the bubble pops, and we get a new president.
EV is a separate thing. Vastly overmarketed for the technology as it exists today.
The input cache hit tokens are incredibly cheap for them, (incredibly high margin too, except for deepseek).
And input tokens are in the middle. Input tokens can be processed very efficiently.
Also his math is wrong. $100k gets you 22.7B output tokens at $4.4/M which is how much GLM 5.2 costs.
At 500/s 22.7B is just 500 days. Or about 1.54 years. Which is much less then the life of the hardware.
concurrency
Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?
The compression is almost certainly in part specific knowledge getting fuzzed.
Likewise, LLMs do not violate the laws of information theory, and therefore the only way to encode X amount of information in Y amount of bits where X > Y is by performing what is effectively lossy compression, and as X grows larger relative to Y the compression ratio must change to lose ever more information.
Yes, for the sake of making chatbots that are "conversational" in that they can interpret natural language as input and produce code as output you can easily benefit in incidental and unintuitive ways by training it on more natural language text. But for a given fixed parameter size, it's possible to produce a better model for a specific task by selectively not muddying its training set in the first place with things that are likely irrelevant to the task.
It's hardly self-evident, and your counter-example is hardly applicable.
The first 10^50 of pi is not the same as having BREADTH of information in the training data, which is the whole point not just any random "information that is irrelevant to your use case".
not to mention that the first 10^50 digits of pi compress to quite small formula, so not much information there to begin with from a shannon/kolmogorov perspective
The memorization of say 100000 world facts through training texts, which enrich model associations all around, is absolutely not the same as rote memorization on 10^50 digits of pi. Not for a human, and even more so, not for an LLM.
An LLM trained with digits of pi and one trained with books and posts, even if they both have the exact same amount of bytes of training input, would not be comparable in any way in utility and reasoning capabilities.
>There's an infinite amount of information that we could shove into a model, and a finite amount of bits with which to store any of that information such that it can be usefully recalled or form useful logical associations.
Which is irrelevant. Anyway, the amount of information that doesn't form useful logical associations is even larger (e.g. actual human books vs possible permutations of characters and spaces). Just like those (random) possible permutations of characters aren't good for LLM input to get logical associations out of it, pi isn't either (logical associations of the kind we care for and expect, not of the kind related to pi's sequences).
Also it's not only not self-evident, it's also apparently wrong.
We don’t understand AI or natural intelligence well enough to make such statements. As for self evidence, cross-domain competence in humans and the rise of generalist models over domain-specific ones (on competence, not cost) seems to pretty directly tank your hypothesis.
assuming demand doesn't keep on increasing. even google has trouble having enough capacity apparently.
GLM 5.2 is already capable enough to assist in self-training which is similar to what we saw happen with frontier models and they appear to be getting there at a significantly lower cost than openai/anthropic.
I expect future Chinese models to introduce even more of this type of bogus "safety" training.
Looks like if you are a white hat, then you will be fighting an uphill battle. Black hats will be fine, they will not care, they can just run a heretic model or specialty trained model.
Care to give more context to this? Seems interesting
Mythos level really doesn't seem that scary. And it would be a great way to take away the American labs international market.
I think it would make strategic sense for them to release more capable models than what American labs are allowed to make available to the world. It would help them grow their global soft-power and be a destabilizing effect on the American economy.
China could not be happier.
The same model is going to apply to the silicon supply chain as well is my guess. 1000th the expenditure in exchange for being a little behind the curve.
I worry it will have a very real chilling effect on research and development, since customers will probably very quickly switch to the thing that costs 1/10th as much, sucking out the ROI.
Not that it would make any sense.
Any prohibition on open source models will do nothing to fix the problem.. since attackers will never feel bound to the law. All advanced models must be available for defensive purposes.
If the real motive is profit, then open source models are likely simply not a viable means to that end.
If huggingface or whatever is forced to take down open source licensed weights, there’s always bittorrent.
Export controls are one thing, but the US doesn’t really have import controls, and there’s no copyright issue, so DMCA, etc don’t come into play.
It’d take the courts years to decide how to contort the law to ban open weight models, and by then, it’ll be too late (and also pointless).
But that's the whole point.
Fall out of favor with the admin and you lose access to the good American models, aren't allowed to use Chinese ones, and fall prey to the attackers and behind your competitors.
Which would be fine, but as we know, people securitize the crap out of their investments these days, and least some people probably leveraged themselves on some US AI companies, so now the risk is spreading outside of the sector to the economy in general, which is made worse by the sheer amount of spending on AI.
I’ll happily pay a 100% tariff on open weight models, and there are no regulatory hurdles for them to jump through (yet).
Yes, you get your free model, but the cost of this is not developing your own capability and tying your fate to a country which may or may not have your best interests as a nation in mind.
This is just the deindustrialization that occurred in my home region (the American Midwest) playing out on a global scale in different sectors. It was originally driven by the Japanese, who, to their credit, acted more as partners than competition. Eventually that desire for larger margins went to China, and now you basically can't build anything of consequence without at least some Chinese parts, because there's "no economic case" for it. This means that you have to play Beijing's game if you want access to any sort of modern market.
You see this happening with Volkswagen's restructuring, next you'll see it with non-American, non-Chinese AI.
Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something.
At least with China, we are very good at winning trade wars with them here in Australia.
Or, on a more local note, an Australian automotive worker who worked for a company that figured out 10 years ago that they wouldn't be able to pay him a decent wage, compete with the then-upcoming Chinese EVs, and remain profitable.
There is no good guys in general, and whataboutism and making the scope bigger doesn't help.
The thing is that if the models you are building on are open source whether hosted on chinese / american / whatever service at least give you an option to switch provider easier vs a fable / chatgpt 5.6 that gets banned for none americans etc...
2 years ago america would have had the branding/perception advantage but right now that is well and truly gone...
Stop pretending there’s some type of moral high ground there isn’t. Disgusting.
man you're gonna be disappointed when you learn where the components for Ukrainian drones come from (spoiler alert, it's China 95% of Ukrainian drone manufacturers use Chinese components. Both Ukrainian and Russian drones are Chinese components glued together, the vendors in China literally stagger Russian and Ukrainian buyers on the factory floors to not have them run into each other). The largest trade partner of Vietnam and the Phillipines is China.
The kind of thinking that assumes that rivalry implies deglobalization or bloc politics is exactly what's 30 years out of date. It's projecting how Americans think on the entire world, but that's not how the world works any more. The rest of the world continues to globalize, even through war.
America is undergoing Sovietization and erecting an Iron Curtain, and China ironically enough is simply doing what the US used to do. If Americans think the rest of the world will follow them into isolation they're going to make the same discovery the Russians did in the last century.
I don't understand what your point is? This seems like a perfect example of comparative advantage - Australia can produce iron ore cheaper than anywhere else in the world and even when China launched a trade war against Australia the Australian economy kept growing.
There wasn't even any bump in unemployment from the closing of the car industry.
Once that trade war was settled, Australia got cheaper cars, China got cheaper iron ore and both economies won.
The rational behavior on both parts there is in stark contrast to current US policy, which is unpredictable and capricious.
> You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021.
This seems like a random list of complaints about China and I agree with them in general.
I think you'll find most major powers have similar complaints. There certainly are against the US - I think you might find that both the Philippines and Vietnam(!) have fairly mixed feelings about the US for example.
I’m sceptical they could find the legal framework to do this even if they wanted to
They have legal authority to (a) prevent export of US goods/services; (b) ban imports of physical goods; (c) ban transactions (including purchasing services or license agreements) with foreign firms
But I’m not aware of any legal authority which lets them ban US firms from running a Chinese-developed open source AI model in the United States, if they are at arms length from the vendor, and aren’t using it for government contracts or regulated applications
Possibly they could order HuggingFace/etc to suspend Chinese accounts. But if someone in the US (or a third country) downloads the model from China then reuploads it to a US server, completely independently of the vendor - where is the legal hook to prohibit that?
This would be extremely heavy handed and probably end up accelerating the loss of the virtual US monopoly of payment network. The reast of the world isn't going to let the US dictate that only they get the frontier models whether their US made or otherwise
Can they actually though? Do they have legal authority to tell a payment processor that it has to block transactions of a legal US company, just because the company is hosting a Chinese-developed open source model? I’m sceptical
And what about companies (e.g. AWS) that let you “bring your own model”?
That's sanctioning specific individuals for specific acts they performed which the US claims contravene its interests and those of its allies.
I don't agree with the ICC sanctions, but it really can't be compared with the proposal "sanction any company, even US domestic entities, which use a Chinese-developed open source model".
In fact, I think part of what enables the US to sanction them (under US law) is the fact they are neither US citizens nor residents; if they were US citizens living in the United States, I don't think the President would have the legal authority to impose those kinds of sanctions.
They could sanction Hetzner–because it is a German firm based in Germany. I don't see how they could sanction a US firm based in the US whose owners and staff were US citizens.
Also, the 5th Circuit Court of Appeal decision Van Loon v Treasury (Nov 2024) is relevant–it held that IEEPA (the law used to sanction ICC officials) couldn't be used to sanction the Tornado Cash smart contract system, since open source code wasn't "foreign property" under IEEPA.
I agree, my only caveat is that the current administration has shown it's willing to go beyond aggressive regulatory interpretations to questionable and outright implausible interpretations. As we've seen recently, the federal courts and SCOTUS are overturning most of these but that can take a year or more to resolve. The one positive light is they seem to push the hardest on certain culture war issues (immigration, voting, districting, etc). AI doesn't seem like a core hot button issue for the White House and there is a strong pro-AI / business faction.
US imposing export restrictions on a model from China?
The weights are already available and downloaded, is it going to be a crime to have them, run them, make them available? Constitutional rights still exist (I hope)
Now you're getting it! Commerce will call it a munition and those harboring it as harboring illegal/foreign munitions.
No business will take the hit, so they will quickly deplatform the models.
No end user has the GPU capacity to use GLM 5.2 or similar models at full precision so the government will call the problem "mostly solved." But they might choose to "make examples" out of a few people using p2p software to download the weights if they choose to.
I'm for making software better instead of banning it based on what the rich and powerful claim.
I suspect the real fear is that open weight models undermine the financials and token prices they thought were going to pay off their ludicrous spending because they have all raced and raised hardware prices.
We're still in the middle of the cambrian explosion.
If Anthropic was capable of developing Opus 4.49-4.5 2H 2025.... then any company with a research team capable of reading all the papers and press releases will be capable of producing Opus 4.8 by the end of 2027, either raw model competency, or in a harness like claude code (or better with both). I guess what I am trying to say is that Opus 4.5 does not represent the edge of agentic capability, merely somewhere in the thick meaty layer of "functional and achievable".
We can draw the line at Sonnet 4.6 in the US but much like encryption export restrictions in the 1980s, the line drawn will be laughably low within a few years and simply unthinkable in a decade.
That would be the rational thing to do.
> financials and token prices
I do not think the government thinks this deeply. Market manipulation might be a rational, if unethical reason to ban open source models.
But this admin banned Anthropic models to "own the libs." They will continue to ban what they want for whatever reason they want. I don't think those reasons will be particularly coherent.
Yeah. Illegal numbers.
If you classify AI as a weapon which seems to be the direction that we are all heading towards, they yes first amendment rights won't likely apply.
The reason GLM-5.2 hasn't been banned is that despite these cherry picked use cases, GLM-5.2 isn't even close to Opus in all use cases. These vibe benchmarks are ran by companies that are not part of the cyber services offered by Anthropic and OpenAI where they can use the models without the safeguards and refusals so their actual cyber capabilities can be utilized.
These guys that wrote the article compared a gimped Opus to GLM-5.2, knew full well it's misleading, and got the clicks regardless. They don't have enough clout to be a part of something like Project Glasswing, GPT Cyber, etc.
The others are a waste of taxpayer money. Extraordinarily low return on investment (kill-on-investment?)
Claude Code is an agent harness, not an LLM.
Claude is a brand (or group of LLMs), not an LLM.
wild guess - I wouldn't be surprised if Opus 4.6 was run quantized for a while, and 4.7/4.8 have QAT for that nerfed size.
But I would like to point out that the overwhelming majority of people using LLMs aren’t programmers, don’t care about coding, and couldn’t even be bothered to “vibe code”.
So we should consider the bias of the output of these open weight models, and what that looks like, outside of the context of writing code.
I don’t agree with “Software development is where money is made for these labs”. Coders will inevitably eat up the most tokens & buy the bigger $200 subscriptions because we want to keep working.
But us coders are still the small minority of users. They aren’t counting on us to get to trillion dollar evaluations.
They are counting on the regular folks to buy the $20/ month subscription. It’s really easy to run out your free tier usage these days, asking questions that have nothing to do with coding.
So my point is what does that output look like for someone asking a question about politics or world news?
I know Google gives me free Gemini AI from my Google Drive plan. Microsoft probably already does too, didn't test. Apple is probably crafting some arrangements if not offering already.
My point is most people wont pay for AI. It will be bundled.
And I think AI is going to be free for all, with ads.
Secondly these are "just" IDORs, arguably the easiest class of vulnerabilities.
Thirdly it compares to GPT 5.5 and Opus 4.8.
No, we don't have Mythos at home.
mythos is <10% ahead of gpt 5.5 on all benchmarks, which it gains by being several times the size of opus. had it been economical to provide, it would've been released to the public on day one instead of the marketing circus those effective altruism clowns had exhibited. admitting that it costs >1000% to run inference on a <10% better model would've been very damning.
do you have a source for this claim? i thought LLM providers earn high margins from inference (charged by token). is this no longer the case?
no one has a source, because no one knows closed model parameter counts. we have only heuristics which strongly indicate that Mythos is simply a big fucking model that any other lab could make an equivalent of.
The only ones who seem to profit are the ones running smaller Chinese models. Even NVIDIA seems to have to "reinvest" their profits into sponsoring companies to buy their cards now.
> No, we don't have Mythos at home.
That's still useful. To paraphrase the kids these days, GLM5.2 is in the room with us, today. Mythos is not. And for us in the EU, it's even more complicated, as Mythos might be with us in the room one day, and go poof the next day, on the whims of political entities that we have 0 control over.
Knowing where open, accessible, local models are is important. We know they're behind. But there comes a time when "good enough" is useful. Even if they're "just IDORs" today, and even if they're behind SotA today.
As someone else said above, GLM5.2 (and other models in the same tier like kimi, dsv4, etc) is / are slowly becoming "good enough" to assist in automated repo prepare work (download, install, test, edit, re-test, etc). And that translates in RL traces ready to be trained into the next generations. That might be more important than x% behind on benchmarks.
So one can see businesses owning their own such cluster, next to their database infra, in the near future.
GPT does way worse than Opus without their harness, but better with it.
Opus 4.7 and 4.8 do way worse than 4.6. (Intentional nerfing?)
Would have been interesting to see GLM in the custom harness.
Would also be interesting to run GLM in Claude Code, which it has presumably been fine tuned on.
Where's the cost per vulnerability for all the other models than GLM?
Also, without code this isn't very trustworthy. Could all be made up as well.
People will use the model with the harness. I know that harness may not be optimized to this model, but it's still more useful to see the numbers from an imperfect harness than from a no harness setup.
[0]: https://aibenchy.com/compare/anthropic-claude-opus-4-8-mediu...
It depends a lot on the task and harness too (using plans and to-do lists, vs one-shot answers), but for simply answering directly to an inquiry, often extra thinking doesn't necessarily improve the answer, especially if the answer is binary, or can be correct or wrong, as opposed to having more time to refine a creative output.
https://aibenchy.com/compare/google-gemini-3-1-flash-lite-hi...
Beats which model in Claude? Whenever a "benchmark" doesn't put precise model numbers in their headlines I am immediately skeptical. Either they don't know the difference (bad) or they are benchmarking against weaker models (misleading, also bad).
It's like when studies say "AI is bad at X" and they used GPT-3.5 in current year.
How are we supposed to stay skeptical of everything if we read anything!?
GPT gets there <5 minutes, GLM 5.2 without context takes ~1H.
Though the harness makes a significant difference. On Pi GLM5.2 dreams for minutes, with OpenCode it's more on the point and gets to editing quicker.
Any good resources about this (also for setup and recommend config)?
Oddly, this is a strong indication of the text being hand-written rather than LLM-assisted; it's very likely that a human made a mistake in creating the table.
What we're they? Also, wouldn't one expect a more recently released coding agent (with a more recent knowledge cut off) to perform better because they have access to more knowledge about vulns in these OSS projects, and even possibly have knowledge of your own "prior research"?
Which I guess makes what semgrep sells obsolete. Unless they have built a pareto-optimal point in terms of capabilities and token usage maybe?
Now I feel like that I'm covered by GLM 5.2 and Minimax M3 (when I need vision or a second pass on something).
It just goes off getting confused about how to design the map for 15 minutes and then times out.
Having used GLM 5.2 for non-security software work, I can say it's better than Sonnet (but not Opus), and cheaper than both (because when you steal someone else's IP, you don't have to amortize the cost of their R&D).
Or grabbing their GLM Coding Plan directly: https://z.ai/subscribe
I went with the second one to try it out, feels pretty okay (with OpenCode, though Claude Code would also work), however it feels like I reach the weekly limits somewhat fast with their 65 USD Pro subscription. They also have that whole peak times thing going on and apparently it will get worse after September:
> Supported models and Visual Understanding MCP share the same usage quota. GLM-5.2 and GLM-5-Turbo consume quota at 3x during peak hours and 2x during off-peak hours. Limited-time benefit: off-peak usage is currently charged at only 1x quota through the end of September. Peak hours: 14:00–18:00 daily (UTC+8).
After installing, do a `n8 build` to build the image, then `n8 --danger --provider opencode interactive` to launch it in a container.
Signup for GLM-5.2 here: https://z.ai
When I posted the comment I was both the first commentor as well as the first person to upvote the submission. That matters. My name is ALSO on the open source repo that allows Opencode to be run in a container.
That's transparency, maybe not here, but on a clickthrough to Github it is immediately obvioius.
Not sure a project nobody knows or uses is much better in this regard?
I think they give $5 trail credits to test with any of the open weight models.
Instead of shilling for the LLM providers.
You need to read the market. Linus didn't read it in 90's, Gates did and that's why Windows is in almost every home.
The only niche where it doesn't utterly dwarf the competition is personal computers and it looks like we're all getting priced out of that anyway
I'd be mostly fine switching to it.
I just can't find a cost effective way to do that. z.AI's coding plan is both overpriced and unreliable. ollama's is also overpriced. Paying by the token for it on openrouter etc is more expensive than just having a Codex or Claude coding plan.
If you have to pay by the token, it's clearly cheaper. It's not competitive with a coding plan though.
Because you would have to switch model.
You can't just say "Oh, button X looks weird see [screenshot]" while coding with GLM. You would need to switch to another model and then maybe back.
This article only talks about detecting vulnerabilities, so it's unclear if it's a true Mythos equivalent.
This is where we are heading and why many closed labs are terrified of this affecting their bottom line and the reason why they want them banned from being released.
99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.
The incentive to develop these Chinese models further is to trash the business case of most American AI labs.
additionally, reliable API, because z.ai can be finicky
also, not for Enterprise use, but I like non-US providers, I don't care if the party happens to be the one reading my information and stealing my trade secrets, if they won't respond to a US subpoena
What explains it?
Is TFA lying? Is the most upvoted comment here lying?
The article itself doesn't say "it's better", basically just says "in this one specific benchmark it beat Claude with Claude code". Mind you with multimodality it Opus still beat GLM 5.2 very handily in that same benchmark.
I can't find any contradiction and I don't see anyone lying directly. At most they lead you to imply false things, but they're not untrue at a literal reading.
There's a number of US providers who also run it, if that is your preference.