qwen4exp: add hc ops by am17an · Pull Request #28901 · ggml-org/llama.cpp
time to re-benchmark Qwen Flash Next again submitted by /u/jacek2023 [link] [comments]
独立 AI 资讯库
AI 前沿动态、原文归档与媒体镜像
从自动抓取数据库实时读取，适合快速浏览、检索和后续扩展成完整资讯站。
11947资讯
11815正文归档
75522媒体
Connected a local model (Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M) to GIMP via MCP tools using llama.cpp - and here's the image result from my first prompt "can you draw a picture of a flower in gimp?". Needs work. Setup follows.
can you draw a picture of a flower in gimp? So I ran through this setup to install the MCP server suite from GITHUB, it's pretty easy to follow the install - and installs the libraries necessary to connect llama.cpp to GIMP. https://github.com/maorcc/gimp-mcp After that's installed and the se
I benchmarked IFM/K2-Horizon-7B on 16GB VRAM
After two days of fighting with benchmarking infrastructure, I finally benchmarked IFM/K2-Horizon-7B on 16GB VRAM. TL/DR: it works, but far behind Qwen-3.8-27B. Setup I compared 3 models - Qwen3.8-27B, Ornith-1.5-9B and new IFM/K2-Horizon-7B - on the following setup: All models were fit compl
Combining RAG with Continued Pretraining
For leaning purposes, I ran an experiment where I trained a model on a new domain using continued pretraining. Then to make it more flexible, I added a RAG step to inject dynamic data to augment the stable training data. The specific example is training qwen 3.5 4B on a fictional subway system to w
Cut Qwen3.8-27B Reasoning Tokens by 40% -- 3.8 'ThinkingCap' benchmarked!
EDIT: Sorry for the unclear title. This model is UkisAI's Swift-Qwen3.8-27B, not a new version of BottleCap AI's 3.6-ThinkingCap. All credit goes to UkisAI for making great fine-tune, and I made this post to celebrate their work. I meant no disrespect by mentioning another model in the title.
10%+ performance improvement on MoE ssd-streaming with expert-lookahead
I implemented expert lookahead, a trick that gets a 10%+ performance improvement on MoE models running on low-memory macs (Qwen 3.8 flash in this case) while using expert-offloading/ssd-streaming (using slotstream). This is 10% on top of several optimizations. I initially trained a small model to pr
ByteShape Qwen 3.8 27B: To KL Diverge or Not to KL Diverge, Part 2: Metric Boogaloo
Hey r/LocalLLaMA, We’ve released our full ShapeLearn GGUFs for Qwen 3.8 27B. Blog / Download models TL;DR 3.84 bpw (GPU-5) reaches 99.63% of BF16’s aggregate score of 8 benchmarks, being the most accurate quant we’ve evaluated; 3.23 bpw (GPU-4) reaches 98.72%. These average BF16-normalized sc
DeepSeek V4.1F Q4 on M3 Ultra with native DSpark MTP (40tps / 800tps)
I liked DeepSeek V4.1 Flash as an agent model but at 16 t/s on ds4 it was painful to sit through a real turn. It seemed some redditors and m3 ultra owners appreciated my glm 53 flash optimizations, so I forked antirez/ds4 for V4.1 Flash to see how much I learned optimizing GLM for the M3 Ultra
Best hardware for qwen 3.8
So I would like to run qwen 3.8 27b locally for my ai agents, maybe even in parallel with other small LLMs (such as qwen3.5 9b, oss 20b etc..). What is the best hardware to do this? Not a video card but I mean as “mini pc ai”. Thank you 🙏 submitted by /u/tombino104 [link] [comments]
jinfer: An open-source AI inference engine for the JVM. Finally, AI in jar.
For years, the JVM has watched the AI revolution from the bench. Every model, AI framework, every breakthrough, built with/for Python. jinfer is an inference engine built for the JVM from first principles: chat, vision, audio transcription, embeddings, reranking, and TTS. No Python runtime, no ONNX,
If you have a 3090, or other 30xx for local LLMs, I have something for you
I have a custom fork of llama.cpp designed around the ampere architecture specifically (though many of the upgrades also translate to faster performance of blackwell + lovelace). The recommended config supports 90+ TPS (for agentic/coding, at temp 1; greedy will of course be faster) through 100K tok
DeepSeek engineer relections on RSI - burying my talent to yesterday
Note - This is translated from the actual blog link right at the bottom. A few days ago, DeepSeek v4.1 was released. It raised the ability of small models to a new level. AI is improving much faster than anyone expected. From the first ChatGPT that could only chat simply with a few thousand tokens o
Coding Agent running entirely in the browser with Pi + MiniCPM5-2B (webGPU)
submitted by /u/paf1138 [link] [comments]
For the GPU poor. K2 Horizon 7B ranks between qwen 3.6 27B and qwen 3.6 35BA3b on the Artificial Analysis Intelligence Index.
From initial testing it seems pretty solid so far. Asked it to compile the latest llama.cpp for CUDA and its doing well so far. If this thing holds up to its score then its SHOCKINGLY good for its size. https://huggingface.co/IFM/K2-Horizon-7B-GGUF submitted by /u/Uncle___Marty [link]
UkisAI Swift-Qwen3.8-27B / -58.3% thinking, x1.95 speed while keeping the accuracy of xhigh
Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without "attacking" the reasoning length directly then fixed the accuracy with a bit of secret sauce (hint On-Policy Distillation) and achieved great res
Are there any organizations that are lobbying in favor of open source AI?
So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out open source. The thing is, there’s so much vested corporate interest in ensuring that open source rem
What actually makes you trust a local coding agent enough to leave it running unattended?
I keep seeing people here saying they can leave local coding agents running for an hour or two and come back to something usable. I’m not really asking which model is best. I’m more interested in what made you trust the setup enough to stop watching it. For me, getting a task right once wouldn’t be
The new k2 horizon models seem like an absolute beast
Especially the 7B one seems very interesting, it casually destroys muse glimmer with a way smaller size. And they open source literally everything, every step of the way. Anyone tried that model? It can be a new milestone if 7b and 3.7b ones are actually good, and not just benchmaxed. subm
Running Qwen 3.8 next on 16vram+32ram - A useful/fun post for the gpu poors
Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community. I have yet to benchmark this specific REAP version versus Qwen 3.8 27B QK4, but my assumption is t
3D viz of how Deepseek Flash v4.1 is different from a typical decode only transformer
https://whip.run/experience/356b3eba-23ce-4dc8-bff2-4f7bc1862966 submitted by /u/tempNull [link] [comments]
DeepSeek V4.1 Flash beats Astra on AA's new benchmark
https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that replaces τ³. Astra was farming a ton of points on it and used those to get even with Fable, but…
Plot twist
From ex Qwen leader https://x.com/JustinLin610/status/2098989619576729924 submitted by /u/jacek2023 [link] [comments]
I have just moved from MacBook M5 pro 48 GB to RTX3090
Using the RTX3090 on a linux machine I built for it and running Qwen 3.8 27B getting average 100t/s compared to my MacBook 20t/s I think I can finally get rid of my Claude subscription, this is good enough for me. I am a software dev and I can get what I need from this set up and be more productive.
Talk me out of buying a 3rd Spark
Does anyone think the gurus on the DGX Spark forum are going to figure out how to magically fit DeepSeek 4.1 Flash on a 2x cluster, or is it only possible on 3 or 4 Sparks? submitted by /u/Porespellar [link] [comments]
Dear 24G owners, try VLLM you might be able to run Qwen3.8 27B INT4, 144K FP8 KV on RTX 3090 with better speed. (TLDR VLLM AOT)
VLLM Benchmark: Prefill, Prompt processing - avg, 871.93 tok/s (3 hours constant running xhigh) - 10K prompt, 1000.26 tok/s (16 runs) - 90K prompt, 743,59 tok/s (16 runs) Decode, tok gen - avg, 38.39 tok/s (3 hours constant running xhigh) - 10K, 42.3 tok/s (16 runs) - 90K, 34 tok/s (16 runs) P
Migration from Claude Code to a private local harness. Questions.
I'll start by saying I'm not talking about the models themselves, I'm aware that I can't come close to something like Fable's intelligence locally. Just wanted to get that out of the way. Basically. Over the last year I've gotten quite comfortable with claude code, and it seems likely there were be
Hoping for Optimized Smarter Upcoming Models .... Like DeepSeek-V4.1-Flash( KVCache + Engram) in Small/Medium/Big sizes
It's still a dream for many folks to run medium size(30B range) models @ Q8 with Unquantized KVCache (256K Context) on their GPUs. It would be awesome to have DeepSeek-V4.1-Flash's KVCache + Engram for all Upcoming models. Even for big models. Engram - Heard that approximately 1/3-1/2 of Model
Qwen 3.8 27B UD-IQ4_XS even faster on 16GB CUDA
This is an evolution on top of Raymond's KV cache streaming fork - all credits to what enabled this goes to him. The basic idea behind what he enabled was a pool of memory in VRAM that is used differently depending on the phase (prompt processing or decoding) and when total used context is lar
My experience building 64GB VRAM AI SWE assistant/agent PC
I'm not fond of having an irreplaceable subscription on some OpenAI and Anthropic and being eventually being unable to do my job without using tools from some sketchy corporation, so I kinda turned my attention to local models. I periodically tried and found some usefulness in them, but this y
internlm/Intern-S2 · Hugging Face
from internlm: We introduce Intern-S2-397B, our most capable multimodal foundation model for scientific intelligence and long-horizon agents. Intern-S2-397B scales along three critical dimensions: pre-training, reinforcement-learning task coverage, and interactive agent environments. By combin