community
Reddit LocalLLaMA
2026-09-26T19:54
正文 thin
I wish Qwen also released dataset and method to fully train a model ourselves but it is what it is. However, I come here with my stupid question because someone can answer it better. And will the model still be an over thinker of faster inference will make up for that. submitted by /u/polite
community
Reddit LocalLLaMA
2026-09-26T17:41
正文 thin
I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken. The basic idea came from looking at how Colibri handles experts across disk/RAM/VRAM so I took inspiration from the general approach. Repo: https://githu
community
Reddit LocalLLaMA
2026-09-26T17:27
正文 thin
On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4_K_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent speculative decoding, a well-implemented prefix cache, and mixed-weight support in a single program.
community
Reddit LocalLLaMA
2026-09-26T14:01
正文 thin
For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. And whenever a quant from Unsloth guys comes out the guide can either be updated or new guide could
community
Reddit LocalLLaMA
2026-09-26T10:24
正文 thin
I know there is nothing new with what I am saying but I recently started with hermes agent (it’s been a while I wanted to but did not have the time). Qwen3.8fn 6bpw exl3 (from turboderp) on a 6x3090 (I assume lower quants on lower number of gpus work same) gives me around 80-120t/s with good pp, and
community
Reddit LocalLLaMA
2026-09-26T09:13
正文 thin
Hello r/localllama once again, it's me your kobold concedo Been a few months since I last posted here, and today I have something new I'd like to share. Specifically, KoboldCpp now ships with a built-in integrated KoboldCpp Agent Harness! I know it's a little late to the game, but I saw people frust
community
Reddit LocalLLaMA
2026-09-26T08:54
正文 thin
https://huggingface.co/internlm/Intern-Decision-0.8B Intern-Decision-4B Demo | Model Weights | GitHub Intern-Decision-4B is a multimodal structured decision model fine-tuned from Qwen3.5-4B. It accepts a shared state, a schema of named questions, and optional images, and returns an answer dist
community
Reddit LocalLLaMA
2026-09-26T08:52
正文 thin
Im running a 5090 and 64gb of ram, so im limited on what I can run. I have currently been able to fit the following - Atomic Q4_k_m 4.27bpw @ 31 layers offload Swift IQ4_xs @ 32 layers offload. Im about to try the Unsloth IQ4_xs as well. I could get a "bigger" (non IQ) quant for atomic because its
community
Reddit LocalLLaMA
2026-09-26T06:54
正文 thin
For the last few days, I've been using two computers to run multiple agents. My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are running on a machine with 2x3060, running/managing smaller models such as Gemma 26B A4B or Qwen
community
Reddit LocalLLaMA
2026-09-26T00:28
正文 thin
The conceited little fuckers love to inundate you with unnecessary details, noisy caveats, what's 'load bearing' and what's not, waste your time with a wall of text every time it reports back to you. They think human PP times are as fast as theirs, but they're not! It takes time to read as a human!
community
Reddit LocalLLaMA
2026-09-25T22:53
正文 thin
I've been building a small decision model for agent loops: gates, routers, "should I ask the user or just act" checks. It's out now as Mica v0.1 4B (Apache-2.0). What it does You give it a state, a question and the allowed answers, and it returns a calibrated probability for each answer: yes/n
community
Reddit LocalLLaMA
2026-09-25T21:24
正文 thin
The jump from Qwen3 Coder 30B A3B to current-day Qwen 3.6 35B A3B is crazy, especially with all the fine-tunes, and that was around 6 months (I didn't care for local AI back then, or AI at all, apart from as a toy so I don't know). Is around a year until I will never need cloud without buying ridicu
community
Reddit LocalLLaMA
2026-09-25T20:44
正文 thin
Hi, human here with rambling thoughts to share. Feel free to skip Overall, I don’t feel like I’m missing much; if anything. On my hardware(M1 ultra w 128Gb) it’s probably not as fast as Claude but I’ve been using opencode for research and other business related tasks and it’s been getting the job do
community
Reddit LocalLLaMA
2026-09-25T20:35
正文 thin
Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read this really interesting paper: Cache-to-Cache: Direct Semantic Communication Between Large Langua
community
Reddit LocalLLaMA
2026-09-25T19:13
正文 thin
I have limited Context (usually around 64k) For local use I don’t only do coding But also want like a personal assistant with memory and such. What is the best option? submitted by /u/Adventurous-Gold6413 [link] [comments]
community
Reddit LocalLLaMA
2026-09-25T16:44
正文 thin
I've been working on clinical speaker attribution at Omi and wanted to compare the current diarization models on the same audio. I used 15 mock doctor–patient consultations from PriMock57, about 2.4 hours. Full recordings, automatic speaker counts, without telling the models there are two peop
community
Reddit LocalLLaMA
2026-09-25T14:14
正文 thin
Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful for? submitted by /u/Ambitious_Fold_2874 [link] [comments]
community
Reddit LocalLLaMA
2026-09-25T06:32
正文 thin
edit: had to change formatting as I’m on mobile noe and the table broke.. I let Codex do a quick eval on Qwen 3.8 Flash Next IQ4_XS (served via vLLM + r9v) and Qwen 3.8 27B FP8 (served via vLLM + Radiance) This was not the full eval. I only ran a reduced quick test because the full version would tak
community
Reddit LocalLLaMA
2026-09-25T02:45
正文 thin
I've been playing around with various models on the M5 Ultra 256GB 80-core Mac Studio. These are the results over many rounds of agentic inferencing. I'm happy with the performance. Glad to have the large amount of RAM. But it does feel like the GPU is underpowered for this amount of RAM. I'm
community
Reddit LocalLLaMA
2026-09-25T01:31
正文 thin
Not asking for anyone’s secrets of the trade, I’m more curious how people are thinking about context now that newer models chew through huge amounts of it for reasoning. the TLDR: I’m starting to think of context less as working memory and more as a temp scratchpad to start each step. I’m running a
community
Reddit LocalLLaMA
2026-09-25T00:50
正文 thin
submitted by /u/piddlefaffle12 [link] [comments]
community
Reddit LocalLLaMA
2026-09-24T23:28
正文 thin
I know questions like this are asked often but I didn’t see this specific one submitted by /u/Borkato [link] [comments]
community
Reddit LocalLLaMA
2026-09-24T22:50
正文 thin
Just wanted to share with someone - don't have much to report yet. I am interested in this new AliceAI model and have been wanting to make a community impact for a while - and releasing an initial agentic version of this model sounds cool. I am training on 3 32gb v100s (which has been fun to g
community
Reddit LocalLLaMA
2026-09-24T21:54
正文 thin
I spent the past day and a half trying to get qwen27b to complete some practical work for me. I have a Bambu h2c I have been wanting to get more use out of so thought this would be a fun experiment. I have 27b running on my 5090 and qwen image 2.1 running on a 3080 10gb with comfyui. I had pi
community
Reddit LocalLLaMA
2026-09-24T21:00
正文 thin
The user is frustrated — I rambled too much and didn't act. Let's just run the test suite and move on. No more forensics. One command, execute, then report. (Original memo is a casual internal monologue in English. Translating faithfully while preserving the informal, stream-of-consciousness registe
community
Reddit LocalLLaMA
2026-09-24T16:32
正文 thin
Hey everyone, Jovan from UkisAI here! Today, we are introducing Swift, a family of efficient reasoning LLMs based on Qwen, trained by penalizing tokens related to pathological overthinking patterns and restoring accuracy via RL (GSPO) and OPD. After amazing feedback and 350k+ downloads in 13 d
community
Reddit LocalLLaMA
2026-09-24T16:11
正文 thin
With the release of ThinkingCap-Qwen3.8-27B, I thought it would be worthwhile to do a comparison between the original Qwen3.8-27B, the new ThinkingCap, and Swift-Qwen3.8-27B. Both Swift which I already reviewed, and ThinkingCap do exactly the same thing: they reduce the excessive reasoning loops tha
community
Reddit LocalLLaMA
2026-09-24T15:44
正文 thin
Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a point where context lengths can get so large that 8192mb is not enough and the whole context needs to be
community
Reddit LocalLLaMA
2026-09-24T14:19
正文 thin
I’m releasing OpenJudgement-4B-Preview, an experimental Qwen-based model fine-tuned on custom datasets for classification, scoring and true/false judgments. It scores answer options directly, and Python formats the results into JSON with probabilities. It still uses an LLM backbone, but doesn’t gene
community
Reddit LocalLLaMA
2026-09-24T14:02
正文 thin
Here's my *first* implementation of KVA projectors on QFN (just the uncensored model for now) the highlights are basically as follows for using the projectors at each different layer: Starting at layer 12, prompt processing speeds up 1.85x [1700 t/s -> 3150 t/s] at the tradeoff of increasing p