独立 AI 资讯库
从自动抓取数据库实时读取，适合快速浏览、检索和后续扩展成完整资讯站。
Link to the article: KV Cache Quantization Benchmarks: KVarN, Precision Tail KLD benchmarks with BeeLlama.cpp v0.4.0, fork of llama.cpp with more KV cache quantization options. Models: Qwen 3.6 27B Q5_K_S 64k context, Gemma 4 31B Q5_K_S 16k context Standard quants, extended: q6_0 and q6_1, an
Following up on yesterday's post about running everyone's faves on 2 x 16gb cards while maximizing performance and KV. Previous post data used abandoned Cu130 VLLM image. Stats here are done on cu129-nightly. Which has the KV cache connector fixes and performance improvements. Highlights - 2 c
Just came across this coding benchmark: SciCode Artificialanalysis.ai reports a ranking which contradicts the feeling we've towards those models in real life coding. Is Gemma 4 really that good, or a benchmarking issue? EDIT: The contribution of this benchmark to the Intelligence index of arti
I love to see these impressive models coming out that compete with the giants from companies like Z.ai, Moonshot, Alibaba, etc. A win for the open source/weight community is always welcome. While I am grateful, I worry we might be seeing the slow death of models smaller than 27B. The ones released p
This is very important for us when considering local hosting. A lot of people decided not to buy expensive hardware because DeepSeek’s prices made it very difficult to break even given that deepseek was soo cheap. Also some of us use DeepSeek in routing, hosting Qwen and routing some hard task
TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to increase -b from 512 to 1024 and -ub from 128 to 512. Prompt processing improved by 2.36×, while generation speed remained unchanged within measurement noise.
Hi all. These are my system specs: dual xeon e5 2696 v2 , 160gb DDR3 ram ECC(1600mhz), 3 gpus: 3060 12gb, p100 16gb, 3050 6gb. And a 400gb nvme sdd RAID0, 3000 mb/s. The model is Deepseek-flash-0731 UD_8_X_XL, loseless, 161gb. Now, I'm not too knowledgeable about llama cpp flags, I wish run it witho
[Fully open source under GPL3, made from the ground up for use with local models, no subscriptions, no corporate backing] When i first started this, it was meant to be a fully lightweight, extremely modular alternative to openclaw, hermes and the like, and it still is! But i noticed people esp
https://preview.redd.it/9zwlpcushphh1.png?width=972&format=png&auto=webp&s=18cb49c738caba9799d22d4916337c772baa830f https://modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B submitted by /u/HugeConsideration211 [link] [comments]
https://preview.redd.it/o6ik6qboeohh1.png?width=1134&format=png&auto=webp&s=4016f26c50c1d93bd3d0c7e880e9b55a2d75310f I have been running Qwen3.6 27b for a little while (mostly coding tasks) and recently trying out V4 flash 0731 in it's place. It was very apparent the new v4 flash will make mor
Prime Agent is an open-source coding and research agent for general and long-running work. A self-improving RLM harness for coding and long-running autonomous tasks. Designed to be both token-efficient and expressive through programmatic tool calling, context as a variable, multi-agent messagi
I mean, Deepseek V4 Flash is an absolutely fantastic model, even though I can't run it on my machine it's so fascinating to see how it performs. Knowing that potentially it could be run at home is really motivating and makes me hopeful that those capabilities will trickle down to more affordable siz
I'm mosty interested in 128-192GB VRAM with 128-256GB RAM to spare, so SSD streaming is basically not even necessary. Seems only FP4 is supported, so older hardware will likely be slow - no Unsloth GGUF supported either. I'd be curious what people are getting with V100s, R9700s, etc, just to have so
Hey everyone! Scenema Audio is now a native ComfyUI custom node. Same model that powers scenema.ai now quantized so it fits on 8GB VRAM. When we first released it a few months ago as an API and Docker stack, the full precision transformers were too heavy for most people to self-host. That's fi
Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. because it worked, and at a quite usable generation speed! background: antirez ds4 DwarfStar has a SSD
In tests: ~80 tok/s decoding 2,500–3,500 tok/s long-input prefilling Smooth use by 3–4 concurrent users Private, on-device inference for coding, agents, and offline batch jobs submitted by /u/niacolhealth [link] [comments]
As you all know the model is 2.69B parameters with a 128K context window and purpose-built for multi-step agent workflows. What you are seeing is the Q4_K_M GGUF running on my own inference engine built from scratch. The TUI is my own device probe suite running through ADB (Android Debug Bridg
TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature: Mixture-of-Experts CPU offload: --n-cpu-moe <N> | -ncmoe <N> Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on the CPU; attention, norm
It's the purple cluster on the top left (the good corner...) I'm running the MXFP4 version from Bartoswski with Dspark at 1K t/s prefill and 90 t/s gen (average). I tried different sampling params, you can check the detail. It's very efficient while scoring the best yet. Too bad it does not ha
Hey everyone, I’ve been building Speechfony - a desktop app for reading PDFs (and EPUBs) with offline text-to-speech. Open a document, listen sentence-by-sentence with highlighting, or export selected pages to an MP3. Everything runs locally: Kokoro for speech, and an on-device embedding model
People may remember the Qwen3-TTS llama.cpp demo from a few months ago. That PR said it probably wouldn’t be merged because llama.cpp was missing some of the graph and API pieces it needed. A new implementation was merged into master yesterday. What works now: - Qwen3-TTS-12Hz-1.7B-Base in GGU
So I'm looking at https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF and I realize my 128GB of DRAM just isn't cutting it for this (incredibly powerful) model. If only I had another 64GB, I thought... EVERYBODY is probably thinking that right this second... I hate to say it, but I imagine DR
So one of yall mentioned that cuda 13.1 or 13.2 is broken for unsloth so I looked in to it, and they were right. I had 13.2 installed, after I switched to 13.3 no more looping!!! Before the cuda update, the model was literally unusable. A few minutes into the run it would start loop then the chat wo
submitted by /u/cafedude [link] [comments]
https://github.com/yhfgyyf/vllm-deepseek-v4-sm89 I couldn't believe that someone actually got vLLM working with this particular set of GPUs, but here it is. The video is from right after I got it working with 64k context, but it is now running with 256k. submitted by /u/dangerous_inferen
It is one of the best local models ever released, in both 20B and 120B versions. I always come back to it, especially the 120B version. Its only competition is, in my opinion, Qwen 3.5 122B, but that model is much slower (A10B) and has not been released in a local-friendly QAT format (such as MXFP4)
Liquid AI released LFM2.5-2.6B today, and this might be more relevant to local AI than another massive model most people cannot run. The model is only 2.69B parameters, has 128K context, supports tool calling and was post-trained specifically for multi-step agent workflows. The official Q4_K_M
Kimi K3 full model running on 16x GB10 cluster at 20+tps average (llama-benchy coherent corpus) 38tps peak, 750tps prefill. This is the first run of full k3 with dspark on my cluster. I will be doing some tests and try tp speed this up. As soon as it looks ready I'll publish the vllm image and
Why nobody is talking about this? Seems pretty significant to the community submitted by /u/MuzafferMahi [link] [comments]
https://x.com/xiong_hui_chen/status/2084695353346117760?s=20 submitted by /u/Capital-Remove-6150 [link] [comments]