GPU/local LLM. more RAM and OSS... everywhere
Bangkok, Thailand
- Tweets 12,345
- Following 1,126
- Followers 36,577
- Likes 10,329
here's the order i'd go into local ai, anon:
> 12gb: learn the stack on a compressed 27b, how a model gets served, how an agent runs overnight with features like hermes agent's /goal. you will see good results and see some light at 50 tok/s fresh.
> 24gb: this is where you meet the king, none other than the one and only 27b dense q4. get 2x 3060 if you want hands-on tensor parallel, or one 24gb card if you want to scale correctly and efficiently. at 24gb i prototype about 30% of my daily work, that's where owning your thinking starts.
> skip macs and the 5090 and the rtx 6000 pro for now, learn the stack small first and understand the difference. it's not a thing that happens overnight, it's a process and a lifestyle, you don't just start doing everything, get disappointed and then never realize how profound this is, that you can think and it stays on your metal.
> dgx spark: go gentle, and if you don't want to learn pcie lanes, power distribution and bifurcation, the dgx spark path is a clean cuda box, plug in and go. you can stack more and with the connectx cable it spits well. i have 2 dgx sparks so i can say for sure scaling with dgx spark is the cleanest. sure you don't get super fast bandwidth for dense models, but moes like qwen 3.8 flash next, ling flash and stepfun 3.7 dance on a single dgx spark and go zoom.
> but if you want to learn real infrastructure, from cable management to pcie lanes at x8 and x16, bifurcation, slimsas cables, backplanes, airflow and power distribution, that's what the dgx spark skips for you, and the price is bandwidth. the ultimate path is the rtx 6000 pro path if you dare touch fire, and the fire doesn't care about dense or moe, it just burns everything, which i only intend to.
rejoice my fren
when you realize your 12gb gaming gpu runs a 27b ai model at 50 tok/s and does overnight agentic tasks
15
13
142
14,405
the ai you pay $20 to $200 a month for is a rental, and whoever you rent it from can swap the model, cap your usage or pull it whenever they want.
in february openai retired gpt 4o from chatgpt, paying users asked for more time to move their work off it, and it was gone anyway. the weights on your own gpu don't get retired. a 12gb 3060 runs a 27b at 50 tok/s today and it's the same model next year, offline, no subscription, no usage cap, and nothing you type into it leaves the machine.
own your thinking anon and keep it on your metal. buy a gpu, start at 12gb, and learn the stack while it still feels like a nerd thing.
i think local ai is inevitable. it starts with businesses realizing they simply cannot send their clients’ data, internal documents, code, research or trade secrets to someone else’s servers.
then institutions start realizing the same thing. they’ll want the intelligence, but they’ll also want control over where the data and compute live. once enough of those organizations start buying their own compute, building private inference infrastructure and running models locally, the hardware gets cheaper, the software gets better and the ecosystem matures.
and then normal people hit the wave. today local ai feels like a nerd thing. eventually it’s just called ai.
7
18
1,123
from a 12gb gaming card to 2x dgx spark, here's the model i'd run at each tier, the speed i measured and the settings that make it work, bookmark this:
> 12gb: bonsai 2 27b ptq1_0 with the mtp head, 50 tok/s fresh on a 3060. kv cache at q4_0, one slot, n-max 1 on the head and reasoning effort medium, because the default xhigh can think through your whole token budget and hand you an empty answer.
> 24gb: qwen 3.8 27b q4_k_m, 41.1 tok/s on a 3090 with mtp off and 65.8 tok/s with it on, at a 131k window. add --spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 to llama-server and keep kv at q4_0, f16 kv hits a wall between 90k and 131k on 24gb.
> 1x dgx spark: moes, since it has 273 gb/s of bandwidth. qwen 3.8 flash next ud-iq4_xs does 44.4 tok/s in llama.cpp and ling 3.0 flash on the official int4 does 38.7 tok/s in vllm, both with mtp on.
> 2x dgx spark over connectx: qwen 3.8 flash next official fp8, tensor parallel across both boxes, 45 tok/s fresh and 35 tok/s mid-build with mtp, full 262k context with vision.
the rule under every tier is the same, decode tops out at your memory bandwidth divided by the bytes read per token, which is why a 1.75 bit file flies on a 3060 and why a 180b moe that only reads its active experts per token flies on a spark.
here's the order i'd go into local ai, anon:
> 12gb: learn the stack on a compressed 27b, how a model gets served, how an agent runs overnight with features like hermes agent's /goal. you will see good results and see some light at 50 tok/s fresh.
> 24gb: this is where you meet the king, none other than the one and only 27b dense q4. get 2x 3060 if you want hands-on tensor parallel, or one 24gb card if you want to scale correctly and efficiently. at 24gb i prototype about 30% of my daily work, that's where owning your thinking starts.
> skip macs and the 5090 and the rtx 6000 pro for now, learn the stack small first and understand the difference. it's not a thing that happens overnight, it's a process and a lifestyle, you don't just start doing everything, get disappointed and then never realize how profound this is, that you can think and it stays on your metal.
> dgx spark: go gentle, and if you don't want to learn pcie lanes, power distribution and bifurcation, the dgx spark path is a clean cuda box, plug in and go. you can stack more and with the connectx cable it spits well. i have 2 dgx sparks so i can say for sure scaling with dgx spark is the cleanest. sure you don't get super fast bandwidth for dense models, but moes like qwen 3.8 flash next, ling flash and stepfun 3.7 dance on a single dgx spark and go zoom.
> but if you want to learn real infrastructure, from cable management to pcie lanes at x8 and x16, bifurcation, slimsas cables, backplanes, airflow and power distribution, that's what the dgx spark skips for you, and the price is bandwidth. the ultimate path is the rtx 6000 pro path if you dare touch fire, and the fire doesn't care about dense or moe, it just burns everything, which i only intend to.
rejoice my fren
5
2
25
1,844
trust me when i say this anon, once you have 2x dgx spark on your desk the ai doom talk stops working on you.
the whole darioverse runs on one idea, that this level of intelligence has to live in their data centers and get handed out carefully. then you run a 180b model in fp8 across two boxes at 45 tok/s fresh with mtp, the full 262k window and vision on, it builds you a 3d game in about 2 hours, and each gpu pulls 30 to 35 w while it generates.
the thing they keep calling too dangerous to hand out is sitting on my desk right now at the power of an old light bulb per gpu, answering to me. dario can't pace it and sam can't retire it. that realization is the endgame, the boxes just get you there.
this is what a 180b model dancing looks like, qwen 3.8 flash next fp8 across 2x dgx spark, and the red boxes are the part that surprise me.
> 10 to 12 w per gpu at idle
> 30 to 35 w per gpu while it generates, peaks around 55 w
> never past 60°c on either box
> 45 tok/s fresh, full 262k context, vision on
i keep my hardware cool and these boxes make it easy. i've run models on everything from 8gb cards to 1tb of ram and nothing i've touched does this much per watt, a 180b moe serving an agent all day on about the power of an old light bulb per gpu.
this clip is 7x, seven minutes of it working while the numbers barely move.
2
10
1,280
my first model on huggingface has 11,792 downloads in 5 days, and it's a 27 billion parameter model doing 50 tok/s on a 5 year old rtx 3060.
new here? it's bonsai 2 27b, qwen 3.8 27b compressed to ternary by prismml into one 7 gb file. it shipped without its mtp head, so i grafted the head back on and wrote a kernel fix for gaming cards, and the same card went from 26 tok/s stock to 50 tok/s with both, all local, nothing leaving your machine.
here's what else that 12gb card does:
> 18.3 tok/s with 77k tokens already in the window, head on
> vision on top, it reads a web page screenshot and answers at 50 tok/s, head on, 131k window, still inside 12gb
> built octopus invaders in 5 hours, 8 files, 2,345 lines, every one written by the model
8gb works too, someone ran it on a 3060 ti and sent the numbers as a pr, 42.8 tok/s at a 96k window with the head off.
tell me your card below and i'll point you to the script for it.
holy shit! my bonsai2 mtp gguf now has 10,756 downloads in under 4 days, 6,780 of them since last night, for a model i packaged for gaming gpus like rtx 3060, 3070, 3080s and 40s. i am still processing this.
if you're new here, this is bonsai 2 27b, qwen 3.8 27b dense compressed to ternary by prismml. it shipped without its mtp head, so i grafted the head back on and shipped a faster kernel for the gaming cards, that took it from 26 tok/s to 50 tok/s on a 3060.
this week it built octopus invaders, a multi file space shooter, on a 12gb card, 5 hours, 8 files, every line written by the model, and the downloads went vertical right after. i'll leave the build link below.
the most common gpu on steam is a 3060 and the whole point was that card, the one already sitting in your pc, running a model this size at home with nothing leaving the machine.
model card and the scripts for your card are below, pull it and tell me what your gpu does.
12
5
66
3,797
model card: huggingface.co/sudoingx/Tern…
scripts for 8gb, 12gb and 16gb cards, the kernel patches and every sweep: github. com/sudoingX/bonsai2-small-gpu
2
9
590
BREAKING: anthropic ceo dario amodei is reportedly concerned that an rtx 3060 12gb now runs a 27 billion parameter model fully locally at 50 tok/s fresh, and has called it a grave danger that gpus this powerful are allowed in homes. sources say the new essay's biggest concern is that a gpu this powerful can burn a house down if it isn't power limited, which requires advanced technical safety skills.
more as it develops.
59
50
863
87,250
this is what a 180b model dancing looks like, qwen 3.8 flash next fp8 across 2x dgx spark, and the red boxes are the part that surprise me.
> 10 to 12 w per gpu at idle
> 30 to 35 w per gpu while it generates, peaks around 55 w
> never past 60°c on either box
> 45 tok/s fresh, full 262k context, vision on
i keep my hardware cool and these boxes make it easy. i've run models on everything from 8gb cards to 1tb of ram and nothing i've touched does this much per watt, a 180b moe serving an agent all day on about the power of an old light bulb per gpu.
this clip is 7x, seven minutes of it working while the numbers barely move.
here anon, watch how qwen 3.8 flash next fp8 builds on 2x dgx sparks at 45 tok/s fresh.
the build is my octopus invaders prompt, which is open, and i'll leave the link below so you can reproduce it. i told it to build a 3d version of the same prompt in voxel and it did it agentically, insane, the result is what you see on screen.
i've run many models from 8gb to 1tb of ram, but this model in fp8 on 2x dgx spark is enough to cancel all your cloud subscriptions. it dances like the frontier and produces like one and talks like an engineer, and i'm hooked.
what's your experience with qwen 3.8 flash next?
4
9
2,911
here anon, watch how qwen 3.8 flash next fp8 builds on 2x dgx sparks at 45 tok/s fresh.
the build is my octopus invaders prompt, which is open, and i'll leave the link below so you can reproduce it. i told it to build a 3d version of the same prompt in voxel and it did it agentically, insane, the result is what you see on screen.
i've run many models from 8gb to 1tb of ram, but this model in fp8 on 2x dgx spark is enough to cancel all your cloud subscriptions. it dances like the frontier and produces like one and talks like an engineer, and i'm hooked.
what's your experience with qwen 3.8 flash next?
qwen 3.8 flash next built this 3d octopus invaders game in about 2 hours on my 2x dgx sparks and i didn't even try hard. the model is soo tasteful you can feel it in responses.
here are the data and specs:
> qwen 3.8 flash next, 180b moe, official fp8
> 45 tok/s fresh, 35 tok/s mid-build with mtp
> 2x dgx spark, 256 gb unified, tensor parallel
> hermes agent, three.js and vite, a 3d voxel shooter
it wrote every file and fixed what i called out. with these hardware tiers you just need to know what good looks like and say it plainly.
this is the state of local ai right now, if you have the right hardware you won't even miss the frontier anymore. since i've been using this model i'm going to the cloud less and less.
watch, this is the game played live, no speed up.
5
26
4,258
here's the prompt, open for anyone who wants to try and build it:
github.com/sudoingX/octopus-…
1
375
meanwhile there are people arguing about whether one used rtx 3090 is worth it.
the men at the top are hoarding compute like it's the only thing that will matter. buy a gpu. buy the cheapest one you can find. just get on the board.
Colossus 1 is 150k H100, 50k H200 and 30k GB200.
Colossus 2 is 110k GB200 and 440k GB300.
Another 220k GB300 will be fully operational next week and another 220k in November. If we get lucky, yet another 220k GB300 by late December.
9
4
52
5,260
don't sleep on this model gamers, you already have the vram it needs, and you can taste the freedom of local ai on the card sitting in your pc right now.
it runs a 27b model at 50 tok/s fresh on gpus like rtx 3060, way faster than you can read, with no account and nothing leaving your machine. and it doesn't just chat, it does agent work, you hand it a task and it opens files, writes the code, runs it, checks it and keeps going while you're away from the desk.
this chart is the part people ask about most, context. with the mtp head on it starts at 50 tok/s and still does 18.3 tok/s with 77k tokens already in the window, faster than you read. the full 262k window fits on a 12gb card with the head off, and the shipped mtp script runs 163k, that's a whole codebase in one prompt. it slows as the window fills, and i'd rather show you the whole curve than crop it.
the card you bought for games sits idle most of the day, give it a job. i packaged all of it for you, the model, a serve script for every vram tier and a prebuilt bundle, so you extract it and run.
it got 10,756 downloads in 4 days already. 3060, 3070, 4060, anything with 8gb and up, pull it and tell me what your card does:
huggingface.co/sudoingx/Tern…
your gaming gpu can run a 27 billion parameter model at home, free, no account, nothing leaving the machine, and since tonight it runs at twice the speed it did yesterday.
i went into the code that serves it, found the kernel leaving two thirds of the card's threads idle on the biggest matrices, fixed that, then grafted the draft head back on top. 25 tok/s stock, 50 tok/s now, on an rtx 3060, a card people bought to play games.
it writes faster than you can read it, it holds a 262k context window, and it is a 7 gb download.
3060, 3070, 4060, 4070, 4080, 4090, 8gb and up, it is yours tonight.
10
6
59
6,685
here's the repo with everything behind this chart, the serve script for every vram tier, the kernel patches and every sweep, so you can check my numbers instead of trusting them: github.com/sudoingX/bonsai2-…
on a 20 or 50 series card, build the fork from source with the steps in there, and stock llama.cpp won't load this file yet, so use the bundle or the fork.
3
759
holy shit! my bonsai2 mtp gguf now has 10,756 downloads in under 4 days, 6,780 of them since last night, for a model i packaged for gaming gpus like rtx 3060, 3070, 3080s and 40s. i am still processing this.
if you're new here, this is bonsai 2 27b, qwen 3.8 27b dense compressed to ternary by prismml. it shipped without its mtp head, so i grafted the head back on and shipped a faster kernel for the gaming cards, that took it from 26 tok/s to 50 tok/s on a 3060.
this week it built octopus invaders, a multi file space shooter, on a 12gb card, 5 hours, 8 files, every line written by the model, and the downloads went vertical right after. i'll leave the build link below.
the most common gpu on steam is a 3060 and the whole point was that card, the one already sitting in your pc, running a model this size at home with nothing leaving the machine.
model card and the scripts for your card are below, pull it and tell me what your gpu does.
2,304 of you pulled this model in under two days. rtx 3060, 3070, 4060, 4070 owners, you are who this was built for.
- a 27 billion parameter model on one gaming card
- 50 tok/s on a rtx 3060, 53 tok/s on code
- the full 256k window fits in 12gb gpu
every gamer on the planet can run this and taste what local ai actually feels like. go pull it.
20
19
318
17,553
and if you want to see what it can build, the octopus invaders prompt is open too, run it on your card and post your numbers:
github.com/sudoingX/octopus-…
2
9
849
here are the octopus invaders results, a 5 hour build sped up to 12 minutes, watch till the end for the full uninterrupted gameplay:
this is what 12gb of vram builds in 2026, absolute magic
> rtx 3060 12gb, #1 gpu on steam
> bonsai 2 27b + mtp, 5.95 gb of weights
> hermes agent, 5 hours, 328k tokens written
> 8 js files, 2,368 lines, zero hand written code
> 50 tok/s fresh, 22 tok/s average, 125k context
watch the full video, 5 hours in 12 minutes of pure dance of a local ai model on rtx 3060 12gb vram, and stay till the end for the full gameplay.
this entire game was built by bonsai2, a qwen 3.8 27b dense compressed to ternary, and this small model is punching way above its weight. it built multi file engineering work using hermes agent, sure it's not fast but perfect for overnights and routine work and the quality is insane, and context holding is another best one, it does not lose the thread.
i ran PrismML bonsai 1 made from qwen 3.6 27b dense and i built things with it, but this time with the latest base model qwen 3.8 these results are insane, and because i loved building with it a lot i thought many more of you would run it because this gpu exists in almost every home.
so i packaged mtp, doubled the speed from 26 tok/s to 50 tok/s, packed the prefill fix in and released it on huggingface, almost 4,000 downloads in 3 days. i'll leave a link below.
2
827