#bigquery #streaminganalytics #logsaremetrics #structuredlogging @elixirlang @elixirphoenix #myelixirstatus

Lumis now lets you choose which languages to load in your package. Need only Elixir and Rust? That's only 2mb vs ~130mb of the full bundle. Just add: {:lumis_wasm_elixir, "~> 0.26"} That also applies to MDEx and MDEx Native. See mdex.hexdocs.pm/lumis.html#p… docs.lumis.sh

The future of programming seems certainly to be systems where change is constant and ongoing via agents, everywhere, and communicating all the time. And we happen to have a runtime perfectly suited to that. Lets go!

🥳 Excited to start revealing what we've been working on in the last few months. First, we decided to reinvent Kubernetes for agentic workloads with statefulness and fast resumption. Secondly, we are building an agentic orchestrator that will be Google's open agentic orchestrator and runtime. github.com/google/ax

I made a DuckDB extension where you can use @typesafeai 's Jev to do quick classification of rows in any csv/parquet file or duckdb table about 10sec for 1k rows ~ better than using an LLM, way more ergonomic than a classifier game-changing for data analysis!

The keynotes at @ElixirConf showed the Elixir community matured a lot when it comes to AI practices within the last year: * Quinn Wilton discussed the importance and accessibility of applying formal methods and static analysis in the age of AI * Chris McCord highlighted that last year coding agents could barely run for 20 minutes and today it can run for 20 hours non-stop. You should be stress testing, optimizing, running verifications way more frequently than before. Furthermore, as coding agents own more of the software development cycle, runtime observability becomes part of the loop (something Erlang/Elixir excels at) and will be super valuable (see screenshots) * Zach Daniel closed with hands on steps to adopt agents throughout and around your software development lifecycle The keynotes are all on YouTube: piped.video/@codesync?si=5fw… enjoy!

Everyone will need their own databases too

@smolmachines make it possible to branch a postgres db for testing or giving it to your agent for a PR.

The RabbitMQ vs Kafka debate was settled since the start, but benchmarketing bent it the wrong way. Now here you have the facts.

my next office

Introducing erlang_wasm: a WebAssembly runtime implemented in Erlang/OTP. No Wasmtime, Wasmer or wasm3 binding. It interprets WebAssembly on the BEAM. With the compiled tier enabled, hot code is compiled to Core Erlang and loaded as a BEAM module. 64,774 spec assertions passing. github.com/benoitc/erlang_wa…

The difficult part is writing the distributed cache that will sit in front of object storage to lower the latency and reduce cost. Getting it to work is trivial. Using object store for WAL or data is trivial to get going. You have to pay the piper. The piper collects on the write/durability path, and the currencies are coupled. Object storage gives you durability at PUT latency (tens to low-hundreds of ms with an ugly tail on standard S3) and bills per request, PUTs roughly 10× GETs. So WAL-on-S3 silently pins everything to the worst-case: commit latency is PUT latency, and every small write is a billed PUT. The cost that bites isn’t $/GB, it’s request count and cross-AZ transfer. Tiny objects blow up the request bill, so you’re forced into compaction/batching to build big objects, but batching has its own read/write amplification against S3, so you’ve relocated the request bill, not killed it. Zone aligned reads end up mattering as much as hit rate. It’s the classic latency/bandwidth tradeoff with cost dimensions.

You don’t write to s3 on the write path. You simply write to a distributed log on nvme. S3 should only be used to store the pages that are applied after processing the wal

Our take with SlateDB is to make the WAL pluggable. We have an S3-native WAL implementation if you can tolerate 100ms latency on writes. If not, go ahead and plugin whatever Kafka/Safekeeper/Paxos shit you want. Systems engineering is about exposing the right tradeoffs.

WHOA

Adding a few others from my experience working on an OLAP database: · Costs: S3 is expensive if you do a lot of read/write operations. I’d say it can be prohibitive for large-scale systems. The workaround is to add batching and other techniques (including another support databases), which adds a lot of complexity if you need to keep transactions atomic. · Rate limits: People think S3 scales as much as you need, but like any other service, it has rate limits that you need to handle. Not new, every storage system has them. · Bucket performance degradation: When you use S3 for a service like this, you do it because you don’t need to worry about designing a distributed system (at least the hard parts). You trust S3 to handle HA and so on. But bucket performance can degrade, and it’s more common than you’d expect. Usually, this means higher error rates and latency, but it can take down your wrapper. So you need to design for that, and that’s not easy. · You are tied to AWS. GCS has a compatibility layer which works "well" but azure does have it. Locally you need to rely on minio (which changed the OS license recently). In onprem installs you need an extra product that serve as the S3. Using AWS S3 from a different provider is too expensive due to egress costs and latency. · Object management. Eventually you need to do batch opeeation to drop/rename/whatever on a lot of objects. Not easy/cheap. Final users don't see this but the service provider needs to account for this.

When your thin S3 wrapper starts to get THICK

I have found the phrase "databases built on S3" misleading. In some of these systems an ACK'd write doesn't even reach S3 so in what sense is S3 the source of truth?

New post! The WebSockets vs. SSE discussion should be about ordering and correctness: dashbit.co/blog/websockets-v… When using SSE+Fetch, it is very easy to receive and render events in the wrong order, resulting in confusing user experiences or misleading users into making the wrong decisions. The article includes a bunch of diagrams to illustrate the issues with different approaches. The article was fully written by a human and proof-read by LLMs. The interactive examples were fully produced by a coding agent under adult supervision.

I too, like every other nerd on this platform, have been playing around with a WAL git implementation on s3, after Cursor's incredible "Git at any scale" blog post. - Written in Elixir/Phoenix - Customizable storage adapter. - Web UI in Phoenix LiveView. Claude took a little bit too much inspiration from someone... - Configurable webhook dispatcher - It's really fast, O(1) lookups Didn't build any auth. I run it in my tailscale network, so no personal need yet. Lots of fun to learn about this implementation, parallelization issues, and the inner workings of git. Give it a spin! But probably don't run it in prod 😅 Code here: github.com/Nickforall/walgit…