ilya watches

My dear friends, I’d like to show you my latest toy project Introducing Handwired: turn your regular JavaScript code into a neural net! - no training - exact outputs vs the JS program - fully differentiable All runnable in a browser, without GPU

High-performance code is simple in principle. Make one big contiguous array where each element in the array contains only what you need for a given "step" of calculation. Ideally this struct is compacted to fit on a 64-byte cache line. The struct should have no pointers in it. All the actual data should be 'flattened' into the struct. When the piece of data loads into L1 cache it takes the whole element with it and subsequent reads of that data come from L1 which can be 100x faster than a read from system RAM. Then write a for loop to iterate over the array processing element by element. CPU Prediction can recognize the contiguous processing occurring and prefetch an upcoming element before the processor reaches it. This makes it so that the bottleneck of the calculation is purely the CPU Hz and not the data bandwidth. Which today both CPU and GPU can calculate multiple fold faster than they can be fed data to do calculation on. Which is why this has such major potential of performance improvement. It is to remove the data bandwidth bottleneck by designing your data access to be optimally predictable so it can all optimally prefetch and the processor is unlikely to wait on data fetches. The calculation per element should have no branches. It should not change based on a bool or tag or other conditional with an if statement or switch statement. If you have a different category of calculation that should be done create a completely separate contiguous array for that data and that calculation to be done next. Branch misprediction can be as expensive as cache miss depending. This is also so that the same set of instructions can be utilized for everything which may need them before needing to load new instructions. Instructions are also data and have their own section of cache on a chip. Ideally, you should load and cache a set of instructions once, then reuse them for all data needed, then go to the next set of instructions. You can encounter real-world scenarios where there is so much instruction cache churn from variation of calculation being done that instruction cache by itself can become a bottleneck. This is where avoiding C++ STL and exceptions like the plague can show real benefits as they tend to easily bloat your instructions multiple fold. The only exception to using a conditional branch in a step of calculation is if the conditional is predictable. Meaning it does not change for the entire cycle of going through the contiguous array. Or it predictably changes in a pattern. Such as every odd or even step does the same thing. The logic of this pattern needs to be implemented as simply as possible to ensure the chip can actually predict it. Like a modulo on two uints. If you need any kind of unpredictable conditional you work the calculation to rely on a cmove. Generally, this means use of the ternary operator. But compilers can convert ternary operators to branches if they are too complex. They can also convert if statements to a cmove if they are simple enough. This is a scenario of where you need to check the assembly to ensure any conditional logic is not a branch unless you expect it to be. Generally, the constraints of ternary syntax will incentivize you to write statements that can be a cmove. Changes to the contents of the array should only occur between cycles of processing it. If you need to write anything from this processing, you should have a separate 'output' contiguous array for the results. You don't write back into the same structs as you read them. Multiple threads can read from the same contiguous array to do different calculations. But multiple threads should not write to the same contiguous array. You would need a different output array for each thread if you were to do this. You never use a mutex, fence or semaphore in any calculation occurring on a contiguous array. Threads can all wait at a single sync point at the end of the cycle if need be. However, if there is no need for a thread to ever sync or wait on other threads, then to pass data between threads you want to use channels. 'Channels' are Single-Producer Single-Consumer queues which use a singular atomic operation on an uint as the synchronization. This is the premise behind ECS but this is not explicitly ECS. This can be the generalized way you program everything. If you don't have contiguous elements of data but do only have one occurrence. Then make sure that struct is also tightly compacted with no pointers and the structs are grouped by access pattern. Then everything needed for one chunk of calculation is in one struct. Then everything needed for the next chunk of calculation is in a struct which comes after that. Ideally, you put all these sections in one big fat struct one after another to ensure their layout in relation to each other. Even though each single struct may have completely different data and different calculations done on it. The sequential layout of one after the other in the order their calculation occurs can still help prefetching of both the data and the instructions. If you notice. Doing this well is completely contradictory to object orientation. It can even be contradictory to functional programming, although it tends to have more overlap with functional programming. Any use of virtual methods, or smart pointers, or any pointers, or malloc/free mid-calculation is contradictory to this. You avoid all of that. There is a whole level of 'optimization' beyond what I just described when you get into specialized CPU intrinsics and SIMD operations. But what I just described is the foundational approach to designing a performance-oriented data-oriented system. The scaffolding and patterns you'd want to set up before drilling into anything too bespoke for a specific chip. The role as a 'System Architect' is to translate a problem into these constraints. Be able to conceptualize a problem into categories that can be optimally arranged for contiguous processing and fed in sequence. Personally, IMO, people who fill their code with Modern C++ features that throw everything over the fence to the heap with no awareness of how their data is laid out in memory. That's not system architecture. They have only followed the instruction manual of C++ or Rust or whatever and know how to get code past a compiler. But there was no 'architecting' of the 'system'. System including the physical system and hardware and making the software optimally fit how the hardware functions. This is the thing which an LLM struggles with. Is the decomposing and conceptualizing of a problem into this kind of scaffolding. Not all problems described in high-level layman prompts carry an obvious conversion to this. You need to conceptualize the problem and describe it in a way which can fit to this. All these fundamentals are very simple. The harder part is how to take a complex set of functionalities needed for a whole application and get it to feed these simple fundamentals. It's not really hard in comparison to any other programming endeavor. It's just most people aren't used to it. Thanks to C++, Object-Orientation, Java and other things. The whole software industry got stuck going a completely different direction trying to describe the higher-level 'intent' of a program. On the promise that someday compilers would get good enough to translate this 'intent' to the system better than any human could do. But that never turned out to be true. They sold the industry on a promise that never materialized. Now 20 to 30 years later when people should be looking back and evaluating. Did some high-level abstraction of "intent" combined with a fancy compiler ever enable someone to outperform what a real system architect could do? No. It never happened. Someone highly knowledgeable with C has always been able to make it faster. In reality the exact opposite of what was expected to happen came to pass. Part of the premise of why abstracting the system was originally seen as so important was because there was expected to be dozens of different systems. Different chipset instructions. Different kinds of memory and how you access the memory. It was expected that writing multi-platform code would be impossible if you had to deal with 'System' details because there would be too many systems. But what really happened is all the systems consolidated. To the point now where today your typical developer only needs to worry about two instruction sets. x86_64 and arm64. Further, these two instruction sets have so much overlap in how they operate that you can translate one to the other in real time. The reality is that being aware of system details, and writing code to fit them, became completely manageable, while the notion that a fancy compiler would be able to outdo a real system architect turned out to be completely false. Everyone got sold snake oil but unfortunately too much of that tech got entrenched in too much of the industry so it's now awkward to point it out. I am finding that if you "flatten" object-oriented C++ code into structs and fixed containers. Then let an LLM analyze it at runtime and freely rearrange everything by access patterns. You can get double digit performance gains compared to when it was using object-oriented features. Ironically C with an LLM is better at doing what we were once promised the compilers would be able to do. LLM's open the door that you could go back over the past few decades automatically reworking code and undoing the approaches that the compilers failed to deliver on.

Generational new Zuck GIF. Thanks @collision