Updating on a PR of mine merged to master: github.com/kaspanet/rusty-ka… TL;DR computing data relating to the pruning proof on the fly while building the proof rather than all the time results in substantial performance optimization on many fronts. A word or two for the unacquainted: theoretically speaking, all you need to know to follow a blockchain consensus are block headers and block bodies, if your node stores those locally, anything else you possibly want (to verify a new block, to know what can be pruned, to answer RPC calls) is implied by that. It is however preposterous to recompute stuff from scratch all the time, so in practice (in all computer science, not just blockchain), you more than often store a lot of “metadata” with the goal of making computations shorter at each step of the way. In Kaspa one prominent examples of such metadata are the various “per block” stores - databases containing certain types of info on every block (indexed by the block’s hash). Moving on: As you know in Kaspa there is a Dag structure and each block on it is assigned a blue score - a parameter somewhat equivalent to block number in btc. What is less famous is that there are actually multiple Dags in Kaspa - derived from the main one. Simplifying a bit, what you should imagine is that in these “higher level” Dags, blocks whose hash difficulty is not rare enough for that level are filtered out, and instead of the now hanging edges, new edges are accordingly shortcutted between great grandparents and great grandchildren. These are known as indirect parents, and they are written out in block headers for every level out of the possible 225 (It is worth noting though that most levels are regularly empty or contain very little parents). Additionally, each such Dag also implies a per level blue score for the blocks in it. Without diving into much detail, these high level POW representative but sparse secondary dags allow us to quickly compare the POW accumulated by two diverging DAGs without being forced to always store all blocks since genesis. For this comparison, what we require is the blue score of the blocks of the secondary Dags. In the far far past precrescendo, we computed the blue score for every block in the network on all possible levels, and stored it in such aforementioned “per block” store. In practice however, most of this data is of no real relevance - the levels in the pruning proof are by design themselves “pruned” to about 2000 headers, as it is the higher and sparser levels that form the significant part of the proof. Ergo a past PR by @coderofstuff_ improved performance by opting to compute the blue score of the higher levels only when generating/comparing a pruning proof. And yet, this on the fly computation of the blue score still left a remnant of per block X per level storage. It was a thing called relations_store. What the (level i) relations_store gives for every block is quick access to its “children” on that level. While the indirect parents of the level are directly derivable from the headers the same does not apply for indirect children. This store facilitated forward traversal of the DAGs, which was useful when calculating the blue score. The point is that children are ultimately still implied from the Dag structure and thus a satisfactory relations_store can be reconstructed on the spot by doing one backward traversal of the DAG on that level. Doing this only while building the pruning proof, and only for a small number of blocks, meant we no longer have to regularly maintain relations at every level. The effects did turn out to be dramatic - first, archival nodes reported a significant decrease in storage usage. Second, header processing became a lot more efficient without the need to write these new stores to the disk. The effect is that the Kaspa node is lighter on your PC - but of no less important is that the header stage of IBD became a lot faster. I am not sure if I should give figures, because every machine is different, but for me it was about an x3 difference. Finally, not storing these higher level relations also meant you don’t need to erase them during pruning. As it turned out - this was a big bottleneck during pruning, which my personal laptop had a hard time with, and is finally relieved from.

I want to share some explanation of a PR of mine freshly merged to master, regarding an optimization on the IBD process: First I’ll explain how stuff worked thus far and what the problem was: Initial block distribution (IBD) is a bombastic term to say syncing the node. If you run a node you know this well - you either connect for the first time, or just reconnect after a short/long while being offline and you start downloading a multitude of stuff before your node is back on track. There were currently two types of IBD, the first, “Sync”, was the simpler one - you reconnect after a short while, download and process the recent headers from a peer, then download and process the corresponding block data (transactions). Depending on your hardware this may be longer or shorter, but the process itself is simple. The second is called “ibd_with_headers_proof”, and can occur in two scenarios - (a) this is the first time you run the node and (b) you have been disconnected for so long that you no longer recognize your peer’s pruning point, and hence you cannot just “Sync” from him as a straightforward sync requires data they pruned. What happens in this type of ibd is more involved: (1) you open a new consensus instance with an empty database. (2) you download in it the MLS proof of your peer and verify it, you also compare it to your own old proof (if such exists) to see it does not contradict it. you proceed to Import pruning points and other storages accordingly. (3) You download some “trusted data” from the peer which serves as a base layer for the sync. No one likes the name “trusted data” but no one came up with something better. Still, it is emphasized that this data is only trusted through the duration of the IBD - by its termination, either this temporary trust will be justified by POW, or this “trusted data” will be thrown in the trash can where it belongs. This trusted data is mostly block data of the pruning point block and its anticone (=”parallel blocks”, kind of), but there are also some other headers there which are not relevant right now. (4)you download all headers from that pruning point to the tip of the syncer (5) you download from the syncer the utxoset corresponding to that pruning point, and verify it. In the far past this used to be rather short - but not anymore since dustman had its way with our utxos for a tumultuous 12 hours. (6) Finally you download and process the block bodies The first problem was that 1-5 was a take it or leave it process - if you had disconnected during any stage this attempt to resync is thrown away and you have to start from the beginning the next time. Given (4) and (5) can be quite long - this was not ideal. I skip the explanation here, but much for the same reason you know there must be valid block bodies in (6), you really know there must be a valid utxoset in (5). So first thing - PR made it so that if you stop during (5), you only need to repeat from (5), not from the very beginning. As a sidenote I already say that taking it to the next step is making (4), (5) themselves be gradual processes. (4) is absolutely on the map (see comment below), for (5) I’m not sure what the right approach is. The second issue is more subtle: There are cases that fall in between “Sync” and “ibd_with_headers_proof. Say you resync normally, but disconnect during the headers state before downloading block data. you come back a day later, and try to continue from where you were - but alas, your peer has pruned away the block data you require. Nevertheless you won’t go into ibd_with_headers_proof because you do know that pruning point’s header - so what will happen is that you will fail to resync, getting stuck in failed ibds until more pruning points surpass or you hard reset the node manually. If you run a weak node - you might have encountered this. Now of course the first measure would be recognizing this scenario, and saying you should go into ibd_with_headers_proof instead of getting stuck in failed IBDs for a while. But really this is a big waste - you already have the syncer’s pruning point header, and potentially many headers beyond it - you don’t need to redownload these headers, you don’t need to verify MLS proof again, you don’t need to break away from your current consensus data you have that it is a valid header from anew. What you really want to do is be convinced that it is a pruning point - which you can do if you will download sufficient headers on top of it - update that to be your new pruning point (and also update a dozen other storages), receive the “trusted data” as your base layer for future processing (that name is even more misleading here you know the data is real), and carry on syncing (including downloading a corresponding new utxoset). So that was what the PR was about, and at the time it sounded simple to me. Welp maybe it is, but it was certainly easier said than done since there was a lot of subtleties and a lot of need to prevent and handle “limbo states”, which the codebase was not designed with in mind. Was very instructive to me about ibd, mls, and the pruning process in general. Happy with the result and hope it eases up the ibd process for us weak noders. github.com/kaspanet/rusty-ka…

A few words on what I'm researching with @hashdag the last few days: A paper from 2017 by Lavi-Sattath-Zohar suggests an alternate fee mechanism for blockchains. The gist is initially unintuitive - miners charge from all transactions on their block the fee of the lowest paying transaction included on that block (even from the transactions that agreed to pay more). The idea is that this causes miners to optimize for number of transactions times lowest value transaction, and naturally exclude transactions paying considerably lower than others. It can be thought of (sortofish) like a smart and adaptive way to do "minimum fee". The desired effect of this mechanism as it pertains to incentivize users to pay higher fees to the miners in a manner independent of the blocksize (i.e. even when the mempool is smaller than the capacity of a single block). First, it's worth mentioning that this scheme was to my knowledge never implemented in the real world. Second, all of the above is true for blockchains. One of the challenges lie in adopting it to DAGS. Before continuing I want to clarify a common misconception I see about Kaspa: ---------------Intermission: inclusive blockchains------- Kaspa's high throughput is not primarily because of the transactions in parallel blocks - even if transactions not on the selected chain were discarded, Kaspa will still be more capacious than traditional blockchains by the mere fact that it can allow a higher blockrate for a given blocksize (...for a given security parameter and a given min hardware, etc.). But it is true that this simple effect diminishes as the number of parallel blocks increase - throughput is wasted if you throw away parallel blocks' transactions. This throughput can be salvaged by something called the inclusive protocol. Simply put - an order is assigned to parallel blocks, and all transaction in them which do not contradict a previous one are included in the ledger as usual. *In Theory*, we can now use every block to its fullest throughput. But why won't all parallel blocks just include the same transactions, wasting away our theoretical gains? well, if they all choose the same transactions, they will sometimes win big, but often get nothing. A miner then choosing transactions other had not is in average better off! Even if those transactions have lower fees. Long story short it turns out that the "correct" ("equilibrium", with some 5 asterisks or so.) thing for them to do is choose transactions in random, but weighted on the fees they pay. Simplifying a lot here fyi. It coincidentally also turns out this generally gives good throughput to the system. This is what happens in Kaspa today, but again, I cannot stress enough this is not the sole source of higher throughput in Kaspa. ---------------------Intermission End--------------------- So basically, while it theoretically was possible to just take a step back and give up "inclusive" (wavering some precious throughput), we wish to merge this new fee mechanism with the insights on inclusive protocols which Kaspa currently uses. With a new fee mechanism the equilibrium changes, and with miners having a different strategy, the original guarantees on user's behavior needs be reexamined as well - so once the miner equilibrium is set we need to ask of the users' equilibrium. Our hope is that after all the analysis we find out that either similar guarantees on the behavior of users as in the original paper can be maintained. Alternatively, we'll try to find a variant of the original scheme such that they will. If anyone with a proper background wants to take a look at this as well they are also invited btw. Long term research decentralization is a goal like everything else. Two final points: a) No fee mechanism change, as good as it may be, will replace the need for an increase in demand. This effort should be in parallel to building new utilities to Kaspa (e.g. vProgs). b) This is a mid term solution (but one long due serious research imo) and requires a HF. There were suggestion from within the community for modifications which could be instantaneous. These suggestions may not be long term theoretically sound but are often non detrimental - miner adjustable min fee for example. These types of solutions are usually prone to prisoner dilemma and as I said I don't think fee policy alone can currently change the big picture - but if someone were to just implement this I personally won't object as it is harmless. Sounds like a nice entry level project into rusty Kaspa dev imo.

And an extra word or two: the last week has been great in seeing people push for and suggest for solutions on their own. I alluded to this above, but I'll restress it - we cannot and shouldn't rely only on a small subset of people to provide solutions. No matter how smart they are, and they are, we as a community must take more responsibility on ourselves, to expand and eventually push out and outsmart the old generation. One of my criticisms over the "Terrah AI" project of Yonatan is that, it would never have made me smarter about crypto the way discourse with real people had. Yonatan said that very well, but Terrah is made not to make people smarter, only more informed. I want people to be smarter, not more informed. Me and Yonatan disagree here, and it is left for the reader to decide whether that's because he is smarter or because he is more informed. Anyway here is my poor attempt at pushing people to be the smartest version of themselves: my suggestion is that those of you who independently devised fee market solutions (and those who read those) to take it even one step further - be a researcher: don't rely on exteriors to criticize your idea and find its weaknesses, try and do so yourselves, see what others had to say on similar ideas, attack it from a thousand angles, find everything wrong with it - and then when you present them, present them alongside its weaknesses for discussion. I emphasize that a solution that has weaknesses is not necessarily a bad one, it can still be argued for and against, but it can give you a lot of insight on the problem, and in turn, help the community as a whole converge on optimal ideas.

“We want a cryptocurrency to ping and be ponged” A small wish list of mine for community devs: 1) Bullet Chess, obviously. 2) Battleships: You can use commitments (hashing) to hide your ships from the other player. 3) Trivia: Kaspa operates in real time, players could compete on who answers correctly first, game show style. 4) The most ambitious of all: I believe a clunky version of pong could run on Kaspa, it will be difficult to pull off well and undoubtedly requires optimizing the infra. 5) Some application which is not a game, surprise me. I am excited to see more people having fun with Kaspa. I secretly hope eventually an application will rise that will make true use of decentralization, but it’s fine that many are gonna be for fun initially. I hope someone does push it further to allow for persistency (see Michael post in comments), and in due time even ZK support.

Just an excerpt on that Litecoin blogpost: taking pride in not having RBF so you could put your trust in the mempool's good will is a testament to having given up any pretense of being mathematically sound money.

I deleted a previous iteration of this post where I insinuated Btc disables RBF by default as well. @OriNewman corrected me that I was malinformed on the resolution of the RBF arguments and that it has been standard for a while now. My apologies to BTC on my mistake.