I have long thought of a way to properly present and explain the ideas behind this new approach to a broader audience. I first thought about doing it in a video to explain all the reasoning behind it but a video would maybe not be concise enough to allow people to properly look at the presented ideas and also “challenge” them. Having “something” written down will definitely help here!
Usually, new consensus mechanisms get presented in form of a white-paper with lots and lots of math analysis and proper formal language but even though I am able to read and understand these formal terms and notions, this is not the way I (and most other people) “think” and I would really like everybody who reads this text to understand how the proposed consensus mechanisms works no matter if he/she/it has a university level math background or not.
Motivation
I want to start with talking a bit about the “motivation”, since I can literally see people asking questions like: “Why do we even need another approach if coordicide is already solving the trilemma?”
The answer is simple:
This is of course a pretty bold statement and before I continue, I want to say that this whole blog post is in no way “approved” by the IOTA Foundation or in any way “massively researched”. It is a very new idea that most probably still has room for improvements (or maybe even some serious flaws) but it anyway feels mature enough to already share it with you.
That being said, what do I really mean with “better”?
- It has less “moving parts” and is much easier to implement.
- It is orders of magnitude more secure.
- It has a smaller message overhead since we use “virtual voting” to reach consensus.
- It gets rid of the need for re-attachments and promotions.
- It is completely partition tolerant and resilient against large scale network splits.
- It works in the IoT and mesh net environment.
- It is independent of the amount of “bad nodes” in the network and does not require a specific network topology like CA.
I could continue this list and name even more things but let’s keep it at that for now and revisit the “features” at the end of part2.
Some last words upfront: This whole idea is heavily influenced by discussions with people who voiced their criticism of the current coordicide proposal like Eric Wall (and others). You guys challenged me and made me think of ways how to deal with large scale network splits and other more exotic attack vectors. Without this criticism, this blog post would most probably not exist. So thank you for voicing your concerns in a constructive way!
The basic idea
The basic idea of this proposal can be described in a single sentence:
If we find an efficient way of “encoding” our votes publicly in the tangle (in form of transactions) and allow nodes to reach consensus using a form of “virtual voting”, then this would make a lot of things much easier.
What do I mean in particular:
- Nodes could never fall out of sync, because their opinion would no longer be “local”. Every node who sees the same tangle, would automatically come to the same conclusion. The chance for an agreement failure would essentially be 0.
- Nodes who newly joined the network, would have an objective way to detect which tangle is the correct one if presented with conflicting versions by an attacker without having to “trust” an external voting layer. They can for example choose the version of the tangle that has the higher economic activity. The same mechanism can accordingly be used to deal with network splits (more on that later).
- We don’t need to waste bandwidth on exchanging additional votes and could massively reduce the message overhead of voting.
- Cheating will be much harder or maybe even impossible if all votes are public and we could eventually get rid of many of the additional attack-detection mechanisms (more on that later).
- The network topology could stay completely secret which would even make the auto peering much easier to implement and eclipse attacks much harder, because we could use readily available protocols.
The benefits seemed to be so overwhelming, that I decided to start thinking about this more.
Virtual voting in a permissionless setting
I started reading papers about virtual voting and analyzed the algorithms of several competitors but it looked like they all required some form of permissioned committee and were therefore not really applicable for IOTA.
I tried to modify the algorithms and brainstorm new algorithms but they all had some serious flaws and attack vectors in the open and permissionless setting, and just when I was about to conclude that it was most probably not possible, I had a moment of revelation:
The original white-paper version of the tangle already uses a form of “virtual voting” and everybody who sees the same tangle will automatically agree on the “heaviest” branch without exchanging any additional votes.
In fact, the very definition of the tangle is that “every transaction approves two other transactions”. A transaction therefore is already a vote by the issuing node about which branch it prefers.
The biggest problem of the white-paper version was not that it failed in its underlying principles, but that it underestimated the potential hashing-power of attackers. If we would consequently find a way to calculate weight based on something that is bounded and not as easily attackable as hashing power, then this would solve the single biggest issue of the original IOTA whitepaper.
Since we already have the concept of mana, it was a relatively straight forward decision to try to build a mana based version of the original white paper tangle and at the same time fix all of the other issues (like splitting attacks, the need for reattachments and promotions and so on).
The tangle as a voting layer
Some readers might have seen the video, where I talk about the fact, that the tangle is most probably the most efficient way to enable reliable decentralized communication over an unreliable medium but what I started to understand now, was the fact that the tangle is even more than that.
It is also a highly efficient virtual voting mechanism: A single transaction is able to vote on a potentially infinite number of other transactions because votes and approval can be inherited. This seems like a minor and kind of obvious detail but understanding this fact (and its importance) changed everything for me:
So far we have always considered the tangle to be a direct representation of the ledger state. Similarly to the block chain world, where blocks can only contain non-conflicting transactions, we envisioned the tangle to also only contain non-conflicting transactions.
This is however in total contrast to the nature of the tangle: Since anybody can attach anything to the tangle, we can not really prevent it from containing conflicts every now and then. The only way to deal with disapproval was accordingly to abandon the bad parts of the tangle and force everybody to reattach their transactions if they happened to attach to the wrong branch.
By trying to force the tangle to work “like a block chain” that has some kind of “validation mechanism” in front of it, we essentially ignored its nature and prevented it from unfolding its true potential of becoming an extremely efficient virtual voting mechanism.
The ledger state: a tangle inside the tangle
If we look at the tangle, we realize pretty fast that a large part of the transactions are not even value transactions. These data transactions have no influence on the ledger state and it is pretty obvious, that the ledger state and the tangle are two completely different things.
I remember having a conversation with Paul Handy who is one of the earliest IOTA developers (together with cfb), where we discussed economic clustering and he told me that the value transactions form an “invisible tangle” inside the tangle as some form of “ontology” on the tangle.
What did he mean with that? He talks about the fact, that funds flow from the genesis towards the addresses that have balances today. This flow of funds forms a “hidden” DAG inside the tangle, that is not connected through the branch and trunk but instead connects transactions that are spending related funds (see image).
This hidden DAG forms the actual ledger state which is not the same thing as the tangle itself.
According to Paul, the “ultimate vision” of IOTA would be that branch and trunk are only used to find related events in the tangle, but if a transaction is accepted or not is supposed to be decided independently of the “attachment location” by another “ontology” (way of reading the tangle).
This happens to be a pretty accurate description of the mechanisms, that I am going to propose.
By decoupling the tangle from the ledger state, and using the tangle purely as a “communication and voting mechanism” we can leverage on all the things that the tangle can do extremely well without having to abandon parts of it, just because they contain a conflicting transaction.
Conflicts will be resolved using virtual voting and every node will be able to independently reconstruct the “correct ledger state” by observing the transactions in the tangle.
Conclusion
The remaining parts will introduce the new voting scheme and describe the concrete algorithms (which are surprisingly simple) that are used to turn the tangle into this “virtual voting machine”.
The key take away from this first part should be, that the ledger state and the tangle are two separate things and that the original white-paper version of the tangle already describes a form of virtual voting, which we will slightly modify and extend, to build a mana-based version of it.
