Node Operator Issues Open Letter of Concern as Hyperliquid's Centralized API Sparks Controversy

Original Article Title: A Letter To The Hyperliquid Core Team
Original Article Author: Kam Benbrik, Researcher @Chorus One
Article Translation: Ashley, BlockBeats

Editor's Note: Node operator Chorus One has detailed multiple issues with the Hyperliquid testnet in an open letter to the X platform, including frequent node shutdowns, operational difficulties due to closed-source code, and risks of centralization due to a single point of failure in the API. They have proposed various improvement suggestions aimed at increasing the chain's transparency and decentralization. In response to these concerns, Hyperliquid founder Jeff has replied, emphasizing that the validator selection criteria have been outlined in an announcement; the official Hyperliquid account has also posted a separate article on the X platform, addressing the issues raised in the letter and stating that node code will be open-sourced under secure conditions.

The following is the original content (lightly edited for readability):

The following letter is from Chorus One to the @HyperliquidX engineering team, hoping the team can take the time to review this feedback on Hyperliquid chain management.

TL;DR

· Due to closed-source code, lack of documentation, and reliance on a centralized API, validators face significant challenges, leading to frequent node jailing and unstable performance.

· The testnet incentive mechanism has resulted in black market trading of the HYPE token, favoring transactions with large holders rather than fair validator selection.

· Low returns for validators on the mainnet fail to cover the high self-staking requirements, limiting decentralization as 81% of stake is controlled by foundation nodes.

· To compete with major layer 1s, Hyperliquid must enhance transparency, decentralize stake delegation, implement a fair validator selection mechanism, and engage more with external validators.

I've been involved with Hyperliquid since December 2023 and find the application to be outstanding. It is user-friendly, with excellent user experience, and offers some unique features not found elsewhere, such as Vaults and the renowned HLP. Currently, HLP manages over $3.5 billion in funds and allows anyone to passively participate in Hyperliquid.

Upon seeing the platform's excellent performance and learning that Hyperliquid operates independently as a layer 1, I would like Chorus One to participate as an operator on the Hyperliquid chain. I am an employee from Chorus One, one of the largest node operators in the industry. Since 2018, Chorus One has been active in the Proof of Stake industry. We have collaborated with many outstanding teams, contributed to the development of various blockchain designs and consensus algorithms, and played key roles in some of the earliest Proof of Stake chains such as Tezos and Cosmos Hub. Currently, Chorus One manages over 50 blockchains, with a total staked asset value exceeding $3 billion, and has been collaborating with all major Proof of Stake blockchains since the early stages.

After being whitelisted on October 17, Chorus One joined the Hyperliquid testnet. I would like to share our overall experience on the testnet with the Hyperliquid engineering team, as even after almost 3 months of running on the testnet, we have not had the opportunity to interact with the team. During this time, we witnessed one of the most successful token launches of 2024—the HYPE token. At the same time, we experienced a testnet environment that was both fun and challenging. I would like to mention some key observations in the hope that they will be considered in the coming days, weeks, or months.

Testnet Experience

The experience on the testnet has been quite challenging so far. Node operators have almost no information on how to run a node, with very limited resources available for reference.

Frequent Node Shutdowns with Unclear Reasons

Initially, we were shut down multiple times without understanding the reasons. Due to the closed-source nature of the code, we could not properly assess why the shutdowns were happening. The only way was to communicate with other validators on Discord and speculate together on possible reasons. After talking to several validators, we learned that other validators were also repeatedly shut down, and they were not entirely clear about the reasons either.

Node Location Issue

Later, we found that the shutdown issue might be because our node was not deployed in Tokyo. Moving the node to Tokyo might be helpful. Unfortunately, the team never explicitly communicated this to us, and we only discovered it after facing issues multiple times.

After moving the node to Tokyo, the situation improved. This might be because many high-stake testnet nodes are also deployed in Tokyo, allowing our node to catch up slightly and reduce missed blocks. However, even after the migration, we still face shutdown issues, and the specific reasons remain unclear. This lack of understanding is primarily due to the closed-source nature of the code.

Dependency on Automatic Unjail Scripts

We acknowledge that maintaining a good uptime on the Hyperliquid testnet relies on the speed of scripts automatically unjailing nodes. The only way to improve uptime is by relying on fast automatic unjailing scripts. Validators cannot fully understand or address potential issues and can only unjail their nodes blindly in situations they do not comprehend.

Centralized Hyperliquid API as a Single Point of Failure

There have been occasions where our unjailing attempts have failed due to the Hyperliquid API being down. Since validators must send requests to the Hyperliquid server to unjail, they are left unable to unjail on their own when the API is down.

The team might already be aware of this, but this design needs to be reconsidered as it centralizes the API as a critical point of failure in the network. If the goal is to build a Byzantine Fault Tolerant system, then no node should have special privileges, such as relying on a centralized API.

Validator Selection on Mainnet

Hyperliquid recently went through a decentralization process for its validator set and chose approximately 16 validators. Previously, there were 4 validators managed by the core team that faced significant criticism. Hyperliquid recently took a significant step by expanding the validator set from 4 to 16.

Regarding the validator selection, the 4 validators were announced in the following Discord post:

These validators are Validao, Bharvest, Hypurrstake, and Prrposefulnode. These validators were selected based on maintaining over 90% uptime in the past 7 or 30 days.

This is a significant achievement on many fronts, mainly because validator performance is still influenced by external factors such as Hyperliquid API downtime, unjailing issues, and continuous crashes of the binary files, all of which have a non-negligible impact on performance.

In addition to the 4 validators selected based on the testnet performance, 5 validators from the Hyperliquid Foundation are also running on the mainnet. Furthermore, 7 additional validators have been chosen to participate in the mainnet, but the reasons for their selection have not been publicly disclosed.

Subsequently, a black market for HYPE Testnet tokens emerged.

The Hyperliquid Testnet initially had a set of 50 validators. Initially, specific entities were whitelisted to join the Testnet, but as of December 12, the validator set became fully open.

The conditions were simple: you needed 10,000 HYPE Testnet tokens to register as a validator. However, to become an active validator, you also needed to be in the top 50, or else validators would remain inactive.

This decision led to a surge in the price of HYPE Testnet tokens. The price initially rose to over 3,000 simulated USDC, and a few days later, it even exceeded 28,000 simulated USDC. At the time of writing, the price is approximately 700 simulated USDC per token.

Unfortunately, the faucet only distributes 100 simulated USDC every 4 hours. To be among the top 50 validators on the Testnet currently requires over 528,747 HYPE Testnet tokens. Assuming a price of 700 simulated USDC per token and relying solely on the faucet, the calculation is as follows:

Days = (528,747 × 700) ÷ (100 × 6) = 616,871.5 days

This means that relying solely on the faucet would take approximately 616,871.5 days, or 1,690 years, to accumulate enough HYPE Testnet tokens to become an active validator on Hyperliquid.

However, those who received a HYPE airdrop on the mainnet also received the same amount of tokens on the Testnet. This provided an opportunity for validators to collaborate with these community members by staking Testnet HYPE tokens, enabling validators to ensure entry into the active set.

Simultaneously, this situation also offered another perspective for those holding Testnet HYPE tokens. Given the competitive nature of joining the Testnet validator set, many validators were eager to acquire as many HYPE Testnet tokens as possible. Consequently, a black market emerged, where whales holding a significant amount of Testnet HYPE tokens began selling their Testnet tokens to validators in exchange for mainnet USDC.

I have never seen such a chaotic situation before. Although the Hyperliquid team clearly does not endorse these practices, they are fully capable of addressing this issue. One potential solution is to implement a proper testnet validator selection process.

In most other PoS networks, the core team usually shares a form that any validator can fill out to express their willingness to run the chain. The team would then review these applications based on various criteria, such as the validator's node operation experience, past contributions, community involvement, or other factors.

This group of pre-selected validators can then participate in the testnet, closely collaborate with the engineering team, provide feedback, and ensure everything runs smoothly. We have tried multiple times to provide feedback, but so far have not been successful.

Mainnet and Decentralization

As mentioned earlier, the current validator set of the Hyperliquid mainnet consists of 16 validators, which can be viewed at the following URL: https://app.hyperliquid.xyz/staking

· 5 validators are from the Hyperliquid Foundation.

· 4 validators were selected based on their performance on the testnet, maintaining over 90% uptime in the last 7 days.

· 7 validators were self-selected by the Hyperliquid team.

Out of the staked 404,495,250 HYPE tokens, approximately 329,578,724 HYPE tokens are staked on Foundation nodes, accounting for approximately 81.4% of the total stake. We know very little about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, the core assumption of most BFT systems is that no more than 33% of the voting power behaves maliciously. If a single entity controls 1/3 of the stake, they can halt the chain. If they control 2/3 of the stake, they have full control of the network.

The Hyperliquid Foundation initially staked 60 million HYPE tokens on each Foundation node. However, many HYPE holders also chose to stake on Foundation nodes, which is not ideal for decentralization. The team should engage more with the community to encourage a more decentralized staking distribution.

There are three potential solutions:

· Educate the community on the importance of staking with external validators to enhance chain security and decentralization.

· Implement a 100% commission rate for foundation nodes to incentivize users to stake with external validators and promote decentralization.

· Reallocate foundation staking to external validators, which is a common practice in many chains.

Decentralizing staking to external validators will also help them achieve economic sustainability. Hyperliquid is a blockchain focused on high throughput, where infrastructure costs can be high, especially when nodes are deployed in Tokyo. Currently, validators at the bottom of the validator set earn between $3,000 and $5,000 annually, which is insufficient to cover costs. This is particularly challenging as they must self-stake the initial 10,000 HYPE tokens (worth around $250,000 at the current price) to validate on the mainnet.

Currently, users interact with Hyperliquid by bridging USDC from Arbitrum to the Hyperliquid chain. Upon reviewing the bridge contract, it appears that the bridge is still managed by 4 validators. These validators do not seem to be associated with the chain's consensus or the 16 validators on the mainnet.

Hyperliquid has a great product, but the team still needs to make several infrastructure improvements to truly compete with major layer 1s. Some improvements are straightforward, such as:

Seek input from experienced validators who operate on multiple networks. While the team's current approach of working independently has been very effective in building its sustainable product, validators are a key pillar of a layer 1. It is equally important to seek their input to ensure everything runs smoothly.

Open-source the code. This will help validators better understand the issues they face when running a node on the Hyperliquid L1 and also help users trust the product. Open-sourcing the code will also enable validators to have more insights into the architecture and consensus algorithm. Currently, information on HyperBFT is very limited, and open-sourcing can provide much-needed transparency and understanding. Chorus One has a network handbook on the importance of open source. Operators should be able to build all software they operate from source code: https://handbook.chorus.one/node-software/open-source.html

Create an appropriate validator selection process to prevent black market trading of the HYPE Testnet token. Selecting validators based on normal operation time is a fair approach, but achieving good normal operation time should also be fair. This should not depend on whether one has a relationship to acquire testnet tokens, purchased testnet tokens, or external factors (such as relying on Hyperliquid API for normal operation time).

Overall, Hyperliquid does not need to make too many changes to compete with major layer 1s. The main focus should be on interacting more with external parties and incorporating their feedback. I look forward to seeing the changes in the coming weeks and months, and our team will be ready to provide support and feedback at any time.

Hyperliquid Founder Jeff and Official Account Respond

In response to this letter, Hyperliquid Founder Jeff has responded on Platform X.

He emphasized that running a validator successfully is not difficult; the key lies in the validator's own setup and specialization. In addition, he pointed out that the validator selection criteria have been explained in the announcement and are based on the high normal operation time performance in the early stages of the testnet. This indicates that Jeff is more inclined to believe that the current issues stem more from the validators' own configuration rather than a flaw in system design.

Furthermore, the Hyperliquid official team has also released further clarifications, stating that the node code will be open-sourced in a secure manner.

· All validators qualify based on testnet performance and cannot obtain a seat through purchase; unfounded remarks denigrate the efforts of validators who have invested time and effort to understand the system; as the blockchain matures, the validator set will gradually expand.

· As previously announced, a foundation delegation program will be launched to support high-performing validators and further decentralize the network.

· Anyone can run an API server pointing to any node; sample client code sends requests to specific API servers, but this is not a fundamental requirement of the network.

· It is unacceptable for users to create a black market for the HYPE testnet; this has been stated multiple times; we will continue to work on improving the testnet onboarding process.

· The node code is currently closed source; open sourcing is very important, and the project will be open sourced once development reaches a stable state; Hyperliquid's development speed is several orders of magnitude faster than most projects, and its scope is also several orders of magnitude larger than most projects; the code will be open sourced under safe conditions.

· Currently, there is only one binary file. Even in a mature network like Solana, the vast majority of validators run a single client.

「Original Article Link」