How (Not) To Store Digital Assets: Lessons From 5 Crypto Custody Disasters

The ownership and control of a digital asset wallet only comes from the knowledge of a secret value such as this:

16 bytes, or 128 bits. It’s the amount of secret data sufficient to obtain all the private keys from many wallets (we recommend 256 bits, but 128-bit secrets are the most common).

If someone finds this secret, they can steal all your cryptocurrency. In practice, consumer wallets often use so-called “seed phrases” or “mnemonics”, as defined by the BIP39 standard, which may look like this:

This representation is equivalent to the raw secret above and has the same entropy, or uncertainty—128 bits, in this case, a 12-word phrase.

Now, you might wonder, how difficult could it be for someone to "guess" this value? Well, let's put it into perspective. There are 2¹²⁸ possible 128-bit strings, and trying them one by one until you hit the correct one doesn’t work in practice because of the sheer size of this number. To give you an idea of the magnitude of 2¹²⁸, it’s a billion times bigger than 2⁸⁸, which is approximately the age of the universe in nanoseconds (there’s a billion nanoseconds in a second).

You need much more than this for a safe custody solution (source)

But there are many other ways to steal secret keys than testing all possible values—what cryptographers call exhaustive search, or brute force. The core security goal of a crypto custody solution is to prevent such attacks, even for attackers with a budget and time on their hands. Unfortunately, the history of crypto custody is riddled with spectacular failures, be it from sub-custodians such as exchanges, or from self-custody. Let’s thus look at 5 remarkable crypto custody disasters and review how their root causes often combine attacker creativity, insufficient security controls, and bad luck.

Please note that we have chosen not to discuss DeFi-related security incidents, such as flash loan attacks and various smart contract bugs, as these do not fall under the category of purely custody failures. If you would like more information on these specific incidents, you can find detailed information here or on this site. We have also chosen not to mention the FTX or LUNA tragedies, as improper custody was not the main issue, although it did contribute to the overall problems.

Exchange hack: Coincheck (2018)

The XEM token value crashed after the Coincheck hack (source) 

What happened

In January 2018, the Tokyo-based cryptocurrency exchange Coincheck suffered one of the largest crypto thefts in history. Attackers stole 523 million NEM tokens (XEM) from Coincheck's hot wallet, amounting to approximately $530 million in value at the time. Coincheck compensated its affected customers for their loss.

After the hack, Coincheck faced intense scrutiny from regulators and the broader community. In response, the exchange temporarily suspended withdrawals and deposits for most cryptocurrencies. Over the next few weeks, the company tried to identify the culprits and uncover the intricacies of the attack. Eventually, it became clear that the hackers reached access to Coincheck's hot wallet by leveraging a vulnerability in its access control protocols.

This high-profile breach also led Japan's Financial Services Agency (FSA) to tighten regulations governing cryptocurrency exchanges operating in the country. As part of the recovery process, Coincheck was eventually acquired by Monex Group, a Japanese online brokerage, with the aim of rebuilding and compensating affected users.

This hack is not isolated; numerous exchange platforms have suffered some breach and loss of funds. As we write this, we just learnt that HTX (ex-Huobi) lost $8 million from a hack. You’ll find online various lists of exchange hacks.

Root causes

• Lack of cold storage: Coincheck stored a significant amount of NEM in a hot wallet (that is, a wallet connected to the internet (see our recent post What’s the Most Secure Crypto Custody Model?) rather than a more secure cold wallet, which is isolated from online access.

• Weak approval measures: The exchange used single-signature wallets with no multi-party approval protocol. The compromise of a single system thus allowed the attackers to issue transactions from the hot wallet interface. Furthermore, no strong whitelisting mechanism seems to have been in place.

• Weak detection measures: No measures seem to have been in place to detect and block unauthorized transactions. For example, a best practice is to set a limit on the amount of assets to be transferred in a transaction or over a specific period of time.

Exchange failure: QuadrigaCX (2019) 

QuadrigaCX’s late CEO (source)

What happened

In 2019, QuadrigaCX, a Canadian cryptocurrency exchange, faced a major crisis following the sudden passing of its CEO, Gerald Cotten. Unfortunately, Cotten was the only person with access to the users' funds stored in cold wallets and did not share the necessary access keys. This resulted in approximately $190 million worth of cryptocurrencies becoming inaccessible after his death.

In response, the exchange filed for creditor protection, sparking investigations and legal proceedings. Ultimately, QuadrigaCX was declared bankrupt, with its assets far from covering users' losses. Fast forward to May 2023, the law firm representing QuadrigaCX creditors announced that each creditor with a verified claim would receive a 13 percent reimbursement, determined by a 2019 valuation, of their initial investment.

To learn more about this story, you can watch this documentary film on Netflix.

Root causes

• Single point of control: Gerald Cotten, the CEO, was the sole individual with access to the exchange's cold wallets containing significant cryptocurrency funds. This created a central point of control, and thus of failure: only Cotten could move funds. Unsupervised and without third-party approval.

• Lack of contingency planning: There was no backup, recovery measures, or disaster recovery plan in place if Cotten became incapacitated or was otherwise unable to access the funds.

• Inadequate operational security: Funds were reportedly transferred to personal accounts, and there was an absence of segregation of duties and internal checks and balances.

• Poor financial management: The company did not have proper accounting and financial tracking mechanisms. This was exacerbated by the use of multiple third-party payment processors due to banking issues.

• Delayed communication and transparency: Following Cotten's death, there was a delay in communicating the liquidity issues to customers and regulators, leading to increased mistrust and speculation. As of 2023, a number of points from the QuadrigaCX case remain unelucidated.

Bridge hack: Wormhole (2022)

One of the first reports of the Wormhole hack (source)

What happened

In February 2022, an attacker exploited a vulnerability in the Wormhole token bridge, resulting in the loss of 120,000 Wrapped Ether (wETH) tokens worth over $320 million at the time. wETH is a distinct token separate from Ethereum's native cryptocurrency (ETH) and is used to facilitate the use of Ethereum's native currency on other blockchains. Yet wETH has the same value as ETH, or at least it should.

A bridge is a protocol implemented as a smart contract program that “connects” two blockchains by allowing a same asset to coexist (such as ETH) on the two chains as distinct tokens. When such a bridge would “move” Ether from Ethereum to Solana, it would “lock” the original ETH and create (mint) wETH on Solana. What could go wrong?

Wormhole is a popular bridge that links Ethereum and Solana blockchain. It is both a token and NFT bridge. In the incident, the hacker minted 120,000 wETH on the Solana blockchain without providing the equivalent amount on the Ethereum side due to a subtle flaw in the bridge’s smart contract code. More precisely, the validation of a privileged account was incomplete, as detailed in this analysis from Kudelski Security.

Root causes

• Bridges are inherently risky because of their complex logic involving different blockchains, the fact that they often have the capability to hold and mint tokens, and the uneven attention to security. Wormhole is not the only bridge that was hacked.

• Securing smart contracts is hard, especially non-trivial ones such as bridges' contracts. Subtle flaws can have disastrous, irreversible consequences, and recovering from an exploit isn’t as simple as patching the code of a web application. In the Wormhole case, the bug consisted of a missing validation of privileged accounts due to the use of a deprecated function. This allowed the attacker to impersonate these accounts and their privilege level to mint tokens.

Vanity address bug: Profanity (2022)

GPT-4 answer about the security of the pseudorandom generator used in the Profanity wallet

What happened

In the context of Ethereum and other blockchains, a so-called vanity address is an address that has a specific set of characters in its representation, usually to make it more recognizable, memorable, or personal to its owner. It's essentially a customized address created by brute-forcing addresses until you find one that matches your desired pattern. To enhance security, some degree of randomization is incorporated to prevent predictability in the addresses.

For instance, an Ethereum address consists of 20 bytes usually represented in hexadecimal, such as in 0xc4c39783d80bb0b57b6f55234e4a662723373248. A vanity address generator, such as the famous Profanity, allows you to search for addresses with a specific pattern. As an example, my friend Sofia might like to have an address starting with 0x50F1A, which fixes 20 bits in the address, and thus will take of the order of 2²⁰ ≈ one million attempts.

Now, what went wrong with the Profanity vanity address generator? In short, the underlying pseudorandom generator was very weak, making addresses predictable, and far from the level of 128-bit security expected. This weak generator was the infamous Mersenne Twister, a popular, super-fast random generator that is cryptographically insecure and has been a repeated source of cryptographic failures.

The most significant consequence was the theft of approximately $160 million worth of crypto assets from the DeFi trading organization Wintermute.

Root causes

• Weak pseudorandom generator: History repeating itself, a non-cryptographic random generator was used where cryptographic security was needed. In short, a cryptographic random generator ensures that the output will have a high level of entropy (that is, a high degree of uncertainty), and also guarantees the unpredictability of its stream of bytes. In stark contrast, the Mersenne Twister was non-cryptographic, with only 32 bits of entropy. Instead, you should use cryptographic interfaces such as Go’s crypto/rand package, or OpenSSL's rand API.

• Widespread adoption without proper expert review: Having done many security audits of cryptographic components, including such key generators, one of the first things an expert would look at when auditing the code is the pseudorandom generator and its source of entropy. Evidently, Profanity had not been reviewed, let alone designed, by cryptography experts. Yet it’s been widely adopted because of its convenience and high speed, with little regard given to security.

Wallet access: Nexo (2023)

AI-created with Midjourney

What happened

The crypto trading platform lost a hardware wallet, specifically a Ledger device, containing around $12 million worth of crypto assets. They claim that it’s been stolen by one of their co-founders, or that it’s in the hands of the Bulgarian government. In any case, Nexo appears to have lost access to the funds, suggesting that that hardware wallet was the only device containing the keys and that no back-ups had been created. Otherwise, Nexo could have transferred the funds to another address and wouldn't have bothered filing a legal complaint.

Root causes

• Lack of back-ups: It’s somewhat striking, but not entirely surprising, that in 2023 organizations claiming to take security seriously don’t have proper back-ups in place, whether for their cold of hot storage. Due to such errors, a significant amount of crypto assets are locked forever, although it is difficult to estimate how much.

• Single point of control: Evidently, one person had the capability to single-handedly access a hardware wallet holding millions, presumably knowing the credentials required to unlock it, without effective third-party approval and validation.

Lessons

The challenges highlighted above can be distilled into several core concerns:

1. The management of assets hinges on tiny piece of data, such as a 16-byte master key or a 12-word passphrase. If this data is unprotected or stolen, assets can be irretrievably lost. In some regulated contexts, however, the loss of a key, even on a public blockchain, does not necessarily mean the loss of assets when owners are registered off-chain. Such frameworks, which eliminate the single point of irreparable loss, seem crucial for broader adoption of decentralized platforms.

2. Ensuring the security of rapidly developed software, especially when scaled, poses significant challenges. While security patches are easily deployed to web or mobile applications, the deployment model of smart contracts can make them harder to mend. Any breach in a wallet or contract is often irreversible. Secure software development lifecycle and independent security assessment must become the norm for critical decentralized systems.

3. The absence of structured oversight, appropriate regulatory frameworks, and mature governance within many blockchain ventures amplifies both internal and external vulnerabilities. This environment can unfortunately pave the way for ill-advised decisions by those at the helm. As the industry matures and regulations evolve, the implementation of controls and audit obligations is expected to reduce that risk.

Addressing point 3 appears crucial for the successful resolution of point 2. While point 1 finds a solution within specific frameworks like the Swiss tokenization legal system, it’s often in contradiction with the principles of decentralized environments. Proponents of full decentralization might thus need to reevaluate their stance to ensure the widespread adoption of their systems, and adopt centralization trade-offs to ensure proper governance, checks and balances, and guarantees that adequate security controls are in place—from software development lifecycle and supply-chain security to key management and disaster recovery.

If you'd like to delve deeper into a comprehensive custody framework, we encourage you to check Banking Grade Digital Asset Custody.