How to speed up the synchronization of Bitcoin, Ethereum and other wallets?

It is better to store cryptocurrencies and manage your coins through client programs. There are many of them created now, they are practically no different. They download and install as standard, but almost all beginners are interested in how long to wait for full synchronization. Sometimes it drags on for several days.

How to speed up the synchronization of Bitcoin, Ethereum and other wallets? The speed depends on how fast your internet is.

Unfortunately, the servers do not always operate at full capacity, so you have to wait a long time for loading. The overall wallet database takes up 20 GB of disk space.

Basic wallet functions

Let's start with the basic functionality of a digital wallet:

• backup of private keys;
• accepting payments;
• balance display;
• displaying transaction history;
• sending payments.

To back up private keys, you only need a module responsible for storing and managing keys. However, in the case of the other four functions, you will definitely need to work with the Bitcoin network. After all, in order to read and write a common transaction database, you must at least become one of the nodes of this network and follow the specifications for processing and exchanging data in it. Thus, it is necessary to have data on all transactions in which the addresses of a certain wallet were involved. Data on the current state of the network will also be needed to create a new transaction and make a payment. The most important thing that a digital wallet user must know reliably when accepting and sending payments is the status of their transactions. Obviously, a transaction is considered incomplete if the transaction has been sent but has not received full confirmation. As a result, the recipient cannot accept payment in order, for example, to give away a product or provide a service.

Bcoin master

Synchronization of the full Bcoin master node (commit 77d8804) on block 601,300 was completed in 18 hours 29 minutes.

If you read last year's report, you may remember that I had to tinker with Bcoin - I tried syncing about ten times before I could find the right settings that didn't cause the NodeJS process to break and hang. Since then, the Bcoin team has made a number of synchronization improvements.

My configuration:

• checkpoints:false
• cache-size:10000
• sigcache-size:1000000
• max-files:5000

Difficulties of working in a distributed network

Working with a distributed network is a rather complex topic in which the following problems are usually encountered.
One of them is the way to verify the received data from the distributed network, and the second is to optimize the data storage of this data. These requirements conflict slightly with each other, since a larger volume of stored data allows for more complete verification of new transactions and blocks. There is no ideal solution to these problems; it is impossible to process transactions reliably and with minimal resources, so we have to look for some balance, a compromise between simplicity and security. Another problem is this. To get the history of transactions for a specific address on a full network node, you will have to go through the entire blockchain (from genesis to last known), which is quite long, since this is a large amount of data, and brute force is an ineffective approach. Therefore, you can use certain approaches to optimize queries. To do this, special add-ons are used on top of a regular network node, which help index the blockchain and cache some data to speed up searches using popular metrics. Any blockchain explorer works on this principle, for example blockchain.info, which quickly returns data about almost any address, any transaction, etc.

So, there are several fundamentally different approaches to interacting and synchronizing a digital wallet with a distributed network. Let's look at them.

Gocoin 1.9.7

It took 19 hours and 56 minutes to fully sync Gocoin 1.9.7 to block 601300 with the secp256k1 library enabled. The CPU load remained stable (at 50%) and no other resources were maxed out.

After reading the documentation, I made the following changes to get the best performance:

Installed the secp256k1 library and compiled gocoin using sipasec.go.

Disabled wallet functionality via AllBalances.AutoLoad:false

My configuration:

• LastTrustedBlock: 00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048;
• AllBalances.AutoLoad:false;
• UTXOSave.SecondsToTake:0.

Gocoin is still quite fast compared to the competition, but I don't think it synced as quickly as it could have. On early blocks my CPU usage was around 20% and never went above 50%. I suspect this may be due to the fact that I'm not using hyperthreading - my machine has 6 physical but 12 virtual processors. I tried increasing the number of peers from 10 to 20 and improved the transition from 1.11 to 1.13, but this did not lead to a significant improvement in efficiency. On the other hand, Gocoin has the best admin panel of all the implementations.

Approaches for synchronizing a wallet with a payment network

There are three main approaches. The first assumes that the wallet itself is a full node of the Bitcoin network. The second approach uses a so-called trusted node. In fact, this is a foreign network node, but the wallet user trusts it. And the third approach involves direct interaction with other network nodes, but in a lightweight version, using the Simplified Payment Verification (SPV) technique. This technique allows you to check the confirmation of transactions with a fairly high degree of reliability, but without resorting to launching a full network node. Let's take a closer look at each of these synchronization approaches.

Working with a full network node

In the first case, everything is quite simple.
Your wallet has a module that stores the entire blockchain, and a module that communicates with other nodes in the Bitcoin network. Therefore, you can always check a new incoming transaction: whether it really complies with the rules of the protocol, whether it is a double spend, etc. You have complete control over data verification. The disadvantage of this approach is the need to allocate a large amount of disk space to store the complete blockchain, maintain the network node in uninterrupted operation and timely software updates.

This method is used by ordinary users who regularly accept and send payments in large amounts, which requires a high level of reliability. In addition, all major services that provide the possibility of centralized storage of bitcoins and online access to a wallet necessarily use this approach. Because there are high requirements for security, as well as for independent work with Bitcoin.

It is important to understand that the user of a centralized bitcoin storage is completely dependent on this service and the correct operation of the network node that is used by this service. In fact, if coins are stored on such a service, then they belong not to the user but to the service.

Note that using this approach is one of the simplest and most reliable, even for the average user. To do this, you just need to take the source code or ready-made binaries, for example, Bitcoin Core and run the software. The only significant drawback is the computer requirements.

Working with a trusted network host

The difference between this approach and the previous one is the separation of the wallet logic and transaction verification logic.
There is a so-called trusted node of the Bitcoin network. Roughly speaking, this is the most common full node of the Bitcoin network, which the user trusts to verify his transactions. For example, it could be a network node that is maintained by a friend, and the user can trust it to verify their transactions. Quite often this approach is used in mobile digital wallets. Let's say a company that develops a mobile wallet supports trusted nodes and promises that it will ensure correct verification of transactions. In this case, users trust the digital wallet developer company and actually believe the promises. But at the same time, it is worth understanding that the user can independently store his personal keys and certify transactions on his device. However, the user does not check the transaction confirmation status himself, but asks the trusted node.

What are the advantages and disadvantages of this approach? As in the previous case, although indirectly, it is possible to influence the verification of transactions in the Bitcoin network, because the user remains a trusted network node. The advantage is that he is freed from the need to store a complete copy of the blockchain on his device. However, depending on a trusted node is a disadvantage because if it fails, the user cannot always switch to another trusted node. Since transaction verification takes place on a remote server, you need to make sure that the data transmission channel between the wallet and the server on which the verification takes place is reliable. With this approach, threats of attacks such as “Man in the middle” appear.

When a trusted host is used, private keys are stored separately from wallet addresses. In this case, the node itself cannot reliably know which addresses it needs to monitor for changes. This means that the digital wallet must periodically provide the trusted node with a list of addresses that it is interested in.

The inconvenience of this approach lies in the dependence of the digital wallet on a trusted node, in particular in terms of access to current payment system data. Once a user loses access to a trusted node or the node denies service, the user loses connectivity to the network. Accordingly, the user loses the ability to receive and send payments. To reduce the risk of losing access to a trusted site, you can select multiple trusted sites. This means that under normal circumstances the user connects and works with one trusted node, but if access is lost, the digital wallet automatically connects to another trusted node. Essentially, with each connection, the wallet randomly selects one node from the list of trusted ones.

This method is mainly used by mobile wallets, because it allows both verification of transactions and simplifies the operation of the mobile application. Examples of such mobile wallets for Bitcoin are Mycelium and Coinomi. Distributed Lab has also developed and maintains a wallet called Bitxfy that uses this approach.

Working with an SPV network node

The third common approach to syncing a digital wallet with the Bitcoin network is a lightweight network node.
We are talking about Simplified Payment Verification (SPV). In this approach, there may be no need to run your own full nodes or select trusted ones. It is assumed that the digital wallet communicates directly with other network nodes: in fact, it selects several dozen other full nodes and maintains a connection with them. The difference is that a lightweight node communicates with other nodes on equal terms, exchanging p2p messages with them, but does not store the entire transaction history. Here, the SPV technique is used to verify incoming transactions. In fact, a lightweight node performs only a fraction of the most important checks that a full node in the network performs. How does work work in this case? Instead of receiving blocks like a full node does, the SPV node only receives the block headers, which are much smaller in size (80 bytes). The header contains the necessary data to verify that a specific transaction has been confirmed in this block. More specifically, the block header has a Merkle Root field, which contains the hash value of all transactions in the block, calculated using the Merkle Tree scheme.

The Merkle Tree scheme allows you to check whether the hash value of a particular transaction is included in the Merkle Root without having to look at all the other transactions in the block. So, by having a transaction, a block header, and a Merkle Branch for that transaction (can be queried from a full node), a digital wallet can verify that a transaction was actually confirmed in a specific block.

Why can’t this be considered a full-fledged transaction verification? Of course, the user gains independence from specific network nodes and can communicate directly with the distributed network. To do this, he does not need to store the entire blockchain. But the downside is that the lightweight client doesn't fully validate the transaction itself because you don't have all the necessary data to do so (such as unspent coins). It only checks that the transaction has been confirmed by validators—that is, it makes sure that this transaction has been fully verified by other nodes and the owners of the majority of the network’s computing power (miners). In this case, reliability again comes down to trust that the majority of the computing power is being used for honest work. In other words, you do not check the transaction yourself, but trust the majority to do so.

A feature of the use of digital wallets that implement the SPV node approach is the need to maintain a network connection with a large number of independent full network nodes. This is necessary to minimize the likelihood that a node will connect and receive data only from attacker nodes. The attacker can then impose an alternative (fake) transaction history state on the user. As a result, the victim can accept payment from a transaction that, according to the attacker, is confirmed, although in reality it is not. To increase the chances of receiving up-to-date data about the real state of the network, the user should maximize the number of independent nodes for communication. The more there are, the less likely it is that they are all conspiring against the user.

As you already understand, SPV nodes are demanding in terms of stable operation of the network connection, however, they are often used even in mobile applications. This approach of synchronizing with the payment network is quite widespread and is used in a number of popular Bitcoin wallets. Among them are Bitcoin Wallet, Electrum, and Bread Wallet.

Download blocks in one archive on Cryptochainer

Data sometimes takes a long time to transfer, not because of poor internet on the user’s part, but because of low speed on the servers.

Therefore, it makes sense to download all the blocks in one archive, then transfer them to the desired wallet directory. An additional update may be required, but it is much faster.

The Cryptochainer project was created especially for this purpose, where blockchains for many cryptocurrencies are offered. All of them are uploaded as archives that can be downloaded for free.

You can really save time, but keep in mind that blocks take up a lot of space, so you still have to wait for them to load:

Everything is done in a simple format; you will need to select the blockchain you need from a huge list. We will not publish a complete list of cryptocurrencies, because there are too many of them and new ones are constantly being added. Here you can download blocks of such popular altcoins as:

• DASH;
• Dogecoin;
• DigitalCoin;
• GoldPieces;
• Hyper;
• Litecoin;
• Peercoin.

This is just a small list of available blocks for download. Downloading archives is much faster than standard wallet synchronization.

Please note that the data is constantly increasing, so we cannot say how much a particular archive for a particular cryptocurrency weighs.

Bitcoin Core - what it is and how it works: creating and using a wallet

During the installation process, the client computer is included in the general network component, since one of the copies is located there.

Approaches to synchronizing a wallet with a payment network (read more...)

1. You need to click on the button to create an online wallet.
2. In the form that comes off, you need to indicate where the wallet will be stored.
3. You need to give it a custom name.
4. You need to enter a security password by following the steps – File – Add Password.
5. Be sure to come up with a key and keep it in a safe place.

Problem solving

Newcomers most often complain about several problems with Bitcoin Core.

The wallet takes a long time to synchronize

How long is it? Normally, the process takes 3 days. If the process stops during the download, you can try restarting the computer, or, as a last resort, delete all information and start again. If the processor is powerful and multi-core, there is enough free space on the disk, and the RAM is not overloaded, then there will be no problems with downloading.

An unknown block version was received

This situation was relevant a year ago. Then many wallet owners received a notification that an unknown version of the block had been received. In the screenshot, the notification is in a yellow frame. The developers said not to worry about this, if you are interested in learning more details, you can read the forum bitcointalk.org/index.php?topic=2027513.0.

results

• Bitcoin Core 0.19: 6 hours, 39 minutes;
• Bcoin: 18 hours, 29 minutes;
• Gocoin 1.9.7: 19 hours, 56 minutes;
• Libbitcoin Node 3.2.0: 1 day, 3 hours, 37 minutes;
• Parity Bitcoin: 2 days, 2 hours, 10 minutes;
• BTCD v0.20.0-beta: 3 days, 3 hours, 12 minutes;
• Stratis 3.0.5.0: Synchronization was not completed; definitely more than a week.

Libbitcoin Node 3.6.0

It took 27 hours and 37 minutes to fully sync Libbitcoin Node 3.6.0 on block 601,300 with 10,000 UTXO cache.

Libbitcoin Node wasn't all that bad - the hardest part was removing the hard-coded block checkpoints. To do this, I had to clone the git repository, check out the "version3" branch, remove all breakpoints (except genesis) from bn.cfg and run the install.sh script to compile the node and its dependencies.

My configuration:

• [database]
• cache_capacity = 100000
• [node]
• block_latency_seconds = 5

I noticed that during the sync, Libbitcoin wrote 3TB and read over 30TB to and from disk. The total number of writes seems reasonable, but the total number of reads is much higher than other implementations. Caching can be improved; by the end of the sync, only 5 GB of RAM was used. There were also a number of short pauses when the node seemed not to be working; presumably the queue of blocks for processing was idle from time to time.

History of Ravencoin

After launching in 2022, the Ravencoin team was funded by Medici Ventures, a blockchain-focused subsidiary of Overstock.com.

Ravencoin was specifically designed to create a mining ecosystem in which the specialized mining hardware used for other cryptocurrencies, called application-specific integrated circuits (ASICs), did not have an advantage.

In January 2022, Tron Black, lead developer of Ravencoin, wrote that ASIC miners had found a way to beat the original RVN algorithms, giving them an advantage over computers with normal processing power.

To fix this, the Ravencoin team has further changed their algorithms so that even if miners decide to use ASICs, their hardware will be forced to behave like regular computer hardware. This effectively leveled the playing field, at least for a time.

What is Bootstrap.dat?

Bootstrap.dat is a file that contains a copy of the blockchain from the genesis block up to a certain point in time. This compressed Bootstrap.dat file is used to speed up the initial loading time of the blockchain.

How does this happen? Your wallet client downloads and verifies each block from the P2P network. This is usually slow, especially if you are using the wallet for the first time, then the synchronization process can take quite a long time.

Instead of using peer-to-peer communication, your wallet client can read blockchain data from this compressed bootstrap file, which contains a copy of the blockchain data up to a certain block height.

Once the wallet client has finished reading the data from the bootstrap file, it will use the P2P connection to download the remaining blocks.

This method is faster and also consumes less bandwidth than the standard synchronization process.

However, the bootstrap method takes some time as your wallet client must validate each individual block.

How to speed up Bitcoin or Ethereum wallet synchronization?

One of the attentive users decided to analyze the synchronization procedure and made their own assumptions about the speed.

After checking the PC resources, he noticed that not much RAM was being used and there was no load on the processor. Internet speed allows you to load faster, but something is slowing down the operation:

It turns out that a regular hard drive is simply not capable of recording data as quickly as it moves on the Internet. Conclusion - we need a more modern and high-quality medium.

The ideal solution is an SSD drive. You can purchase it at any computer store or order it on Aliexpress.