"Head State Missing" Error on Restarting "geth"

This section describes the issue with 'Head State Missing' error on restarting 'geth' on a private network.

After growing the blockchain in my private Ethereum block, I closed the "geth" node and started it again. The "geth" node is running with no problem, but the blockchain was truncated back to the genesis block.

Here is what happened:

1. Check the block count and to close the "geth" node:

> eth.blockNumber
7

> exit
C:\>

2. Restart the "geth" node and check the block count:

C:\>"\Program Files\geth\geth" 
   --datadir \herong\Ethereum\personal
   console 2>> geth-personal.log

> eth.blockNumber
0

All blocks are gone, except the genesis block. Why?

3. Check the log file:

INFO  Commit new mining work                number=7 txs=0 uncles=0 
INFO  Successfully sealed new block         number=7 hash=ab8263…a0533c
INFO  block reached canonical chain         number=2 hash=56ced2…2c1765
INFO  mined potential block                 number=7 hash=ab8263…a0533c
INFO  IPC endpoint closed                   endpoint=\\\\.\\pipe\\geth.ipc
ERROR Dangling trie nodes after full cleanup 
INFO  Blockchain manager stopped 
INFO  Stopping Ethereum protocol 
INFO  Ethereum protocol stopped 
INFO  Transaction pool stopped 
INFO  Database closed                          
   database=C:\\herong\\Ethereum\\personal\\geth\\chaindata
   ("geth" is closed now)

   ("geth" is started again)
INFO  Maximum peer count                     ETH=25 LES=0 total=25
INFO  Starting peer-to-peer node             instance=Geth/v1.8.1-...
INFO  Allocated cache and file handles       
   database=C:\\herong\\Ethereum\\personal\\geth\\chaindata 
INFO  Initialised chain configuration        config="{ChainID: 314 ...
INFO  Disk storage enabled for ethash caches dir=C:\\herong\\Ethereum...
INFO  Disk storage enabled for ethash DAGs   dir=C:\\Users\\herong\\...
INFO  Initialising Ethereum protocol         versions="[63 62]" network=1
WARN  Head state missing, repairing chain    number=7 hash=ab8263…a0533c
INFO  Rewound blockchain to past state       number=0 hash=1019b1…220d59
INFO  Loaded most recent local header        number=7 hash=ab8263…a0533c
INFO  Loaded most recent local full block    number=0 hash=1019b1…220d59
INFO  Loaded most recent local fast block    number=7 hash=ab8263…a0533c
INFO  Loaded local transaction journal       transactions=0 dropped=0
INFO  Regenerated local transaction journal  transactions=0 accounts=0
INFO  Starting P2P networking 
...
INFO  Commit new mining work                 number=1 txs=0 uncles=0

Based on the log entries, I lost all blocks, because "Head state missing" on restarting the "geth" node. But that might be related to the error during the "geth" shutdown process:

ERROR Dangling trie nodes after full cleanup 

After searching Internet on this issue, I found a comment that says: "It seems a bug in geth. It works perfectly in geth 1.8.2..."

Table of Contents

 About This Book

 Introduction of Ethereum

 Ethereum Blockchain

 Ethereum Mist Wallet

 geth - Go Ethereum

 Testnet - Ropsten network

 Private Ethereum Network

 64-Bit "geth" for Private Ethereum Network

Private Network with Custom Genesis Block

 Transferring Funds between Ether Accounts

 MetaMask - Browser Based Ethereum Wallet

 Managing Ethereum Account

 Mining Ether with Your Computer

 Archived Tutorials

 References

 Full Version in PDF/EPUB