By Gustavo Grieco
Cosmos is a platform enabling the creation of blockchains in Go (or other languages). Its reference implementation, Cosmos SDK, leverages strong fuzz testing extensively, following two approaches: smart fuzzing for low-level code, and dumb fuzzing for high-level simulation.
In this blog post, we explain the differences between these approaches and show how we added smart fuzzing on top of the high-level simulation framework. As a bonus, our smart fuzzer integration led us to identify and fix three minor issues in Cosmos SDK.
Laying low
The first approach to Cosmos code fuzzing leverages well-known smart fuzzers such as AFL, go-fuzz, or Go native fuzzing for specific parts of the code. These tools rely on source code instrumentation to extract useful information to guide a fuzzing campaign. This is essential to explore the input space of a program efficiently.
Using fuzzing for low-level testing of Go functions in Cosmos SDK is very straightforward. First, we select a suitable target function, usually stateless code, such as testing the parsing of normalized coins:
func FuzzTypesParseCoin(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
_, _ = types.ParseCoinNormalized(string(data))
})
}
Figure 1: A small fuzz test for testing the parsing of normalized coins
Smart fuzzers can quickly find issues in stateless code like this; however, it is clear that the limitations of being applied only to low-level code will not help uncover more complex and interesting issues in the cosmos-sdk execution.
Moving up!
If we want to catch more interesting bugs, we need to go beyond low-level fuzz testing in Cosmos SDK. Fortunately, there is already a high-level approach for testing: this works from the top down, instead of the bottom up. Specifically, cosmos-sdk provides the Cosmos Blockchain Simulator, a high-level, end-to-end transaction fuzzer, to uncover issues in Cosmos applications.
This tool allows executing random operation transactions, starting either from a random genesis state or a predefined one. To get this tool to work, application developers must implement several important functions that will generate both a random genesis state and transactions. Fortunately for us, this is fully implemented for all the cosmos-sdk features.
For instance, to test the MsgSend operation from the x/nft module, the developers defined the SimulateMsgSend function to generate a random NFT transfer:
// SimulateMsgSend generates a MsgSend with random values.
func SimulateMsgSend(
cdc *codec.ProtoCodec,
ak nft.AccountKeeper,
bk nft.BankKeeper,
k keeper.Keeper,
) simtypes.Operation {
return func(
r *rand.Rand, app *baseapp.BaseApp, ctx sdk.Context, accs []simtypes.Account, chainID string,
) (simtypes.OperationMsg, []simtypes.FutureOperation, error) {
sender, _ := simtypes.RandomAcc(r, accs)
receiver, _ := simtypes.RandomAcc(r, accs)
…
Figure 2: Header of the SimulateMsgSend function from the x/nft module
While the simulator can produce end-to-end execution of transaction sequences, there is an important difference with the use of smart fuzzers such as go-fuzz. When the simulator is invoked, it will use only a single source of randomness for producing values. This source is configured when the simulation starts:
func SimulateFromSeed(
tb testing.TB,
w io.Writer,
app *baseapp.BaseApp,
appStateFn simulation.AppStateFn,
randAccFn simulation.RandomAccountFn,
ops WeightedOperations,
blockedAddrs map[string]bool,
config simulation.Config,
cdc codec.JSONCodec,
) (stopEarly bool, exportedParams Params, err error) {
// in case we have to end early, don't os.Exit so that we can run cleanup code.
testingMode, _, b := getTestingMode(tb)
fmt.Fprintf(w, "Starting SimulateFromSeed with randomness created with seed %d\n", int(config.Seed))
r := rand.New(rand.NewSource(config.Seed))
params := RandomParams(r)
…</pre>
Figure 3: Header of the SimulateFromSeed function
Since the simulation mode will only loop through a number of purely random transactions, it is pure random testing (also called dumb fuzzing).
Why don’t we have both?
It turns out, there is a simple way to combine these approaches, allowing the native Go fuzzing engine to randomly explore the cosmos-sdk genesis, the generation of transactions, and the block creation. The first step is to create a fuzz test that invokes the simulator. We based this code on the unit tests in the same file:
func FuzzFullAppSimulation(f *testing.F) {
f.Fuzz(func(t *testing.T, input [] byte) {
…
config.ChainID = SimAppChainID
appOptions := make(simtestutil.AppOptionsMap, 0)
appOptions[flags.FlagHome] = DefaultNodeHome
appOptions[server.FlagInvCheckPeriod] = simcli.FlagPeriodValue
db := dbm.NewMemDB()
logger := log.NewNopLogger()
app := NewSimApp(logger, db, nil, true, appOptions, interBlockCacheOpt(), baseapp.SetChainID(SimAppChainID))
require.Equal(t, "SimApp", app.Name())
// run randomized simulation
_,_, err := simulation.SimulateFromSeed(
t,
os.Stdout,
app.BaseApp,
simtestutil.AppStateFn(app.AppCodec(), app.SimulationManager(), app.DefaultGenesis()),
simtypes.RandomAccounts,
simtestutil.SimulationOperations(app, app.AppCodec(), config),
BlockedAddresses(),
config,
app.AppCodec(),
)
if err != nil {
panic(err)
}
})</pre>
Figure 4: Template of a Go fuzz test running a full simulation of cosmos-sdk
We still need a way to let the fuzzer control possible inputs. A simple approach would be to let the smart fuzzer directly control the seed of the random value generator:
func FuzzFullAppSimulation(f *testing.F) {
f.Fuzz(func(t *testing.T, input [] byte) {
config.Seed = IntFromBytes(input)
…
Figure 5: A fuzz test that receives a single seed as input
func SimulateFromSeed(
…
config simulation.Config,
…
) (stopEarly bool, exportedParams Params, err error) {
…
r := rand.New(rand.NewSource(config.Seed))
…
Figure 6: Lines modified in SimulateFromSeed to load a seed from the fuzz test
However, there is an important flaw in this: changing the seed directly will give the fuzzer a very limited amount of control over the input, so their smart mutations will be very ineffective. Instead, we need to allow the fuzzer to better control the input from the random number generator but without refactoring every simulated function from every module. 
Against all odds
The Go standard library already ships a variety of general functions and data structs. In that sense, Go has “batteries included.” In particular, it provides a random number generator in the math/rand module:
// A Rand is a source of random numbers.
type Rand struct {
src Source
s64 Source64 // non-nil if src is source64
// readVal contains remainder of 63-bit integer used for bytes
// generation during most recent Read call.
// It is saved so next Read call can start where the previous
// one finished.
readVal int64
// readPos indicates the number of low-order bytes of readVal
// that are still valid.
readPos int8
}
…
// Seed uses the provided seed value to initialize the generator to a deterministic state.
// Seed should not be called concurrently with any other Rand method.
func (r *Rand) Seed(seed int64) {
if lk, ok := r.src.(*lockedSource); ok {
lk.seedPos(seed, &r.readPos)
return
}
r.src.Seed(seed)
r.readPos = 0
}
// Int63 returns a non-negative pseudo-random 63-bit integer as an int64.
func (r *Rand) Int63() int64 { return r.src.Int63() }
// Uint32 returns a pseudo-random 32-bit value as a uint32.
func (r *Rand) Uint32() uint32 { return uint32(r.Int63() >> 31) }
…
Figure 7: Rand data struct and some of its implementation code
However, we can’t easily provide an alternative implementation of this because Rand was declared as a type and not as an interface. But we can still provide our custom implementation of its randomness source (Source/Source64):
// A Source64 is a Source that can also generate
// uniformly-distributed pseudo-random uint64 values in
// the range [0, 1<<64) directly.
// If a Rand r's underlying Source s implements Source64,
// then r.Uint64 returns the result of one call to s.Uint64
// instead of making two calls to s.Int63.
type Source64 interface {
Source
Uint64() uint64
}
Figure 8: Source64 data type
Let’s replace the default Source with a new one that uses the input from the fuzzer (e.g., an array of int64) as a deterministic source of randomness (arraySource):
type arraySource struct {
pos int
arr []int64
src *rand.Rand
}
// Uint64 returns a non-negative pseudo-random 64-bit integer as an uint64.
func (rng *arraySource) Uint64() uint64 {
if (rng.pos >= len(rng.arr)) {
return rng.src.Uint64()
}
val := rng.arr[rng.pos]
rng.pos = rng.pos + 1
if val < 0 {
return uint64(-val)
}
return uint64(val)
}
Figure 9: An implementation of uint64() to get signed integers from our deterministic source of randomness
This new type of source either pops a number from the array or produces a random value from a standard random source if the array was fully consumed. This allows the fuzzer to continue even if all the deterministic values were consumed.
Ready, Set, Go!
Once we have modified the code to properly control the random source, we can leverage Go fuzzing like this:
$ go test -mod=readonly -run=_ -fuzz=FuzzFullAppSimulation -GenesisTime=1688995849 -Enabled=true -NumBlocks=2 -BlockSize=5 -Commit=true -Seed=0 -Period=1 -Verbose=1 -parallel=15
fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
fuzz: elapsed: 1s, gathering baseline coverage: 1/1 completed, now fuzzing with 15 workers
fuzz: elapsed: 3s, execs: 16 (5/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 6s, execs: 22 (2/sec), new interesting: 0 (total: 1)
…
fuzz: elapsed: 54s, execs: 23 (0/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 57s, execs: 23 (0/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 1m0s, execs: 23 (0/sec), new interesting: 0 (total: 1)
fuzz: elapsed: 1m3s, execs: 23 (0/sec), new interesting: 5 (total: 6)
fuzz: elapsed: 1m6s, execs: 30 (2/sec), new interesting: 10 (total: 11)
fuzz: elapsed: 1m9s, execs: 38 (3/sec), new interesting: 11 (total: 12)
Figure 10: A short fuzzing campaign using the new approach
After running this code for a few hours, we collected a number of low-severity bugs in this small trophy case:
- https://github.com/cosmos/cosmos-sdk/pull/16951
- https://github.com/cosmos/cosmos-sdk/pull/18542
- https://github.com/cosmos/cosmos-sdk/pull/16978
We provided the Cosmos SDK team with our patch for improving the simulation tests, and we are in the process of discussing how to better integrate this into the master.
Article Link: Improving the state of Cosmos fuzzing | Trail of Bits Blog