Share append-only databases with Hyperbee

Hyperbee is an append-only B-tree based on Hypercore. It provides a key/value-store API with methods to insert and get key/value pairs, perform atomic batch insertions, and create sorted iterators. This guide uses Corestore and Hyperswarm to manage and replicate the underlying core; see Work with many Hypercores using Corestore if those concepts are unfamiliar.

This guide consists of three applications:

bee-writer-app - stores 100k entries from a given dictionary file into a Hyperbee instance.
bee-reader-app - queries the Hyperbee instance for key/value pairs.
core-reader-app - inspects the Hyperbee as a Hypercore.

Create the bee writer app

The bee-writer-app stores 100k entries from a given dictionary file into a Hyperbee instance. The Corestore instance used to create the Hyperbee instance is replicated using Hyperswarm. This enables other peers to replicate their Corestore instance and sparsely (on-demand) download the dictionary data into their local Hyperbee instances.

Create the `bee-writer-app` directory and add dependencies

Start the bee-writer-app project with the following commands:

mkdir bee-writer-app
cd bee-writer-app
npm init -y
npm pkg set type="module"
npm install corestore hyperswarm hyperbee b4a bare-fs bare-process

This will install the following dependencies:

bare-fs: A module for working with file systems.
bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
corestore: A module for working with Corestore.
hyperbee: A module for working with Hyperbee.
b4a: A module for working with buffers.

Add the `bee-writer-app` logic

Create the bee-writer-app/index.js file with the following content:

import fsp from 'bare-fs/promises'
import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import Hyperbee from 'hyperbee'
import b4a from 'b4a'
// create a corestore instance with the given location
const store = new Corestore('./bee-writer-storage')

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// replication of corestore instance
swarm.on('connection', conn => store.replicate(conn))

// creation of Hypercore instance (if not already created)
const core = store.get({ name: 'my-bee-core' })

// creation of Hyperbee instance using the core instance 
const bee = new Hyperbee(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'utf-8'
})

// wait till all the properties of the hypercore are initialized
await core.ready()

// join a topic
const discovery = swarm.join(core.discoveryKey)

// Only display the key once the Hyperbee has been announced to the DHT
discovery.flushed().then(() => {
  console.log('bee key:', b4a.toString(core.key, 'hex'))
})

// Only import the dictionary the first time this script is executed
// The first block will always be the Hyperbee header block
if (core.length <= 1) {
  console.log('importing dictionary...')
  const dict = JSON.parse(
    await fsp.readFile(new URL('./dict.json', import.meta.url))
  )
  const batch = bee.batch()
  for (const { key, value } of dict) {
    await batch.put(key, value)
  }
  await batch.flush()
} else {
  // Otherwise just seed the previously-imported dictionary
  console.log('seeding dictionary...')
}

Save the `dict.json` file

Save the dict.json file to the bee-writer-app directory. The dict.json file contains 100,000 dictionary words. This file holds the data that will be stored in the Hyperbee instance, and will be imported into the Hyperbee instance if it is the first time the script is run.

Click here to save dict.json.

Run the `bee-writer-app`

In one terminal, run bee-writer-app with bare.

bare bee-writer-app

Create the bee reader app

The bee-reader-app creates a Corestore instance and replicates it using the Hyperswarm instance to the same topic as bee-writer-app. On every word entered in the command line, it will download the respective data to the local Hyperbee instance.

Create the `bee-reader-app` directory and add dependencies

Create the bee-reader-app project with the following commands:

mkdir bee-reader-app
cd bee-reader-app
npm init -y
npm pkg set type="module"
npm install corestore hyperswarm hyperbee b4a bare-pipe bare-process

This will install the following dependencies:

bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
corestore: A module for working with Corestore.
hyperbee: A module for working with Hyperbee.
b4a: A module for working with buffers.

Add the `bee-reader-app` logic

Create the bee-reader-app/index.js file with the following content:

Alter the generated bee-reader-app/index.js file to the following

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import Hyperbee from 'hyperbee'
import Pipe from 'bare-pipe'
import b4a from 'b4a'

const key = Bare.argv[2]

if (!key) throw new Error('provide a key')

// creation of a corestore instance 
const store = new Corestore('./bee-reader-storage')

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// replication of the corestore instance on connection with other peers
swarm.on('connection', (conn) => store.replicate(conn))

// create or get the hypercore using the public key supplied as command-line argument
const core = store.get({ key: b4a.from(key, 'hex') })

// create a hyperbee instance using the hypercore instance
const bee = new Hyperbee(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'utf-8'
})

// wait till the hypercore properties to be initialized
await core.ready()

// logging the public key of the hypercore instance
console.log('core key here is:', core.key.toString('hex'))

// Attempt to connect to peers
swarm.join(core.discoveryKey)

const stdin = new Pipe(0)

stdin.on('data', (data) => {
  const word = data.toString().trim()
  if (!word.length) return
  bee.get(word).then(node => {
    if (!node || !node.value) console.log(`No dictionary entry for ${word}`)
    else console.log(`${word} -> ${node.value}`)
    setImmediate(console.log) // flush hack
  }, console.error)
})

Run the `bee-reader-app`

In another terminal, run the bee-reader-app with bare and pass it the core key from the bee-writer-app.

bare bee-reader-app <SUPPLY THE KEY HERE>

Query the database by entering a key from the dict file to lookup into the bee-reader-app terminal and hitting return.

Each application has dedicated storage at Pear.config.storage. Try logging out Pear.config.storage for the bee-reader-app and then look at the disk space for that storage path after each query. Notice that it's significantly smaller than bee-writer-app. This is because Hyperbee only downloads the Hypercore blocks it needs to satisfy each query, a feature called sparse downloading.

Hyperbee is a Hypercore with the tree nodes stored as Hypercore blocks. This means that the Hyperbee instance is a Hypercore instance, and the tree nodes are stored as Hypercore blocks.

Inspect the Hyperbee as a Hypercore

Create the `core-reader-app` directory and add dependencies

Finally create the core-reader-app project with the following commands:

mkdir core-reader-app
cd core-reader-app
npm init -y
npm pkg set type="module"
npm install corestore hyperswarm hyperbee b4a bare-process

This will install the following dependencies:

bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
corestore: A module for working with Corestore.
hyperbee: A module for working with Hyperbee.
b4a: A module for working with buffers.

Add the `core-reader-app` logic

Create the core-reader-app/index.js file with the following content:

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Corestore from 'corestore'
import b4a from 'b4a'
import { Node } from 'hyperbee/lib/messages.js'

const key = Bare.argv[2]

if (!key) throw new Error('provide a key')

// creation of a corestore instance 
const store = new Corestore('./reader-storage')

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// replication of the corestore instance on connection with other peers
swarm.on('connection', conn => store.replicate(conn))

// create or get the hypercore using the public key supplied as command-line argument
const core = store.get({ key: b4a.from(key, 'hex') })
// wait till the properties of the hypercore instance are initialized
await core.ready()

// join a topic
swarm.join(core.discoveryKey)
await swarm.flush()

// update the meta-data information of the hypercore instance
await core.update()

const seq = core.length - 1
const lastBlock = await core.get(core.length - 1)

// print the information about the last block or the latest block of the hypercore instance
console.log(`Raw Block ${seq}:`, lastBlock)
console.log(`Decoded Block ${seq}`, Node.decode(lastBlock))

Run the `core-reader-app`

In another terminal, run the core-reader-app with bare, passing the core key from the bee-writer-app:

bare core-reader-app <SUPPLY KEY HERE>

You can now examine the Hyperbee as if it were just a Hypercore.

The core-reader-app will continually download and log the last block of the Hypercore containing the Hyperbee data. Note that these blocks are encoded using Hyperbee's Node encoding, which has been imported directly from Hyperbee for the purposes of explanation.

Share append-only databases with Hyperbee

Create the bee writer app

Create the `bee-writer-app` directory and add dependencies

Add the `bee-writer-app` logic

Save the `dict.json` file

Run the `bee-writer-app`

Create the bee reader app

Create the `bee-reader-app` directory and add dependencies

Add the `bee-reader-app` logic

Run the `bee-reader-app`

Inspect the Hyperbee as a Hypercore

Create the `core-reader-app` directory and add dependencies

Add the `core-reader-app` logic

Run the `core-reader-app`

See also

On this page