Create a full peer-to-peer filesystem with Hyperdrive

This guide will show you how to create a full peer-to-peer filesystem with Hyperdrive.

Hyperdrive is a wrapper around two Hypercores:

one is a Hyperbee index for storing file metadata, and
the other is used to store file contents.

In this guide, you will create three Pear Terminal Applications:

drive-writer-app that mirrors a local directory into a Hyperdrive,
drive-reader-app that replicates the Hyperdrive with a reader peer, and
drive-bee-reader-app that inspects the Hyperdrive as a Hyperbee

When the writer modifies its drive — adding, removing, or changing files — the reader's local copy updates to match. To do this, you need two additional tools:

MirrorDrive: a tool for syncing changes between a local directory and a Hyperdrive, and
LocalDrive: a tool for syncing changes between a Hyperdrive and a local directory.

These tools handle all interactions between Hyperdrives and the local filesystem.

Create the `drive-writer-app`

Create the `drive-writer-app` directory and add dependencies

Start by creating the drive-writer-app project with these commands:

mkdir drive-writer-app
cd drive-writer-app
npm init -y
npm pkg set type="module"
npm install corestore localdrive hyperswarm hyperdrive debounceify b4a bare-process

This will install the following dependencies:

bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
hyperdrive: A module for working with Hyperdrive.
localdrive: A module for working with Localdrive.
corestore: A module for working with Corestore.
debounceify: A module for debouncing functions.
b4a: A module for working with buffers.

Add the `drive-writer-app` logic

Create the drive-writer-app/index.js file with the following content:

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Hyperdrive from 'hyperdrive'
import Localdrive from 'localdrive'
import Corestore from 'corestore'
import debounce from 'debounceify'
import b4a from 'b4a'

// create a Corestore instance 
const store = new Corestore('./drive-writer-storage')
const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// replication of the corestore instance on connection with other peers
swarm.on('connection', conn => store.replicate(conn))

// A local drive provides a Hyperdrive interface to a local directory
const local = new Localdrive('./writer-dir')

// A Hyperdrive takes a Corestore because it needs to create many cores
// One for a file metadata Hyperbee, and one for a content Hypercore
const drive = new Hyperdrive(store)

// wait till the properties of the hyperdrive instance are initialized
await drive.ready()

// Import changes from the local drive into the Hyperdrive
const mirror = debounce(mirrorDrive)

const discovery = swarm.join(drive.discoveryKey)
await discovery.flushed()

console.log('drive key:', b4a.toString(drive.key, 'hex'))

// start the mirroring process (i.e copying) of content from writer-dir to the drive
// whenever something is entered (other than '/n' or Enter )in the command-line
process.stdin.setEncoding('utf-8')
process.stdin.on('data', (data) => {
  if (!data.match('\n')) return
  mirror()
})

// this function copies the contents from writer-dir directory to the drive
async function mirrorDrive () {
  console.log('started mirroring changes from \'./writer-dir\' into the drive...')
  const mirror = local.mirror(drive)
  await mirror.done()
  console.log('finished mirroring:', mirror.count)
}

Run the drive-writer-app with:

bare drive-writer-app

The drive-writer-app creates a LocalDrive instance for a local directory and then mirrors the LocalDrive into the Hyperdrive instance.

The store used to create the Hyperdrive instance is replicated using Hyperswarm to make the data of Hyperdrive accessible to other peers.

It outputs a key which will be passed to drive-reader-app upon execution.

Create the `drive-reader-app`

Create the `drive-reader-app` directory and add dependencies

Leave the drive-writer-app running and in a new terminal create the drive-reader-app project with the following commands:

mkdir drive-reader-app
cd drive-reader-app
npm init -y
npm pkg set type="module"
npm install corestore localdrive hyperswarm hyperdrive debounceify b4a bare-process

This will install the following dependencies:

bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
hyperdrive: A module for working with Hyperdrive.
localdrive: A module for working with Localdrive.
corestore: A module for working with Corestore.
debounceify: A module for debouncing functions.
b4a: A module for working with buffers.

Add the `drive-reader-app` logic

Create the drive-reader-app/index.js file with the following content:

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Hyperdrive from 'hyperdrive'
import Localdrive from 'localdrive'
import Corestore from 'corestore'
import debounce from 'debounceify'
import b4a from 'b4a'

const key = Bare.argv[2]

if (!key) throw new Error('provide a key')

// create a Corestore instance
const store = new Corestore('./drive-reader-storage')

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

// replication of store on connection with other peers
swarm.on('connection', conn => store.replicate(conn))

// create a local copy of the remote drive
const local = new Localdrive('./reader-dir')

// create a hyperdrive using the public key passed as a command-line argument
const drive = new Hyperdrive(store, b4a.from(key, 'hex'))

// wait till all the properties of the drive are initialized
await drive.ready()

const mirror = debounce(mirrorDrive)

// call the mirror function whenever content gets appended 
// to the Hypercore instance of the hyperdrive
drive.core.on('append', mirror)

// join a topic
swarm.join(drive.discoveryKey, { client: true, server: false })

// start the mirroring process (i.e copying the contents from remote drive to local dir)
mirror()

async function mirrorDrive () {
  console.log('started mirroring remote drive into \'./reader-dir\'...')
  const mirror = drive.mirror(local)
  await mirror.done()
  console.log('finished mirroring:', mirror.count)
}

The drive-reader-app creates a LocalDrive instance for a local directory and then mirrors the contents of the local Hyperdrive instance into the LocalDrive instance, which will write the contents to the local directory. The drive-reader-app will output a message when the mirroring process is complete.

Run the `drive-reader-app`

Run the drive-reader-app with bare, passing the key printed by the drive-writer-app:

bare drive-reader-app <SUPPLY_KEY_HERE>

LocalDrive does not create the directory passed to it until something has been written, so create writer-dir (mkdir writer-dir) beside the app directories. Then add, remove, or modify files inside writer-dir and press Enter in the writer's terminal to import those local changes into the drive. Observe that the new changes mirror into reader-dir.

Just as a Hyperbee is just a Hypercore, a Hyperdrive is just a Hyperbee - which is just a Hypercore.

Test the filesystem

A a third terminal, add, remove, or modify files inside writer-dir and observe that the new changes mirror into reader-dir.

For example, create one file in the writer directory:

mkdir -p writer-dir
printf 'hello from hyperdrive\n' > writer-dir/hello.txt

Inspect the Hyperdrive as a Hyperbee

Hyperdrive exposes its metadata index as drive.db, which is the underlying Hyperbee backing the file structure. You can inspect that Hyperbee directly when you want to see the raw file-entry metadata that drive.entry() wraps, or when you want to access the metadata index directly.

Create the `drive-bee-reader-app` directory and add dependencies

In a new terminal, create the drive-bee-reader-app project with these commands:

mkdir drive-bee-reader-app
cd drive-bee-reader-app
npm init -y
npm pkg set type="module"
npm install corestore hyperswarm hyperdrive b4a bare-process

This will install the following dependencies:

bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
corestore: A module for working with Corestore.
hyperdrive: A module for working with Hyperdrive.
b4a: A module for working with buffers.

Add the `drive-bee-reader-app` logic

Create the drive-bee-reader-app/index.js file with the following content:

import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Hyperdrive from 'hyperdrive'
import Corestore from 'corestore'
import b4a from 'b4a'

const key = Bare.argv[2]

if (!key) throw new Error('provide a key')

const store = new Corestore('./drive-bee-reader-storage')

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

swarm.on('connection', conn => store.replicate(conn))

const drive = new Hyperdrive(store, b4a.from(key, 'hex'))
await drive.ready()
await drive.db.ready()

// Hyperdrive stores file metadata in the "files" sub-bee.
const files = drive.db.sub('files', { keyEncoding: 'utf-8' })

const discovery = swarm.join(drive.discoveryKey, { client: true, server: false })
await discovery.flushed()

let dbEntry = null
for (let attempt = 0; attempt < 60; attempt++) {
  await drive.update()
  dbEntry = await files.peek()
  if (dbEntry) break
  await new Promise(resolve => setTimeout(resolve, 500))
}

if (!dbEntry) throw new Error('expected at least one file entry to appear in drive.db')

const driveEntry = await drive.entry(dbEntry.key)

console.log('hyperbee entry:', JSON.stringify({
  key: dbEntry.key,
  value: dbEntry.value
}))

console.log('drive entry:', JSON.stringify({
  key: driveEntry.key,
  value: driveEntry.value
}))

await swarm.destroy()
await drive.close()

This app joins the same drive as the other readers, then inspects drive.db directly as a Hyperbee. It fetches the first available file entry from the metadata index and logs the raw Hyperbee entry alongside the higher-level drive.entry() view of the same file.

Run the `drive-bee-reader-app`

Run the drive-bee-reader-app with bare, passing the key printed by the drive-writer-app:

bare drive-bee-reader-app <SUPPLY_KEY_HERE>

Create a full peer-to-peer filesystem with Hyperdrive

Create the `drive-writer-app`

Create the `drive-writer-app` directory and add dependencies

Add the `drive-writer-app` logic

Create the `drive-reader-app`

Create the `drive-reader-app` directory and add dependencies

Add the `drive-reader-app` logic

Run the `drive-reader-app`

Test the filesystem

Inspect the Hyperdrive as a Hyperbee

Create the `drive-bee-reader-app` directory and add dependencies

Add the `drive-bee-reader-app` logic

Run the `drive-bee-reader-app`

See also

On this page