Replicate and persist with Hypercore

In this guide you'll extend the ephemeral chat example in Connect Many Peers by using Hypercore to add two significant new features:

Persistence: The owner of the Hypercore can add messages at any time, and they'll be persisted to disk. Whenever they come online, readers can replicate these messages over Hyperswarm.
Many Readers: New messages added to the Hypercore will be broadcast to interested readers. The owner gives each reader a reading capability (core.key) and a corresponding discovery key (core.discoveryKey). The former is used to authorize the reader, ensuring that they have permission to read messages, and the latter is used to discover the owner (and other readers) on the swarm.

Hypercore is a secure, distributed append-only log. It is built for sharing enormous datasets and streams of real-time data. It has a secure transport protocol, making it easy to build fast and scalable peer-to-peer applications.

The following example consists of two Pear Terminal Applications: writer-app and reader-app. When these two applications are opened, two peers are created and connected to each other. Hypercore stores the data entered into the command line.

Create the writer app

The writer-app stores command-line input to a Hypercore instance and replicates that instance to other peers over Hyperswarm.

Create the writer-app directory and add dependencies

Create the writer-app project with these commands:

mkdir writer-app
cd writer-app
npm init -y
npm pkg set type="module"
npm install bare-path bare-process hypercore hyperswarm b4a

Ths commnad installs the following dependencies:

bare-path: A module for working with paths.
bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
hypercore: A module for working with Hypercore.
b4a: A module for working with buffers.

Add the app logic

Create the writer-app/index.js file with the following content:

import path from 'bare-path'
import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Hypercore from 'hypercore'
import b4a from 'b4a'

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

const core = new Hypercore(path.join('./storage', 'writer-storage'))

// core.key and core.discoveryKey will only be set after core.ready resolves
await core.ready()
console.log('hypercore key:', b4a.toString(core.key, 'hex'))

// Append all stdin data as separate blocks to the core
process.stdin.on('data', (data) => core.append(data))

// core.discoveryKey is *not* a read capability for the core
// It's only used to discover other peers who *might* have the core
swarm.join(core.discoveryKey)
swarm.on('connection', conn => core.replicate(conn))

Create the reader app

The reader-app uses Hyperswarm to connect to the writer peer and synchronize its local Hypercore with the writer's Hypercore.

Create the reader-app directory and add dependencies

Create the reader-app project with these commands:

mkdir reader-app
cd reader-app
npm init -y
npm pkg set type="module"
npm install bare-path bare-process hypercore hyperswarm

Ths command installs the following dependencies:

bare-path: A module for working with paths.
bare-process: A module for working with processes.
hyperswarm: A module for working with Hyperswarm.
hypercore: A module for working with Hypercore.

Add the reader-app logic

Create the reader-app/index.js file with the following content:

import path from 'bare-path'
import process from 'bare-process'
import Hyperswarm from 'hyperswarm'
import Hypercore from 'hypercore'

const swarm = new Hyperswarm()
process.once('SIGINT', () => swarm.destroy().then(() => process.exit(0)))

const core = new Hypercore(path.join('./storage', 'reader-storage'), Bare.argv[2])
await core.ready()

swarm.join(core.discoveryKey)
swarm.on('connection', conn => core.replicate(conn))

// swarm.flush() will wait until *all* discoverable peers have been connected to
await swarm.flush()

await core.update()

let position = core.length
console.log(`Skipping ${core.length} earlier blocks...`)
for await (const block of core.createReadStream({ start: core.length, live: true })) {
  console.log(`Block ${position++}: ${block}`)
}

Run the writer and reader

In one terminal, run writer-app with bare:

bare writer-app

The writer-app will output the Hypercore key.

In another terminal, open the reader-app and pass it the key:

bare reader-app <SUPPLY THE KEY HERE>

As inputs are made to the terminal running the writer application, outputs should be shown in the terminal running the reader application.