snarkOPs

Snops Quickstart | snarkos-aot Quickstart

This repository is home to the snops (snarkOS operations) ecosystem, and snarkos-aot, a crate for performing ahead-of-time ledger actions, transaction authorizations, and various tools for helping with developing Aleo programs.

snops is a suite of tools that can be used to maintain Aleo network environments. The environments are defined in a manner similar to infrastructure-as-code, which means you can create repeatable infrastructure using the environment schema.

This can be used to create devnets, simulate events on the network like outages and attacks, and guarantee metrics.

To learn more about snops we recommend checking out the mdbook here.

Snops Quickstart

Easy Setup

snops contains several components that can be compiled separately. The release-big build profile allows for faster near-release profile performance with faster near-debug profile build times.

Install rust
Clone the repo
Start + build the control plane: ./scripts/control_plane.sh

The controlplane is the webserver that communicates to agents how to run snarkOS, or what transactions to execute.
In another terminal, install the cli: cargo install --path ./crates/snops-cli, or build with cargo xtask build cli and use from target/release-big/snops-cli.

The cli is used to interact with the controlplane and manage environments. It provides JSON based output. We recommend pairing our cli with jq when leveraging other scripts and tools
Build the agent: cargo xtask build agent

The agent is a lightweight service that starts up snarkos-aot which automatically configures snarkos nodes, or executes transactions.
Build snarkos-aot (for running nodes): cargo xtask build aot

snarkos-aot is an alternative snarkOS CLI providing more developer-oriented features as well as tooling for distributed transaction generation and execution.
In separate terminals, start up some agents with the following commands:
```
./script/agent.sh 0
./script/agent.sh 1
./script/agent.sh 2
./script/agent.sh 3
```
Each of these can be dynamically configured as snarkos nodes. The default agent configuration should connect to a locally operated controlplane.

Local Isolated Networks (Isonets)

This example requires 4 agents and the control plane to be running. It allows you to run a devnet with a custom genesis block.

Start the environment: snops-cli env apply specs/test-4-validators.yaml
Check the current network height: snops-cli env height
Look at the latest block: snops-cli env block
Look at the genesis block: snops-cli env block 0
Stop the environment: snops-cli env delete

Isonet Transfers

Using the setup for a Local Isonet, executing Aleo programs has never been more convenient. Snops aliases locally generated keys automatically from configuration and reduces the need to keep track of individual key files.

Start an environment (see previous example)
Start a compute agent: ./scripts/agent_compute.sh 0

Compute agents are able to distribute the execution of transactions, offloading the compute task to different servers.

Check the balance of committee member 1's key:

$ snops-cli env balance committee.1
10000000000000

Transfer 1 credit to committee member 1 from committee member 0 (look on snarkos this command would require MANY more flags):
```
snops-cli env action execute transfer_public committee.1 1_000_000u64
```
Check the balance of committee member 1's key (it may take a few seconds for the block to advance):
```
snops-cli env balance committee.1
10000001000000
```

Isonet Program Deployments

Deploying and executing Aleo programs on your isonets is easiest with snops. You do not need any extra json files or tooling, only a standalone .aleo file.

Use the same setup as before

Create a file mapping_example.aleo with the following contents:

program mapping_example.aleo;

mapping mymapping:
  key as u8.public;
  value as u8.public;

function store:
    input r0 as u8.public;
    input r1 as u8.public;
    async store r0 r1 into r2;
    output r2 as mapping_example.aleo/store.future;

finalize store:
    input r0 as u8.public;
    input r1 as u8.public;
    set r1 into mymapping[r0];

Deploy the program: snops-cli env action deploy ./mapping_example.aleo
Verify the program is on the chain: snops-cli env program mapping_example.aleo

Check an example mapping

$ snops-cli env mapping mapping_example.aleo mymapping 0u8
{
  "value": null
}

Execute the store function on-chain: snops-cli env action execute mapping_example.aleo/store 0u8 5u8

Check the example mapping for its updated value

$ snops-cli env mapping mapping_example.aleo mymapping 0u8
{
  "value": "5u8"
}

SnarkOS-aot Quickstart

snarkos-aot provides various CLI tools to help with developing and executing Aleo programs as well as interact with snarkOS ledgers.

Build snarkos-aot with: cargo xtask build aot. The compiled binary can be found in target/release-big/snarkos-aot.

Use the NETWORK environment variable to specify mainnet (default), testnet, or canary.

Transaction Authorizations

Typically when executing snarkOS transactions you specify the program and inputs and receive a proof. Behind the scenes, snarkVM is using a private key to "authorize" the proof for the program execution, and "authorize" the proof for the fee.

Creating authorizations is much quicker than executing the transaction, and the process can be done airgapped or even on web clients.

# Create an authorization for a executing transfer_public to the "example.aleo" program for 1 microcredit
NETWORK=testnet snarkos-aot auth program --private-key <PK> credits.aleo/transfer_public example.aleo 1u64 > auth.json
# The same as above but with a different private key
NETWORK=testnet snarkos-aot auth program --private-key <PK> --fee-private-key <PK> credits.aleo/transfer_public example.aleo 1u64 > auth.json

# Create an authorization for deploying a program (from my_program.aleo)
NETWORK=testnet snarkos-aot auth deploy --private-key <PK> my_program.aleo
# Determine the cost to deploy a program (from stdin)
cat my_program.aleo | NETWORK=testnet snarkos-aot auth deploy --private-key <PK> - | NETWORK=testnet snarkos-aot auth cost -

# Execute an authorization from auth.json without broadcasting it (the - means stdin)
cat auth.json | NETWORK=testnet snarkos-aot auth execute - --query https://api.explorer.aleo.org/v1

# Get the cost of an authorization (program or deployment) without executing the transaction
cat auth.json | NETWORK=testnet snarkos-aot auth cost -

# Derive the transaction id from an authorization without executing it
cat auth.json | NETWORK=testnet snarkos-aot auth id -

Program Helpers

snarkos-aot contains various tools to programmatically interact with aleo program.

If you don't have any Aleo programs you can download a program (credits.aleo) from Aleo Explorer with the following commands. You can find a list of programs on Aleo Explorer's Programs List.

NETWORK=testnet
PROGRAM=credits.aleo
# Download a program, then un-jsonify it
curl https://api.explorer.aleo.org/v1/$NETWORK/program/$PROGRAM | jq -r > $PROGRAM

Below are some snarkos-aot commands for interacting with programs.

# Get a program's id from a text file
$ snarkos-aot program id ./credits.aleo
credits.aleo

# Calculate the cost of deploying a program
$ snarkos-aot program cost ./example.aleo
2568400

# Get a list of imports for a program (output in a json format with --json)
$ snarkos-aot program imports ./staking_v1.aleo --json
["credits.aleo"]

# Get a list of functions (and respective inputs/outputs) for a program (jq for formatting)
$ snarkos-aot program functions ./domainnames.aleo --json | jq
{
  "validate_name": {
    "inputs": [
      "[u128; 4u32].private"
    ],
    "outputs": [
      "boolean.private"
    ]
  }
}

Account Helpers

Need to generate a bunch of vanity accounts for testing?

# Generate 5 accounts that have addresses that start with `aleo1f00`
snarkos-aot accounts 5 --vanity f00

Contributing

snops is free and open source. You can find the source code on GitHub and issues and feature requests can be posted on the GitHub issue tracker. If you'd like to contribute, please read the CONTRIBUTING guide and consider opening a pull request.

License

The snops source and documentation are released under the MIT License .

Architecture

A snops "instance" is composed of multiple parts:

A Control Plane
Agents
A runner which is: snarkos-aot: our own modified runner and aot wrapper around the official snarkOS.

Agent

The agent is responsible for:

Communicating with the running control plane.
Launching and running the snarkos-aot binary.
Communicating to the running snarkos-aot.

Running the Agent

The agent can be run on the same machine as the control plane or a separate one.

Depending on the environment you specified, you will need the number agents listed in it, unless they are external.

Startup Options

The binary has several options you can run mess with at launch and some are required that we will go over here.

However, for a more in depth information you can read about the CLI options here.

endpoint

Is an optional argument that can be provided via the CLI or the SNOPS_ENDPOINT environment variable.

NOTE: If both the CLI flag and the ENV variable are present, the ENV variable takes precedence over the CLI flag.

If not provided it will default to 127.0.0.1:1234.

This is the endpoint of the Control Plane.

If you want it to be a secure connection please specify https:// or wss:// at the beginning of the endpoint or it will default to http and ws.

id

The field where you can give this agent a specific ID, so it is identifiable within the snops ecosystem.

labels

Optional comma separated list of labels you can apply to the agent, which are used for filtering and grouping.

private-key-file

An optional private key file. If provided is used when starting snarkOS.

path

Optional path to the directory containing the stored data and configuration for the agent.

By default it is snops-data local to where the agent was run from.

external

TODO

internal

TODO

bind_addr

The optional address for the agent to bind to when running.

Defaults to 0.0.0.0.

node

Optional port for the snarkOS node server to run on.

Defaults to 4130.

bft

Optional port for the snarkOS BFT to run on.

Defaults to 5000.

rest

Optional port for the snarkOS REST API to run on.

Defaults to 3030.

metrics

Optional port for the snarkOS metrics to run on.

Defaults to 9000.

validator

Enables validator mode as an option for the agent's snarkOS node.

prover

Enables prover mode as an option for the agent's snarkOS node.

client

Enables client mode as an option for the agent's snarkOS node.

compute

Enables compute mode as an option for the agent to be able to run transactions fired from within snops.

quiet

Run the agent in quiet mode which prevents snarkOS node output.

How it Works

The agent once running will connect to the control plane. If the control plane goes/offline or isn't online yet that's okay! The agent will continuously try to reconnect to the control plane endpoint provided.

For running snarkOS the agent will download that binary from the control plane.

Updating

You don't have to worry about updating the individual agents.

The control plane will serve the updated agent binary to them once there is an update.

Control Plane

The control plane is responsible for:

Communicating to agents.
Delegating work to appropriate agents.
Various actions you can perform on agents and within an environment.
Searching among agents.
Forwarding logs and metrics.
Serving snarkos-aot and agent binaries.
Storing and querying environment information.

The binary has several options you can run mess with at launch and some are required that we will go over here.

However, for a more in depth information you can read about the CLI options here.

Running the Control Plane

The control plane is the critical part of snops as it delegates and handles all incoming information from agents.

Startup Options

The binary has several options you can run mess with at launch and some are required that we will go over here.

However, for a more in depth information you can read about the CLI options here.

bind_addr

The optional address for the control plane to bind to when running.

Defaults to 0.0.0.0.

port

The optional address for the control plane to bind to when running.

Defaults to 1234.

prometheus

Is a optional argument that can be provided via the CLI or the PROMETHEUS_URL environment variable.

NOTE: If both the CLI flag and the ENV variable are present, the ENV variable takes precedence over the CLI flag.

If provided it will send metric data to prometheus.

loki

Is a optional argument that can be provided via the CLI or the LOKI_URL environment variable.

NOTE: If both the CLI flag and the ENV variable are present, the ENV variable takes precedence over the CLI flag.

If provided it will send log data to Loki.

prometheus_location

Optional value of where prometheus is located.

This option only matters if you are running the agents local to the control plane.

The default is docker.

The other two options are external and internal.

WARNING: For locally run agents an external prometheus server is not yet supported.

path

Optional path to the directory containing the stored data and configuration for the agent.

By default it is snops-control-data local to where the control plane was run from.

hostname

The optional hostname(IP or FQDN) for an external cannon/compute.

It must include http:// or https://.

Updating

To update the control plane simply stop the current one, and replace the binary.

When you run the control plane again it will read it's data back from the path specified during running.

Metrics and Logging

TODO

Cli Help

This section contains documentation on all the different CLIs we have.

Command-Line Help for `snarkos-aot`

This document contains the help content for the snarkos-aot command-line program.

Command Overview:

`snarkos-aot`

The different AOT commands

Usage: snarkos-aot [OPTIONS] <COMMAND>

Subcommands:

genesis — This command helps generate a custom genesis block given an initial private key, seed, and committee size
accounts — Given a seed and a count, generate a number of accounts
ledger — Commands for interacting with the ledger
auth — A command to help generate various different types of authorizations and execute them
program — A command to help gather information about a program, including its cost and imports
man — For generating cli manpages. Only with the mangen feature enabled
md — For generating cli markdown. Only with the clipages feature enabled
run — A wrapper around the snarkos node run commands that provide additional logging and configurability

Options:

--enable-profiling
--log <LOG> — The path to the log file
--verbosity <VERBOSITY> — The verbosity level of the logs

Default value: 4
--loki <LOKI> — The optional loki url to send logs to

`snarkos-aot genesis`

This command helps generate a custom genesis block given an initial private key, seed, and committee size

Usage: snarkos-aot genesis [OPTIONS]

Options:

-g, --genesis-key <GENESIS_KEY> — The private key to use when generating the genesis block. Generates one randomly if not passed
-o, --output <OUTPUT> — Where to write the genesis block to

Default value: genesis.block
--committee-size <COMMITTEE_SIZE> — The committee size. Not used if --bonded-balances is set

Default value: 4
--committee-output <COMMITTEE_OUTPUT> — A place to optionally write out the generated committee private keys JSON
--additional-accounts <ADDITIONAL_ACCOUNTS> — Additional number of accounts that aren't validators to add balances to

Default value: 0
--additional-accounts-balance <additional-accounts-balance> — The balance to add to the number of accounts specified by additional-accounts

Default value: 100000000
--additional-accounts-record-balance <ADDITIONAL_ACCOUNTS_RECORD_BALANCE> — If --additional-accounts is passed you can additionally add an amount to give them in a record
--additional-accounts-output <ADDITIONAL_ACCOUNTS_OUTPUT> — A place to write out the additionally generated accounts by --additional-accounts
--seed <SEED> — The seed to use when generating committee private keys and the genesis block. If unpassed, uses DEVELOPMENT_MODE_RNG_SEED (1234567890u64)
--bonded-balance <BONDED_BALANCE> — The bonded balance each bonded address receives. Not used if --bonded-balances is passed

Default value: 10000000000000
--bonded-balances <BONDED_BALANCES> — An optional map from address to bonded balance. Overrides --bonded-balance and --committee-size
--bonded-withdrawal <BONDED_WITHDRAWAL> — An optional to specify withdrawal addresses for the genesis committee
--bonded-commission <BONDED_COMMISSION> — The bonded commission each bonded address uses. Not used if --bonded-commissions is passed. Defaults to 0. Must be 100 or less

Default value: 0
--bonded-commissions <BONDED_COMMISSIONS> — An optional map from address to bonded commission. Overrides --bonded-commission. Defaults to 0. Must be 100 or less
--ledger <LEDGER> — Optionally initialize a ledger as well

`snarkos-aot accounts`

Given a seed and a count, generate a number of accounts

Usage: snarkos-aot accounts [OPTIONS] [COUNT]