Using Sawtooth with the SGX Implementation of PoET

Note

These instructions have been tested on Ubuntu 16.04 only.

Prerequisites

BIOS Update

Important

You may need to update your BIOS with a security fix before running Hyperledger Sawtooth with PoET. Affected versions and instructions for updating can be found on Intel’s website. If you’re running an affected version, you must update the BIOS to ensure that these installation instructions work correctly.

You can verify the BIOS version after the machine has booted by running:

$ sudo lshw| grep -A5 *-firmware
     *-firmware
          description: BIOS
          vendor: Intel Corp.
          physical id: 0
          version: BNKBL357.86A.0050.2017.0816.2002
          date: 08/16/2017

Install SGX/PSW

Install the prerequisites for SGX/PSW:

$ sudo apt-get update &&
  sudo apt-get install -q -y \
      alien \
      autoconf \
      automake \
      build-essential \
      cmake \
      libcurl4-openssl-dev \
      libprotobuf-dev \
      libssl-dev \
      libtool \
      libxml2-dev \
      ocaml \
      pkg-config \
      protobuf-compiler \
      python \
      unzip \
      uuid-dev \
      wget

Download and install the SGX driver:

$ mkdir ~/sgx && cd ~/sgx
$ wget https://download.01.org/intel-sgx/linux-2.0/sgx_linux_x64_driver_eb61a95.bin
$ chmod +x sgx_linux_x64_driver_eb61a95.bin
$ sudo ./sgx_linux_x64_driver_eb61a95.bin

Download and install the Intel Capability Licensing Client. This is presently available only as an .rpm, so you must convert it to a .deb package with alien:

$ wget http://registrationcenter-download.intel.com/akdlm/irc_nas/11414/iclsClient-1.45.449.12-1.x86_64.rpm
$ sudo alien --scripts iclsClient-1.45.449.12-1.x86_64.rpm
$ sudo dpkg -i iclsclient_1.45.449.12-2_amd64.deb

Download and install the Dynamic Application Loader Host Interface (JHI):

$ wget https://github.com/01org/dynamic-application-loader-host-interface/archive/master.zip -O jhi-master.zip
$ unzip jhi-master.zip && cd dynamic-application-loader-host-interface-master
$ cmake .
$ make
$ sudo make install
$ sudo systemctl enable jhi

Download and install the Intel SGX Platform Software (PSW):

$ cd ~/sgx
$ wget https://download.01.org/intel-sgx/linux-2.0/sgx_linux_ubuntu16.04.1_x64_psw_2.0.100.40950.bin
$ chmod +x sgx_linux_ubuntu16.04.1_x64_psw_2.0.100.40950.bin
$ sudo ./sgx_linux_ubuntu16.04.1_x64_psw_2.0.100.40950.bin

Check to make sure the kernel module is loaded:

$ lsmod | grep sgx
isgx                   36864  2

If the output does not show the isgx module, make sure that SGX is set to “Enabled” in the BIOS.

If you’re still having trouble, the SGX software may need to be reinstalled:

$ sudo /opt/intel/sgxpsw/uninstall.sh
$ cd ~/sgx
$ sudo ./sgx_linux_x64_driver_eb61a95.bin
$ sudo ./sgx_linux_ubuntu16.04.1_x64_psw_2.0.100.40950.bin

After ensuring that the SGX kernel module is loaded, go to the next section to install and configure Sawtooth.

Configuring Sawtooth to Use SGX

Install Sawtooth

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8AA7AF1F1091A5FD
$ sudo add-apt-repository 'deb http://repo.sawtooth.me/ubuntu/1.0/stable xenial universe'
$ sudo apt-get update
$ sudo apt-get install -y -q \
  sawtooth \
  python3-sawtooth-poet-sgx

Certificate File

The configuration process requires an SGX certificate file in PEM format (.pem), which you will need before continuing.

Instructions for creating your own service provider certificate can be found here.

After your certificate is created you’ll need to register it with the attestation service. Click here for the registration form.

Configure the Validator to Use SGX PoET

After installing Sawtooth, add config settings so PoET will work properly.

Create the file /etc/sawtooth/poet_enclave_sgx.toml with your favorite editor (such as vi):

$ sudo vi /etc/sawtooth/poet_enclave_sgx.toml

Add the following lines, replacing [example] with the spid value provided by Intel:

# Service Provider ID. It is linked to the key pair used to authenticate with
# the attestation service.

spid = '[example]'

# ias_url is the URL of the Intel Attestation Service (IAS) server.

ias_url = 'https://test-as.sgx.trustedservices.intel.com:443'

# spid_cert_file is the full path to the PEM-encoded certificate file that was
# submitted to Intel in order to obtain a SPID

spid_cert_file = '/etc/sawtooth/sgx-certificate.pem'

Next, install the .pem certificate file that you downloaded earlier. Replace [example] in the path below with the path to the certificate file on your local system:

$ sudo install -o root -g sawtooth -m 640 \
/[example]/sgx-certificate.pem /etc/sawtooth/sgx-certificate.pem

Create validator keys:

$ sudo sawadm keygen

Note

If you’re configuring multiple validators, the steps below are required for the first validator only. For additional validators, you can skip the rest of this procedure. Continue with Change the Validator Config File.

Become the sawtooth user and change to /tmp. In the following commands, the prompt [sawtooth@system] shows the commands that must be executed as the sawtooth user.

$ sudo -u sawtooth -s
[sawtooth@system]$ cd /tmp

Create a genesis batch:

[sawtooth@system]$ sawset genesis --key /etc/sawtooth/keys/validator.priv -o config-genesis.batch

Create and submit a proposal:

[sawtooth@system]$ sawset proposal create -k /etc/sawtooth/keys/validator.priv \
sawtooth.consensus.algorithm=poet \
sawtooth.poet.report_public_key_pem="$(cat /etc/sawtooth/ias_rk_pub.pem)" \
sawtooth.poet.valid_enclave_measurements=$(poet enclave --enclave-module sgx measurement) \
sawtooth.poet.valid_enclave_basenames=$(poet enclave --enclave-module sgx basename) \
sawtooth.poet.enclave_module_name=sawtooth_poet_sgx.poet_enclave_sgx.poet_enclave \
-o config.batch

When the sawset proposal command runs, you should see several lines of output showing that the SGX enclave has been initialized:

[12:03:58 WARNING poet_enclave] SGX PoET enclave initialized.
[12:03:59 WARNING poet_enclave] SGX PoET enclave initialized.

Note

There’s quite a bit going on in the previous sawset proposal command, so let’s take a closer look at what it accomplishes:

sawtooth.consensus.algorithm=poet
Changes the consensus algorithm to PoET.
sawtooth.poet.report_public_key_pem="$(cat /etc/sawtooth/ias_rk_pub.pem)"
Adds the public key that the validator registry transaction processor uses to verify attestation reports.
sawtooth.poet.valid_enclave_measurements=$(poet enclave --enclave-module sgx measurement)
Adds the enclave measurement for your enclave to the blockchain for the validator registry transaction processor to use to check signup information.
sawtooth.poet.valid_enclave_basenames=$(poet enclave --enclave-module sgx basename)
Adds the enclave basename for your enclave to the blockchain for the validator registry transaction processor to use to check signup information.
sawtooth.poet.enclave_module_name
Specifies the name of the Python module that implements the PoET enclave. In this case, sawtooth_poet_sgx.poet_enclave_sgx.poet_enclave is the SGX version of the enclave; it includes the Python code as well as the Python extension.

Create a poet-genesis batch:

[sawtooth@system]$ poet registration create -k /etc/sawtooth/keys/validator.priv \
  --enclave-module sgx -o poet_genesis.batch
Writing key state for PoET public key: 0387a451...9932a998
Generating poet_genesis.batch

Create a genesis block:

[sawtooth@system]$ sawadm genesis config-genesis.batch config.batch poet_genesis.batch

You’ll see some output indicating success:

Processing config-genesis.batch...
Processing config.batch...
Processing poet_genesis.batch...
Generating /var/lib/sawtooth/genesis.batch

Genesis configuration is complete! Log out of the sawtooth account:

[sawtooth@system]$ exit
$

Change the Validator Config File

You must specify some networking information so that the validator advertises itself properly and knows where to search for peers. Create the file /etc/sawtooth/validator.toml:

$ sudo vi /etc/sawtooth/validator.toml

Add the following content to the file:

#
# Hyperledger Sawtooth -- Validator Configuration
#

# This file should exist in the defined config directory and allows
# validators to be configured without the need for command line options.

# The following is a possible example.

# Bind is used to set the network and component endpoints. It should be a list
# of strings in the format "option:endpoint", where the options are currently
# network and component.
bind = [
  "network:tcp://eno1:8800",
  "component:tcp://127.0.0.1:4004"
]

# The type of peering approach the validator should take. Choices are 'static'
# which only attempts to peer with candidates provided with the peers option,
# and 'dynamic' which will do topology buildouts. If 'dynamic' is provided,
# any static peers will be processed first, prior to the topology buildout
# starting.
peering = "dynamic"

# Advertised network endpoint URL.
endpoint = "tcp://[external interface]:[port]"

# Uri(s) to connect to in order to initially connect to the validator network,
# in the format tcp://hostname:port. This is not needed in static peering mode
# and defaults to None.
seeds = ["tcp://[seed address 1]:[port]",
         "tcp://[seed address 2]:[port]"]

# A list of peers to attempt to connect to in the format tcp://hostname:port.
# It defaults to None.
peers = []

# The type of scheduler to use. The choices are 'serial' or 'parallel'.
scheduler = 'serial'

# A Curve ZMQ key pair are used to create a secured network based on side-band
# sharing of a single network key pair to all participating nodes.
# Note if the config file does not exist or these are not set, the network
# will default to being insecure.
#network_public_key = ''
#network_private_key = ''

Next, locate the endpoint section in this file. Replace the external interface and port values with either the publicly addressable IP address and port or the NAT values for your validator.

endpoint = "tcp://[external interface]:[port]"

Find the seeds section in the config file. Replace the seed address and port values with either the publicly addressable IP address and port or the NAT values for the other nodes in your network.

seeds = ["tcp://[seed address 1]:[port]",
         "tcp://[seed address 2]:[port]"]

If necessary, change the network bind interface in the bind section.

bind = [
  "network:tcp://eno1:8800",
  "component:tcp://127.0.0.1:4004"
]

The default network bind interface is “eno1”. If this device doesn’t exist on your machine, change the network definition to specify the correct bind interface.

Important

If the bind interface doesn’t exist, you may see a ZMQ error in the sawtooth-validator systemd logs when attempting to start the validator, as in this example:

Jun 02 14:50:37 ubuntu validator[15461]:   File "/usr/lib/python3.5/threading.py", line 862, in run
...
Jun 02 14:50:37 ubuntu validator[15461]:   File "zmq/backend/cython/socket.pyx", line 487, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5156)
Jun 02 14:50:37 ubuntu validator[15461]:   File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535)
Jun 02 14:50:37 ubuntu validator[15461]: zmq.error.ZMQError: No such device
Jun 02 14:50:37 ubuntu systemd[1]: sawtooth-validator.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 14:50:37 ubuntu systemd[1]: sawtooth-validator.service: Unit entered failed state.
Jun 02 14:50:37 ubuntu systemd[1]: sawtooth-validator.service: Failed with result 'exit-code'.

Restrict permssions on validator.toml to protect the network private key.

$ sudo chown root:sawtooth /etc/sawtooth/validator.toml
$ sudo chown 640 /etc/sawtooth/validator.toml

Start the Sawtooth Services

Use these commands to start the Sawtooth services:

$ sudo systemctl start sawtooth-rest-api.service
$ sudo systemctl start sawtooth-poet-validator-registry-tp.service
$ sudo systemctl start sawtooth-validator.service
$ sudo systemctl start sawtooth-settings-tp.service
$ sudo systemctl start sawtooth-intkey-tp-python.service

You can follow the logs by running:

$ sudo journalctl -f \
-u sawtooth-validator \
-u sawtooth-tp_settings \
-u sawtooth-poet-validator-registry-tp \
-u sawtooth-rest-api \
-u sawtooth-intkey-tp-python

Additional logging output can be found in /var/log/sawtooth/.

To verify that the services are running:

$ sudo systemctl status sawtooth-rest-api.service
$ sudo systemctl status sawtooth-poet-validator-registry-tp.service
$ sudo systemctl status sawtooth-validator.service
$ sudo systemctl status sawtooth-settings-tp.service
$ sudo systemctl status sawtooth-intkey-tp-python.service

Stop or Restart the Sawtooth Services

If you need to stop or restart the Sawtooth services for any reason, use the following commands:

Stop Sawtooth services:

$ sudo systemctl stop sawtooth-rest-api.service
$ sudo systemctl stop sawtooth-poet-validator-registry-tp.service
$ sudo systemctl stop sawtooth-validator.service
$ sudo systemctl stop sawtooth-settings-tp.service
$ sudo systemctl stop sawtooth-intkey-tp-python.service

Restart Sawtooth services:

$ sudo systemctl restart sawtooth-rest-api.service
$ sudo systemctl restart sawtooth-poet-validator-registry-tp.service
$ sudo systemctl restart sawtooth-validator.service
$ sudo systemctl restart sawtooth-settings-tp.service
$ sudo systemctl restart sawtooth-intkey-tp-python.service