etcd – Documentation versions

Docs: etcd client architecture

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

etcd server has proven its robustness with years of failure injection testing. Most complex application logic is already handled by etcd server and its data stores (e.g. cluster membership is transparent to clients, with Raft-layer forwarding proposals to leader). Although server components are correct, its composition with client requires a different set of intricate protocols to guarantee its correctness and high availability under faulty conditions. Ideally, etcd server provides one logical cluster view of many physical machines, and client implements automatic failover between replicas. This documents client architectural decisions and its implementation details.

Glossary

clientv3 — etcd Official Go client for etcd v3 API.

clientv3-grpc1.0 — Official client implementation, with grpc-go v1.0.x, which is used in latest etcd v3.1.

clientv3-grpc1.7 — Official client implementation, with grpc-go v1.7.x, which is used in latest etcd v3.2 and v3.3.

clientv3-grpc1.14 — Official client implementation, with grpc-go v1.14.x, which is used in latest etcd v3.4.

Balancer — etcd client load balancer that implements retry and failover mechanism. etcd client should automatically balance loads between multiple endpoints.

Endpoints — A list of etcd server endpoints that clients can connect to. Typically, 3 or 5 client URLs of an etcd cluster.

Pinned endpoint — When configured with multiple endpoints, <= v3.3 client balancer chooses only one endpoint to establish a TCP connection, in order to conserve total open connections to etcd cluster. In v3.4, balancer round-robins pinned endpoints for every request, thus distributing loads more evenly.

Client Connection — TCP connection that has been established to an etcd server, via gRPC Dial.

Sub Connection — gRPC SubConn interface. Each sub-connection contains a list of addresses. Balancer creates a SubConn from a list of resolved addresses. gRPC ClientConn can map to multiple SubConn (e.g. example.com resolves to 10.10.10.1 and 10.10.10.2 of two sub-connections). etcd v3.4 balancer employs internal resolver to establish one sub-connection for each endpoint.

Transient disconnect — When gRPC server returns a status error of code Unavailable.

Client requirements

Correctness — Requests may fail in the presence of server faults. However, it never violates consistency guarantees: global ordering properties, never write corrupted data, at-most once semantics for mutable operations, watch never observes partial events, and so on.

Liveness — Servers may fail or disconnect briefly. Clients should make progress in either way. Clients should never deadlock waiting for a server to come back from offline, unless configured to do so. Ideally, clients detect unavailable servers with HTTP/2 ping and failover to other nodes with clear error messages.

Effectiveness — Clients should operate effectively with minimum resources: previous TCP connections should be gracefully closed after endpoint switch. Failover mechanism should effectively predict the next replica to connect, without wastefully retrying on failed nodes.

Portability — Official client should be clearly documented and its implementation be applicable to other language bindings. Error handling between different language bindings should be consistent. Since etcd is fully committed to gRPC, implementation should be closely aligned with gRPC long-term design goals (e.g. pluggable retry policy should be compatible with gRPC retry). Upgrades between two client versions should be non-disruptive.

Client overview

The etcd client implements the following components:

balancer that establishes gRPC connections to an etcd cluster,
API client that sends RPCs to an etcd server, and
error handler that decides whether to retry a failed request or switch endpoints.

Languages may differ in how to establish an initial connection (e.g. configure TLS), how to encode and send Protocol Buffer messages to server, how to handle stream RPCs, and so on. However, errors returned from etcd server will be the same. So should be error handling and retry policy.

For example, etcd server may return "rpc error: code = Unavailable desc = etcdserver: request timed out", which is transient error that expects retries. Or return rpc error: code = InvalidArgument desc = etcdserver: key is not provided, which means request was invalid and should not be retried. Go client can parse errors with google.golang.org/grpc/status.FromError, and Java client with io.grpc.Status.fromThrowable.

clientv3-grpc1.0: Balancer Overview

clientv3-grpc1.0 maintains multiple TCP connections when configured with multiple etcd endpoints. Then pick one address and use it to send all client requests. The pinned address is maintained until the client object is closed (see Figure 1). When the client receives an error, it randomly picks another and retries.

clientv3-grpc1.0: Balancer Limitation

clientv3-grpc1.0 opening multiple TCP connections may provide faster balancer failover but requires more resources. The balancer does not understand node’s health status or cluster membership. So, it is possible that balancer gets stuck with one failed or partitioned node.

clientv3-grpc1.7: Balancer Overview

clientv3-grpc1.7 maintains only one TCP connection to a chosen etcd server. When given multiple cluster endpoints, a client first tries to connect to them all. As soon as one connection is up, balancer pins the address, closing others (see Figure 2).

The pinned address is to be maintained until the client object is closed. An error, from server or client network fault, is sent to client error handler (see Figure 3).

The client error handler takes an error from gRPC server, and decides whether to retry on the same endpoint, or to switch to other addresses, based on the error code and message (see Figure 4 and Figure 5).

Stream RPCs, such as Watch and KeepAlive, are often requested with no timeouts. Instead, client can send periodic HTTP/2 pings to check the status of a pinned endpoint; if the server does not respond to the ping, balancer switches to other endpoints (see Figure 6).

clientv3-grpc1.7: Balancer Limitation

clientv3-grpc1.7 balancer sends HTTP/2 keepalives to detect disconnects from streaming requests. It is a simple gRPC server ping mechanism and does not reason about cluster membership, thus unable to detect network partitions. Since partitioned gRPC server can still respond to client pings, balancer may get stuck with a partitioned node. Ideally, keepalive ping detects partition and triggers endpoint switch, before request time-out (see issue #8673 and Figure 7).

clientv3-grpc1.7 balancer maintains a list of unhealthy endpoints. Disconnected addresses are added to “unhealthy” list, and considered unavailable until after wait duration, which is hard coded as dial timeout with default value 5-second. Balancer can have false positives on which endpoints are unhealthy. For instance, endpoint A may come back right after being blacklisted, but still unusable for next 5 seconds (see Figure 8).

clientv3-grpc1.0 suffered the same problems above.

Upstream gRPC Go had already migrated to new balancer interface. For example, clientv3-grpc1.7 underlying balancer implementation uses new gRPC balancer and tries to be consistent with old balancer behaviors. While its compatibility has been maintained reasonably well, etcd client still suffered from subtle breaking changes. Furthermore, gRPC maintainer recommends not relying on the old balancer interface. In general, to get better support from upstream, it is best to be in sync with latest gRPC releases. And new features, such as retry policy, may not be backported to gRPC 1.7 branch. Thus, both etcd server and client must migrate to latest gRPC versions.

clientv3-grpc1.14: Balancer Overview

clientv3-grpc1.7 is so tightly coupled with old gRPC interface, that every single gRPC dependency upgrade broke client behavior. Majority of development and debugging efforts were devoted to fixing those client behavior changes. As a result, its implementation has become overly complicated with bad assumptions on server connectivities.

The primary goal of clientv3-grpc1.14 is to simplify balancer failover logic; rather than maintaining a list of unhealthy endpoints, which may be stale, simply roundrobin to the next endpoint whenever client gets disconnected from the current endpoint. It does not assume endpoint status. Thus, no more complicated status tracking is needed (see Figure 8 and above). Upgrading to clientv3-grpc1.14 should be no issue; all changes were internal while keeping all the backward compatibilities.

Internally, when given multiple endpoints, clientv3-grpc1.14 creates multiple sub-connections (one sub-connection per each endpoint), while clientv3-grpc1.7 creates only one connection to a pinned endpoint (see Figure 9). For instance, in 5-node cluster, clientv3-grpc1.14 balancer would require 5 TCP connections, while clientv3-grpc1.7 only requires one. By preserving the pool of TCP connections, clientv3-grpc1.14 may consume more resources but provide more flexible load balancer with better failover performance. The default balancing policy is round robin but can be easily extended to support other types of balancers (e.g. power of two, pick leader, etc.). clientv3-grpc1.14 uses gRPC resolver group and implements balancer picker policy, in order to delegate complex balancing work to upstream gRPC. On the other hand, clientv3-grpc1.7 manually handles each gRPC connection and balancer failover, which complicates the implementation. clientv3-grpc1.14 implements retry in the gRPC interceptor chain that automatically handles gRPC internal errors and enables more advanced retry policies like backoff, while clientv3-grpc1.7 manually interprets gRPC errors for retries.

clientv3-grpc1.14: Balancer Limitation

Improvements can be made by caching the status of each endpoint. For instance, balancer can ping each server in advance to maintain a list of healthy candidates, and use this information when doing round-robin. Or when disconnected, balancer can prioritize healthy endpoints. This may complicate the balancer implementation, thus can be addressed in later versions.

Client-side keepalive ping still does not reason about network partitions. Streaming request may get stuck with a partitioned node. Advanced health checking service need to be implemented to understand the cluster membership (see issue #8673 for more detail).

Currently, retry logic is handled manually as an interceptor. This may be simplified via official gRPC retries.

Docs: Install

Mon, 01 Jan 0001 00:00:00 +0000

System requirements

The etcd performance benchmarks run etcd on 8 vCPU, 16GB RAM, 50GB SSD GCE instances, but any relatively modern machine with low latency storage and a few gigabytes of memory should suffice for most use cases. Applications with large v2 data stores will require more memory than a large v3 data store since data is kept in anonymous memory instead of memory mapped from a file. For running etcd on a cloud provider, see the Example hardware configuration documentation.

Download the pre-built binary

The easiest way to get etcd is to use one of the pre-built release binaries which are available for OSX, Linux, Windows, appc, and Docker. Instructions for using these binaries are on the GitHub releases page.

Build the latest version

For those wanting to try the very latest version, build etcd from the master branch. Go version 1.9+ is required to build the latest version of etcd. To ensure etcd is built against well-tested libraries, etcd vendors its dependencies for official release binaries. However, etcd’s vendoring is also optional to avoid potential import conflicts when embedding the etcd server or using the etcd client.

To build etcd from the master branch without a GOPATH using the official build script:

$ git clone https://github.com/etcd-io/etcd.git
$ cd etcd
$ ./build

To build a vendored etcd from the master branch via go get:

# GOPATH should be set
$ echo $GOPATH
/Users/example/go
$ go get -v go.etcd.io/etcd
$ go get -v go.etcd.io/etcd/etcdctl

Test the installation

Check the etcd binary is built correctly by starting etcd and setting a key.

Starting etcd

If etcd is built without using go get, run the following:

$ ./bin/etcd

If etcd is built using go get, run the following:

$ $GOPATH/bin/etcd

Setting a key

Run the following:

$ ./bin/etcdctl put foo bar
OK

(or $GOPATH/bin/etcdctl put foo bar if etcdctl was installed with go get)

If OK is printed, then etcd is working!

Docs: Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.

Go libraries

etcd/client/v3 - the officially maintained Go client for v3
etcd/client/v2 - the officially maintained Go client for v2
go-etcd - the deprecated official client. May be useful for older (<2.0.0) versions of etcd.
encWrapper - encWrapper is an encryption wrapper for the etcd client Keys API/KV.

Java libraries

coreos/jetcd - Supports v3
boonproject/etcd - Supports v2, Async/Sync and waits
justinsb/jetcd
diwakergupta/jetcd - Supports v2
jurmous/etcd4j - Supports v2, Async/Sync, waits and SSL
AdoHe/etcd4j - Supports v2 (enhance for real production cluster)
cdancy/etcd-rest - Uses jclouds to provide a complete implementation of v2 API.

Scala libraries

maciej/etcd-client - Supports v2. Akka HTTP-based fully async client
eiipii/etcdhttpclient - Supports v2. Async HTTP client based on Netty and Scala Futures.

Perl libraries

hexfusion/perl-net-etcd - Supports v3 grpc gateway HTTP API
robn/p5-etcd - Supports v2

Python libraries

kragniz/python-etcd3 - Client for v3
jplana/python-etcd - Supports v2
russellhaering/txetcd - a Twisted Python library
cholcombe973/autodock - A docker deployment automation tool
lisael/aioetcd - (Python 3.4+) Asyncio coroutines client (Supports v2)
txaio-etcd - Asynchronous etcd v3-only client library for Twisted (today) and asyncio (future)
dims/etcd3-gateway - etcd v3 API library using the HTTP grpc gateway
aioetcd3 - (Python 3.6+) etcd v3 API for asyncio
Revolution1/etcd3-py - (python2.7 and python3.5+) Python client for etcd v3, using gRPC-JSON-Gateway

Node libraries

stianeikeland/node-etcd - Supports v2 (w Coffeescript)
lavagetto/nodejs-etcd - Supports v2
deedubs/node-etcd-config - Supports v2

Ruby libraries

C libraries

apache/celix/etcdlib - Supports v2
jdarcy/etcd-api - Supports v2
shafreeck/cetcd - Supports v2

C++ libraries

edwardcapriolo/etcdcpp - Supports v2
suryanathan/etcdcpp - Supports v2 (with waits)
nokia/etcd-cpp-api - Supports v2
nokia/etcd-cpp-apiv3 - Supports v3

Clojure libraries

Erlang libraries

marshall-lee/etcd.erl - Supports v2
zhongwencool/eetcd - Supports v3+ (GRPC only)

.Net Libraries

wangjia184/etcdnet - Supports v2
drusellers/etcetera
shubhamranjan/dotnet-etcd - Supports v3+ (GRPC only)

PHP Libraries

linkorb/etcd-php
activecollab/etcd
ouqiang/etcd-php - Client for v3 gRPC gateway

Haskell libraries

wereHamster/etcd-hs

R libraries

ropensci/etseed

Nim libraries

etcd_client

Tcl libraries

efrecon/etcd-tcl - Supports v2, except wait.

Rust libraries

jimmycuadra/rust-etcd - Supports v2

Gradle Plugins

gradle-etcd-rest-plugin - Supports v2

Chef Integration

coderanger/etcd-chef

Chef Cookbook

spheromak/etcd-cookbook

BOSH Releases

Projects using etcd

etcd Raft users - projects using etcd’s raft library implementation.
apache/celix - an implementation of the OSGi specification adapted to C and C++
binocarlos/yoda - etcd + ZeroMQ
blox/blox - a collection of open source projects for container management and orchestration with AWS ECS
calavera/active-proxy - HTTP Proxy configured with etcd
chain/chain - software designed to operate and connect to highly scalable permissioned blockchain networks
derekchiang/etcdplus - A set of distributed synchronization primitives built upon etcd
go-discover - service discovery in Go
gleicon/goreman - Branch of the Go Foreman clone with etcd support
garethr/hiera-etcd - Puppet hiera backend using etcd
mattn/etcd-vim - SET and GET keys from inside vim
mattn/etcdenv - “env” shebang with etcd integration
kelseyhightower/confd - Manage local app config files using templates and data from etcd
configdb - A REST relational abstraction on top of arbitrary database backends, aimed at storing configs and inventories.
kubernetes/kubernetes - Container cluster manager introduced by Google.
mailgun/vulcand - HTTP proxy that uses etcd as a configuration backend.
duedil-ltd/discodns - Simple DNS nameserver using etcd as a database for names and records.
skynetservices/skydns - RFC compliant DNS server
xordataexchange/crypt - Securely store values in etcd using GPG encryption
spf13/viper - Go configuration library, reads values from ENV, pflags, files, and etcd with optional encryption
lytics/metafora - Go distributed task library
ryandoyle/nss-etcd - A GNU libc NSS module for resolving names from etcd.
Gru - Orchestration made easy with Go
Vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
lclarkmichalek/etcdhcp - DHCP server that uses etcd for persistence and coordination.
openstack/networking-vpp - A networking driver that programs the FD.io VPP dataplane to provide OpenStack cloud virtual networking
OpenStack - OpenStack services can rely on etcd as a base service.
CoreDNS - CoreDNS is a DNS server that chains plugins, part of CNCF and Kubernetes
Uber M3 - M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus
Rook - Storage Orchestration for Kubernetes
Patroni - A template for PostgreSQL High Availability with ZooKeeper, etcd, or Consul
Trillian - Trillian implements a Merkle tree whose contents are served from a data storage layer, to allow scalability to extremely large trees.

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

The naming of metrics follows the suggested Prometheus best practices. A metric name has an etcd or etcd_debugging prefix as its namespace and a subsystem prefix (for example wal and etcdserver).

etcd namespace metrics

The metrics under the etcd prefix are for monitoring and alerting. They are stable high level metrics. If there is any change of these metrics, it will be included in release notes.

Metrics that are etcd2 related are documented v2 metrics guide.

Server

These metrics describe the status of the etcd server. In order to detect outages or problems for troubleshooting, the server metrics of every production etcd cluster should be closely monitored.

All these metrics are prefixed with etcd_server_

Name	Description	Type
has_leader	Whether or not a leader exists. 1 is existence, 0 is not.	Gauge
leader_changes_seen_total	The number of leader changes seen.	Counter
proposals_committed_total	The total number of consensus proposals committed.	Gauge
proposals_applied_total	The total number of consensus proposals applied.	Gauge
proposals_pending	The current number of pending proposals.	Gauge
proposals_failed_total	The total number of failed proposals seen.	Counter

has_leader indicates whether the member has a leader. If a member does not have a leader, it is totally unavailable. If all the members in the cluster do not have any leader, the entire cluster is totally unavailable.

leader_changes_seen_total counts the number of leader changes the member has seen since its start. Rapid leadership changes impact the performance of etcd significantly. It also signals that the leader is unstable, perhaps due to network connectivity issues or excessive load hitting the etcd cluster.

proposals_committed_total records the total number of consensus proposals committed. This gauge should increase over time if the cluster is healthy. Several healthy members of an etcd cluster may have different total committed proposals at once. This discrepancy may be due to recovering from peers after starting, lagging behind the leader, or being the leader and therefore having the most commits. It is important to monitor this metric across all the members in the cluster; a consistently large lag between a single member and its leader indicates that member is slow or unhealthy.

proposals_applied_total records the total number of consensus proposals applied. The etcd server applies every committed proposal asynchronously. The difference between proposals_committed_total and proposals_applied_total should usually be small (within a few thousands even under high load). If the difference between them continues to rise, it indicates that the etcd server is overloaded. This might happen when applying expensive queries like heavy range queries or large txn operations.

proposals_pending indicates how many proposals are queued to commit. Rising pending proposals suggests there is a high client load or the member cannot commit proposals.

proposals_failed_total are normally related to two issues: temporary failures related to a leader election or longer downtime caused by a loss of quorum in the cluster.

Disk

These metrics describe the status of the disk operations.

All these metrics are prefixed with etcd_disk_.

Name	Description	Type
wal_fsync_duration_seconds	The latency distributions of fsync called by wal	Histogram
backend_commit_duration_seconds	The latency distributions of commit called by backend.	Histogram

A wal_fsync is called when etcd persists its log entries to disk before applying them.

A backend_commit is called when etcd commits an incremental snapshot of its most recent changes to disk.

High disk operation latencies (wal_fsync_duration_seconds or backend_commit_duration_seconds) often indicate disk issues. It may cause high request latency or make the cluster unstable.

Network

These metrics describe the status of the network.

All these metrics are prefixed with etcd_network_

Name	Description	Type
peer_sent_bytes_total	The total number of bytes sent to the peer with ID `To`.	Counter(To)
peer_received_bytes_total	The total number of bytes received from the peer with ID `From`.	Counter(From)
peer_sent_failures_total	The total number of send failures from the peer with ID `To`.	Counter(To)
peer_received_failures_total	The total number of receive failures from the peer with ID `From`.	Counter(From)
peer_round_trip_time_seconds	Round-Trip-Time histogram between peers.	Histogram(To)
client_grpc_sent_bytes_total	The total number of bytes sent to grpc clients.	Counter
client_grpc_received_bytes_total	The total number of bytes received to grpc clients.	Counter

peer_sent_bytes_total counts the total number of bytes sent to a specific peer. Usually the leader member sends more data than other members since it is responsible for transmitting replicated data.

peer_received_bytes_total counts the total number of bytes received from a specific peer. Usually follower members receive data only from the leader member.

gRPC requests

These metrics are exposed via go-grpc-prometheus.

etcd_debugging namespace metrics

The metrics under the etcd_debugging prefix are for debugging. They are very implementation dependent and volatile. They might be changed or removed without any warning in new etcd releases. Some of the metrics might be moved to the etcd prefix when they become more stable.

Snapshot

Name	Description	Type
snapshot_save_total_duration_seconds	The total latency distributions of save called by snapshot	Histogram

Abnormally high snapshot duration (snapshot_save_total_duration_seconds) indicates disk issues and might cause the cluster to be unstable.

Prometheus supplied metrics

The Prometheus client library provides a number of metrics under the go and process namespaces. There are a few that are particularly interesting.

Name	Description	Type
process_open_fds	Number of open file descriptors.	Gauge
process_max_fds	Maximum number of open file descriptors.	Gauge

Heavy file descriptor (process_open_fds) usage (i.e., near the process’s file descriptor limit, process_max_fds) indicates a potential file descriptor exhaustion issue. If the file descriptors are exhausted, etcd may panic because it cannot create new WAL files.

Generated list of metrics

v3.3.0	v3.3.1	v3.3.10	v3.3.11	v3.3.12	v3.3.13
v3.3.14	v3.3.15	v3.3.16	v3.3.17	v3.3.18	v3.3.19
v3.3.2	v3.3.20	v3.3.21	v3.3.22	v3.3.23	v3.3.24
v3.3.3	v3.3.4	v3.3.5	v3.3.6	v3.3.7	v3.3.8
v3.3.9

Docs: How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

curl https://discovery.etcd.io/new?size=3
https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

# grab this token
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
DISCOVERY=https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Now etcd is ready! To connect to etcd with etcdctl:

export ETCDCTL_API=3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379

etcdctl --endpoints=$ENDPOINTS member list

Docs: How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

curl https://discovery.etcd.io/new?size=3
https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

# grab this token
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
DISCOVERY=https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Now etcd is ready! To connect to etcd with etcdctl:

export ETCDCTL_API=3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379

etcdctl --endpoints=$ENDPOINTS member list

Docs: How to access etcd

Mon, 01 Jan 0001 00:00:00 +0000

put command to write:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

get to read from etcd:

etcdctl --endpoints=$ENDPOINTS get foo
etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo

Docs: How to access etcd

Mon, 01 Jan 0001 00:00:00 +0000

put command to write:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

get to read from etcd:

etcdctl --endpoints=$ENDPOINTS get foo
etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo

Docs: How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

Docs: How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

Docs: How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

Docs: How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

Docs: How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

txn to wrap multiple requests into one transaction:

etcdctl --endpoints=$ENDPOINTS put user1 bad
etcdctl --endpoints=$ENDPOINTS txn --interactive

compares:
value("user1") = "bad"

success requests (get, put, delete):
del user1

failure requests (get, put, delete):
put user1 good

Docs: How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

txn to wrap multiple requests into one transaction:

etcdctl --endpoints=$ENDPOINTS put user1 bad
etcdctl --endpoints=$ENDPOINTS txn --interactive

compares:
value("user1") = "bad"

success requests (get, put, delete):
del user1

failure requests (get, put, delete):
put user1 good

Docs: How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

Docs: How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

Docs: How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

Docs: How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

Docs: How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS lock mutex1

Docs: How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS lock mutex1

Docs: How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

elect for leader election:

etcdctl --endpoints=$ENDPOINTS elect one p1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS elect one p2

Docs: How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

elect for leader election:

etcdctl --endpoints=$ENDPOINTS elect one p1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS elect one p2

Docs: Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd --logger=zap
{"level":"info","ts":"2021-09-20T08:19:31.340-0400","caller":"etcdmain/etcd.go:110","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:
```
$ etcdctl put greeting "Hello, etcd"
OK
```

From the same terminal, retrieve the key:

$ etcdctl get greeting
greeting
Hello, etcd

What’s next?

Learn about more ways to configure and use etcd from the following pages:

Explore the gRPC API.
Set up a multi-machine cluster.
Learn how to configure etcd.
Find language bindings and tools.
Use TLS to secure an etcd cluster.
Tune etcd.

Docs: Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd
{"level":"info","ts":"2021-09-17T09:19:32.783-0400","caller":"etcdmain/etcd.go:72","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:
```
$ etcdctl put greeting "Hello, etcd"
OK
```

From the same terminal, retrieve the key:

$ etcdctl get greeting
greeting
Hello, etcd

What’s next?

Learn about more ways to configure and use etcd from the following pages:

Explore the gRPC API.
Set up a multi-machine cluster.
Learn how to configure etcd.
Find language bindings and tools.
Use TLS to secure an etcd cluster.
Tune etcd.

Docs: Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd
{"level":"info","ts":"2021-09-17T09:19:32.783-0400","caller":"etcdmain/etcd.go:72","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:
```
$ etcdctl put greeting "Hello, etcd"
OK
```

From the same terminal, retrieve the key:

$ etcdctl get greeting
greeting
Hello, etcd

What’s next?

Learn about more ways to configure and use etcd from the following pages:

Explore the gRPC API.
Set up a multi-machine cluster.
Learn how to configure etcd.
Find language bindings and tools.
Use TLS to secure an etcd cluster.
Tune etcd.

Docs: How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Specify the initial cluster configuration for each machine:

etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.5.0 | 45 kB | true | false | 4 | 16726 | 16726 | |
| 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.5.0 | 45 kB | false | false | 4 | 16726 | 16726 | |
| 10.240.0.19:2379 | 94df724b66343e6c | 3.5.0 | 45 kB | false | false | 4 | 16726 | 16726 | |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------|

etcdctl --endpoints=$ENDPOINTS endpoint health

10.240.0.17:2379 is healthy: successfully committed proposal: took = 3.345431ms
10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.767967ms
10.240.0.18:2379 is healthy: successfully committed proposal: took = 4.025451ms

Docs: How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Specify the initial cluster configuration for each machine:

etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.5.0 | 45 kB | true | false | 4 | 16726 | 16726 | |
| 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.5.0 | 45 kB | false | false | 4 | 16726 | 16726 | |
| 10.240.0.19:2379 | 94df724b66343e6c | 3.5.0 | 45 kB | false | false | 4 | 16726 | 16726 | |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------|

etcdctl --endpoints=$ENDPOINTS endpoint health

10.240.0.17:2379 is healthy: successfully committed proposal: took = 3.345431ms
10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.767967ms
10.240.0.18:2379 is healthy: successfully committed proposal: took = 4.025451ms

Docs: Overview

Mon, 01 Jan 0001 00:00:00 +0000

The etcd v3 API is designed to give users a more efficient and cleaner abstraction compared to etcd v2. There are a number of semantic and protocol changes in this new API.

To prove out the design of the v3 API the team has also built a number of example recipes, there is a video discussing these recipes too.

Design

Flatten binary key-value space
Keep the event history until compaction
- access to old version of keys
- user controlled history compaction
Support range query
- Pagination support with limit argument
- Support consistency guarantee across multiple range queries
Replace TTL key with Lease
- more efficient/ low cost keep alive
- a logical group of TTL keys
Replace CAS/CAD with multi-object Txn
- MUCH MORE powerful and flexible
Support efficient watching with multiple ranges
RPC API supports the completed set of APIs.
- more efficient than JSON/HTTP
- additional txn/lease support
HTTP API supports a subset of APIs.
- easy for people to try out etcd
- easy for people to write simple etcd application

Notes

Request Size Limitation

The max request size is around 1MB. Since etcd replicates requests in a streaming fashion, a very large request might block other requests for a long time. The use case for etcd is to store small configuration values, so we prevent user from submitting large requests. This also applies to Txn requests. We might loosen the size in the future a little bit or make it configurable.

Protobuf Defined API

api protobuf

kv protobuf

Examples

Put a key (foo=bar)

// A put is always successful
Put( PutRequest { key = foo, value = bar } )
PutResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 1,
raft_term = 0x1,
}

Get a key (assume we have foo=bar)

Get ( RangeRequest { key = foo } )
RangeResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 1,
raft_term = 0x1,
kvs = {
{
key = foo,
value = bar,
create_revision = 1,
mod_revision = 1,
version = 1;
},
},
}

Range over a key space (assume we have foo0=bar0… foo100=bar100)

Range ( RangeRequest { key = foo, end_key = foo80, limit = 30 } )
RangeResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 100,
raft_term = 0x1,
kvs = {
{
key = foo0,
value = bar0,
create_revision = 1,
mod_revision = 1,
version = 1;
},
...,
{
key = foo30,
value = bar30,
create_revision = 30,
mod_revision = 30,
version = 1;
},
},
}

Finish a txn (assume we have foo0=bar0, foo1=bar1)

Txn(TxnRequest {
// mod_revision of foo0 is equal to 1, mod_revision of foo1 is greater than 1
compare = {
{compareType = equal, key = foo0, mod_revision = 1},
{compareType = greater, key = foo1, mod_revision = 1}}
},
// if the comparison succeeds, put foo2 = bar2
success = {PutRequest { key = foo2, value = success }},
// if the comparison fails, put foo2=fail
failure = {PutRequest { key = foo2, value = failure }},
)
TxnResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 3,
raft_term = 0x1,
succeeded = true,
responses = {
// response of PUT foo2=success
{
cluster_id = 0x1000,
member_id = 0x1,
revision = 3,
raft_term = 0x1,
}
}
}

Watch on a key/range

Watch( WatchRequest{
key = foo,
end_key = fop, // prefix foo
start_revision = 20,
end_revision = 10000,
// server decided notification frequency
progress_notification = true,
}
… // this can be a watch request stream
)
// put (foo0=bar0) event at 3
WatchResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 3,
raft_term = 0x1,
event_type = put,
kv = {
key = foo0,
value = bar0,
create_revision = 1,
mod_revision = 1,
version = 1;
},
}
…
// a notification at 2000
WatchResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 2000,
raft_term = 0x1,
// nil event as notification
}
…
// put (foo0=bar3000) event at 3000
WatchResponse {
cluster_id = 0x1000,
member_id = 0x1,
revision = 3000,
raft_term = 0x1,
event_type = put,
kv = {
key = foo0,
value = bar3000,
create_revision = 1,
mod_revision = 3000,
version = 2;
},
}
…

Docs: Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

curl 'https://discovery.etcd.io/new?size=3'
https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

# grab this token
TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
DISCOVERY=https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
 --discovery ${DISCOVERY} \
 --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Now etcd is ready! To connect to etcd with etcdctl:

export ETCDCTL_API=3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379

etcdctl --endpoints=$ENDPOINTS member list

Access etcd

put command to write:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

get to read from etcd:

etcdctl --endpoints=$ENDPOINTS get foo
etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo

Get by prefix

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

Delete

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

Transactional write

txn to wrap multiple requests into one transaction:

etcdctl --endpoints=$ENDPOINTS put user1 bad
etcdctl --endpoints=$ENDPOINTS txn --interactive

compares:
value("user1") = "bad"

success requests (get, put, delete):
del user1

failure requests (get, put, delete):
put user1 good

Watch

watch to get notified of future changes:

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

Lease

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

Distributed locks

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS lock mutex1

Elections

elect for leader election:

etcdctl --endpoints=$ENDPOINTS elect one p1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS elect one p2

Cluster status

Specify the initial cluster configuration for each machine:

etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.4.15 | 45 kB | true | false | 4 | 16726 | 16726 | |
| 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.4.15 | 45 kB | false | false | 4 | 16726 | 16726 | |
| 10.240.0.19:2379 | 94df724b66343e6c | 3.4.15 | 45 kB | false | false | 4 | 16726 | 16726 | |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------|

etcdctl --endpoints=$ENDPOINTS endpoint health

10.240.0.17:2379 is healthy: successfully committed proposal: took = 3.345431ms
10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.767967ms
10.240.0.18:2379 is healthy: successfully committed proposal: took = 4.025451ms

Snapshot

snapshot to save point-in-time snapshot of etcd database:

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

ENDPOINTS=$HOST_1:2379
etcdctl --endpoints=$ENDPOINTS snapshot save my.db

Snapshot saved at my.db

etcdctl --write-out=table --endpoints=$ENDPOINTS snapshot status my.db

+---------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+---------+----------+------------+------------+
| c55e8b8 | 9 | 13 | 25 kB |
+---------+----------+------------+------------+

Migrate

migrate to transform etcd v2 to v3 data:

# write key in etcd version 2 store
export ETCDCTL_API=2
etcdctl --endpoints=http://$ENDPOINT set foo bar

# read key in etcd v2
etcdctl --endpoints=$ENDPOINTS --output="json" get foo

# stop etcd node to migrate, one by one

# migrate v2 data
export ETCDCTL_API=3
etcdctl --endpoints=$ENDPOINT migrate --data-dir="default.etcd" --wal-dir="default.etcd/member/wal"

# restart etcd node after migrate, one by one

# confirm that the key got migrated
etcdctl --endpoints=$ENDPOINTS get /foo

Member

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

# get member ID
export ETCDCTL_API=3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list

# remove the member
MEMBER_ID=278c654c9a6dfd3b
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 \
 member remove ${MEMBER_ID}

# add a new member (node 4)
export ETCDCTL_API=3
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_4=10.240.0.16 # new member
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379 \
 member add ${NAME_4} \
 --peer-urls=http://${HOST_4}:2380

Next, start the new member with --initial-cluster-state existing flag:

# [WARNING] If the new member starts from the same disk space,
# make sure to remove the data directory of the old member
#
# restart with 'existing' flag
TOKEN=my-etcd-token-1
CLUSTER_STATE=existing
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_4=10.240.0.16 # new member
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_4}=http://${HOST_4}:2380

THIS_NAME=${NAME_4}
THIS_IP=${HOST_4}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Docs: Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Docs: Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Docs: How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

snapshot to save point-in-time snapshot of etcd database:

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

ENDPOINTS=$HOST_1:2379
etcdctl --endpoints=$ENDPOINTS snapshot save my.db

Snapshot saved at my.db

etcdutl --write-out=table snapshot status my.db

+---------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+---------+----------+------------+------------+
| c55e8b8 | 9 | 13 | 25 kB |
+---------+----------+------------+------------+

Docs: How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

snapshot to save point-in-time snapshot of etcd database:

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

ENDPOINTS=$HOST_1:2379
etcdctl --endpoints=$ENDPOINTS snapshot save my.db

Snapshot saved at my.db

etcdutl --write-out=table snapshot status my.db

+---------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+---------+----------+------------+------------+
| c55e8b8 | 9 | 13 | 25 kB |
+---------+----------+------------+------------+

Docs: Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.4.30 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

From a shell, test that etcd is in your path:

$ etcd --version
etcd Version: 3.4.30
...

Build from source

If you have Go version 1.2+, you can build etcd from source by following these steps:

Download the etcd repo as a zip file and unzip it, or clone the repo using the following command.
```
$ git clone -b v3.4.30 https://github.com/etcd-io/etcd.git
```
To build from main@HEAD, omit the -b v3.4.30 flag.
Change directory:
```
$ cd etcd
```
Run the build script:
```
$ ./build
```
The binaries are under the bin directory.
Add the full path to the bin directory to your path, for example:
```
$ export PATH="$PATH:`pwd`/bin"
```
Test that etcd is in your path:
```
$ etcd --version
```

Installation check

For a slightly more involved sanity check of your installation, see Quickstart.

Docs: Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.5.12 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

From a shell, test that etcd is in your path:

$ etcd --version
etcd Version: 3.5.12
...

Build from source

If you have Go version 1.2+, you can build etcd from source by following these steps:

Download the etcd repo as a zip file and unzip it, or clone the repo using the following command.
```
$ git clone -b v3.5.12 https://github.com/etcd-io/etcd.git
```
To build from main@HEAD, omit the -b v3.5.12 flag.
Change directory:
```
$ cd etcd
```
Run the build script:
```
$ ./build.sh
```
The binaries are under the bin directory.
Add the full path to the bin directory to your path, for example:
```
$ export PATH="$PATH:`pwd`/bin"
```
Test that etcd is in your path:
```
$ etcd --version
```

Installation via OS packages

Disclaimer: etcd installations through OS package managers can deliver outdated versions since they are not being automatically maintained nor officially supported by etcd project. Therefore use OS packages with caution.

There are various ways of installing etcd on different operating systems and these are just some examples how it can be done.

MacOS (Homebrew)

Update homebrew:

$ brew update

Install etcd:

$ brew install etcd

Verify install

$ etcd --version

Linux

TBD

Installation as part of Kubernetes installation

TBD—Help Wanted

Installation on Kubernetes, using a statefulset or helm chart

The etcd project does not currently maintain a helm chart, however you can follow the instructions provided by Bitnami’s etcd Helm chart.

Installation check

For a slightly more involved sanity check of your installation, see Quickstart.

Docs: Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.6.0-alpha.0 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

From a shell, test that etcd is in your path:

$ etcd --version
etcd Version: 3.6.0-alpha.0
...

Build from source

If you have Go version 1.21+, you can build etcd from source by following these steps:

Download the etcd repo as a zip file and unzip it, or clone the repo using the following command.
```
$ git clone -b v3.6.0-alpha.0 https://github.com/etcd-io/etcd.git
```
To build from main@HEAD, omit the -b v3.6.0-alpha.0 flag.
Change directory:
```
$ cd etcd
```
Run the build script:
```
$ ./scripts/build.sh
```
The binaries are under the bin directory.
Add the full path to the bin directory to your path, for example:
```
$ export PATH="$PATH:`pwd`/bin"
```
Test that etcd is in your path:
```
$ etcd --version
```

Installation via OS packages

There are various ways of installing etcd on different operating systems and these are just some examples how it can be done.

MacOS (Homebrew)

Update homebrew:

$ brew update

Install etcd:

$ brew install etcd

Verify install

$ etcd --version

Linux

TBD

Installation as part of Kubernetes installation

TBD—Help Wanted

Installation on Kubernetes, using a statefulset or helm chart

The etcd project does not currently maintain a helm chart, however you can follow the instructions provided by Bitnami’s etcd Helm chart.

Installation check

For a slightly more involved sanity check of your installation, see Quickstart.

Docs: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

Do clients have to send requests to the etcd leader?

Raft is leader-based; the leader handles all client requests which need cluster consensus. However, the client does not need to know which node is the leader. Any request that requires consensus sent to a follower is automatically forwarded to the leader. Requests that do not require consensus (e.g., serialized reads) can be processed by any cluster member.

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

listen-client-urls and listen-peer-urls specify the local addresses etcd server binds to for accepting incoming connections. To listen on a port for all interfaces, specify 0.0.0.0 as the listen IP address.

advertise-client-urls and initial-advertise-peer-urls specify the addresses etcd clients or other etcd members should use to contact the etcd server. The advertise addresses must be reachable from the remote machines. Do not advertise addresses like localhost or 0.0.0.0 for a production setup since these addresses are unreachable from remote machines.

Why doesn’t changing `--listen-peer-urls` or `--initial-advertise-peer-urls` update the advertised peer URLs in `etcdctl member list`?

A member’s advertised peer URLs come from --initial-advertise-peer-urls on initial cluster boot. Changing the listen peer URLs or the initial advertise peers after booting the member won’t affect the exported advertise peer URLs since changes must go through quorum to avoid membership configuration split brain. Use etcdctl member update to update a member’s peer URLs.

Deployment

System requirements

Since etcd writes data to disk, its performance strongly depends on disk performance. For this reason, SSD is highly recommended. To assess whether a disk is fast enough for etcd, one possibility is using a disk benchmarking tool such as fio. For an example on how to do that, read here. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a configurable storage size quota set to 2GB by default. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. 8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD at the very least. Note that performance is intrinsically workload dependent; please test before production deployment. See hardware for more recommendations.

Most stable production environment is Linux operating system with amd64 architecture; see supported platform for more.

Why an odd number of cluster members?

An etcd cluster needs a majority of nodes, a quorum, to agree on updates to the cluster state. For a cluster with n members, quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for quorum. Although adding a node to an odd-sized cluster appears better since there are more machines, the fault tolerance is worse since exactly the same number of nodes may fail without losing quorum but there are more nodes that can fail. If the cluster is in a state where it can’t tolerate any more failures, adding a node before removing nodes is dangerous because if the new node fails to register with the cluster (e.g., the address is misconfigured), quorum will be permanently lost.

What is maximum cluster size?

Theoretically, there is no hard limit. However, an etcd cluster probably should have no more than seven nodes. Google Chubby lock service, similar to etcd and widely deployed within Google for many years, suggests running five nodes. A 5-member etcd cluster can tolerate two member failures, which is enough in most cases. Although larger clusters provide better fault tolerance, the write performance suffers because data must be replicated across more machines.

What is failure tolerance?

An etcd cluster operates so long as a member quorum can be established. If quorum is lost through transient network failures (e.g., partitions), etcd automatically and safely resumes once the network recovers and restores quorum; Raft enforces cluster consistency. For power loss, etcd persists the Raft log to disk; etcd replays the log to the point of failure and resumes cluster participation. For permanent hardware failure, the node may be removed from the cluster through runtime reconfiguration.

It is recommended to have an odd number of members in a cluster. An odd-size cluster tolerates the same number of failures as an even-size cluster but with fewer nodes. The difference can be seen by comparing even and odd sized clusters:

Cluster Size	Majority	Failure Tolerance
1	1	0
2	2	0
3	2	1
4	3	1
5	3	2
6	4	2
7	4	3
8	5	3
9	5	4

Adding a member to bring the size of cluster up to an even number doesn’t buy additional fault tolerance. Likewise, during a network partition, an odd number of members guarantees that there will always be a majority partition that can continue to operate and be the source of truth when the partition ends.

Does etcd work in cross-region or cross data center deployments?

Deploying etcd across regions improves etcd’s fault tolerance since members are in separate failure domains. The cost is higher consensus request latency from crossing data center boundaries. Since etcd relies on a member quorum for consensus, the latency from crossing data centers will be somewhat pronounced because at least a majority of cluster members must respond to consensus requests. Additionally, cluster data must be replicated across all peers, so there will be bandwidth cost as well.

With longer latencies, the default etcd configuration may cause frequent elections or heartbeat timeouts. See tuning for adjusting timeouts for high latency deployments.

Operation

How to backup a etcd cluster?

etcdctl provides a snapshot command to create backups. See backup for more details.

Should I add a member before removing an unhealthy member?

When replacing an etcd node, it’s important to remove the member first and then add its replacement.

etcd employs distributed consensus based on a quorum model; (n/2)+1 members, a majority, must agree on a proposal before it can be committed to the cluster. These proposals include key-value updates and membership changes. This model totally avoids any possibility of split brain inconsistency. The consequence of permanent quorum loss is catastrophic.

How this applies to membership: If a 3-member cluster has 1 downed member, it can still make forward progress because the quorum is 2 and 2 members are still live. However, adding a new member to a 3-member cluster will increase the quorum to 3 because 3 votes are required for a majority of 4 members. Since the quorum increased, this extra member buys nothing in terms of fault tolerance; the cluster is still one node failure away from being unrecoverable.

Additionally, that new member is risky because it may turn out to be misconfigured or incapable of joining the cluster. In that case, there’s no way to recover quorum because the cluster has two members down and two members up, but needs three votes to change membership to undo the botched membership addition. etcd will by default reject member add attempts that could take down the cluster in this manner.

On the other hand, if the downed member is removed from cluster membership first, the number of members becomes 2 and the quorum remains at 2. Following that removal by adding a new member will also keep the quorum steady at 2. So, even if the new node can’t be brought up, it’s still possible to remove the new member through quorum on the remaining live members.

Why won’t etcd accept my membership changes?

etcd sets strict-reconfig-check in order to reject reconfiguration requests that would cause quorum loss. Abandoning quorum is really risky (especially when the cluster is already unhealthy). Although it may be tempting to disable quorum checking if there’s quorum loss to add a new member, this could lead to full fledged cluster inconsistency. For many applications, this will make the problem even worse (“disk geometry corruption” being a candidate for most terrifying).

Why does etcd lose its leader from disk latency spikes?

This is intentional; disk latency is part of leader liveness. Suppose the cluster leader takes a minute to fsync a raft log update to disk, but the etcd cluster has a one second election timeout. Even though the leader can process network messages within the election interval (e.g., send heartbeats), it’s effectively unavailable because it can’t commit any new proposals; it’s waiting on the slow disk. If the cluster frequently loses its leader due to disk latencies, try tuning the disk settings or etcd time parameters.

What does the etcd warning “request ignored (cluster ID mismatch)” mean?

Every new etcd cluster generates a new cluster ID based on the initial cluster configuration and a user-provided unique initial-cluster-token value. By having unique cluster ID’s, etcd is protected from cross-cluster interaction which could corrupt the cluster.

Usually this warning happens after tearing down an old cluster, then reusing some of the peer addresses for the new cluster. If any etcd process from the old cluster is still running it will try to contact the new cluster. The new cluster will recognize a cluster ID mismatch, then ignore the request and emit this warning. This warning is often cleared by ensuring peer addresses among distinct clusters are disjoint.

What does “mvcc: database space exceeded” mean and how do I fix it?

The multi-version concurrency control data model in etcd keeps an exact history of the keyspace. Without periodically compacting this history (e.g., by setting --auto-compaction), etcd will eventually exhaust its storage space. If etcd runs low on storage space, it raises a space quota alarm to protect the cluster from further writes. So long as the alarm is raised, etcd responds to write requests with the error mvcc: database space exceeded.

To recover from the low space quota alarm:

Compact etcd’s history.
Defragment every etcd endpoint.
Disarm the alarm.

What does the etcd warning “etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:43020: read: connection reset by peer” mean?

This is gRPC-side warning when a server receives a TCP RST flag with client-side streams being prematurely closed. For example, a client closes its connection, while gRPC server has not yet processed all HTTP/2 frames in the TCP queue. Some data may have been lost in server side, but it is ok so long as client connection has already been closed.

Only old versions of gRPC log this. etcd >=v3.2.13 by default log this with DEBUG level, thus only visible with --debug flag enabled.

Performance

How should I benchmark etcd?

Try the benchmark tool. Current benchmark results are available for comparison.

What does the etcd warning “apply entries took too long” mean?

After a majority of etcd members agree to commit a request, each etcd server applies the request to its data store and persists the result to disk. Even with a slow mechanical disk or a virtualized network disk, such as Amazon’s EBS or Google’s PD, applying a request should normally take fewer than 50 milliseconds. If the average apply duration exceeds 100 milliseconds, etcd will warn that entries are taking too long to apply.

Usually this issue is caused by a slow disk. The disk could be experiencing contention among etcd and other applications, or the disk is too simply slow (e.g., a shared virtualized disk). To rule out a slow disk from causing this warning, monitor backend_commit_duration_seconds (p99 duration should be less than 25ms) to confirm the disk is reasonably fast. If the disk is too slow, assigning a dedicated disk to etcd or using faster disk will typically solve the problem.

The second most common cause is CPU starvation. If monitoring of the machine’s CPU usage shows heavy utilization, there may not be enough compute capacity for etcd. Moving etcd to dedicated machine, increasing process resource isolation cgroups, or renicing the etcd server process into a higher priority can usually solve the problem.

Expensive user requests which access too many keys (e.g., fetching the entire keyspace) can also cause long apply latencies. Accessing fewer than a several hundred keys per request, however, should always be performant.

If none of the above suggestions clear the warnings, please open an issue with detailed logging, monitoring, metrics and optionally workload information.

What does the etcd warning “failed to send out heartbeat on time” mean?

etcd uses a leader-based consensus protocol for consistent data replication and log execution. Cluster members elect a single leader, all other members become followers. The elected leader must periodically send heartbeats to its followers to maintain its leadership. Followers infer leader failure if no heartbeats are received within an election interval and trigger an election. If a leader doesn’t send its heartbeats in time but is still running, the election is spurious and likely caused by insufficient resources. To catch these soft failures, if the leader skips two heartbeat intervals, etcd will warn it failed to send a heartbeat on time.

Usually this issue is caused by a slow disk. Before the leader sends heartbeats attached with metadata, it may need to persist the metadata to disk. The disk could be experiencing contention among etcd and other applications, or the disk is too simply slow (e.g., a shared virtualized disk). To rule out a slow disk from causing this warning, monitor wal_fsync_duration_seconds (p99 duration should be less than 10ms) to confirm the disk is reasonably fast. If the disk is too slow, assigning a dedicated disk to etcd or using faster disk will typically solve the problem. To tell whether a disk is fast enough for etcd, a benchmarking tool such as fio can be used. Read here for an example.

The second most common cause is CPU starvation. If monitoring of the machine’s CPU usage shows heavy utilization, there may not be enough compute capacity for etcd. Moving etcd to dedicated machine, increasing process resource isolation with cgroups, or renicing the etcd server process into a higher priority can usually solve the problem.

A slow network can also cause this issue. If network metrics among the etcd machines shows long latencies or high drop rate, there may not be enough network capacity for etcd. Moving etcd members to a less congested network will typically solve the problem. However, if the etcd cluster is deployed across data centers, long latency between members is expected. For such deployments, tune the heartbeat-interval configuration to roughly match the round trip time between the machines, and the election-timeout configuration to be at least 5 * heartbeat-interval. See tuning documentation for detailed information.

If none of the above suggestions clear the warnings, please open an issue with detailed logging, monitoring, metrics and optionally workload information.

What does the etcd warning “snapshotting is taking more than x seconds to finish …” mean?

etcd sends a snapshot of its complete key-value store to refresh slow followers and for backups. Slow snapshot transfer times increase MTTR; if the cluster is ingesting data with high throughput, slow followers may livelock by needing a new snapshot before finishing receiving a snapshot. To catch slow snapshot performance, etcd warns when sending a snapshot takes more than thirty seconds and exceeds the expected transfer time for a 1Gbps connection.

Docs: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

What is etcd?

etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.

How do you pronounce etcd?

etcd is pronounced /ˈɛtsiːdiː/, and means “distributed etc directory.”

Do clients have to send requests to the etcd leader?

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

Why doesn’t changing `--listen-peer-urls` or `--initial-advertise-peer-urls` update the advertised peer URLs in `etcdctl member list`?

Deployment

System requirements

Most stable production environment is Linux operating system with amd64 architecture; see supported platform for more.

Why an odd number of cluster members?

What is maximum cluster size?

What is failure tolerance?

Cluster Size	Majority	Failure Tolerance
1	1	0
2	2	0
3	2	1
4	3	1
5	3	2
6	4	2
7	4	3
8	5	3
9	5	4

Does etcd work in cross-region or cross data center deployments?

With longer latencies, the default etcd configuration may cause frequent elections or heartbeat timeouts. See tuning for adjusting timeouts for high latency deployments.

Operation

How to backup a etcd cluster?

etcdctl provides a snapshot command to create backups. See backup for more details.

Should I add a member before removing an unhealthy member?

When replacing an etcd node, it’s important to remove the member first and then add its replacement.

Why won’t etcd accept my membership changes?

Why does etcd lose its leader from disk latency spikes?

What does the etcd warning “request ignored (cluster ID mismatch)” mean?

What does “mvcc: database space exceeded” mean and how do I fix it?

To recover from the low space quota alarm:

Compact etcd’s history.
Defragment every etcd endpoint.
Disarm the alarm.

What does the etcd warning “etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:43020: read: connection reset by peer” mean?

Only old versions of gRPC log this. etcd >=v3.2.13 by default log this with DEBUG level, thus only visible with --log-level=debug flag enabled.

Performance

How should I benchmark etcd?

Try the benchmark tool. Current benchmark results are available for comparison.

What does the etcd warning “apply entries took too long” mean?

If none of the above suggestions clear the warnings, please open an issue with detailed logging, monitoring, metrics and optionally workload information.

What does the etcd warning “failed to send out heartbeat on time” mean?

The second most common cause is CPU starvation. If monitoring of the machine’s CPU usage shows heavy utilization, there may not be enough compute capacity for etcd. Moving etcd to dedicated machine, increasing process resource isolation with cgroups, or renicing the etcd server process into a higher priority can usually solve the problem.

If none of the above suggestions clear the warnings, please open an issue with detailed logging, monitoring, metrics and optionally workload information.

What does the etcd warning “snapshotting is taking more than x seconds to finish …” mean?

Docs: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

What is etcd?

etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.

How do you pronounce etcd?

etcd is pronounced /ˈɛtsiːdiː/, and means “distributed etc directory.”

Do clients have to send requests to the etcd leader?

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

Why doesn’t changing `--listen-peer-urls` or `--initial-advertise-peer-urls` update the advertised peer URLs in `etcdctl member list`?

Deployment

System requirements

Most stable production environment is Linux operating system with amd64 architecture; see supported platform for more.

Why an odd number of cluster members?

What is maximum cluster size?

What is failure tolerance?

Cluster Size	Majority	Failure Tolerance
1	1	0
2	2	0
3	2	1
4	3	1
5	3	2
6	4	2
7	4	3
8	5	3
9	5	4

Does etcd work in cross-region or cross data center deployments?

With longer latencies, the default etcd configuration may cause frequent elections or heartbeat timeouts. See tuning for adjusting timeouts for high latency deployments.

Operation

How to backup a etcd cluster?

etcdctl provides a snapshot command to create backups. See backup for more details.

Should I add a member before removing an unhealthy member?

When replacing an etcd node, it’s important to remove the member first and then add its replacement.

Why won’t etcd accept my membership changes?

Why does etcd lose its leader from disk latency spikes?

What does the etcd warning “request ignored (cluster ID mismatch)” mean?

What does “mvcc: database space exceeded” mean and how do I fix it?

To recover from the low space quota alarm:

Compact etcd’s history.
Defragment every etcd endpoint.
Disarm the alarm.

What does the etcd warning “etcdserver/api/v3rpc: transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:43020: read: connection reset by peer” mean?

Only old versions of gRPC log this. etcd >=v3.2.13 by default log this with DEBUG level, thus only visible with --log-level=debug flag enabled.

Performance

How should I benchmark etcd?

Try the benchmark tool. Current benchmark results are available for comparison.

What does the etcd warning “apply entries took too long” mean?

If none of the above suggestions clear the warnings, please open an issue with detailed logging, monitoring, metrics and optionally workload information.

What does the etcd warning “failed to send out heartbeat on time” mean?

The second most common cause is CPU starvation. If monitoring of the machine’s CPU usage shows heavy utilization, there may not be enough compute capacity for etcd. Moving etcd to dedicated machine, increasing process resource isolation with cgroups, or renicing the etcd server process into a higher priority can usually solve the problem.

If none of the above suggestions clear the warnings, please open an issue with detailed logging, monitoring, metrics and optionally workload information.

What does the etcd warning “snapshotting is taking more than x seconds to finish …” mean?

Docs: How to migrate etcd from v2 to v3

Mon, 01 Jan 0001 00:00:00 +0000

migrate to transform etcd v2 to v3 data:

# write key in etcd version 2 store
export ETCDCTL_API=2
etcdctl --endpoints=http://$ENDPOINT set foo bar

# read key in etcd v2
etcdctl --endpoints=$ENDPOINTS --output="json" get foo

# stop etcd node to migrate, one by one

# migrate v2 data
export ETCDCTL_API=3
etcdctl --endpoints=$ENDPOINT migrate --data-dir="default.etcd" --wal-dir="default.etcd/member/wal"

# restart etcd node after migrate, one by one

# confirm that the key got migrated
etcdctl --endpoints=$ENDPOINTS get /foo

Docs: How to migrate etcd from v2 to v3

Mon, 01 Jan 0001 00:00:00 +0000

migrate to transform etcd v2 to v3 data:

# write key in etcd version 2 store
export ETCDCTL_API=2
etcdctl --endpoints=http://$ENDPOINT set foo bar

# read key in etcd v2
etcdctl --endpoints=$ENDPOINTS --output="json" get foo

# stop etcd node to migrate, one by one

# migrate v2 data
export ETCDCTL_API=3
etcdctl --endpoints=$ENDPOINT migrate --data-dir="default.etcd" --wal-dir="default.etcd/member/wal"

# restart etcd node after migrate, one by one

# confirm that the key got migrated
etcdctl --endpoints=$ENDPOINTS get /foo

Docs: How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

# get member ID
export ETCDCTL_API=3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list

# remove the member
MEMBER_ID=278c654c9a6dfd3b
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 \
 member remove ${MEMBER_ID}

# add a new member (node 4)
export ETCDCTL_API=3
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_4=10.240.0.16 # new member
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379 \
 member add ${NAME_4} \
 --peer-urls=http://${HOST_4}:2380

Next, start the new member with --initial-cluster-state existing flag:

# [WARNING] If the new member starts from the same disk space,
# make sure to remove the data directory of the old member
#
# restart with 'existing' flag
TOKEN=my-etcd-token-1
CLUSTER_STATE=existing
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_4=10.240.0.16 # new member
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_4}=http://${HOST_4}:2380

THIS_NAME=${NAME_4}
THIS_IP=${HOST_4}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

Docs: How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

# get member ID
export ETCDCTL_API=3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 member list

# remove the member
MEMBER_ID=278c654c9a6dfd3b
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379,${HOST_3}:2379 \
 member remove ${MEMBER_ID}

# add a new member (node 4)
export ETCDCTL_API=3
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_4=10.240.0.16 # new member
etcdctl --endpoints=${HOST_1}:2379,${HOST_2}:2379 \
 member add ${NAME_4} \
 --peer-urls=http://${HOST_4}:2380

Next, start the new member with --initial-cluster-state existing flag:

# [WARNING] If the new member starts from the same disk space,
# make sure to remove the data directory of the old member
#
# restart with 'existing' flag
TOKEN=my-etcd-token-1
CLUSTER_STATE=existing
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_4=etcd-node-4
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_4=10.240.0.16 # new member
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_4}=http://${HOST_4}:2380

THIS_NAME=${NAME_4}
THIS_IP=${HOST_4}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
 --listen-peer-urls http://${THIS_IP}:2380 \
 --advertise-client-urls http://${THIS_IP}:2379 \
 --listen-client-urls http://${THIS_IP}:2379 \
 --initial-cluster ${CLUSTER} \
 --initial-cluster-state ${CLUSTER_STATE} \
 --initial-cluster-token ${TOKEN}

Docs: Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Note that third-party libraries and tools (not hosted on https://github.com/etcd-io) mentioned below are not tested or maintained by the etcd team. Before using them, users are recommended to read and investigate them.

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcdhelper - An intellij platform plugin for etcd.

Go libraries

etcd/client/v3 - the officially maintained Go client for v3
etcd/client/v2 - the officially maintained Go client for v2
go-etcd - the deprecated official client. May be useful for older (<2.0.0) versions of etcd.
encWrapper - encWrapper is an encryption wrapper for the etcd client Keys API/KV.

Java libraries

coreos/jetcd - Supports v3
boonproject/etcd - Supports v2, Async/Sync and waits
justinsb/jetcd
diwakergupta/jetcd - Supports v2
jurmous/etcd4j - Supports v2, Async/Sync, waits and SSL
AdoHe/etcd4j - Supports v2 (enhance for real production cluster)
cdancy/etcd-rest - Uses jclouds to provide a complete implementation of v2 API.

Scala libraries

maciej/etcd-client - Supports v2. Akka HTTP-based fully async client
eiipii/etcdhttpclient - Supports v2. Async HTTP client based on Netty and Scala Futures.
mingchuno/etcd4s - Supports v3 using gRPC with optional Akka Stream support.

Perl libraries

hexfusion/perl-net-etcd - Supports v3 grpc gateway HTTP API
robn/p5-etcd - Supports v2

Python libraries

kragniz/python-etcd3 - Client for v3
jplana/python-etcd - Supports v2
russellhaering/txetcd - a Twisted Python library
cholcombe973/autodock - A docker deployment automation tool
lisael/aioetcd - (Python 3.4+) Asyncio coroutines client (Supports v2)
txaio-etcd - Asynchronous etcd v3-only client library for Twisted (today) and asyncio (future)
dims/etcd3-gateway - etcd v3 API library using the HTTP grpc gateway
aioetcd3 - (Python 3.6+) etcd v3 API for asyncio
Revolution1/etcd3-py - (python2.7 and python3.5+) Python client for etcd v3, using gRPC-JSON-Gateway

Node libraries

microsoft/etcd3 - Supports v3
stianeikeland/node-etcd - Supports v2 (w Coffeescript)
lavagetto/nodejs-etcd - Supports v2
deedubs/node-etcd-config - Supports v2

Ruby libraries

C libraries

apache/celix/etcdlib - Supports v2
jdarcy/etcd-api - Supports v2
shafreeck/cetcd - Supports v2

C++ libraries

edwardcapriolo/etcdcpp - Supports v2
suryanathan/etcdcpp - Supports v2 (with waits)
nokia/etcd-cpp-api - Supports v2
nokia/etcd-cpp-apiv3 - Supports v3

Clojure libraries

Erlang libraries

marshall-lee/etcd.erl - Supports v2
zhongwencool/eetcd - Supports v3+ (GRPC only)

Elixir

team-telnyx/etcdex - Supports v3+ (GRPC only)

.Net Libraries

wangjia184/etcdnet - Supports v2
drusellers/etcetera
shubhamranjan/dotnet-etcd - Supports v3+ (GRPC only)

PHP Libraries

linkorb/etcd-php
activecollab/etcd
ouqiang/etcd-php - Client for v3 gRPC gateway

Haskell libraries

wereHamster/etcd-hs

R libraries

ropensci/etseed

Nim libraries

etcd_client

Tcl libraries

efrecon/etcd-tcl - Supports v2, except wait.

Rust libraries

jimmycuadra/rust-etcd - Supports v2

Gradle Plugins

gradle-etcd-rest-plugin - Supports v2

Chef Integration

coderanger/etcd-chef

Chef Cookbook

spheromak/etcd-cookbook

BOSH Releases

Projects using etcd

etcd Raft users - projects using etcd’s raft library implementation.
apache/celix - an implementation of the OSGi specification adapted to C and C++
binocarlos/yoda - etcd + ZeroMQ
blox/blox - a collection of open source projects for container management and orchestration with AWS ECS
calavera/active-proxy - HTTP Proxy configured with etcd
chain/chain - software designed to operate and connect to highly scalable permissioned blockchain networks
derekchiang/etcdplus - A set of distributed synchronization primitives built upon etcd
go-discover - service discovery in Go
gleicon/goreman - Branch of the Go Foreman clone with etcd support
garethr/hiera-etcd - Puppet hiera backend using etcd
mattn/etcd-vim - SET and GET keys from inside vim
mattn/etcdenv - “env” shebang with etcd integration
kelseyhightower/confd - Manage local app config files using templates and data from etcd
configdb - A REST relational abstraction on top of arbitrary database backends, aimed at storing configs and inventories.
kubernetes/kubernetes - Container cluster manager introduced by Google.
mailgun/vulcand - HTTP proxy that uses etcd as a configuration backend.
duedil-ltd/discodns - Simple DNS nameserver using etcd as a database for names and records.
skynetservices/skydns - RFC compliant DNS server
xordataexchange/crypt - Securely store values in etcd using GPG encryption
spf13/viper - Go configuration library, reads values from ENV, pflags, files, and etcd with optional encryption
lytics/metafora - Go distributed task library
ryandoyle/nss-etcd - A GNU libc NSS module for resolving names from etcd.
Gru - Orchestration made easy with Go
Vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
lclarkmichalek/etcdhcp - DHCP server that uses etcd for persistence and coordination.
openstack/networking-vpp - A networking driver that programs the FD.io VPP dataplane to provide OpenStack cloud virtual networking
OpenStack - OpenStack services can rely on etcd as a base service.
CoreDNS - CoreDNS is a DNS server that chains plugins, part of CNCF and Kubernetes
Uber M3 - M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus
Rook - Storage Orchestration for Kubernetes
Patroni - A template for PostgreSQL High Availability with ZooKeeper, etcd, or Consul
Trillian - Trillian implements a Merkle tree whose contents are served from a data storage layer, to allow scalability to extremely large trees.

Docs: Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcd-defrag - An easier to use and smarter etcd defragmentation tool.
etcdhelper - An intellij platform plugin for etcd.

Libraries

The sections below list etcd client libraries by language.

Go

etcd/client/v3 - the officially maintained Go client for v3
etcd/client/v2 - the officially maintained Go client for v2
go-etcd - the deprecated official client. May be useful for older (<2.0.0) versions of etcd.
encWrapper - encWrapper is an encryption wrapper for the etcd client Keys API/KV.

Java

coreos/jetcd - Supports v3
boonproject/etcd - Supports v2, Async/Sync and waits
justinsb/jetcd
diwakergupta/jetcd - Supports v2
jurmous/etcd4j - Supports v2, Async/Sync, waits and SSL
AdoHe/etcd4j - Supports v2 (enhance for real production cluster)
cdancy/etcd-rest - Uses jclouds to provide a complete implementation of v2 API.
IBM/etcd-java

Scala

maciej/etcd-client - Supports v2. Akka HTTP-based fully async client
eiipii/etcdhttpclient - Supports v2. Async HTTP client based on Netty and Scala Futures.
mingchuno/etcd4s - Supports v3 using gRPC with optional Akka Stream support.

Perl

hexfusion/perl-net-etcd - Supports v3 grpc gateway HTTP API
robn/p5-etcd - Supports v2

Python

kragniz/python-etcd3 - Client for v3
jplana/python-etcd - Supports v2
russellhaering/txetcd - a Twisted Python library
cholcombe973/autodock - A docker deployment automation tool
lisael/aioetcd - (Python 3.4+) Asyncio coroutines client (Supports v2)
txaio-etcd - Asynchronous etcd v3-only client library for Twisted (today) and asyncio (future)
aioetcd3 - (Python 3.6+) etcd v3 API for asyncio
Revolution1/etcd3-py - (python2.7 and python3.5+) Python client for etcd v3, using gRPC-JSON-Gateway

Node

mixer/etcd3 - Supports v3
stianeikeland/node-etcd - Supports v2 (w Coffeescript)
lavagetto/nodejs-etcd - Supports v2
deedubs/node-etcd-config - Supports v2

Ruby

C

apache/celix/etcdlib - Supports v2
jdarcy/etcd-api - Supports v2
shafreeck/cetcd - Supports v2

C++

edwardcapriolo/etcdcpp - Supports v2
suryanathan/etcdcpp - Supports v2 (with waits)
nokia/etcd-cpp-api - Supports v2
nokia/etcd-cpp-apiv3 - Supports v3

Clojure

Erlang

marshall-lee/etcd.erl - Supports v2
zhongwencool/eetcd - Supports v3+ (GRPC only)

Elixir

team-telnyx/etcdex - Supports v3+ (GRPC only)

Tcl

efrecon/etcd-tcl - Supports v2, except wait.

Rust

jimmycuadra/rust-etcd - Supports v2

Gradle

gradle-etcd-rest-plugin - Supports v2

Lua

api7/lua-resty-etcd - Supports v2 and v3 (grpc gateway HTTP API)

Deployment tools

Chef integrations

coderanger/etcd-chef

Chef cookbooks

spheromak/etcd-cookbook

BOSH releases

Projects using etcd

etcd Raft users - projects using etcd’s raft library implementation.
apache/celix - an implementation of the OSGi specification adapted to C and C++
binocarlos/yoda - etcd + ZeroMQ
blox/blox - a collection of open source projects for container management and orchestration with AWS ECS
calavera/active-proxy - HTTP Proxy configured with etcd
chain/chain - software designed to operate and connect to highly scalable permissioned blockchain networks
derekchiang/etcdplus - A set of distributed synchronization primitives built upon etcd
go-discover - service discovery in Go
gleicon/goreman - Branch of the Go Foreman clone with etcd support
garethr/hiera-etcd - Puppet hiera backend using etcd
mattn/etcd-vim - SET and GET keys from inside vim
mattn/etcdenv - “env” shebang with etcd integration
kelseyhightower/confd - Manage local app config files using templates and data from etcd
configdb - A REST relational abstraction on top of arbitrary database backends, aimed at storing configs and inventories.
kubernetes/kubernetes - Container cluster manager introduced by Google.
mailgun/vulcand - HTTP proxy that uses etcd as a configuration backend.
duedil-ltd/discodns - Simple DNS nameserver using etcd as a database for names and records.
skynetservices/skydns - RFC compliant DNS server
xordataexchange/crypt - Securely store values in etcd using GPG encryption
spf13/viper - Go configuration library, reads values from ENV, pflags, files, and etcd with optional encryption
lytics/metafora - Go distributed task library
ryandoyle/nss-etcd - A GNU libc NSS module for resolving names from etcd.
Gru - Orchestration made easy with Go
Vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
lclarkmichalek/etcdhcp - DHCP server that uses etcd for persistence and coordination.
openstack/networking-vpp - A networking driver that programs the FD.io VPP dataplane to provide OpenStack cloud virtual networking
OpenStack - OpenStack services can rely on etcd as a base service.
CoreDNS - CoreDNS is a DNS server that chains plugins, part of CNCF and Kubernetes
Uber M3 - M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus
Rook - Storage Orchestration for Kubernetes
Patroni - A template for PostgreSQL High Availability with ZooKeeper, etcd, or Consul
Trillian - Trillian implements a Merkle tree whose contents are served from a data storage layer, to allow scalability to extremely large trees.
Apache APISIX - Apache APISIX is a dynamic, real-time, high-performance API gateway.
purpleidea/mgmt - Next generation distributed, event-driven, parallel config management!
Portworx/kvdb - The internal kvdb for storing Portworx cluster configuration.
Apache Pulsar - Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud.

Docs: Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcd-defrag - An easier to use and smarter etcd defragmentation tool.
etcdhelper - An intellij platform plugin for etcd.

Libraries

The sections below list etcd client libraries by language.

Go

etcd/client/v3 - the officially maintained Go client for v3
go-etcd - the deprecated official client. May be useful for older (<2.0.0) versions of etcd.
encWrapper - encWrapper is an encryption wrapper for the etcd client Keys API/KV.

Java

coreos/jetcd - Supports v3
boonproject/etcd - Supports v2, Async/Sync and waits
justinsb/jetcd
diwakergupta/jetcd - Supports v2
jurmous/etcd4j - Supports v2, Async/Sync, waits and SSL
AdoHe/etcd4j - Supports v2 (enhance for real production cluster)
cdancy/etcd-rest - Uses jclouds to provide a complete implementation of v2 API.
IBM/etcd-java

Scala

maciej/etcd-client - Supports v2. Akka HTTP-based fully async client
eiipii/etcdhttpclient - Supports v2. Async HTTP client based on Netty and Scala Futures.
mingchuno/etcd4s - Supports v3 using gRPC with optional Akka Stream support.

Perl

hexfusion/perl-net-etcd - Supports v3 grpc gateway HTTP API
robn/p5-etcd - Supports v2

Python

kragniz/python-etcd3 - Client for v3
jplana/python-etcd - Supports v2
russellhaering/txetcd - a Twisted Python library
cholcombe973/autodock - A docker deployment automation tool
lisael/aioetcd - (Python 3.4+) Asyncio coroutines client (Supports v2)
txaio-etcd - Asynchronous etcd v3-only client library for Twisted (today) and asyncio (future)
dims/etcd3-gateway - etcd v3 API library using the HTTP grpc gateway
aioetcd3 - (Python 3.6+) etcd v3 API for asyncio
Revolution1/etcd3-py - (python2.7 and python3.5+) Python client for etcd v3, using gRPC-JSON-Gateway

Node

mixer/etcd3 - Supports v3
stianeikeland/node-etcd - Supports v2 (w Coffeescript)
lavagetto/nodejs-etcd - Supports v2
deedubs/node-etcd-config - Supports v2

Ruby

C

apache/celix/etcdlib - Supports v2
jdarcy/etcd-api - Supports v2
shafreeck/cetcd - Supports v2

C++

edwardcapriolo/etcdcpp - Supports v2
suryanathan/etcdcpp - Supports v2 (with waits)
nokia/etcd-cpp-api - Supports v2
nokia/etcd-cpp-apiv3 - Supports v3

Clojure

Erlang

marshall-lee/etcd.erl - Supports v2
zhongwencool/eetcd - Supports v3+ (GRPC only)

Elixir

team-telnyx/etcdex - Supports v3+ (GRPC only)

.NET

wangjia184/etcdnet - Supports v2
drusellers/etcetera
shubhamranjan/dotnet-etcd - Supports v3+ (GRPC only)

PHP

linkorb/etcd-php
activecollab/etcd
ouqiang/etcd-php - Client for v3 gRPC gateway

Haskell

wereHamster/etcd-hs

R

ropensci/etseed

Nim

etcd_client

Tcl

efrecon/etcd-tcl - Supports v2, except wait.

Rust

jimmycuadra/rust-etcd - Supports v2

Gradle

gradle-etcd-rest-plugin - Supports v2

Lua

api7/lua-resty-etcd - Supports v2 and v3 (grpc gateway HTTP API)

Deployment tools

Chef integrations

coderanger/etcd-chef

Chef cookbooks

spheromak/etcd-cookbook

BOSH releases

Projects using etcd

etcd Raft users - projects using etcd’s raft library implementation.
apache/apisix - A dynamic, real-time, high-performance API gateway
apache/celix - an implementation of the OSGi specification adapted to C and C++
binocarlos/yoda - etcd + ZeroMQ
blox/blox - a collection of open source projects for container management and orchestration with AWS ECS
calavera/active-proxy - HTTP Proxy configured with etcd
chain/chain - software designed to operate and connect to highly scalable permissioned blockchain networks
derekchiang/etcdplus - A set of distributed synchronization primitives built upon etcd
go-discover - service discovery in Go
gleicon/goreman - Branch of the Go Foreman clone with etcd support
garethr/hiera-etcd - Puppet hiera backend using etcd
mattn/etcd-vim - SET and GET keys from inside vim
mattn/etcdenv - “env” shebang with etcd integration
kelseyhightower/confd - Manage local app config files using templates and data from etcd
configdb - A REST relational abstraction on top of arbitrary database backends, aimed at storing configs and inventories.
kubernetes/kubernetes - Container cluster manager introduced by Google.
mailgun/vulcand - HTTP proxy that uses etcd as a configuration backend.
duedil-ltd/discodns - Simple DNS nameserver using etcd as a database for names and records.
skynetservices/skydns - RFC compliant DNS server
xordataexchange/crypt - Securely store values in etcd using GPG encryption
spf13/viper - Go configuration library, reads values from ENV, pflags, files, and etcd with optional encryption
lytics/metafora - Go distributed task library
ryandoyle/nss-etcd - A GNU libc NSS module for resolving names from etcd.
Gru - Orchestration made easy with Go
Vitess - Vitess is a database clustering system for horizontal scaling of MySQL.
lclarkmichalek/etcdhcp - DHCP server that uses etcd for persistence and coordination.
openstack/networking-vpp - A networking driver that programs the FD.io VPP dataplane to provide OpenStack cloud virtual networking
OpenStack - OpenStack services can rely on etcd as a base service.
CoreDNS - CoreDNS is a DNS server that chains plugins, part of CNCF and Kubernetes
Uber M3 - M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus
Rook - Storage Orchestration for Kubernetes
Patroni - A template for PostgreSQL High Availability with ZooKeeper, etcd, or Consul
Trillian - Trillian implements a Merkle tree whose contents are served from a data storage layer, to allow scalability to extremely large trees.
Apache APISIX - Apache APISIX is a dynamic, real-time, high-performance API gateway.
purpleidea/mgmt - Next generation distributed, event-driven, parallel config management!
Portworx/kvdb - The internal kvdb for storing Portworx cluster configuration.
Apache Pulsar - Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud.

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

etcd namespace metrics

The metrics under the etcd prefix are for monitoring and alerting. They are stable high level metrics. If there is any change of these metrics, it will be included in release notes.

Metrics that are etcd2 related are documented v2 metrics guide.

Server

These metrics describe the status of the etcd server. In order to detect outages or problems for troubleshooting, the server metrics of every production etcd cluster should be closely monitored.

All these metrics are prefixed with etcd_server_

Name	Description	Type
has_leader	Whether or not a leader exists. 1 is existence, 0 is not.	Gauge
leader_changes_seen_total	The number of leader changes seen.	Counter
proposals_committed_total	The total number of consensus proposals committed.	Gauge
proposals_applied_total	The total number of consensus proposals applied.	Gauge
proposals_pending	The current number of pending proposals.	Gauge
proposals_failed_total	The total number of failed proposals seen.	Counter

proposals_pending indicates how many proposals are queued to commit. Rising pending proposals suggests there is a high client load or the member cannot commit proposals.

proposals_failed_total are normally related to two issues: temporary failures related to a leader election or longer downtime caused by a loss of quorum in the cluster.

Disk

These metrics describe the status of the disk operations.

All these metrics are prefixed with etcd_disk_.

Name	Description	Type
wal_fsync_duration_seconds	The latency distributions of fsync called by wal	Histogram
backend_commit_duration_seconds	The latency distributions of commit called by backend.	Histogram

A wal_fsync is called when etcd persists its log entries to disk before applying them.

A backend_commit is called when etcd commits an incremental snapshot of its most recent changes to disk.

High disk operation latencies (wal_fsync_duration_seconds or backend_commit_duration_seconds) often indicate disk issues. It may cause high request latency or make the cluster unstable.

Network

These metrics describe the status of the network.

All these metrics are prefixed with etcd_network_

Name	Description	Type
peer_sent_bytes_total	The total number of bytes sent to the peer with ID `To`.	Counter(To)
peer_received_bytes_total	The total number of bytes received from the peer with ID `From`.	Counter(From)
peer_sent_failures_total	The total number of send failures from the peer with ID `To`.	Counter(To)
peer_received_failures_total	The total number of receive failures from the peer with ID `From`.	Counter(From)
peer_round_trip_time_seconds	Round-Trip-Time histogram between peers.	Histogram(To)
client_grpc_sent_bytes_total	The total number of bytes sent to grpc clients.	Counter
client_grpc_received_bytes_total	The total number of bytes received to grpc clients.	Counter

peer_received_bytes_total counts the total number of bytes received from a specific peer. Usually follower members receive data only from the leader member.

gRPC requests

These metrics are exposed via go-grpc-prometheus.

etcd_debugging namespace metrics

Snapshot

Name	Description	Type
snapshot_save_total_duration_seconds	The total latency distributions of save called by snapshot	Histogram

Abnormally high snapshot duration (snapshot_save_total_duration_seconds) indicates disk issues and might cause the cluster to be unstable.

Prometheus supplied metrics

The Prometheus client library provides a number of metrics under the go and process namespaces. There are a few that are particularly interesting.

Name	Description	Type
process_open_fds	Number of open file descriptors.	Gauge
process_max_fds	Maximum number of open file descriptors.	Gauge

Generated list of metrics

v3.4.0	v3.4.1	v3.4.10	v3.4.11	v3.4.12	v3.4.13
v3.4.2	v3.4.3	v3.4.4	v3.4.5	v3.4.6	v3.4.7
v3.4.8	v3.4.9

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

etcd namespace metrics

The metrics under the etcd prefix are for monitoring and alerting. They are stable high level metrics. If there is any change of these metrics, it will be included in release notes.

Metrics that are etcd2 related are documented in the v2 metrics guide.

Server

These metrics describe the status of the etcd server. In order to detect outages or problems for troubleshooting, the server metrics of every production etcd cluster should be closely monitored.

All these metrics are prefixed with etcd_server_

Name	Description	Type
has_leader	Whether or not a leader exists. 1 is existence, 0 is not.	Gauge
leader_changes_seen_total	The number of leader changes seen.	Counter
proposals_committed_total	The total number of consensus proposals committed.	Gauge
proposals_applied_total	The total number of consensus proposals applied.	Gauge
proposals_pending	The current number of pending proposals.	Gauge
proposals_failed_total	The total number of failed proposals seen.	Counter

proposals_pending indicates how many proposals are queued to commit. Rising pending proposals suggests there is a high client load or the member cannot commit proposals.

proposals_failed_total are normally related to two issues: temporary failures related to a leader election or longer downtime caused by a loss of quorum in the cluster.

Disk

These metrics describe the status of the disk operations.

All these metrics are prefixed with etcd_disk_.

Name	Description	Type
wal_fsync_duration_seconds	The latency distributions of fsync called by wal	Histogram
backend_commit_duration_seconds	The latency distributions of commit called by backend.	Histogram

A wal_fsync is called when etcd persists its log entries to disk before applying them.

A backend_commit is called when etcd commits an incremental snapshot of its most recent changes to disk.

High disk operation latencies (wal_fsync_duration_seconds or backend_commit_duration_seconds) often indicate disk issues. It may cause high request latency or make the cluster unstable.

Network

These metrics describe the status of the network.

All these metrics are prefixed with etcd_network_

Name	Description	Type
peer_sent_bytes_total	The total number of bytes sent to the peer with ID `To`.	Counter(To)
peer_received_bytes_total	The total number of bytes received from the peer with ID `From`.	Counter(From)
peer_sent_failures_total	The total number of send failures from the peer with ID `To`.	Counter(To)
peer_received_failures_total	The total number of receive failures from the peer with ID `From`.	Counter(From)
peer_round_trip_time_seconds	Round-Trip-Time histogram between peers.	Histogram(To)
client_grpc_sent_bytes_total	The total number of bytes sent to grpc clients.	Counter
client_grpc_received_bytes_total	The total number of bytes received to grpc clients.	Counter

peer_received_bytes_total counts the total number of bytes received from a specific peer. Usually follower members receive data only from the leader member.

gRPC requests

These metrics are exposed via go-grpc-prometheus.

etcd_debugging namespace metrics

Snapshot

Name	Description	Type
snapshot_save_total_duration_seconds	The total latency distributions of save called by snapshot	Histogram

Abnormally high snapshot duration (snapshot_save_total_duration_seconds) indicates disk issues and might cause the cluster to be unstable.

Prometheus supplied metrics

The Prometheus client library provides a number of metrics under the go and process namespaces. There are a few that are particularly interesting.

Name	Description	Type
process_open_fds	Number of open file descriptors.	Gauge
process_max_fds	Maximum number of open file descriptors.	Gauge

Generated list of metrics

latest

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

etcd namespace metrics

The metrics under the etcd prefix are for monitoring and alerting. They are stable high level metrics. If there is any change of these metrics, it will be included in release notes.

Metrics that are etcd2 related are documented in the v2 metrics guide.

Server

These metrics describe the status of the etcd server. In order to detect outages or problems for troubleshooting, the server metrics of every production etcd cluster should be closely monitored.

All these metrics are prefixed with etcd_server_

Name	Description	Type
has_leader	Whether or not a leader exists. 1 is existence, 0 is not.	Gauge
leader_changes_seen_total	The number of leader changes seen.	Counter
proposals_committed_total	The total number of consensus proposals committed.	Gauge
proposals_applied_total	The total number of consensus proposals applied.	Gauge
proposals_pending	The current number of pending proposals.	Gauge
proposals_failed_total	The total number of failed proposals seen.	Counter

proposals_pending indicates how many proposals are queued to commit. Rising pending proposals suggests there is a high client load or the member cannot commit proposals.

proposals_failed_total are normally related to two issues: temporary failures related to a leader election or longer downtime caused by a loss of quorum in the cluster.

Disk

These metrics describe the status of the disk operations.

All these metrics are prefixed with etcd_disk_.

Name	Description	Type
wal_fsync_duration_seconds	The latency distributions of fsync called by wal	Histogram
backend_commit_duration_seconds	The latency distributions of commit called by backend.	Histogram

A wal_fsync is called when etcd persists its log entries to disk before applying them.

A backend_commit is called when etcd commits an incremental snapshot of its most recent changes to disk.

High disk operation latencies (wal_fsync_duration_seconds or backend_commit_duration_seconds) often indicate disk issues. It may cause high request latency or make the cluster unstable.

Network

These metrics describe the status of the network.

All these metrics are prefixed with etcd_network_

Name	Description	Type
peer_sent_bytes_total	The total number of bytes sent to the peer with ID `To`.	Counter(To)
peer_received_bytes_total	The total number of bytes received from the peer with ID `From`.	Counter(From)
peer_sent_failures_total	The total number of send failures from the peer with ID `To`.	Counter(To)
peer_received_failures_total	The total number of receive failures from the peer with ID `From`.	Counter(From)
peer_round_trip_time_seconds	Round-Trip-Time histogram between peers.	Histogram(To)
client_grpc_sent_bytes_total	The total number of bytes sent to grpc clients.	Counter
client_grpc_received_bytes_total	The total number of bytes received to grpc clients.	Counter

peer_received_bytes_total counts the total number of bytes received from a specific peer. Usually follower members receive data only from the leader member.

gRPC requests

These metrics are exposed via go-grpc-prometheus.

etcd_debugging namespace metrics

Snapshot

Name	Description	Type
snapshot_save_total_duration_seconds	The total latency distributions of save called by snapshot	Histogram

Abnormally high snapshot duration (snapshot_save_total_duration_seconds) indicates disk issues and might cause the cluster to be unstable.

Prometheus supplied metrics

The Prometheus client library provides a number of metrics under the go and process namespaces. There are a few that are particularly interesting.

Name	Description	Type
process_open_fds	Number of open file descriptors.	Gauge
process_max_fds	Maximum number of open file descriptors.	Gauge

Generated list of metrics

latest

Docs: Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

If any part of the etcd project has bugs or documentation mistakes, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).
Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please includes the steps that might lead to the problem. If possible, please attach the affected etcd data dir and stack strace to the bug report.
Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report. Debugging external systems that rely on etcd is out of scope, but we are happy to provide guidance in the right direction or help with using etcd itself.
Unique. Do not duplicate existing bug report.
Scoped. One bug per report. Do not follow up with another bug inside one report.

It may be worthwhile to read Elika Etemad’s article on filing good bug reports before creating a bug report.

We might ask for further information to locate a bug. A duplicated bug report will be closed.

Frequently asked questions

How to get a stack trace

$ kill -QUIT $PID

How to get etcd version

$ etcd --version

How to get etcd configuration and log when it runs as systemd service ‘etcd2.service’

$ sudo systemctl cat etcd2
$ sudo journalctl -u etcd2

Due to an upstream systemd bug, journald may miss the last few log lines when its processes exit. If journalctl says etcd stopped without fatal or panic message, try sudo journalctl -f -t etcd2 to get full log.