Servers for Hackers

Customize your Login Screen via Linux's Message of the Day (Ubuntu/CentOS)

See how you can customize the screen you see when you login to give you important information quickly.

Docker for Gulp Build Tasks

I run my development environments inside of a Vagrant virtual machine. Most of my projects use a `gulp` pipeline to build static assets. **Generally, you can decide to run Gulp in one of two places:** 1. **On the host machine:** Install Node (via NVM or not) on your host machine (perhaps your Mac computer) and run `gulp watch` from the host machine. 2. **On the guest machine** Install Node on the guest (the virtual machine) and run `gulp watch` from there. Here are some pros and cons of that. Some of these cons are purely subjective (my personal preference). You can disagree with them, I won't mind. ## NodeJS On the Host **The Pro:** NodeJS on the host machine doesn't miss file changes. Since the files are actually on the host machine file system (not shared into the VM), inotify and friends are generally working as they should. This means that `gulp watch` doesn't bat an eye. **The Con:** NodeJS is one of those tools I often fight with - either `npm install` in inexplicably failing, or NodeJS versions aren't matching up, or, even if everything should work, it just refuses to install correctly. Therefore I HATE having it installed directly on my Mac. NVM doesn't really solves this for me. I instead prefer to install development related software on a virtual machine, as these can be destroyed and re-created very easily. NodeJS, especially, has a fast-moving community. It's always changing. ## NodeJS on the Guest **The Pro:** NodeJS on the guest let's me worry a LOT less about testing and installing things. VM's are servers that I can destroy and re-create easily. I don't mind experimenting with tools, or throwing multiple versions of binaries around inside of it willy-nilly. (Personal preference, remember!) **The Con**: The default file sharing that Virtualbox uses can easily miss file changes, making `gulp watch` frustrating to work with. NFS make this situation better, but is **not** infallible. It should be noted that I've tried LOTS of various NFS configurations, including using [`cachefilesd`](https://github.com/fideloper/Vaprobash/commit/677e19d61337b11bc64eb829016028f1047c7d76#diff-23b6f443c01ea2efcb4f36eedfea9089)). ### Anything else we can try?

Picard management tip: There is always a non-obvious third option. When caught between a rock and a hard place, find a jackhammer.
— Picard Tips (@PicardTips) April 22, 2015

Agreed! ## Docker I've recently been getting reacquainted with Docker and it's [new tools](https://blog.docker.com/2016/06/docker-1-12-built-in-orchestration/). I realized I could use it effectively here. **My goals in using Docker to run Gulp build steps were:** 1. Have NodeJS in something super ephemeral - something I wouldn't mind making mistakes on. 2. Have `gulp watch` work more reliably. The following setup does both of these things! Note that point 2 works because Docker's volume mounting is much better than either NFS or Virtualhost's default file share. > This is possible on my Mac because of the [Docker app (in beta)](https://docs.docker.com/engine/installation/mac/), which is solving the sticky bits of running Docker on Mac's kernel. It no longer runs in a Virtualbox VM thanks to [*magic*](https://github.com/mist64/xhyve/). Let's see how Docker helped me here: ### Dockerfile First, we'll create new Docker image that has NodeJS in it. I install Node 5, since that's what my project happened to expect. Node 6 is the latest and might be what you want instead. ``` FROM ubuntu:16.04 MAINTAINER Chris Fidao WORKDIR /opt ADD setup_5.x /tmp/setup_5.x RUN bash /tmp/setup_5.x RUN apt-get update RUN apt-get install -y build-essential RUN apt-get install -y nodejs RUN /usr/bin/npm install -g gulp RUN /usr/bin/npm install -g bower VOLUME ["/opt"] CMD ["gulp", "watch"] ``` All we do here is grab the latest Ubuntu LTS and install NodeJS. I then globally install Gulp and Bower. This is taken straight out of the [Homestead install script](https://github.com/laravel/settler/blob/9ad04c61eed8bbb163b567d4f8f9015ad6636a46/scripts/provision.sh#L170-L174). **Three things to note:** 1. I add a file called `setup_5.x`. This is directly from the [Nodesource Node 5.x installer](https://deb.nodesource.com/setup_5.x), which I downloaded and saved as a file first. 2. The working directory is set to `/opt`, so any commands run will be relative to that directory 3. Using `CMD ["gulp", "watch"]` means that if I don't specify any other command, this container will run `gulp watch`. I can, however, specify any other command I want. And I will! You'll see why. Once this Dockerfile is saved, we can build a new image. I'll name (tag) it simply `gulp`: ```bash $ cd /path/to/dir/with/Dockerfile $ docker build -t gulp . # ...it builds... $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE gulp latest 59fe57f1d14a 17 hours ago 460.6 MB ubuntu 16.04 2fa927b5cdd3 5 weeks ago 122 MB ``` ### Building node_modules There's two steps to actually using this: 1. Use the container to build the `node_modules` directory 2. Use the container to run `gulp watch` Let's first assume your project has not had `npm install` run yet. (I backed up and deleted my old `node_modules` directory). ```bash # Get old node_modules dir out of the way $ cd ~/Sites/some-project $ cp node_modules ~/some-project-node_modules $ rm -rf node_modules # everyone's familiar with this command # Use our container to build a new node_modules dir $ docker run --rm -v ~/Sites/some-project:/opt gulp npm install ``` Let's cover what's happening there: 1. `docker run` - run a container 2. `--rm` - Delete the container when it exits. We don't need this container to persist. 3. `-v ~/Sites/some-project:/opt` - Share our host computer's `some-project` dir with the `/opt` dir, which we set as the working directory of the image (and will continue to be the working directory when we use this image to make a new container) 4. `gulp` - specify the image from which the container should be created 5. `npm install` - The default CMD set was `gulp watch`, but we're over-riding it in this case so we can have it install our npm dependencies first. This way, anything specific to the container's node version and architecture (Node 5.x, Ubuntu 16.04 x64) will be created correctly. ### Running "Gulp Watch" Once that's done, a new `node_modules` directory will exist! We're ready to run `gulp watch` now: ```bash $ docker run --rm -v ~/Sites/some-project:/opt gulp ``` This is almost the exact same command, except we didn't give the container a command to run. Since we defined `CMD ["gulp", "watch"]` in the Dockerfile, the `gulp watch` command will be run by default. > Don't get the fact that I named the image `gulp` confused with the commands we're running in that container. The container first ran `npm install`, and our second container ran `gulp watch`. I use `--rm` on both containers since the containers don't need to persist - the results of their actions are saved on our host computer thanks to the volume mounting. When you're done with that container, you can open a new shell and run `docker ps` to get the docker ID or name, and then run `docker stop ` to stop it.

Letsencrypt for Free & Easy SSL Certificates

See how to easy it is to use letsencrypt to create and automatically renew FREE SSL certificates!

Logrotate for Forge

Laravel Forge (and Envoyer) keep your Laravel application's `storage` directory persistent through deployments. One side effect of this is that your `storage/logs/laravel.log` file is always growing. Linux systems come with [Logrotate, which can help us manage your growing log files](https://serversforhackers.com/managing-logs-with-logrotate). It's really simple - we can basically set it and forget it. > **Note:** This is best used when your logs are set to `single` rather than `daily` within your `config/app.php` file. ## Logrotate for Forge First, we need to know the application's log storage directory. If my application is `foo.example.com`, that location is likely `/home/forge/foo.example.com/storage/logs`. Knowing that, we can make a logrotate configuration for it at `/etc/logrotate.d/foo.example.com`: ``` /home/forge/foo.example.com/storage/logs/*.log { su forge forge weekly missingok rotate 24 compress notifempty create 755 forge forge } ``` Once you save that file, you're done! Let's go over the options we set there: * First we set the file path, and tell it to find any file ending in `.log` * **su forge forge** tells logrotate to use user/group `forge` to process these logs * **weekly** tells logrotate to rotate logs weekly (versus daily, monthly, yearly) * **missingok** - If no *.log files are found, don't freak out * **rotate 24** - Keep 24 files before deleting old log archived (deletes oldest first). This is 24 weeks, or 6 months * **compress** - Compress archived log files so they take less disk space * **notifempty** - Don't rotate the log if it's empty * **create 755 forge forge** - Create new log files with these permissions/owner settings ## Testing To test this command out, we can force the running of logrotate with the following command: ```bash # Forge logrotate to run against config file foo.example.com sudo logrotate --force foo.example.com ``` This will tell you if there are any errors. If you get no output, you should be able to check your log directory and see some newly rotated logs! ## Resources * See more about [logrotate here](https://serversforhackers.com/managing-logs-with-logrotate)

Mysqldump with Modern MySQL

Mysqldump has *many* options ([I count 111 options](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#idm140518991203200) 😁). Most of us are likely keeping it simple. Here's how I've typically exported a single database: ```bash mysqldump some_database > some_database.sql # Or with user auth mysqldump -u some_user -p some_database > some_database.sql # Or with gzip compression mysqldump some_database | gzip > some_database.sql.gz # Or with the "pv" tool, which let's us know how much data is # flowing between our pipes - useful for knowing if the msyqldump # has stalled mysqldump some_database | pv | gzip > some_database.sql.gz # 102kB 0:01:23 [1.38MB/s] [ <=> ``` However, it's worth digging into this command a bit to learn what's going on. If you're using mysqldump against a production database, it's usage can cause real issues for your users while it's running. ## Defaults First, let's cover mysqldump's defaults. Unless we explicitly tell it not to, mysqldump is using the [`--opt`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_opt) flag. The `opt` option is an alias for the following flags: * [--add-drop-table](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_add-drop-table) - Write a DROP TABLE statement before each CREATE TABLE statement, letting you re-use the resulting .sql file over and over with idempotence. * [--add-locks](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_add-locks) - This applies when you're importing your dump file (not when running mysqldump). Surrounds each table dump with LOCK TABLES and UNLOCK TABLES statements. This results in faster inserts when the dump file is reloaded. This means that while you're importing data, each table will be locked from reads and writes while it's (re-)creating a table. * [--create-options](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_create-options) - Include all MySQL-specific table options in the CREATE TABLE statements. In testing this (turn it off using `-create-options=false`), I found that the main/most obvious difference was the absense of `AUTO_INCREMENT` on primary keys when setting this option to false. * [--disable-keys](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_disable-keys) - *This option is effective only for nonunique indexes of **MyISAM** tables.* This makes loading the dump file faster *(for MyISAM tables)* because the indexes are created after all rows are inserted. * [--extended-insert](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_extended-insert) - Write INSERT statements using multiple-row syntax that includes several VALUES lists. Not using this option may be required for tables with large columns (usually blobs) that cause queries to go higher than client/server "max_allowed_packet" configuration, but generally always use this option. Using a single-query per insert slows down imports considerably. * [--lock-tables](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_lock-tables) - Unline `add-locks`, this applies to when you're running mysqldump. This locks all tables for the duration of the mysqldump, making it a bad option to use on a live environment. Primarily it's used for protection of data integrity when dumping MyISAM tables. Since InnoDB is rightly the default table storage engine now-a-days, this option usually should be over-ridden by using `--skip-lock-tables` to stop the behavior and `--single-transaction` to run mysqldump within a transaction, which I'll cover in a bit. * [--quick](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_quick) - Reads out large tables in a way that doesn't require having enough RAM to fit the full table in memory. * [--set-charset](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_set-charset) - Write SET NAMES default_character_set to the output. This DOES NOT perform any character set conversion (mysqldump won't do with that *any* flag). Instead, it's just saying that you want the character set info added in so it's set when re-importing the dump file. So the default are pretty good, with the exception of **--lock-tables**. This causes the database to become unusable while mysqldump is running, but it doesn't need to be this way! We can use mysqldump more intelligently. ## Mysqldump and Table Locks When using mysqldump, there's a trade off to be made between halting/affecting database performance and ensuring data integrity. Your strategy will largely be determined by what storage engine(s) your using in your database tables. > Since each table can have a separate storage engine, this can get interesting :D By default, mysqldump locks all the tables it's about to dump. This ensure the data is in a **consistent state** during the dump. ### Data Consistency A "consistent state" means that the data is in an expected state. More specifically, all relationships should match up. Imagine if mysqldump exports the first 5 tables out of 20. If table 1 and table 20 got new rows related to eachother by primary/foreign keys after mysqldump dumped table 1 but before it dumped table 20, then we're in an inconsistent state. Table 20 has data relating to a row in table 1 that did not make it into the dump file. MyISAM tables require this locking because they don't support transactions. However, InnoDB (the default storage engine as of MySQL 5.5.5) supports transactions. Mysqldump defaults to a conservative setting of locking everything, but we don't need to use that default - we an avoid locking tables completely. ### Mysqldump with Transactions As a rule of thumb, **unless you are using MyISAM for a specific reason, you should be using the InnoDB storage engine** on all tables. If you've been porting around a database to various MySQL servers for years (back when MyISAM used to be the default storage engine), check to make sure your tables are using InnoDB. This is the important one: **Assuming you are using InnoDB tables, your mysqldump should look something like this:** ```bash mysqldump --single-transaction --skip-lock-tables some_database > some_database.sql ``` The [`--single-transaction`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_single-transaction) flag will start a transaction before running. Rather than lock the entire database, this will let mysqldump read the database in the current state at the time of the transaction, making for a consistent data dump. > The `single-transaction` options uses the default transaction isolation mode: [REPEATABLE READ](https://dev.mysql.com/doc/refman/5.7/en/set-transaction.html#isolevel_repeatable-read). Note that if you have a mix of MyISAM and InnoDB tables, using the above options can leave your MyISAM (or Memory tables, for that matter) in an inconsistent state, since it does not lock reads/writes to MyISAM tables. In that case, I suggest dumping your MyISAM tables separately from InnoDB tables. However, if that still results in inconsistent state (if the MyISAM table has PK/FK relationships to InnoDB tables), then using the [`--lock-tables`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_lock-tables) option becomes the only way to guarantee the database is in a consistent state when using mysqldump. This means that in that situation, you'll have to be careful about when you run mysqldump on a live database. Perhaps run it on a replica database instead of a master one, or investigate options such as Xtrabackup, which copies the mysql data directory and does not cause down time. ## Replication If you're using replication, you already have a backup on your replica servers. That's awesome! However, off-site backups are still a good thing to have. In such a setup, I try to run mysqldump on the replica server instead of a master server. In terms of mysqldump, this has as few implications: 1. Running mysqldump on a replica server means the data it receives might be slightly behind the master server. - For regular backups, this is likely fine. If you need the data to be at a certain point, then you need to wait until that data has reached the replica server. 2. Running mysqldump on a replica is prefered (IMO) since in theory, there is already a built-in assumption that the replica servers will be behind anyway - adding a bit of "strain" of a mysqldump shouldn't be a big deal. In any case, there are some useful flags to use when replication is in place (or when binlogs are enabled in general). ### Master Data The [`--master-data`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_master-data) flag adds output to a dump file which allows it to be used to set up another server as a replica of the master. The replica needs the master data to know where to start replication. The `--master-data` option automatically turns off `--lock-tables`, since the included binlog position will say where to start replication off, letting you not lose queries if the dump ends up in an inconsistent state. (Again, that's only a consideration if you have MyISAM tables). If `--single-transaction` is also used, a global read lock is acquired only for a short time at the beginning of the dump. Use this when dumping from a master server. ### Dump Replica The [`--dump-slave`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_dump-slave) option is very similar to the `--master-data` except it's use case is: 1. Instead of being a dump from the master server, it's meant to be a dump of a replica server 2. It will contain the same master information as the replica server being dumped, where as `--master-data` set itself as the master Use this when dumping from a replica server. > From the docs: "This option should not be used if the server where the dump is going to be applied uses gtid_mode=ON and MASTER_AUTOPOSITION=1." > > GTID is a newer way to do MySQL replication as of MySQL 5.6. It's a nicer method, so --dump-slave in theory can be one to ignore. ## Dump more than one (all) database I generally dump specific databases, which lets me more easily recover a specific database if I need to. However you can dump multiple databases: ```bash mysqldump --single-transaction --skip-lock-tables --databases db1 db2 db3 \ > db1_db2_and_db3.sql ``` You can also dump specific tables from a single database: ```bash mysqldump --single-transaction --skip-lock-tables some_database table_one table_two table_three \ > some_database_only_three_tables.sql ``` You can also dump the entire database. Note that this likely includes the internal `mysql` database as well: ```bash mysqldump --single-transaction --skip-lock-tables --flush-privileges --all-databases > entire_database_server.sql ``` The above command used the [`--all-databases`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_all-databases) option along with the [`--flush-privileges`](http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_flush-privileges) option. Since we'll get the internal `mysql` database, which includes mysql users and privileges, the `--flush-privileges` option adds a `FLUSH PRIVILEGES` query at the end of the dump, needed since the dump may change users and privileges when being imported. ## That's it! There are many, many options you can use with mysqldump. However, we covered what I think are the most important for using mysqldump in a modern implementation of MySQL.

Side note, If you're interested in a service to help you manage MySQL-optimized, backup and (eventually) replication-enabled database servers, sign up here to let me know! The idea is to allow you to better manage your MySQL servers, taking advantage of many of MySQL's more advanced options, especially around backup and recovery.

MySQL Network Security

On Digital Ocean, Linode and similar clouds, private networks are open to the entire data center (e.g. Newark or NYC3). They are **NOT** private to your specific account. My Linode server in Newark can potentially communicate to your Linode server in Newark over their private network address. If your database requires access from other servers, it needs to be protected. ## Our Available Tools MySQL comes with plenty of network security tools. Here are the essential ones: 1. MySQL Bind-Address 2. MySQL User security 3. Firewall ### MySQL Bind Address * MySQL can bind to no networks (connect over `localhost` only, e.g. a unix socket) * MySQL can bind to all networks (`0.0.0.0`) * MySQL can bind to a specific network, e.g. a public network that the whole internet can reach, or a private network that can only be reached from within a data center > MySQL usually listens to a `localhost` unix socket **in addition to** a network location, so localhost connections should be just fine still no matter what setup you use. The more restrictive we can be, the better. If our application is on the same server as the database, we can close mysql from binding to any network (defaulting instead to listening only on the local unix socket). More common is to also bind to the loopbackn etwork address `127.0.0.1` so both `localhost` (unix socket) and `127.0.0.1` (tcp socket) connections work. ### MySQL User Security In addition to setting what networks MySQL listens on, we can set **where** users are allowed to connect **from**. This means we can say "*user `my_app_user` can only connect to MySQL from the server whose address is `192.168.33.10`*". Let's see how that looks in MySQL. #### What users exist Run the following to see what users exist on the MySQL server: ```sql mysql> SELECT User, Host from mysql.user; +-----------+--------------------+ | User | Host | +-----------+--------------------+ | root | 127.0.0.1 | | root | ::1 | | mysql.sys | localhost | | root | localhost | +-----------+--------------------+ 4 rows in set (0.00 sec) ``` We can see that we have three `root` users and one system user. * `root@127.0.0.1` - Can connect using the loopback ipv4 network `127.0.0.1` * `root@::1` - Can connect using the loopback ipv6 network `::1` * `root@localhost` - Can connect using the unix socket Note that `localhost` in MySQL will mean connecting over the Unix socket, even if the hostname `localhost` resolves to IP address `127.0.0.1`. #### Application Users We need to make MySQL users for our applications to use. Let's pretend that our MySQL server is in a single region with networks: * **Public Ipv4:** 159.203.81.145 * **Private Ipv4:** 10.132.30.23 And an application server in the same data center with networks: * **Public Ipv4:** 104.131.100.163 * **Private Ipv4:** 10.132.51.34 Since these two servers are within the same private network (`10.132.*.*`), they can communicate to each other. Let's set the application server to be able to connect to the MySQL server. We have a few tools we can use: * Hostnames (`example.com`) * Explicit IP addresses (`192.168.10.10`) * Wildcards (`192.168.10.%`) * Netmasks Here we use an explicit IP address, some wildcards, and then some netmasks to see how they can be used: ```sql -- From the MySQL Server, create a new user -- Create a user that can only connect from this one server CREATE USER 'my_app_user'@'10.132.51.34' IDENTIFIED BY 'some-strong-password'; -- Create a user that can connect from any IP starting in `10.132.51.` CREATE USER 'my_app_user'@'10.132.51.%' IDENTIFIED BY 'some-strong-password'; -- Create a user that can connect from any IP starting in `10.132.` CREATE USER 'my_app_user'@'10.132.%' IDENTIFIED BY 'some-strong-password'; -- Create a user that can connect from any IP starting in `10.` CREATE USER 'my_app_user'@'10.%' IDENTIFIED BY 'some-strong-password'; -- Create a user that can connect from `10.132.51.*` CREATE USER 'my_app_user'@'10.132.51.0/255.255.255.0' IDENTIFIED BY 'some-strong-password'; -- Create a user that can connect from `10.132.*.*` CREATE USER 'my_app_user'@'10.132.0.0/255.255.0.0' IDENTIFIED BY 'some-strong-password'; -- Create a user that can connect from `10.*.*.*` CREATE USER 'my_app_user'@'10.0.0.0/255.0.0.0' IDENTIFIED BY 'some-strong-password'; ``` The first example there is very specific - the user can only connect to MySQL if they are connecting from that one server at ip address `10.132.51.34`. The next few examples use wildcards (`%`). Wildcards use the same principles as `LIKE` matching in MySQL , and are best used to match hostnames (e.g. `'my_app_user'@'%.example.com'`). Finally we have some netmask examples. Netmasks are specific to IPv4 network addresses (they don't work for IPv6 currently). > In a cloud like Digital Ocean or Linode, I create multiple users who can only connect from a specific IP address (no wildcard, no netmask), since **all other options leave open the possibility that someone else's server can connect to your MySQL instance**. #### Public Networks The concepts here work for a public network as well. If your database needs to be connected from other regions, then the private network won't work. In this case, you "should" look into setting up a VPN or SSL Tunnel, so a private network can be setup across regions. However, we can get fairly secure without a VPN, which are complex to setup and maintain. You can bind your MySQL to either all networks (if you also need private network access), `0.0.0.0`, or to the public network, `159.203.81.145` in our example. User setup is then the same - you can define the IP address or hostname the user can connect from. ```sql -- Create a user that can only connnect from this one server CREATE USER 'my_app_user'@'104.131.100.163' IDENTIFIED BY 'some-strong-password'; ``` ### Firewall On top of what MySQL provides, we can (should) also use our firewalls to protect us. For example, we can set our firewall to only accept connections to port 3306 if they are coming to our private network (the destination is our private network): ```bash # Append rule to the end of the INPUT chain sudo iptables -A INPUT -p tcp -m tcp --dport 3306 -d 10.132.30.23 -j ACCEPT # Or, insert it into the middle of our chain (position 3 here) sudo iptables -I INPUT 3 -p tcp -m tcp --dport 3306 -d 10.132.30.23 -j ACCEPT ``` The `-d` flag sets the destination network. Alternatively, we can set what network interface allows traffic to port 3306: ```bash # Append rule to the end of the INPUT chain sudo iptables -A INPUT -p tcp -m tcp --dport 3306 -i eth1 -j ACCEPT # Or, insert it into the middle of our chain (position 3 here) sudo iptables -I INPUT 3 -p tcp -m tcp --dport 3306 -i eth1 -j ACCEPT ``` This works well on Digital Ocean where the `eth1` interface (`-i`) is your private network. Linode, however, doesn't setup a network interface for the private traffic, so this firewall rule wouldn't work there. And of course we can set rules to only allow traffic from specific IP addresses or networks, using both netmask and CIDR notation. ```bash # Allow traffic from a specific IP sudo iptables -A INPUT -p tcp -m tcp --dport 3306 -s 10.132.51.34 -j ACCEPT # Allow traffic from a netmask range of IP addresses 10.132.51.* sudo iptables -A INPUT -p tcp -m tcp --dport 3306 -s 10.132.51.0/255.255.255.0 -j ACCEPT # Allow traffic from a CIDR range of IP addresses 10.132.51.* sudo iptables -A INPUT -p tcp -m tcp --dport 3306 -s 10.132.51.0/24 -j ACCEPT ``` ## TL;DR 1. Bind MySQL to `0.0.0.0` if you want it to listen on all networks, private and public. Bind to the private network Ip address of the MySQL to listen only on the private network. 2. MySQL will listen on a local unix socket as well, used when connecting to `localhost` 3. Setup new MySQL users to be restrictive in where they can connect from - a specific IP address or as small a range of IPs as possible. 4. Use firewalls in conjunction with MySQL user restrictions ## Resources * More on [MySQL User security](https://serversforhackers.com/video/mysql-user-security) * [Connecting to a remote MySQL server securely over SSH](https://serversforhackers.com/video/connecting-to-mysql-via-ssh) from your workstation * [MySQL bind-address, firewall and mysql users](https://serversforhackers.com/video/application-servers-and-mysql)

Side note, If you're interested in a service to help you manage MySQL-optimized, backup and (eventually) replication-enabled database servers, sign up here to let me know! The idea is to allow you to better manage your MySQL servers, taking advantage of many of MySQL's more advanced options, especially around backup and recovery.

Running Ansible 2 Programmatically

Ansible 2 is out, and that means it's time to upgrade the previous article on [Running Ansible Programmatically](https://serversforhackers.com/running-ansible-programmatically) for Ansible 2, which has significant API changes under the hood. ## Use Case At work, we are spinning up hosted trials for a historically on-premise product (no multi-tenancy). To ensure things run smoothly, we need logging and reporting of Ansible runs while these trials spin up or are updated. Each server instance (installation of the application) has unique data (license, domain configuration, etc). Running Ansible programmatically gives us the most flexibility and has proven to be a reliable way to go about this. At the cost of some code complexity, we gain the ability to avoid generating host and variable files on the system (although dynamic host generations may have let us do this - this is certainly not THE WAY™ to solve this problem). > There are ways to accomplish all of this without diving into Ansible's internal API. The trade-off seems to be control vs "running a CLI command from another program", which always feels dirty. > > Learning some of Ansible's internals was fun, so I went ahead and did it. Overall, there's just more control when calling the Ansible API programmatically. ## Install Dependencies Ansible 2 is the latest stable, so we don't need to do anything fancy to get it. We can get a virtual environment (Python 2.7, because I live in the stone-age) up and running and install dependencies into it. I'm using an Ubuntu server in this case: ```bash # Get pip sudo apt-get install -y python-pip # Get/udpate pip and virtualenv sudo pip install -U pip virtualenv # Create virtualenv cd /path/to/runner/script virtualenv ./.env source ./env/bin/activate # Install Ansible into the virtualenv pip install ansible ``` Then, with the virtual environment active, we call upon Ansible from our Python scripts. ## Ansible Config In the Ansible 1 series, I first tried and eventually gave up pon trying to set the path to the ansible.cfg file in code. I didn't even try again in Ansible 2, opting instead to (again) set the environmental variable `ANSIBLE_CONFIG`. That variable looks something like `ANSIBLE_CONFIG=/path/to/ansible.cfg`; The `ansible.cfg` file looks something like this: ```ini [defaults] log_path = /var/log/ansible/ansible.log callback_plugins = /path/to/project/callback_plugins:~/.ansible/plugins/callback_plugins/:/usr/share/ansible_plugins/callback_plugins [ssh_connection] ssh_args = -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s control_path = /home/your_username/.ansible/cp/ansible-ssh-%%h-%%p-%%r ``` Here's what is set here: * `log_path` - Uses the internal log plugin to log all actions to a file. I may turned this off in production in favor of our own log plugin which will send logs to another source (database or a log aggregator). * `callback_plugins` - The file path a custom logging plugin (or any plugin!) would be auto-loaded from, if any. **I add the project's path in first**, and then append the default paths after. * `ssh_args` - Sets SSH options as you would normally set with the `-o` flag. This controls how Ansible connects to the host server. The ones I used will prevent the prompt that asks if this is a trusted host (be cautious with doing that!) and ensure Ansible uses our private key to access the server. Check out the video on Logging in with SSH to see some examples of those options used with the SSH command. * `control_path` - I set the SSH control path to a directory writable by the user running the script/Ansible code (and thus using SSH to connect to the remote server). This is likely optional in reality. ## The Script(s) Like in our previous article, we'll dive right into the main script. ```python import os from tempfile import NamedTemporaryFile from ansible.inventory import Inventory from ansible.vars import VariableManager from ansible.parsing.dataloader import DataLoader from ansible.executor import playbook_executor from ansible.utils.display import Display class Options(object): """ Options class to replace Ansible OptParser """ def __init__(self, verbosity=None, inventory=None, listhosts=None, subset=None, module_paths=None, extra_vars=None, forks=None, ask_vault_pass=None, vault_password_files=None, new_vault_password_file=None, output_file=None, tags=None, skip_tags=None, one_line=None, tree=None, ask_sudo_pass=None, ask_su_pass=None, sudo=None, sudo_user=None, become=None, become_method=None, become_user=None, become_ask_pass=None, ask_pass=None, private_key_file=None, remote_user=None, connection=None, timeout=None, ssh_common_args=None, sftp_extra_args=None, scp_extra_args=None, ssh_extra_args=None, poll_interval=None, seconds=None, check=None, syntax=None, diff=None, force_handlers=None, flush_cache=None, listtasks=None, listtags=None, module_path=None): self.verbosity = verbosity self.inventory = inventory self.listhosts = listhosts self.subset = subset self.module_paths = module_paths self.extra_vars = extra_vars self.forks = forks self.ask_vault_pass = ask_vault_pass self.vault_password_files = vault_password_files self.new_vault_password_file = new_vault_password_file self.output_file = output_file self.tags = tags self.skip_tags = skip_tags self.one_line = one_line self.tree = tree self.ask_sudo_pass = ask_sudo_pass self.ask_su_pass = ask_su_pass self.sudo = sudo self.sudo_user = sudo_user self.become = become self.become_method = become_method self.become_user = become_user self.become_ask_pass = become_ask_pass self.ask_pass = ask_pass self.private_key_file = private_key_file self.remote_user = remote_user self.connection = connection self.timeout = timeout self.ssh_common_args = ssh_common_args self.sftp_extra_args = sftp_extra_args self.scp_extra_args = scp_extra_args self.ssh_extra_args = ssh_extra_args self.poll_interval = poll_interval self.seconds = seconds self.check = check self.syntax = syntax self.diff = diff self.force_handlers = force_handlers self.flush_cache = flush_cache self.listtasks = listtasks self.listtags = listtags self.module_path = module_path class Runner(object): def __init__(self, hostnames, playbook, private_key_file, run_data, become_pass, verbosity=0): self.run_data = run_data self.options = Options() self.options.private_key_file = private_key_file self.options.verbosity = verbosity self.options.connection = 'ssh' # Need a connection type "smart" or "ssh" self.options.become = True self.options.become_method = 'sudo' self.options.become_user = 'root' # Set global verbosity self.display = Display() self.display.verbosity = self.options.verbosity # Executor appears to have it's own # verbosity object/setting as well playbook_executor.verbosity = self.options.verbosity # Become Pass Needed if not logging in as user root passwords = {'become_pass': become_pass} # Gets data from YAML/JSON files self.loader = DataLoader() self.loader.set_vault_password(os.environ['VAULT_PASS']) # All the variables from all the various places self.variable_manager = VariableManager() self.variable_manager.extra_vars = self.run_data # Parse hosts, I haven't found a good way to # pass hosts in without using a parsed template :( # (Maybe you know how?) self.hosts = NamedTemporaryFile(delete=False) self.hosts.write("""[run_hosts] %s """ % hostnames) self.hosts.close() # This was my attempt to pass in hosts directly. # # Also Note: In py2.7, "isinstance(foo, str)" is valid for # latin chars only. Luckily, hostnames are # ascii-only, which overlaps latin charset ## if isinstance(hostnames, str): ## hostnames = {"customers": {"hosts": [hostnames]}} # Set inventory, using most of above objects self.inventory = Inventory(loader=self.loader, variable_manager=self.variable_manager, host_list=self.hosts.name) self.variable_manager.set_inventory(self.inventory) # Playbook to run. Assumes it is # local to this python file pb_dir = os.path.dirname(__file__) playbook = "%s/%s" % (pb_dir, playbook) # Setup playbook executor, but don't run until run() called self.pbex = playbook_executor.PlaybookExecutor( playbooks=[playbook], inventory=self.inventory, variable_manager=self.variable_manager, loader=self.loader, options=self.options, passwords=passwords) def run(self): # Results of PlaybookExecutor self.pbex.run() stats = self.pbex._tqm._stats # Test if success for record_logs run_success = True hosts = sorted(stats.processed.keys()) for h in hosts: t = stats.summarize(h) if t['unreachable'] > 0 or t['failures'] > 0: run_success = False # Dirty hack to send callback to save logs with data we want # Note that function "record_logs" is one I created and put into # the playbook callback file self.pbex._tqm.send_callback( 'record_logs', user_id=self.run_data['user_id'], success=run_success ) # Remove created temporary files os.remove(self.hosts.name) return stats ``` We'll cover what all this is doing - in particular, what that huge, ugly `Options` class is doing there. ### Some Assumptions I have a directory called `roles` in the same directory as this script. However, you can set the roles path in your `ansible.cfg` file. Playbooks are assumed to be in the same directory as this script as well. That's hard-coded above via `pb_dir = os.path.dirname(__file__)`. Lastly, as noted, when this code is run, ensure the `ANSIBLE_CONFIG` environment variable is set with the full path to the `ansible.cfg` file. Now let's start from the top to cover the file. ### Imports We import the Ansible objects that we'll use to run everything. The only non-Ansible imports are `os` (to get environment variables and the current file path of this script) and `NamedTemporaryFile`, useful for generating files with dynamic content for when Ansible expects a file. Note that Ansible has refactored a lot of code in Ansible 2 so things like the DataLoader and VariableMapper are in one class, making things like variable loading precedence much more consistent. ### Options We made a monster `Options` class, which is basically a glorified dict. During a regular CLI call to Ansible, an option parser gets available options (via `optparse`). These are options like "ask vault password", "become user", "limit hosts" and other common options we would pass to Ansible. Since we're not calling it via CLI, we need something to provide options. In it's place, we provide an object with the same properties, painfully taken from the [Github code](https://github.com/ansible/ansible/blob/v2.0.1.0-1/lib/ansible/cli/playbook.py) and some experimentation. > Note that this is similar to how Ansible itself runs Ansible programmatically in their [Python API docs](http://docs.ansible.com/ansible/developing_api.html). The use `options = Options(connection='local', module_path='/path/to/mymodules', forks=100, ... )`. This sets almost all the options (and more!) that you might pass a CLI call to `ansible` or `ansible-playbook`. We'll later fill out some of these options (not all, some just need to exist, even if they have a None value) and our Options object to the `PlaybookExecutor`. ### Runner Here's the magic - I made a `Runner` class, responsible for collecting needed data and running the Ansible Playbook executor. The runner needs a few bits of information, which of course you can customize to your needs. As noted, it uses an instance of the **Options** object and sets the important bits to it, such as the `become` options, `verbosity`, `private_key_file` location and more. In this case, we can pass our desired verbosity to the **Display** object, which will set how much is output to stdout when we run this. We create a **DataLoader** instance, which will load in YAML/JSON data from our roles. This gets passed a Vault password as well, in case you are encrypting any variable data using Ansible Vault. **Note** we have a second environmental variable, `VAULT_PASS`. You may want to pass that in instead of use an environmental variable - whatever works for you. Then the script creates a **VariableManager** object, which is responsible for adding in all variables from the various sources, and keeping [variable precedence](http://docs.ansible.com/ansible/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable) consistent. We pass in any `extra_vars` to this object. **This is the main way in which I've chosen to add in variable data for the roles to use.** It skips using Jinja2 or other methods of passing in host variables, although those methods are also available to you. Since our use case was to have a lot of custom data per customer, this method of passing variables to Ansible made sense to us. After that, we create a **NamedTemporaryFile** and create a small hosts file entry (I'm assuming one host at a time, you don't have to!). I avoided using Jinja2 there, but you could easily do that just like in the [previous article on running Ansible 1 programmatically](https://serversforhackers.com/running-ansible-programmatically). Next, we create an **Inventory** object and pass it the items it needs. **Finally** we create an instance of **PlaybookExecutor** with all our objects. That's then ready to run! The actual execution of the playbook is in a `run` method, so we can call it when we need to. The `__init__` method just sets everything up for us. This should run your roles against your hosts! It will still output the usual data to Stderr/Stdout. ## Callback Module I needed a callback module to log Ansible runs to a database. Here's how to do that! First, the callback module's path is set in the `ansible.cfg` file. The following callback module is in that defined directory location. Second, a note: You may have noticed that on the bottom of the Runner object, we reach deep into the PlaybookExecutor's Task Queue Manager object and tell it to send a (custom) callback. This object is meant to be a private property of the PlaybookExecutor, but Python's "We're all adults here" philosophy makes adding a custom callback possible. Here's the callback module: ```python from datetime import datetime from ansible.plugins.callback import CallbackBase from some_project.storage import Logs # A custom object to store to the database class PlayLogger: """Store log output in a single object. We create a new object per Ansible run """ def __init__(self): self.log = '' self.runtime = 0 def append(self, log_line): """append to log""" self.log += log_line+"\n\n" def banner(self, msg): """Output Trailing Stars""" width = 78 - len(msg) if width < 3: width = 3 filler = "*" * width return "\n%s %s " % (msg, filler) class CallbackModule(CallbackBase): """ Reference: https://github.com/ansible/ansible/blob/v2.0.0.2-1/lib/ansible/plugins/callback/default.py """ CALLBACK_VERSION = 2.0 CALLBACK_TYPE = 'stored' CALLBACK_NAME = 'database' def __init__(self): super(CallbackModule, self).__init__() self.logger = PlayLogger() self.start_time = datetime.now() def v2_runner_on_failed(self, result, ignore_errors=False): delegated_vars = result._result.get('_ansible_delegated_vars', None) # Catch an exception # This may never be called because default handler deletes # the exception, since Ansible thinks it knows better if 'exception' in result._result: # Extract the error message and log it error = result._result['exception'].strip().split('\n')[-1] self.logger.append(error) # Remove the exception from the result so it's not shown every time del result._result['exception'] # Else log the reason for the failure if result._task.loop and 'results' in result._result: self._process_items(result) # item_on_failed, item_on_skipped, item_on_ok else: if delegated_vars: self.logger.append("fatal: [%s -> %s]: FAILED! => %s" % (result._host.get_name(), delegated_vars['ansible_host'], self._dump_results(result._result))) else: self.logger.append("fatal: [%s]: FAILED! => %s" % (result._host.get_name(), self._dump_results(result._result))) def v2_runner_on_ok(self, result): self._clean_results(result._result, result._task.action) delegated_vars = result._result.get('_ansible_delegated_vars', None) if result._task.action == 'include': return elif result._result.get('changed', False): if delegated_vars: msg = "changed: [%s -> %s]" % (result._host.get_name(), delegated_vars['ansible_host']) else: msg = "changed: [%s]" % result._host.get_name() else: if delegated_vars: msg = "ok: [%s -> %s]" % (result._host.get_name(), delegated_vars['ansible_host']) else: msg = "ok: [%s]" % result._host.get_name() if result._task.loop and 'results' in result._result: self._process_items(result) # item_on_failed, item_on_skipped, item_on_ok else: self.logger.append(msg) def v2_runner_on_skipped(self, result): if result._task.loop and 'results' in result._result: self._process_items(result) # item_on_failed, item_on_skipped, item_on_ok else: msg = "skipping: [%s]" % result._host.get_name() self.logger.append(msg) def v2_runner_on_unreachable(self, result): delegated_vars = result._result.get('_ansible_delegated_vars', None) if delegated_vars: self.logger.append("fatal: [%s -> %s]: UNREACHABLE! => %s" % (result._host.get_name(), delegated_vars['ansible_host'], self._dump_results(result._result))) else: self.logger.append("fatal: [%s]: UNREACHABLE! => %s" % (result._host.get_name(), self._dump_results(result._result))) def v2_runner_on_no_hosts(self, task): self.logger.append("skipping: no hosts matched") def v2_playbook_on_task_start(self, task, is_conditional): self.logger.append("TASK [%s]" % task.get_name().strip()) def v2_playbook_on_play_start(self, play): name = play.get_name().strip() if not name: msg = "PLAY" else: msg = "PLAY [%s]" % name self.logger.append(msg) def v2_playbook_item_on_ok(self, result): delegated_vars = result._result.get('_ansible_delegated_vars', None) if result._task.action == 'include': return elif result._result.get('changed', False): if delegated_vars: msg = "changed: [%s -> %s]" % (result._host.get_name(), delegated_vars['ansible_host']) else: msg = "changed: [%s]" % result._host.get_name() else: if delegated_vars: msg = "ok: [%s -> %s]" % (result._host.get_name(), delegated_vars['ansible_host']) else: msg = "ok: [%s]" % result._host.get_name() msg += " => (item=%s)" % (result._result['item']) self.logger.append(msg) def v2_playbook_item_on_failed(self, result): delegated_vars = result._result.get('_ansible_delegated_vars', None) if 'exception' in result._result: # Extract the error message and log it error = result._result['exception'].strip().split('\n')[-1] self.logger.append(error) # Remove the exception from the result so it's not shown every time del result._result['exception'] if delegated_vars: self.logger.append("failed: [%s -> %s] => (item=%s) => %s" % (result._host.get_name(), delegated_vars['ansible_host'], result._result['item'], self._dump_results(result._result))) else: self.logger.append("failed: [%s] => (item=%s) => %s" % (result._host.get_name(), result._result['item'], self._dump_results(result._result))) def v2_playbook_item_on_skipped(self, result): msg = "skipping: [%s] => (item=%s) " % (result._host.get_name(), result._result['item']) self.logger.append(msg) def v2_playbook_on_stats(self, stats): run_time = datetime.now() - self.start_time self.logger.runtime = run_time.seconds # returns an int, unlike run_time.total_seconds() hosts = sorted(stats.processed.keys()) for h in hosts: t = stats.summarize(h) msg = "PLAY RECAP [%s] : %s %s %s %s %s" % ( h, "ok: %s" % (t['ok']), "changed: %s" % (t['changed']), "unreachable: %s" % (t['unreachable']), "skipped: %s" % (t['skipped']), "failed: %s" % (t['failures']), ) self.logger.append(msg) def record_logs(self, user_id, success=False): """ Special callback added to this callback plugin Called by Runner objet :param user_id: :return: """ log_storage = Logs() return log_storage.save_log(user_id, self.logger.log, self.logger.runtime, success) ``` One thing not shown here is the `some_project.storage.Logs` object that has some boiler plate for saving log output to a database. We have a `PlayLogger` object that does two things: 1. Concatenates log output string together 2. Times how long the Ansible run takes Then we have the callback module object which is basically copy and paste boiler plate from [the default callback module](https://github.com/ansible/ansible/blob/v2.0.0.2-1/lib/ansible/plugins/callback/default.py) with a few tweaks. In particular, I edit how Exceptions are handled (ignoring verbosity settings) and remove calls to "Display", since we're saving output to a log string rather than outputting data to Stderr/Stdout. The most interesting part here is the added method `record_logs`. This is the custom callback method we call from the `Runner` object. It Just Works™ and is amazing! In that method, we collect a `user_id` to give this Ansible run context (that's specific to our use case and we pass it a bunch more information in reality, including the ID of the server it was run on). ## Running It Here's how to use it, assuming we have a playbook called `run.yaml`. ```python from task import Runner # You may want this to run as user root instead # or make this an environmental variable, or # a CLI prompt. Whatever you want! become_user_password = 'foo-whatever' run_data: { 'user_id': 12345, 'foo': 'bar', 'baz': 'cux-or-whatever-this-one-is' } runner = task.Runner( hostnames='192.168.10.233' playbook='run.yaml', private_key='/home/user/.ssh/id_whatever', run_data=run_data, become_pass=become_user_password, verbosity=0 ) stats = runner.run() # Maybe do something with stats here? If you want! return stats ``` That's it! You can run Ansible 2 programmatically now!

Using New Relic's Free Server Monitoring

## Why New Relic? Purely due to random circumstances at work (and on the side), I hadn't used New Relic in quite some time (~3 years). Recently I took another look and found that their Server monitoring tier was separated from their other offerings - [APM](https://newrelic.com/application-monitoring), [Insights](https://newrelic.com/insights), Mobile, Browser and others. This isn't how I remembered it, but perhaps the distinction was just in the messaging/marketing. In any case, the exciting part is that **the Server monitoring tier is free**! While in terms of your web application, you'll get the most value out of New Relic with their paid offerings, I've found the free Server monitoring to still be an instant life saver, especially when monitoring many (70+) servers. > The free server monitoring tier is a lead-in to make it easier on your to decide to buy into the paid products, of course. Whether or not that stays free remains to be seen! At work, we run customer trials and paid hosting through AWS. Since the main application is an on-premise application (and not the typical multi-tenant code/database structure), we have a server per application instance. That makes for a lot of running servers! We've setup Cloudwatch with SNS for alarms and notifications, but it's still hard to use Cloudwatch to get a good overview of all the servers. We decided to out New Relic on a subset of the servers to see what insights we could find. We were immediately happy that we did. ## Install Installing the server monitoring daemon is really easy, particularly because they hand you OS-specific instructions on a silver platter when you choose to add a new server within their web console. For Debian/Ubuntu servers, these instructions are as follows (only really slightly tweaked for those not running as user root): ```bash # Add Repository and Signing Key echo deb http://apt.newrelic.com/debian/ newrelic non-free | sudo tee /etc/apt/sources.list.d/newrelic.list wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add - # Update repos and install their package sudo apt-get update sudo apt-get install newrelic-sysmond # Configure the local install with your license key: sudo nrsysmond-config --set license_key=1234567890abcdefghijklmnopqrstuvwxyz # Start the service: sudo service newrelic-sysmond start ``` Their instructions will conveniently give you your license key. **Don't blindly copy and paste** the license key above 🚨. ## Configure You technically don't need any more configuration, but I do like to change one more thing. The server name as shown in New Relic will match the server's host name, which you can find by running the command `hostname`. This might not be the way you want to identify it, so you can either wait for the server to appear in New Relic, or configure the hostname ahead of time in the New Relic configuration installed on your server. I choose the latter approach. Edit the `/etc/newrelic/nrsysmond.cfg` file and change the `hostname` section to whatever is appropriate for your server. ``` # You'll find your license key in here, which you can set manually # instead of running the `nrsysmond-config` command license_key=1234567890abcdefghijklmnopqrstuvwxyz # Find the hostname entry and set it hostname=some-server.example.com ``` After making any changes, start (or restart) the `newrelic-sysmond` service: ```bash sudo service newrelic-sysmond restart ``` You'll soon see that server appear in your New Relic account under the Server section, without any further work on your part. ## What We Watch This monitoring on New Relic isn't super advanced, but it has some really nice features: ### Processes and process count. Note that PHP is a bit out of control there - that was one thing that we were alerted to, as the server has a high CPU usage over their default threshold of 15 minutes. ![server process list and count](https://s3.amazonaws.com/sfh-assets/process_list.png) We found some zombie PHP processes, which we killed off quickly. We immediately saw CPU reduction back to normal levels. ```bash # Find a process $ ps aux | grep php www-data 5889 0.0 3.3 438856 34004 ? S Jan12 0:17 php: /path/to/file.php # Kill it based on process ID $ sudo kill 5889 ``` ![cpu usage reduction](https://s3.amazonaws.com/sfh-assets/cpu_reduction.png) We can get a more detailed view of each process type as well, sorting by memory and cpu usage: ![process memory usage](https://s3.amazonaws.com/sfh-assets/process_memory2.png) ### Disk Usage New Relic also gave us alerts about disk usage. One server had excess files taking up a lot of space. We were able to reduce the disk spaced used by manually deleting extra media files, giving us time to provision more hard drive space. ![disk drive usage](https://s3.amazonaws.com/sfh-assets/disk_available.png) ### Alerts Alerts may not send you alerts off the bat, but can be easily configured. In any case, when viewing a list of all your servers, you will see some on the sidebar worth paying attention to. ![new relic alerts](https://s3.amazonaws.com/sfh-assets/recent_events.png) ## Automated with Ansible I automated this installation with a super simple Ansible Playbook (specific to Debian/Ubuntu). The basics are as follows: File `tasks/main.yml`: ```yaml --- - name: Add New Relic Repository apt_repository: repo: deb http://apt.newrelic.com/debian/ newrelic non-free state: present - name: Add New Relic Signing Key apt_key: url: https://download.newrelic.com/548C16BF.gpg state: present - name: Update Repositories apt: name: newrelic-sysmond update_cache: yes state: present - name: Add New Relic Config template: src: nrsysmond.cfg.j2 dest: /etc/newrelic/nrsysmond.cfg owner: newrelic group: newrelic - name: Restart New Relic service: name: newrelic-sysmond state: restarted ``` File `templates/nrsysmond.cfg.j2`, with 2 variables: ``` # # Option : license_key # Value : 40-character hexadecimal string provided by New Relic. This is # required in order for the server monitor to start. # Default: none # license_key={{ new_relic_key }} # Much taken out for brevity.... # # Setting: hostname # Type : string # Purpose: Sets the name of the host (max 64 characters) that you wish to use # for reporting. This is usually determined automatically on startup # but you may want to change it if, for example, you have machine # generated hostnames that are not visually useful (for example, the # names generated by Docker containers). # Default: The system configured host name # hostname={{ domain }} # Much taken out for brevity.... ``` Then you can set your defaults or `vars/main.yml` file with your variable definitions: ```yaml new_relic_key: 1234567890abcdefghijklmnopqrstuvwxyz domain: whatever.example.com ``` > When using Ansible for this, I actually provided the domain variable has a [host variable](http://docs.ansible.com/ansible/intro_inventory.html#host-variables), included in the hosts inventory file for each host being updated.

TCP load balancing with Nginx (SSL Pass-thru)

Learn to use Nginx 1.9.* to load balance TCP traffic. In this case, we'll setup SSL Passthrough to pass SSL traffic received at the load balancer onto the web servers.

Installing PHP-7 with Memcached

Let's install PHP7 and Nginx on a new Ubuntu 14.04 server, and manually build the (not yet packaged) memcached module for PHP7.

HTTP 2.0 With Nginx

Installing the latest Nginx from the MAINLINE branch to get http2 support.

Curl With HTTP2 Support

Installing HTTP2 support with the curl command

Using apt-get

Using apt-get to install, search, query and remove software on Debian and Ubuntu servers.

Creating and Using SSH Keys

Generate an SSH key and use it to log into a user on a new server.

Amazon RDS

I've recently been [investigating interest levels](http://sqlops.launchrock.com/) in a small application that spins up master/replica MySQL servers (with back-ups, yadda yadda yadda). Tangentially from this, I've also gotten a few questions about how such a set would work within Amazon's [Managed Relational Database Service](https://aws.amazon.com/rds/) (RDS). Amazon has a few concepts going on which makes the picture of how everything works together a little hazy at first. Here's my attempt at clearing up how the RDS service works. ## Regions and Availability Zones First, we should cover [Regions vs Availability Zones](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html). AWS has **~11 regions**. These are separate physical locations, very geographically distant. **For example, there is:** * US-East (N. Virginia) * US-West (N. Cali & Oregon) * EU-West (Ireland) * AP-Southeast (Singapore, Sydney) * and 6 others spread across the world Each region has **multiple Availability Zones** (AZ). These zones are also physically different locations, but within the same general geographical area. > These availability zones are close together and connected by high-bandwidth networks, making the network latency between them pretty low. ![google maps of aws locations](https://s3.amazonaws.com/sfh-assets/google_map.png) In theory, **us-east-1d** can explode without affecting **us-east-1e**. However, **practical experience says otherwise**. About twice a year, we've seen large service interruptions which tend to affect entire regions (looking at you, **us-east-1**). **Some generalisms about regions and availability zones:** 1. Network connections between AZ's are usually quite fast, often fast enough to not worry about inter-AZ communication for most web applications. YMMV. 2. Network connections between Regions can/will be slower due to geographical distance. 3. Services (EC2, SES, etc) have different prices in different regions. **Us-east-1** tends to be the cheapest (but most over-used). With that out of the way, let's actually talk about RDS. Specifically, we'll talk about MySQL. Some details will be different per database. ## The Basics RDS is a managed database service. In return for paying an arm and a leg (although it's still cheaper than hiring a human being!), you get a managed database. ### Server Size One of your first decisions in setting up an RDS instance will be to decide how large the database is. Because the cheaper [T instance type uses a system of CPU credits](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html#t2-instances-cpu-credits) to minimize cost for burst work loads, I tend to stay away from it with RDS, which often has a more consistent work load. The T series of instances will throttle available CPU after credits are used. Of course, depending on your work load, using a server type that's better for burst usage can make sense. ![rds server sizes](https://s3.amazonaws.com/sfh-assets/rds_server_sizes.png) I like the **m3.large** (2 vCPU, 7.5GiB RAM) instance size to start with, although that's probably overkill for small/starting applications. > If you're in a position where you can fit your application and database workload into a 1-2 CPU server/2-4GB RAM, I don't suggest using AWS at all. Linode/Digital Ocean is much more cost-effective and simple. In **us-east-1**, with 100gb of storage, that's **~$144/mo** (not counting bandwidth usage) with no failover (Multi-AZ) nor replication. ![aws price rds single instance](https://s3.amazonaws.com/sfh-assets/rds_single.png) **For this price, you get:** * A managed database instance that (probably?) won't suffer too many issues. * Backups back for n (selectable) days. These allow point-in-time recovery, **granularity up to the second** 🎉! ![restore rds point in time backup](https://s3.amazonaws.com/sfh-assets/rds_restore.png) ## Multi-AZ The next decision to make is whether or not to enable a [Multi-AZ deployment](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html). Amazon pushes hard on this in their documentation, but **this is not replication**. At least, not the replication you're thinking of. **Multi-AZ is about two things:** 1. Durability 2. Fast failover In addition to Amazon's default durability metrics, Multi-AZ deployments will make a **synchronous** copy of your database to *another* AZ within the region you're creating the database. If your RDS instance is in **us-east-1d**, the copy might end up in **us-east-1c**. *Synchronous* is the interesting part there. Amazon is guaranteeing that any change that is made on the primary database is also made in the copy. This isn't something that a database will usually guarantee without a lot of potential slowdown, as it's typically done via transactional locking - the database waits for all replicas to report that it's completed the transaction before moving on to the next query. Since AZ's in Amazon are physically different locations, my best guess on how this is accomplished is by proxying TCP connections so they go to both AZ's at once. I don't think this is a copy done on the hard disk level (e.g. raid array) due to the physical locations being different. ### What's it good for? The best reason for this setup is so **your application can failover fast** in the event that the AZ goes down. According to the [opening announcement of Multi-AZ](https://aws.amazon.com/blogs/aws/amazon-rds-multi-az-deployment/) (from 2010), this failover takes ~3 minutes. Amazon takes care of this for you - the database host doesn't even change, so your application should switch over without too much interruption. Other than that phantom 3 minutes, I suppose. I haven't had this happen to me yet to find out. The other nice thing is that backups are done on the replica database, so it doesn't cause any IO congestion on your production database. Note that **you can't use this replica** unless the production server fails over. Also, this considerably increases your cost. Selecting this brings it up to **~$293/mo**, a little more than double the cost. ![rds multi-az pricing](https://s3.amazonaws.com/sfh-assets/rds_multi_az.png) ### Considerations The only thing to watch out for here is if you are putting RDS in the same AZ as your application servers. If your application servers are also down, then having a database failing over to another AZ isn't really going to help you. That's why it's also recommended to use at least 2 AZ's for any application. This means using a load balanced environment for any application where it isn't OK for it to go down once in a while. I don't really worry about network lag between AZ's, and AWS is essentially setup with the assumption that you will not either. They want you to use multiple AZ's for RDS and for your applications. ## Read Replication [RDS does indeed offer replication](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.MySQL) that you can use to distribute **read** queries (there isn't any option for multiple write replicas, because databases are hard). Like regular old database replication, this is asynchronous. Your replica is not guaranteed to be up to speed with your master database. Other than the obvious benefits of distributing expensive read queries, AWS does some other smart things. For example, databases go into maintenance occasionally for updates. In Multi-AZ deployments, the database can fall over to a backup. If you use replication, your application can continue to make read queries to a replication database. You need to be careful, however, as **it's possible to accidentally send write queries to the replication database** (I believe this is an aspect of MySQL, were a super user can override a read-only configuration). Read replication databases can be [promoted to a "regular" database](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote). In fact, this is how you get [cross-region replication](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.XRgn) with RDS. For "serious" (important, larger budget) deployments, this may be an important step in keeping your application available and/or keeping low response time for read queries around the world. Note that **replication lag will increase using cross-region deployment**, due to the large geographic distance between regions. Read replicas are charged just like a regular database instance, so if you're using a database with Multi-AZ and one read-replica, you're paying **~438/mo**, roughly triple what you pay for just one instance. ![rds multi-az with replication price](https://s3.amazonaws.com/sfh-assets/rds_multi_az_replication.png) Although you can **save big money overall by using reserved instances**: ![rds price reduction with reserved instances](https://s3.amazonaws.com/sfh-assets/rds_reserved.png) Let's see that math for one instance, with Multi-AZ and a read replica: * $438.46/mo for 3 years = **$15,784.56** * Reserved for 3 yrs with partial up-front payment is $237.32/mo + $2,158.62 = **$10,702.14** If you know you'll have the database for that long, reserved instances gives you a significant savings. Keep in mind you can, if workload and compliance permits, create many individual databases with an RDS instance as well. ## The rent is too damned high! Those plugged into the Entrepreneurial Scene™ may "know" that products "should" (all those air quotes!) be priced based on the value the product brings: 1. Revenue brought in 2. Time savings 3. Money savings AWS, in theory, replaces the technical know-how of a consultant or full-time DBA, and Amazon charges based on (under!) that value. With RDS, you're paying to relieve administrative burden and disaster recovery. It's not cheap.

Side note, If you're interested in a less expensive, but unmanaged MySQL-optimized, backup and (eventually) replication service, sign up here to let me know! The idea with this application is it will allow you to better manage your own MySQL instances, while taking advantage of many RDS-like features.

PHP-FPM: Process Management

Learn how to manage how PHP-FPM creates and uses PHP processes to get the most out of your server.

PHP-FPM: Configuration the Listen Directive

PHP-FPM can listen on multiple sockets. I also listen on Unix sockets, or TCP sockets. See how this works and how to ensure Nginx is properly sending requests to PHP-FPM.

PHP-FPM: Multiple Resource Pools

Split PHP-FPM into multiple resource pools to separate applications and enhance system security!

Apache and PHP-FPM

Learn to hook Apache up to PHP-FPM using Apache's proxy modules.

Automatic Security Updates: CentOS

On CentOS servers, we can enable the automatic download and installation of security updates. Let's see how to protect our servers by installing the `yum-cron` package!

Backup and Restore MySQL with mysqldump

Let's see how to backup and restore MySQL databases, with a few extra tricks, using the `mysqldump` tool.

MySQL User Security

Learn how to setup users in MySQL to be accessible from various hosts, remote or local, as well as ranges of IP addresses. We'll cover giving users specific grants to determine what they can do to the databases they are assigned.

LEMP on RedHat/CentOS

Let's install Nginx, PHP and MySQL (MariaDB) on a RedHat or CentOS 7 server!

LEMP on Debian

Learn how to install & configure Nginx, MySQL and PHP on a Debian server, which has minor differences from the Ubuntu install video.

LEMP on Ubuntu

Learn how to install & configure Nginx, MySQL and PHP on an Ubuntu server.

Your First LAMP Server on CentOS/RedHat 7

Get up and running with a LAMP server for your PHP applications. * Use `yum search` and `yum install` commands to find and install packages * Install Apache and MariaDB (MySQL) * Install PHP and all the modules you'll need to get started

Automatic Security Updates

On Debian/Ubuntu servers, we can enable the automatic download and installation of security updates. Let's see how to protect our servers by enabling unattended upgrades!

Sudo and Sudoers Configuration

We can configure who can use the sudo command and how. You may have noticed that the Vagrant user on your development server can use sudo without a password. Similarly, AWS servers allow the same thing. Find out how that's done, and much more!

Users and SSH Security

Many server providers only give us a root login when we spin up a server. Other ones create a user which can use "sudo" without needing a password (just as dangerous). In this video we'll see how to manage users and their server access abilities.

Process Monitoring with Systemd

For most Linux distributions, Systemd will be the officially support init process. This will, among many other things, monitor our processes like SysVInit and Upstart did. Get a leg up and learn how to use it now!