Today's system update was mostly infrastructure changes and security fixes, but there's one big visible change -- we now support Python 3.6!

The new version is only available for accounts using the new dangermouse image, so if you'd like to be switched over and get not just the latest Python, but also new shiny versions of all of our installed system images, just send us a message using the "Send feedback" link.

A week ago, one of the sites we were hosting was reported to us by the Russian authorities (specifically, the Federal Service for Supervision in the Sphere of Telecom, Information Technologies and Mass Communications [ROSKOMNADZOR]) for hosting illegal content. They said that we must take it down, or risk having the associated IP address blocked in Russia.

In the current political climate, it's very important to note that the site in question was offering services that are illegal not just in Russia, but in the UK, the United States, and almost every other country we can think of. Our terms and conditions do not allow sites engaging in "any activities that are illegal". We were glad to have it brought to our attention on that basis. If the situation had been different, and (say) we'd been told to take down a website expressing political views by the government of a country hostile to those views, our response would have been very different.

But in this case, the situation was clear-cut. We took the site down, notified its owner that we'd done so and explained why, and responded to the original notification saying that we had done this.

Last night, we discovered that the site had been re-created, and we took it down again and blocked all access for the user. But it seems that this was too late; today a different Russian customer contacted us and said that they could not access their own site, or PythonAnywhere. Further investigation determined that the IP address in question had indeed been blocked by the Russian government. It's accessible from everywhere else in the world -- but not from Russia. This is a problem, because the address in question serves our own site, and a large portion of our customers'. And we have a lot of customers in Russia!

We do have the capability to move sites from IP address to IP address (that's part of why we strongly recommend that customers with custom domains use CNAMEs instead of A records to set up their websites), but this was not designed to be done under emergency circumstances, so it takes time. We felt that in this case, a rapid response was required.

As such, we've implemented a secondary IP address that will work for any PythonAnywhere-hosted website. Over the course of this weekend, we will be testing it. If it is essential that your website be visible in Russia, please contact us on [email protected] and we can move you across during this testing period. If it all looks good early next week, we'll move all other sites on the old IP address over, including our own.

[update] As of 17:15 on 2017-03-07, we've migrated all websites over to the new, unblocked IP address. The only sites that should still be affected are those configured via A records (or similar) to route directly to the old address (a small number). We'll be scanning the sites we host and emailing the affected customers tomorrow.

Honestly, it's strange. We'll work on a bunch of features, care about them deeply for a few days or weeks, commit them to the CI server, and then when we come to deploy them a little while later, we'll have almost forgotten about them. Take today, when Glenn and I were discussing writing the blog post for the release

-- "Not much user-visible stuff in this one was there? Just infrastructure I think..."

-- "Let's have a look. Oh yes, we fixed the ipad text editor. And we did the disable-webapps-on-downgrade thing. Oh yeah, and change-webapp-python-version, people have been asking for that. Oh, wow, and shared files! I'd almost totally forgotten!"

So actually, dear users, lots of nice things to show you.

File sharing

People have been asking us since forever about whether they could use PythonAnywhere to share code with their friends. or to show off that famous text-based guessing game we've all made early on in our programming careers. And, after years of saying "I keep telling people there's no demand for it", we've finally managed to make a start.

If you open up the Editor on PythonAnywhere you'll see a new button marked Share.

You'll be able to get a link that you can share with your friends, who'll then be able to view your code, and, if they dare, copy it into their own accounts and run it.

We're keen to know what you think, so do send feedback!

Change Python version for a web app

Another feature request, more minor this time; you'll also see a new button that'll let you change the version of Python for an existing web app. Sadly the button won't magically convert all your code from Python 2 to Python 3 though, so that's still up to you...

More debugging info on the "Unhandled Exception" page.

When there's a bug, or your code raises an exception for whatever reason, your site will return our standard "Unhandled Exception" page. We've now enhanced it so that, if it notices you're currently logged into PythonAnywhere and are the owner of the site, it will show you some extra debugging info, that's not visible to other users.

Why not introduce some bugs into your code and see for yourself?

ipad fix, and other bits and pieces

We finally gave up on using the fully-featured syntax-highlighting editor on ipads (it seemed like it worked but it really didn't, once you tried to do anything remotely complicated) and have reverted to using a simple textarea.

If you're trying to use PythonAnywhere on a mobile device and notice any problems with the editor, do let us know, and we'll see if we can do the same for your platform.

Other than that, nothing major! A small improvement to the workflow for people who downgrade and re-upgrade their accounts, a fix to a bug with __init__.py in django projects,

Keep your suggestions and comments coming, thanks for being our users and customers, and speak soon!

Harry + the team.

There's an explosion of chat apps and bots at the moment, and it's easy to see why. They're a useful new way of interacting with computer systems, they're interesting to code, and they're actually surprisingly easy to create.

This blog post shows how you can get a simple bot up and running, using Telegram. Telegram isn't as popular a messaging platform as WhatsApp or Skype, but it's much easier to build bots for. You'll need a normal computer and also a phone on which you can install the Telegram app. When you've finished working through the steps here, you'll have a bot that can have an almost-plausible conversation with you.

It uses PythonAnywhere, which probably isn't very surprising given the name of this blog ;-) You can do everything in here using a free PythonAnywhere account, and the bot you wind up with will be fully-functional. You'll only need a paid-for account if your bot starts getting lots of users -- of the order of thousands of messages a day.

So, without further ado, let's get started!

Creating a bot

The first thing you need to do is tell Telegram that you want to create a bot. For this, you'll need a Telegram account -- install their app on your phone, and get it set up.

Next, start a conversation with the "BotFather". This is a bot that Telegram themselves run, and it controls the creation and registration of bots on their platform. On the Android version of their app, here's what you do (other platforms are similar)

• Tap on the start conversation button in the bottom right to start a new conversation.
• Tap the magnifying glass "Search" icon near the top right.
• Type "botfather".
• Tap on the "@BotFather" that appears. Make sure it has a blue checkmark next to it
• It will display a welcome message.
• Click the "Start" button.
• Send it a message "/newbot"
• It will ask for a name for your bot. I'm going to call mine "GTDemo" but you should choose a name that's unique to you.
• Next it will ask for a username; I'll use "GTDemoBot"
• If all goes well, it will print out a message telling you that the bot was created. There's one important bit of information in there that you'll need for later: the token to access the HTTP API. It will be a long string of alphanumeric characters, maybe with a colon in it. To keep this for later, I copied the message on my phone and then emailed it to myself -- not super-secure, but probably safe enough if you're not going to be putting anything secret into your bot.

Right, so let's check that your bot is created, even if it's currently not very talkative. Start a conversation with it, using the same method to start a chat as you did with the BotFather. Hopefully you'll be able to find it and start a chat, but when you click the "Start" button, nothing will happen.

No big surprise there. Let's make it do something.

A first simple bot

On your computer:

• I recommend you create a new PythonAnywhere account for this tutorial -- we'll be creating a website later, and if you already have a website, we don't want to get confused between the two. You can have multiple free accounts, even using the same email address, so there's no harm in signing up for a new one. Just create a free "Beginner" one.
• Start a "Bash" console

• In there, run

  pip3.5 install --user telepot
  

  this will install (for your own PythonAnywhere account) the excellent telepot Python library, which hides some of the complexities of talking to Telegram's API. Wait for the process to complete.

• Next, click the PythonAnywhere logo to the top left to go back to the PythonAnywhere dashboard.

• Go to the "Files" tab.
• In the "Enter new file name", type a filename ending with ".py" for your bot's code -- say, firstsimplebot.py -- and click the "New file" button.

• Enter the following code, replacing "YOUR_AUTHORIZATION_TOKEN" with the token that the BotFather gave you earlier:

  import telepot
  import time
  import urllib3
  
  # You can leave this bit out if you're using a paid PythonAnywhere account
  proxy_url = "http://proxy.server:3128"
  telepot.api._pools = {
      'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
  }
  telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))
  # end of the stuff that's only needed for free accounts
  
  bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')
  
  def handle(msg):
      content_type, chat_type, chat_id = telepot.glance(msg)
      print(content_type, chat_type, chat_id)
  
      if content_type == 'text':
          bot.sendMessage(chat_id, "You said '{}'".format(msg["text"]))
  
  bot.message_loop(handle)
  
  print ('Listening ...')
  
  # Keep the program running.
  while 1:
      time.sleep(10)
  

• Click the ">>> Run this file" button at the bottom of the page.

• A console will appear at the bottom of the page. After a little while, it should display "Listening ..."

Now go back to your phone. In the chat with your bot, type "Hello". It should almost immediately reply "You said 'Hello'".

If you take another look at the console on PythonAnywhere, you'll see that it will have printed out some information about the message -- probably something like

text private 321518746

Woo! A working bot :-)

Let's work through that code bit by bit.

    import telepot
    import time
    import urllib3

This bit just imports the Python modules that we're going to use.

    # You can leave this bit out if you're using a paid PythonAnywhere account
    proxy_url = "http://proxy.server:3128"
    telepot.api._pools = {
        'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
    }
    telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))
    # end of the stuff that's only needed for free accounts

Like the comments say, this stuff is only needed if you're using a free "Beginner" PythonAnywhere account -- we are, of course, for this tutorial, but you can remove it if you want to reuse the code in a paid account later. It's there because free accounts can only connect outwards to particular external websites, and those connections have to go through a proxy server. Many APIs pick up the details of the proxy server automatically from their system environment when they're running, but telepot doesn't. It's not a problem, it just means we have to be a bit more explicit and say "use this proxy over here".

    bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')

Now we get to the core of the code. This line uses telepot to connect to Telegram's server.

Next, we define a function that knows how to handle messages from Telepot.

    def handle(msg):
        content_type, chat_type, chat_id = telepot.glance(msg)

The first thing we do is pull the useful information out of the message, using telepot's glance utility function.

        print(content_type, chat_type, chat_id)

...we print out some of the information, just for debugging purposes.

        if content_type == 'text':
            bot.sendMessage(chat_id, "You said '{}'".format(msg["text"]))

We only handle text messages for the time being; speech recognition is a bit outside the bounds of this tutorial... When we get a text message, we simply reply back telling the person what they said.

So that's the end of the message-handler function. Back to the main code:

    bot.message_loop(handle)

This tells telepot to start running a message loop. This is a background thread that will keep running until the program exits; it listens on the connection that was opened to Telegram and waits for incoming messages. When they come in, it calls our handle function with the details.

    print ('Listening ...')

So we print out a message to our own console to show that we're up and running...

    # Keep the program running.
    while 1:
        time.sleep(10)

And then we wait forever. Like I said, the telepot message loop will only keep running until our program exits, so we want to stop it from exiting.

So now we have a working bot and we know how it works. Let's make it better.

Moving to webhooks

The bot that you have right now is just running inside the console underneath your editor. It will actually keep running for quite a while, but if PythonAnywhere do any system maintenance work that requires restarting the server it's on, it will stop and not restart. That's obviously not much good for a bot, so let's fix it.

What we'll use is Telegram's "webhooks" API. Webhooks are a different way of connecting to Telegram. Our previous code made an out-bound connection from PythonAnywhere to Telegram, then relied on Telegram sending messages down that connection for processing. With webhooks, things are reversed. We essentially tell Telegram, "when my bot receives a message, connect to PythonAnywhere and pass on the message". And the "connect to PythonAnywhere" bit is done by creating a web application to run inside your PythonAnywhere account that will serve a really simple API.

If any of that sounds daunting, don't worry. It's actually pretty simple, and the instructions are detailed :-)

• Click on the PythonAnywhere logo to go back to the PythonAnywhere dashboard.
• On the "Consoles" tab, click on the small "X" next to the "firstsimplebot.py" console. This is important -- it will kill the running bot that we've already created so that it doesn't interfere with the the new one we're about to create.
• Go to the "Web" tab.
• Click the "Add a new web app" button.
• The first page will just tell you that the web app will be hosted at your-pythonanywhere-username.pythonanywhere.com. Click next.
• On the next page, choose the "Flask" web framework. Flask is a great choice for simple websites that are designed for APIs.
• On the next step, choose "Python 3.5". That's the version we installed telepot for.
• On the next page, just accept the default location for your Flask app. It will be something like /home/your-pythonanywhere-username/mysite/flask_app.py
• After a short wait, you'll see an "All done!" message and your website will be set up. There will be a link to it -- follow the link and you should see a message saying "Hello from Flask!"

So now you have a simple website running that just displays one message. What we need to do next is configure it so that instead, it's running an API that Telegram can connect to. And we also need to tell Telegram that it's there, and which bot it's there to handle.

• Click your browser's "Back" button to go back to the "Web" tab.
• Look down the page a bit, and you'll see the "Code" section.
• In that section, open the "Go to directory" link for the "Source code" in a new browser tab. (It'll be useful to keep the "Web" tab around for later.)
• In the new tab, you'll see the "Files" page. One file will be called "flask_app.py"; click on it to go to the editor.

Enter the following code. Don't worry about what it does yet, we'll go through that in a second. But don't forget to replace YOUR_AUTHORIZATION_TOKEN with your Telegram HTTP API token, and YOUR_PYTHONANYWHERE_USERNAME with your PythonAnywhere username. Also replace A_SECRET_NUMBER with a number that only you know; a good way to get one that's properly random is to go to this online GUID generator, which will generate a unique number like "c04a4995-a7e2-4bf5-b8ab-d7599105d1d1".

from flask import Flask, request
import telepot
import urllib3

proxy_url = "http://proxy.server:3128"
telepot.api._pools = {
    'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
}
telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))

secret = "A_SECRET_NUMBER"
bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')
bot.setWebhook("https://YOUR_PYTHONANYWHERE_USERNAME.pythonanywhere.com/{}".format(secret), max_connections=1)

app = Flask(__name__)

@app.route('/{}'.format(secret), methods=["POST"])
def telegram_webhook():
    update = request.get_json()
    if "message" in update:
        text = update["message"]["text"]
        chat_id = update["message"]["chat"]["id"]
        bot.sendMessage(chat_id, "From the web: you said '{}'".format(text))
    return "OK"

Once you've entered the code and made sure you've made the three substitutions:

• Save the file
• Switch to the browser tab with your web app setup in it.
• Click the green "Reload" button near the top.
• Wait for the "spinner" to finish.

Back on your phone, send another message. This time you should get a message back saying clearly that it came from the web. So now we have a bot using webhooks!

Let's work through the code now:

from flask import Flask, request
import telepot
import urllib3

So again, we import some Python modules. This time as well as the telepot and the urllib3 stuff that we need to talk to Telegram, we use some stuff from Flask.

proxy_url = "http://proxy.server:3128"
telepot.api._pools = {
    'default': urllib3.ProxyManager(proxy_url=proxy_url, num_pools=3, maxsize=10, retries=False, timeout=30),
}
telepot.api._onetime_pool_spec = (urllib3.ProxyManager, dict(proxy_url=proxy_url, num_pools=1, maxsize=1, retries=False, timeout=30))

Once again, the stuff we need to access Telegram from a free PythonAnywhere account.

secret = "A_SECRET_NUMBER"

Now, this is a bit of best-practice for Telegram bots using webhooks. Your bot is running as a publicly-accessible website. Anyone in the world could connect to it. And of course we really don't want random people to be able to connect, pretending to be Telegram, and make it say inappropriate things... so, we're going to say that the website only serves up one page, and the URL for that page is unguessable. This should make things reasonably safe. You'll see the code for that in a moment.

bot = telepot.Bot('YOUR_AUTHORIZATION_TOKEN')

We connect to Telegram using telepot, just like we did before.

bot.setWebhook("https://YOUR_PYTHONANYWHERE_USERNAME.pythonanywhere.com/{}".format(secret), max_connections=1)

We use telepot to send a message to Telegram saying "when my bot gets a message, this is the URL to send stuff to". This, of course, not only contains the host name for your website with your PythonAnywhere username, it also includes the hopefully-unguessable secret that we defined earlier. It's also worth noting that it uses secure HTTPS rather than HTTP -- all websites on PythonAnywhere, even free ones, get HTTPS by default, and Telegram (quite sensibly) will only send webhooks over HTTPS.

app = Flask(__name__)

Now we create a Flask application to handle requests.

@app.route('/{}'.format(secret), methods=["POST"])
def telegram_webhook():

This is some Flask code to say "when you get a POST request on the secret URL, run the following function". If you want to learn more about how Flask works, we have a tutorial on that too.

    update = request.get_json()

Telegram sends stuff to bots using JSON encoding, so we decode it to get a Python dictionary.

    if "message" in update:

If the thing we received from Telegran was a message...

        text = update["message"]["text"]
        chat_id = update["message"]["chat"]["id"]

...extract the text of the message, and the ID of the chat session which it forms a part of...

        bot.sendMessage(chat_id, "From the web: you said '{}'".format(text))

...then send the reply back using telepot...

    return "OK"

...and return something to Telegram to say that all is OK.

So now we have, and hopefully understand, a simple Telegram bot that will keep running pretty much forever! Websites on PythonAnywhere free accounts last for three months, and then you can extend them for another three months -- and three months later you can extend again, and so on, as many times as you like. So as long as you're willing to log in to PythonAnywhere four times a year, you're all set :-)

But the bot is pretty boring at the moment. Let's make it a little more interesting.

Introducing Eliza

Sorry, Hamilton fans, not Angelica and Peggy's sister. Eliza is an early natural language processing system, and the normal implementation simulates a Rogerian psychotherapist -- a kind of therapist who simply turns every question back on the patient. That makes it an easy one to implement and use in a bot like this.

Doubly conveniently, the Python nltk package provides an implementation of Eliza, so we don't even need to code it ourselves.

Let's check out how it works. Go to the PythonAnywhere dashboard, and start a new Bash console. In it, try out Eliza in a Python 3.5 interpreter like this (the answers it gives you may vary):

19:20 ~ $ python3.5
Python 3.5.2 (default, Jul 17 2016, 00:00:00) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nltk.chat.eliza import eliza_chatbot
>>> eliza_chatbot.respond("Hello")
'Hi there... how are you today?'
>>> eliza_chatbot.respond("I'm well")
"Why do you think you're well?"
>>> eliza_chatbot.respond("I'm not sure")
'How does being not sure make you feel?'
>>> eliza_chatbot.respond("A little confused, to be honest")
'Very interesting.'
>>> eliza_chatbot.respond("Is it?")
'Why do you ask that?'

OK, that should give you the feel of how it works. Let's code it up.

• Go back to the dashboard, to the "Files" tab, then to your Flask app's code (probably inside mysite/flask_app.py).

• Add a new import to the top:

  from nltk.chat.eliza import eliza_chatbot
  

• Inside the telegram_webhook function, replace this line:

  bot.sendMessage(chat_id, "From the web: you said '{}'".format(text))
  

  with this:

  if text == "/start":
      bot.sendMessage(chat_id, "Hello, I'm the therapist.  How can I help?")
  else:
      bot.sendMessage(chat_id, eliza_chatbot.respond(text))
  

  When someone first connects to a Telegram bot, the app sends you a text message saying "/start", so we have a special case for that so that Eliza doesn't say something weird like "Why do you say that /start?". But all other messages we simply send to Eliza for processing, then return.

• Go to the "Web" tab and hit the green "Reload" button.

Back on your phone, let's start a new session so that we can chat with Eliza afresh.

• In the old session, click the "Hamburger" menu to the top right and select "Delete chat"
• Click on the button to start a chat again, then search for your bot again, click it.
• Start the chat by clicking the "Start" button.

And now you should be able to talk to your chatbot! Many happy hours of not-very-useful therapy to be had :-)

That's it!

That's all for this tutorial. If you hit any problems, leave a comment below. And if you have any thoughts on how we could extend it, just let us know. Have fun, and happy botting!

Welcome to our Christmas newsletter! Featuring a selection of the Internet's very worst Christmas-themed animated Gifs!

Here's what we've been up to:

Is it called "Reindeer Shaving" at this time of year?

One change you might have noticed is that the Consoles page now has little Xs for killing consoles. The original idea was just to change them from being links that cause a page refresh to being ajax calls, which would let you kill multiple consoles at the same time. Somehow though, that small user interface tweak turned into the whole office deciding to treat that How it feels to learn JavaScript in 2016 comedy blog post as if it were an instruction manual, and we have now spent several days knee deep in React, ES6, promises, webpack, npm, Enzyme, fetch, promises, promises, and many, many more. Still, by the end of it, it all worked, and we have to conclude that ES6 is much nicer to work with than horrible old javascript.

Deep Tarpit

Santa needs some names for his "naughty or nice" list, and we've got some ideas for him!

Our tarpit has always been a soft limit, so going over your CPU allowance isn't the end of the world, it just slows your processes down, so that people who are within quota get first dibs on the precious cpu-cycles. Some people are taking the mick though, and we've now built some tools to detect these "deep tarpit" users, and kill their processes. So, more CPU-time for the good users, and naughty users now get the lowest priority of all, "kill".

The inside scoop from the forums

• out of disk quota? something taking up your space? here's how to track it down

• django version mismatches and the correct wsgi settings

• some tips for setting up a telegram bot

• yay non-English keyboard layout compatibility

• giles talks about multitenancy

New modules

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that we have plenty of batteries included.

• We have been switching people over to the new dangermouse image! Send us an email if you want to switch.

• You may have noticed a InsecurePlatformWarning when using pip > 9.0. This is a bug/regression in the pip code for python versions 2.7.9 or lower. We are keeping track of the issue and here are some ways to work around it for now.

New whitelisted sites

Paying PythonAnywhere customers get unrestricted Internet access, but if you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist to stop hackers from creating dummy accounts to hide their identities when breaking into other people's websites.

But we encourage you to suggest new sites that should be on the whitelist! Our rule is, if it's got an official public API (which means that the site's owners are encouraging automated access to their server) then we'll whitelist it. Just drop us a line with a link to the API docs.

Here are some sites we've added since our last newsletter:

.okta.com
.unwiredlabs.com
app.datadoghq.com
api.discogs.com
api.test.netbanx.com
api.test.paysafe.com
api.tinychat.com
apis.vworld.kr
cps.combain.com
epayment.bbs.no
epayment-test.bbs.no
repo.continuum.io
services.arcgisonline.com
steamcommunity.com
timesofindia.indiatimes.com
vizier.u-strasbg.fr
www.datapowerlearning.com
www.nbrb.by
www.rediff.com

What's coming up next year?

Over Christmas we're working on a few small tweaks and improvements, and for early next year, we're looking to start work on a feature that many of you have been requesting for a long time. But shush! No spoilers ;)

Thanks for reading our newsletter! Tune in the same time next month (ish) for more news from PythonAnywhere.

This morning we deployed a new version of PythonAnywhere -- we've blogged about the new stuff that you can see in it, and this post is a run-down on why it took longer than we were expecting.

Our normal system updates tend to take between 15 and 20 minutes. Today's was meant to take about 40 minutes -- more about why later -- but it wound up taking over an hour and forty minutes. That definitely warrants an explanation.

Situation normal

It's worth starting by explaining how a normal system update works. There are three main classes of servers that make up a PythonAnywhere cluster:

• File storage servers -- two types of server, one of which stores the files in your private file space, and one that stores and processes system logs (like the ones you can access from "Web" tab).
• Database servers -- for MySQL and Postgres.
• "Ephemeral" servers -- basically, everything else. Loadbalancers, web servers, console server, servers for IPython/Jupyter notebooks, a "housekeeping" server that does admin stuff like sending out emails, and so on.

In a normal system update, we replace just the ephemeral servers. The file storage and database servers are pretty simple -- they run the minimum code required to do what they do, and rarely need updating.

Our process for that is:

• The day before, start up a complete new cluster of ephemeral servers, hooked up to the existing file and database storage servers, but without anything being routed to it.
• When we get in early for the deploy, we start up a server that just responds to all requests with a "system down for maintenance" response, then switch all of our incoming traffic over to it.
• Then we stop the old ephemeral servers, and run any database migrations we need to apply to the Django code that runs the main PythonAnywhere website.
• We set things up so that when we access PythonAnywhere from our own office network, we bypass the "system maintenance" server and see the new cluster, and then we sanity-check to make sure everything's OK.
• If it is, then we reroute all traffic to the new cluster, and we're done.

All of that tends to take between 15 and 20 minutes.

Sometimes, we need to update the file storage servers. That's a slightly more complicated arrangement.

• When we start up the new cluster the day before, we create completely new storage servers as well as ephemeral servers. But we don't hook the new storage servers up to the storage itself. Storage is contained on Amazon EC2 Elastic Block Storage (EBS), which can be moved from machine to machine but can only be attached to one machine at any given time -- so we leave it on the old storage servers initially.
• When we do the deploy, after we've stopped the old ephemeral servers, we need to:
  • disconnect the old EBS devices from the old storage servers
  • connect them to the new storage servers
  • start up the various system services that turn those block devices into a usable filesystem
  • run some system checks to make sure everything's OK and the new ephemeral servers can see the storage on the new storage servers.
• Once all that is done, then we check out the new system end-to-end, and, if all is well, go live.

Because all of these steps are scripted, the extra work doesn't normally take all that long; maybe half an hour in total. Worst case, 40 minutes.

Today's system update was even larger than that, however.

Virtual Private Clouds

We've recently started hitting some limits to what we can do on AWS. When we started using the service, they had just one kind of server virtualization -- for those who are familiar with this stuff, they only supported paravirtualization (PV). But now they support a new kind, Hardware Virtual Machines (HVM). This new kind has a large number of benefits in terms of the kinds of servers supported and their performance characteristics in an environment like PythonAnywhere.

The problem is, in order to use HVM, we had to move PythonAnywhere out of what AWS call "EC2-classic" and into what they call a Virtual Private Cloud. (Use of a VPC is not normally technically required for HVM, but there was a nice little cascade of dependencies that meant that in our case, it was.)

We'd been testing PythonAnywhere-in-a-VPC in our development and integration environments for some time, and by this system update we felt we were as ready as we ever would be to go ahead with it.

We decided we'd move all of the servers into the VPC in one go. There's a thing called "ClassicLink" which could have allowed us to leave some servers in EC2-classic and move others to the VPC, but using it would have been complex, easy to get wrong, and would only have stored up trouble for the future. So we decided we'd do one of our updates where we'd replace the file storage server as well as the ephemeral ones. The night before, we'd start a full set of new servers inside a VPC, and then when we switched over to them, we'd have moved into it.

But there was a wrinkle -- as well as moving the file storage servers, we'd need to move the database servers. We (rightly) didn't think moving Postgres would be a huge deal, because we manage our own Postgres infrastructure. But moving the MySQL databases would require us to convert them over to the VPC using Amazon's interface.

We timed this with a number of test database servers, and found it took between five and ten minutes. We checked with Amazon to make sure that this was a typical time, and they confirmed.

So -- half an hour for a full cluster replacement, plus ten minutes at the worst case for the MySQL move, forty minutes in total.

What could possibly go wrong?

What went wrong

There were a few delays in today's update, but nothing too major. The code to migrate the EBS storage for the storage servers wasn't quite right, due to a change we'd made but hadn't tested enough, but we have detailed checklists for doing that process (after all, keeping people's data safe is the top priority in anything like this) so we were able to work around them. Everything was looking OK, and we were ready to go live after about 50 minutes -- a bit of a delay, but not too bad.

It was when we run the script to make the new cluster live that we hit the real problem.

Earlier on I glossed over how we route traffic away from the old cluster and over to the site-down server, and then back again when we go live with the new one. Now's the time for an explanation. AWS has a great feature called an "Elastic IP address", or EIP. An EIP is an IP address that's associated with an AWS account, which can be associated with any server that you're running. Our load-balancer endpoints all use EIPs. So to switch into maintenance mode, we simply change all of our EIPs so that instead of being associated with the old cluster, they're associated with the site-down system. To switch back, we move them from the site-down system to the new cluster.

So we ran the script to switch from the site-down system to the new cluster, and got a swathe of errors. For each EIP it said

You must specify an allocation id when mapping an address to a VPC instance

A bit of frantic googling, and we discovered something: EIPs are either associated with the EC2-classic system or with the VPC system. You can't use an EC2-classic EIP in a VPC, or vice versa.

There is a way to convert an EC2-classic EIP to a VPC one, however, so we started looking in to that. Weirdly, the boto API that we normally use to script our interactions with AWS doesn't support that particular API call. And there's no way to do it in the AWS web interface. However, we found that the AWS command-line tool (which we've never used before) does have a way to do it. So, a quick pip install aws-cli, then we started IPython and ran a script to convert all of our EIPs to VPC:

In [4]: for name, eip in eips.items():
   ...:     print name
   ...:     !aws ec2 move-address-to-vpc --public-ip $eip
   ...:     
web-9
{
    "Status": "MoveInProgress"
}
web-3
{
    "Status": "MoveInProgress"
}
consoles-4.pythonanywhere.com
{
    "Status": "MoveInProgress"
}
web-10
{
    "Status": "MoveInProgress"
}
consoles-6.pythonanywhere.com
{
    "Status": "MoveInProgress"
}
web-5

An error occurred (AddressLimitExceeded) when calling the MoveAddressToVpc operation: The maximum limit on Elastic IP addresses has been reached.

Uh-oh. We have a couple of dozen EIPs associated with our account -- we had a vague recollection of having had to increase the limit in the past. But it looked like (and we confirmed) that for some reason there's a completely separate limit of EIPs for VPCs -- the one we had previously increased only applied to EC2-classic.

AWS tech support to the rescue

The advantage of spending thousands of dollars a month on AWS, and paying extra for their premium support package, is that when stuff like this goes wrong, there's always someone you can reach out to. We logged a support case with "production systems down" priority, and were on a chat with a kindly support team member called George within five minutes. He was able to confirm that our VPC EIP limit was just five, and bumped it up enough to cover the EIPs we were moving across.

Unfortunately limit increases like that take some time to propagate across the AWS system, so while that was happening, we took a look at the details of the error that we'd got originally -- again, it was

You must specify an allocation id when mapping an address to a VPC instance

The boto API call that we were using took two parameters -- the IP address we wanted to move, and the instance ID of the server we wanted to move it to. Looking at the EIPs that we had successfully moved into the VPC world, they had a new number associated with them -- an "Allocation ID". And it appeared that boto now required this ID as well as the EIP's actual IP address when it was asked to associate an EIP.

So we reworked the code so that it could do that, and waited for the limit increase to come through.

Finally, it did, so we reran our little IPython script. All of the EIPs moved across. A bit of further scripting and we had allocation IDs ready for all of the EIPs, and could re-run the script to switch everything over and make the new cluster live. And everything worked.

Phew

Lessons learned

It's hard to draw much in the way of lessons from this. One obvious problem is that we didn't know that EIPs have to be moved into the VPC environment, and we didn't know that because our testing environment doesn't use EIPs. That's clearly something we need to fix. But it's unlikely that we would have spotted the fact that there was a different limit of inside-VPC EIPs associated with our account -- and because we had to move the EIPs over while the system was down, we wouldn't have discovered that until it was too late; this morning's update would have taken an hour and a half rather than an hour and forty minutes, which isn't a huge improvement.

I suppose the most important lesson is that large system updates sometimes go wrong -- so we should schedule them to happen later. The quietest time of day across our systems as a whole is from about 5am to 8am UTC.
Things ramp up pretty quickly after that, as people come in to work across western Europe. For a normal update, taking half an hour, 7am is a perfectly OK time to do it. But for future big changes, we'll start off at 5 or 6.

The main driver for our release this morning was a move, behind the scenes, to put our servers into a "VPC", and despite the fact that it'll have no visible impact, it was a significant change to the infrastructure, and not without its challenges, as you'll hear in more detail from Giles later :)

One Yak, Fully Shaved.

One change you might notice is that the Consoles page has changed, and includes some little red Xs for killing consoles. The original idea was just to change them from being links that cause a page refresh to being ajax calls, which would let you kill multiple consoles at the same time. Somehow though, that small user interface tweak turned into the whole office deciding to treat that How it feels to learn JavaScript in 2016 comedy blog post as if it were an instruction manual, and we have now spent several days knee deep in React, ES6, promises, webpack, npm, Enzyme, fetch, promises, promises, and many, many more. Still, by the end of it, it all worked, and we have to conclude that ES6 is much nicer to work with than horrible old javascript.

The Deep Tarpit

The tarpit is one of the key ways we balance the resource needs of our various users. What happens when you exceed your CPU quota is that your processes still run, but they get a lower priority compared to people who are still within the amount they paid for. That's been working fairly well, but as with all things, we notice there's a power law at work, and there are a small number of users who regularly go massively over their tarpit limit. We've added some code that will automatically kill processes of these kinds of users, and send them a friendly notification email. Bad programmer! No biscuit.

We've also added some code to limit the amount of output in consoles, so that kids (and adults) whose first Python program is while True: print("farts") will have less of an impact on the system. Although plenty of farts will still be printed, fear ye not.

The upshot of all that should be that console performance will hopefully be a little more consistent from now on.

Better support for non-English keyboards

We do our best to avoid the classic Anglocentric, parochial laziness of imagining that the world ends with ASCII, but it takes work! For a while we've known that users on certain operating systems with certain keyboard types & layouts would have difficulties entering certain text into our consoles. So we've rolled out the ability to switch from hterm to xterm.js for our client-side terminal emulation.

If you'd like to try it out, give us a shout and we can switch it on for you. NB - keyboard shortcuts for copy + paste will be different, it'll be Ctrl+Ins / Shift+Ins instead of Ctrl-C & Ctrl-V.

General bugfixes and security fixes

And the usual retinue of bug fixes. Some of which were (minor) security fixes, incidentally, as reported by some enthusiastic security researchers. Find out more about our bug bounty if that describes you!

We try to get a newsletter out every month, but sometimes we just get too distracted working on our latest and greatest features to manage it. It wasn't that we were all out in Norway doing an opera, honest :-)

Here's what we were up to:

Two Factor Auth

Something you know, something you own, something borrowed, something blue...

We were very pleased to roll out two-factor authentication, meaning that you can now add a second step to your account login if you want extra security. We support the Google Authenticator token generator. More details on your accounts tab.

The inside scoop

• A couple of people were being caught by an error in FileZilla SFTP, which happens if anything in your .bashrc echoes anything to stdout -- a particularly sneaky bug to track down. (although the most common problem with SSH is still the case-sensitive nature of usernames...)

• Ping! Our own Harry gives some tips on disabling console chimes

• Bossman Giles gives a quick rundown of how to do blue green deployment on PythonAnywhere

• willpaycoin was worried about the Dirty Cow (geddit? a copy-on-write vulnerability. harhar). But he needn't have, our ever-vigilant cow security brigade were on it.

New batteries included? A whole new image more like!

Although you can install Python packages on PythonAnywhere yourself, we like to make sure that our preinstalled batteries included are nice and up-to-date. A few weeks ago we released a whole new system image which we're calling "dangermouse", which is the default for new users. If you are still on the "classic" image (see? it's alphabetical!) and want to switch, drop us an email and we'll upgrade you.

New whitelisted sites

Paying PythonAnywhere customers get unrestricted Internet access, but if you're a free PythonAnywhere user, you may have hit problems when writing code that tries to access sites elsewhere on the Internet. We have to restrict you to sites on a whitelist to stop hackers from creating dummy accounts to hide their identities when breaking into other people's websites.

But we really do encourage you to suggest new sites that should be on the whitelist. Our rule is, if it's got an official public API, which means that the site's owners are encouraging automated access to their server, then we'll whitelist it. Just drop us a line with a link to the API docs.

Here are some sites we've added since our last newsletter:

• api.mailgun.net
• auth0.com
• botframework.com, pandorabots.com
• api.easypost.com
• api.hipchat.com
• api.skype.net/com
• api.wikimapia.org
• backend.deviantart.com
• login.microsoftonline.com
• strawpoll.me
• www.hipchat.com
• xboxapi.com

So if you've ever dreamed of building a weather-forecasting chatbot that posts deviantart images on skype directly from your xbox, now's the time!

A few minor things

Behind the scenes we made some fairly hefty infrastructure upgrades to the way our fileservers and web servers balance load, but that shouldn't be visible, except in increased reliability perhaps. There were a couple of minor security patches, and we got print preview working on Ipython Notebooks, which I'm sure everyone was just dying to see.

That's about it! Thanks for reading, and tune in at the same time next month (ish) for more exciting news from your favourite Python PaaS.

We were sad to hear about the sudden departure of our frenemies* at Nitrous.

Whenever something like this happens, there's a conversation about the reliability of SaaS services in the face of VC pressure -- do you really want to trust your business to a company that has VCs breathing down their neck setting deadlines for exits, and ready to pull the plug if things take a dip?

So we want to reassure you that we haven't taken on any big VC money. And although we do occasionally wonder, from our cramped little London office above a Burrito shop**, what the lives of developers in startups with 7-figure investments and swanky San Francisco offices with their table-tennis tables and elaborate trousers, we mostly don't resent it much. Considering we're now cashflow-positive and our investor (Hi Robert!) has just guaranteed the sole cash injection we need to achieve actual-for-reals-profitability, and that that investment is two orders of magnitude less than what Nitrous took on, well, let's say we hope we're a reasonably low-risk bet as far as our customers are concerned, and we plan to be around for a while.

Anyways, so much to say that, if you were using Nitrous to code Python, and you still fancy having a place in THE CLOUD to store your stuff, head on over to our site, and we'll be happy to see you.

So, in honour of our defunct friends at Nitrous, why not enjoy this naughties dark DnB classic by the unmistakeable Bad Company. Rinseout!

* yes, "frenemies", its a thing. if you're cringing, rest assured that, being British, I cringed the first time I heard this word too. So. American. But like all things Silicon Valley, there's a nugget of truth in it, and more than that. competitors can be friends (someone should remind politicians of this), and particularly when you're out creating new markets, a competitor doing a good job can genuinely increase the size of your own customer pool. even when they don't die!.

** actually it's now a Vietnamese. Nice Banh-mi, delicious roast belly pork, could do with being a little more generous on the meat portions though

This morning's system update went smoothly, and we've made a couple of great changes :-)

Improved website routing

This one should be pretty much transparent to you, but we've revamped the way we route requests for the websites that we host; this should speed things up for some people.

Noisy neighbours always cause problems, in the real world and on the Internet. When someone writes a website that hogs system resources on PythonAnywhere, sometimes it can impact other people who happen to be on the same server. Naturally, we monitor the system, and when we find a particularly badly-behaved website we notify its owner by email and ask them to fix it -- or in extreme cases, if it's causing serious problems, we shut it down. But that's far from ideal.

Today's update makes that all a lot better. We've given ourselves, the system administrators, fine-grained control over where websites run. So now, if we see a website that's causing slowdowns for other users, as well as notifying the owner so that they can fix it, we can move it right away onto a server where it won't impact other people. We're calling it "putting them in the sin bin"...

Security is important

...as people have reminded us frequently in suspiciously-similar Tweets. And they're right! So we've implemented two-factor authentication, using Google Authenticator (or any other TOTP app). It's currently going through a short internal-only testing process (in other words, we've switched it on for our own accounts to see if it breaks anything) and if all is well, we'll provide it as an option for everyone next week.

On the subject of security, we've also fixed a couple of bugs: Nikhil Mittal reported a CSRF issue on PythonAnywhere that would have allowed an attacker who knew both your username and the internal database ID of one of your scheduled tasks to delete that task, if they tricked you into visiting a web page that they controlled while you were logged in to PythonAnywhere. It wouldn't have given the attacker access to any of your data, but it could have been really irritating, and we're glad it was reported so that we could fix it. Bug: fixed. Bug bounty: paid. Nikhil also reported some issues around our email confirmation system, which we've also fixed.

...and the rest

As always, we've put in a number of user interface tweaks, including fixing the print preview on IPython notebooks.

That's it!

Thanks for reading, and for using PythonAnywhere :-)