Bear's Log

a journal writ one beer at a time

Bear's Feed

Python Testing with Flask

On Monday, 15th February 2016, I gave a talk for Developer Week 2016 in San Francisco about best-practices for testing Python. For the talk I created a functional Python Flask app so that the talk would be grounded in reality.

A lot of research went into this talk so I wanted to make sure that all of the resources were available - the full talk, talk notes and the example project are all on GitHub:

This research is still on going! Just today I learned about an even better way to use uWSGI to run the Flask app that uses the PyEnv virtual environment without a lot of tricks – I’m working that up into another article and I’ll be updating Tenki to reflect this new method.

None

IndieWeb Public Timeline Project

I was working on a talk I’m going to be giving next week (2016-02-16) at Developer Week 2016 about Python Testing when I noticed the conversation happening in the IndieWeb IRC Channel:

# 08:35 tantek showed that stream of notes somewhere on the home page
# 08:35 tantek just the notes
# 08:35 aaronpk ah
# 08:35 tantek so it had that old school twitter feel
# 08:36 aaronpk that'd be fun
# 08:52 benwerd tantek aaronpk: I really love that idea
# 08:52 benwerd A hub for people posting notes to their indieweb homepages

This caught my attention because i’ve had this on my personal setup for years as a stream of all of the items/feeds that I have interest in - my own personal stream of events. The code I use for that is extremely ugly and very tied to my personal data so I couldn’t just pop it onto a public site but it is something that I could use as a framework tho!

So I spent the weekend setting up a Python Flask application and started pulling over the various parts that made sense (to me) - so far it does the following:

  • Displays an h-feed of items that have been submitted to it
  • Uses Server Sent Events (SSE) to send new data to the page. Big thanks to Aaron for reminding me about Nginx Stream Push Module!
  • Uses the Bridgy Publish method of having a hidden /publish URL pointing to the site to trigger getting the post into the stream
  • Has the beginnings of an API to pull JSON data for the stream based on domain and date range
  • Will soon have an IndieAuth login page for each domain to allow people to opt-out of being included in case of indirect mentions causing their posts to be displayed.

I need to finish some deploy work, like getting a SSL Certificate in place and standing up the code some place other than my personal server, and then I will start sharing the URL :)

code, indieweb

Working with Nginx logs from bash

Earlier today on the IndieWeb IRC Channel Aaron was talking about writing some code to parse his Nginx logs to get a list of 404s and I commented that it could be done easily enough by setting Nginx to emit JSON formatted logs.

Now that is definitely an option, but then I remembered that I had written a quick-n-dirty parser to spit out a JSON blob per logline and that with the amazingly handy jq tool the JSON could quickly be filtered or acted on!

So I took a break from work to refactor the quick-n-dirty parser to use apache-log-parser to increase it’s flexibility (the old version was hardcoded for a specific log line format) and added some helpful comments.

logs.py will generate via stdout a stream of JSON for each line of the log file:

./logs.py < /var/log/nginx/bear.im.log | jq '. | { method, status, request_first_line }'

code

IndieWeb 2016 Commitment Completed

This holiday break I settled on a couple of items for my IndieWeb 2016 Commitment but it seems that the gods were yet again quirky and I ended up being sick or out-of-sorts for the holidays. I did eventually get a chance to work on the items and a couple bikeshed paint jobs later I am happy with the result!

I refactored my code to be even more event driven and pulled out all of the generation specific code from the Flask app and into my site’s generation code - which now has a helper daemon that listens to redis pubsub events to know when a post or webmention event has happened.

Yep, I should now be able to receive webmentions and have them added to the proper post properly (don’t ask how many small bugs were in the old code that basically crippled it - ugh!)

Now of course the itch list is going to grow as I now need to add edit/update, different post notes and also start adding in the fancier salmention items.

2016 is going to be fun!

mutterings,indieweb

Indie-Stats Update

Tonight I worked thru quite a few issues in the bug list for Indie-Stats

I also took a pass at fixing the auth related errors for the /login flow, but that is still buggy :(

By far the biggest item tonight was adding a very basic API to the code which, for the moment, consists of only two endpoints.

Domain List

GET https://indie-stats.com/api/v1/domains will return a JSON formatted list of the domains being tracked.

Domain Information

GET https://indie-stats.com/api/v1/domains/<domain> will return a JSON formatted hash of all of the meta-data stored for the given domain.

Super simple, yep, but it’s a start and it also (finally!) will start other folks down the path to being able to make use of this data.

Please do file an issue in the bug tracker for anything you want me to add to the API.

mutterings, code

Kenkou Update v0.5

I was pinged at work with a question - do you know of any tool that can do health checks from a cronjob?

Well I replied … I just happen to have one! I created Kenkou a while ago as a tool for that very thing - to run thru a list of resources and check on them.

After gathering some of the requirements they had I realized that I needed to bring Kenkou up to date with some of the changes I had started but never published at &yet and, since I do love yak shaving, I also decided to refactor some of the internals :)

Last night I refactored a whole slew of things, added tests, created support to create a command line script and also added a feature where it would return a json blob as the result to STDOUT. Now you can check on HTTP, DNS and even TLS Certificates from the command line with optional reporting to PagerDuty and/or email via PostageApp.

Kenkou v0.5.2 is now published and also available on PyPI.

mutterings, code

IndieWeb 2016 Commitment

Last year I started my IndieWeb 2015 Commitment kinda late, so I’m working on what that itch-to-scratch will be early to allow myself the time to work on it over the holidays.

I’ll post an updates here and to my IndieWeb Users page as I add (and complete!) them.

  • Finish micropub endpoint
    • enable delete
    • enable update
  • Finish wiring up plumbing needed to generate a page once a micropub event happens

mutterings

Purposeful Change

Spend more than a couple of days around me in a work setting and you will surely hear me say “purposeful” or “mindful” when talking about making decisions or choices. I use the phrase to help others work through the multiple choices that are available so they can select the one that works for them in the moment they are currently in.

This is not necessarily the best choice or the popular choice.

Two weeks ago I was layed off from &yet because the management team made the choice to stop trying to develop a product (which is why I was hired, to help on the Operations side of that decision) and refocus on their core skills. For a couple of days I went thru the normal emotional swings that anyone leaving a wonderful and amazing team like &yet has, and worked to change my thoughts and actions back to what I wanted instead of what my now former team needed.

The past few years saw me move even more from a more hands-on Ops focus to managing the Ops team and then onto managing the development team in order to help them keep maximum productivity and focus on the billable hours – which for consultants is the crank that needs to be turned continuously. The &yet developers all were such a joy to work with and were so understanding when I tripped-up or flubbed the myriad of small things that any fast moving team has to track.

All of the above came into my mind as I started to figure out what could be next for me.

I had already started to receive emails and pings from friends and peers about possible options but one quickly rose to the top of my mind as it was from a former OSAF (Open Source Applications Foundation) co-worker Travis (@tvachon) asking if I would be interested in joining the company he was currently at: Circle CI. The timing of his ping was a couple of days before I was layed off, so my reply was a quick “nope, very happy here, but thanks!” which is funny and odd only because that is also what happened six years ago at OSAF! At that time I had been approached by the CTO of Seesmic during my holiday time in 2009 because of some bugs I found in the XMPP server he had setup; had replied “nope, very happy here, but thanks!” and the very next week all of the OSAF team was layed off…

Two events does not a pattern make, but still very interesting :)

I reached out to Travis asking for more information and after the interviews, meetings and three days of on-site work with the SRE Team I was offered a position!

The challenges and work being done by the Circle CI team is, in my mind, at the cutting edge of the Continuous Delivery realm. This is exciting to me because of my Build/Release roots but also for my Operations and Dev/Ops side – the meta conversation involved when your talking about continuously delivering a product that is used by a lot of folks to enable their own agile operations, is in it’s own right worthy of attention. Add on to that the directions and changes they are looking to make to enable future growth… OMGBBQ yea!

So while the path before me may have changed in direction, I don’t fear or worry because I know that I try to walk my path in a mindful way with an open mind to learning new things.

['mutterings']

Kaku A Micropub Toolkit

One of the challenges to having a pure static site is that you don’t easily get some of the fancier features that other Indieweb sites get, like Micropub endpoints and some of the newer Salmentions work to allow comment threads. The majority of these features are normally implemented using Javascript, something that I don’t allow here. Instead I run a Python Flask application that handles those requests and shuffles them off to a task queue for processing.

Tonight I just got done with a lot of little fiddly bits of infrastructure and plumbing for the toolkits that make up my Indieweb Services endpoints:

  • Ninka is the toolkit to work with Indieauth, it handles the discovery of authentication endpoints and the validation of auth tokens,
  • Ronkyuu handles discovery and processing of Webmentions,
  • Kaku is the Flask app itself that sits behind the web server and handles the /micropub, /token, /login and /webmention routes for the site

Some of the above is still “opinionated” as it may contain configuration values that are specific to my environment, but i’m working on clearing all of those out. But first I need to finish the refactoring of the task worker to take the events generated by the endpoints and generate the proper files and trigger a rebuild.

['mutterings']

Static Site Hybrid

My $dayjob has kept me quite busy and this leads to me often catching up on how things in various communities I lurk/inhabit after I’ve been AFK for a couple weeks…

The latest is the Indieweb Community and in particular an article posted by Kyle that asks:

Looking for web frameworks that provide 1. server-rendered HTML, 2. a background work queue, 3. support for websockets. My current patchwork of Flask, Tornado, Celery, and Redis is working but feels somehow not right.

This is exactly what I’m working towards with my own site - it’s purely static except for a minimal event handling Flask Application.

To be sure, Kyle’s current setup is by far much more advanced than mine, but this is an area that I truly want to help out in.

sigh - time to go add more items to my personal “itch” list :/

['mutterings']