After starting the year with curses work, I decided to do some more!
Currently we lack Soft Label Key functions as specified by POSIX. What are these? Glad you asked! Soft Label Keys ripoff a line from the bottom of the screen and allow for up to 8 keys to be displayed there. You can define lab...
The LLVM toolchain internally uses several std::call_once calls, within LLVM and LLDB codebase. The GNU libstdc++ library implements C++11 std::call_once with wrapping pthread_once(3) and a function pointer passed as a TLS (Thread Local Storage) variable. Currently this setup isn't functional with shared linking on NetBSD and apparently on few other systems and hardware platforms. This issue is still unresolved and its status is traced in PR 51139. As of now, there is a walkaround originally prepared for NetBSD, and adapted for others - namely llvm::call_once.
The PT_GET_SIGINFO interface is designed to read signal information sent to tracee. A program can also fake a signal to be passed to a debugged process, with included destination thread or the whole process. A debugger requires detailed signal information in order to distinguish the exact trap reason easily and precisely, for example whether a program stopped on a software defined breakpoint or a single step trap.
This interface is modeled after the Linux specific calls: PTRACE_GETSIGINFO and PTRACE_SETSIGINFO, but adapted for the NetBSD use-case. FreeBSD currently has no way to fake signal information.
I have modeled this interface to be most efficient in terms of determination what exact thread received the signal. Thanks to it, a process does not need to examine each thread separately and performs only a single ptrace(2) call. This makes a significant difference in a massively threaded software.
The signal information accessors introduce a new structure ptrace_siginfo:
/*
* Signal Information structure
*/
typedef struct ptrace_siginfo {
siginfo_t psi_siginfo; /* signal information structure */
lwpid_t psi_lwpid; /* destination LWP of the signal
* value 0 means the whole process
* (route signal to all LWPs) */
} ptrace_siginfo_t;
I've included in the <sys/ptrace.h> header required file to define the siginfo_t type - <sys/siginfo.h>.
This makes sure that no existing software will break during build due to a missing type definition.
Forking a process is one of the fundamental design features of a UNIX operating system. There are two basic types of forks: fork(2) and vfork(2) ones. The fork(2) one is designed to spawn a mostly independent child from parent's memory layout, however this operation is costy. In order to address it and optimize the forking operation there is vfork(2) which shares address space with its parent. From the ptrace(2) point of view, the difference between fork(2) and vfork(2) calls is whether a parent giving birth to its child is suspended waiting on child termination or exec() operation or not.
Currently there is an interface to monitor fork(2) operations - PTRACE_FORK - however NetBSD missed the vfork(2) ones. I've added two new functions: PTRACE_VFORK and PTRACE_VFORK_DONE. The former is supposed to notify why a parent gives a vfork(2)-like birth to its child and later when a child is born with vfork(2) from its parent - perfoming a handshake with two SIGTRAP signals emitted from the kernel. Once the parent is resuming it gets a notification for vfork(2) done.
These events throw SIGTRAP signal with the TRAP_CHLD property. Currently PTRACE_VFORK is a stub and it's planned to be fully implemented later.
A debugger requires an interface to monitor thread birth and termination. This is needed in order to properly track the thread list of a process and ensure that every thread has for example applied watchpoints.
I've added two events:
This interface reuses the EVENT_MASK and PROCESS_STATE interface. It means that it shares these calls with PTRACE_FORK, PTRACE_VFORK and PTRACE_VFORK_DONE.
To achieve this goal, I've changed the following structure:
typedef struct ptrace_state {
int pe_report_event;
pid_t pe_other_pid;
} ptrace_state_t;
to
typedef struct ptrace_state {
int pe_report_event;
union {
pid_t _pe_other_pid;
lwpid_t _pe_lwp;
} _option;
} ptrace_state_t;
#define pe_other_pid _option._pe_other_pid
#define pe_lwp _option._pe_lwp
This change keeps the size of ptrace_state_t unchanged as both pid_t and lwpid_t
are defined as an int32_t-like integer.
New struct form should not break existing software and be source and binary compatible with it.
I've introduced a new SIGTAP type for thread events: TRAP_LWP.
I've introduced a few changes to the current interface. One of them is allowing to mix single-step operation with enabled hardware assisted watchpoints. The other one was added new extension pw_type to the ptrace_watchpoint structure.
Features in ELF, DWARF, CTF, DTrace are out of scope for the above list.
After research and testing the current watchpoint interface, I've realized that it's impossible (impractically complicated) to pretend to have a "safe" watchpoint interface inside the kernel, as the current one isn't safe from undefined behavior even on stock amd64. I've decided to revert this code and introduce PT_GETDBREGS and PT_SETDBREGS restricted to INSECURE secure level mode. The good side of this change is that there is already part of the code needed in the kernel, I have a local draft introducing this interface and it will be easier to integrate with LLDB, as Linux and FreeBSD keep having the same interface.
I work on the LLDB port inside the pkgsrc-wip repository, in the lldb-netbsd package.
To summarize the changes, there were so far 84 commits in this directory. The overall result is a list of 26 patched or added files. The overall diff's length is 3539 lines.
$ wc -l patches/patch-*
12 patches/patch-cmake_LLDBDependencies.cmake
17 patches/patch-cmake_modules_AddLLDB.cmake
14 patches/patch-include_lldb_Host_netbsd_HostThreadNetBSD.h
30 patches/patch-include_lldb_Host_netbsd_ProcessLauncherNetBSD.h
12 patches/patch-source_CMakeLists.txt
12 patches/patch-source_Host_CMakeLists.txt
60 patches/patch-source_Host_common_Host.cpp
13 patches/patch-source_Host_common_NativeProcessProtocol.cpp
21 patches/patch-source_Host_netbsd_HostThreadNetBSD.cpp
175 patches/patch-source_Host_netbsd_ProcessLauncherNetBSD.cpp
23 patches/patch-source_Host_netbsd_ThisThread.cpp
24 patches/patch-source_Initialization_SystemInitializerCommon.cpp
803 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.cpp
141 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.h
12 patches/patch-source_Plugins_Process_CMakeLists.txt
13 patches/patch-source_Plugins_Process_NetBSD_CMakeLists.txt
1392 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.cpp
188 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.h
393 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.cpp
92 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.h
13 patches/patch-tools_lldb-mi_MICmnBase.cpp
13 patches/patch-tools_lldb-mi_MICmnBase.h
13 patches/patch-tools_lldb-mi_MIDriver.cpp
13 patches/patch-tools_lldb-mi_MIUtilString.cpp
13 patches/patch-tools_lldb-mi_MIUtilString.h
27 patches/patch-tools_lldb-server_CMakeLists.txt
3539 total
I've created the initial code for the Native Process NetBSD Plugin.
The MonitorCallback function supports now the following events:
During this work segment I've completed the following tasks:
The number of passing tests increased by 45% between devel/lldb 3.9.1 and lldb-netbsd 2017-01-21.
The above graphs renders test results for:
It's demo time!
In this example, I'm calling a hello world application that is triggering a software breakpoint. It's implemented by embedding int3 call on amd64. The debugger is capable of catching this and resuming till correct process termination.
$ lldb ./int3
(lldb) target create "./int3"
Current executable set to './int3' (x86_64).
(lldb) r
Hello world!
Process 29578 launched: './int3' (x86_64)
Process 29578 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0:
(lldb) c
Process 29578 resuming
Process 29578 exited with status = 0 (0x00000000)
(lldb)
In this example we set trap on thread events - creation and termination. The executed program incepts a thread and terminates afterwards, this is caught by the MonitorCallback with appropriate thread list update. After the end, program terminates correctly and passes proper exit status to the debugger.
$ lldb ./lwp_create
(lldb) target create "./lwp_create"
Current executable set to './lwp_create' (x86_64).
(lldb) r
Process 27331 launched: './lwp_create' (x86_64)
Hello world!
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
frame #0:
thread #2, stop reason = SIGTRAP has been caught with Process LWP Trap type
frame #0:
(lldb) thread list
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
thread #2: tid = 0x0002, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
frame #0:
(lldb) thread list
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
It works
Process 27331 exited with status = 0 (0x00000000)
(lldb)
I've listed the following goals for the next milestone.
Previous report "Summary of the ptrace(2) project".
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
I've recently installed NetBSD 7.0.2, and I'm having trouble setting my hostname right.
I'm using DHCP, so I got dhclient=YES in my /etc/rc.conf.
Now, the documentation says, that there are several ways of setting the hostname.
The prefered one is to put the hostname into a file. So /etc/myname contains a single line with the fully qualified domain name, etc.
This didn't help so I also added hostname=<my fqdn> into /etc/rc.conf, but it still doesn't honor my hostname settings.
This didn't help either, so I ended up editing /etc/defaults/rc.conf and set the appropriate settings there.
At this point I've run out of ideas. It's still not honering my selection, instead selecting a hostname for me, which is the name of my router plus a number.
pkgsrc’s collectd does not support the thermal plugin, so in order to publish thermal information I had to use the exec plugin:
1 | LoadPlugin exec |
And write this simple script that reads CPUs temperature from NetBSD’s envstat command:
1 | $ cat bin/temp.sh |
I then send those values to an influxdb server:
1 | LoadPlugin network |
And display them using grafana:
HTH!
Accidental theme this week: books.
What are the techniques generally people follow to dump full core dump if the size of core dump is more than the RAM and flash. Say, kernel core is of 2GB size but we have exactly 2GB of RAM and 1GB of disk space.
I am aware external USB and tftp options. But, reliability and stability matters when we choose these options. How do embedded people handle these type of issues? and what are the techniques available?
Platform: NetBSD, ARM7
Thanks,
Paavaanan
This is the sixth in a series of “Nifty and Minimally Invasive qmail Tricks”, following
When my Mac mini’s hard drive died in the Great Crash of Fall 2008, taking this website and my email offline with it, I was already going through a rough time, and my mental bandwidth was extremely limited. I expended some of it explaining to friends what they could do about their hosted domains until such time as my brain became available again (as I assumed and/or hoped it eventually would). I expended a bit more asking a friend to do a small thing to keep my email flowing somewhere I could get it. And then I was spent.
The years where I used Gmail and had no website felt like years in the wilderness. That feeling could mostly have been about how I missed the habit of reflecting about my life now and again, writing about it, and sharing. But when the website returned four years ago (in order to remember Aaron Swartz), the feeling didn’t go away. All I got was a small sense of relief that my writings and recordings were available and that I could safely revive my old habit. After a year and half of reflecting, writing, and sharing, the feels-needle hadn’t rebounded much further.
It was only after painstakingly restoring all my old email (from
Mail.app’s cache, using
emlx2maildir),
moving it up to my IMAP server, carefully merging six years’ worth of
Gmail into that,
accepting SMTP deliveries for @schmonz.com,
and
not needing Gmail at all
for several weeks that I noticed my long, strange sojourn had ended.
If it so happened that I’d instead fixed email first, I’d also have felt a tiny bit weird till my website was back. But only a tiny bit. When my web server’s down, you might not hear from me; when my mail server’s down, I can’t hear from you — or, as happened in 2008, from my professors during finals week. So while web hosting can be interesting, mail hosting keeps me attached to what it feels like to be responsible for a production service.
I value this firsthand understanding very, very highly. I started as a sysadmin, I’m often still a developer, and that’s part of why I’m sometimes helpful to others. But since I’m always in danger of forgetting lessons I learned by doing it, I’m always in danger of being harmful when I try to help others do it.
As a coach, one of my meta-jobs is to remind myself what it takes to know the risks, decide to ship it, live with the consequences, tighten the shipping-it loop until it’s tight enough, and notice when that stops being true.
And that’s why I run my own mail server.
My 2014 mail server was configured just about identically with my 2008 one, for which it was handy to consult the earlier articles in this series.
Then, recently, my weekly build broke on the software I’ve been using to send mail. It was a trivial breakage, easy to fix, but it reminded me about a non-trivial future risk that I didn’t want hanging over my head anymore. (For more details, see my previous post.)
Now I’m sending mail another way. Clients are unchanged, the server no longer needs TMDA or its dependencies, and I no longer have a specific expectation for how this aspect of my mail service will certainly break in the future. (Just some vague guesses, like a newly discovered compromise in the TLS protocol or OpenSSL’s implementation thereof, or STARTTLS or Stunnel’s implementation thereof.)
First, I tried the smallest change that might work:
tmda-ofmipd
with the original ofmipd from
mess822
(by the author of
qmail,
the software around which my mail service is built),spamdyke
(new use of an existing tool),stunnel (as before).It worked!
TMDA no longer needed.
I
committed an update
to my
qmail-run package
with a new
shell script to manage this ofmipd service,
uninstalled TMDA, and removed its configuration files.
Next, I tried a change that might shorten the chain of executables:
mess822 package
adding a build-time option to apply
John R. Levine’s SMTP AUTH patch,ofmipd handle SMTP AUTH on its own.It worked!
Second instance of spamdyke no longer needed.
To start a mail submission service on localhost port 26, these are the lines I added to /etc/rc.conf:
qmailofmipd=YES
qmailofmipd_datalimit="160000000"
qmailofmipd_postofmipd="'' `cat /etc/qmail/control/me` /usr/pkg/bin/checkpassword true"
To make the service available on the network, this is the config from /etc/stunnel/stunnel.conf:
[submission]
accept = submission
connect = localhost:26
protocol = smtp
(It already had this stanza, but with 8025 where tmda-ofmipd was listening.
I simply changed the port number and restarted stunnel.)
I’m still relying on spamdyke for other purposes, but I’m comfortable
with those.
I’m still relying on stunnel for STARTTLS, but I’m relatively
comfortable keeping OpenSSL contained in its own address space and
user account.
The present configuration is a refactoring: no externally visible change to email clients, yes internally visible change to email administrator (moi). I believe this refactoring was one of the best kind, able to be performed safely and reducing the risk I was worried about. The current configuration is much more likely to meet my future need to not have a production outage that interrupts my work for arbitrary duration while I scramble to understand and fix it. I don’t have any more cheap ideas for lowering my risk, and it feels low enough anyway. So I’m comfortable that this is the right place to stop.
Want to learn to see the consequences of your choices and/or help other people do the same? Consider productionizing something important to you.
[Update, 2017-01-16 20:48 UTC: All issues have been resolved and NetBSD.org should be functioning normally now.]
As some of you may have noticed, the netbsd.org DNS records are broken right now. Two independent incidents caused this to happen:
We have sent a request to our registrar to update the delegations to make ns.netbsd.org primary and ns.netbsd.de secondary. We asked them to expedite the process, but owing to the weekend, when the humans who process these requests are off work, it may take another day for that to take effect, and another day after that for the cached TTLs on the old NS delegations to Berkeley's nameservers to expire.
As a provisional stop gap, if you need access to the netbsd.org zone and you can prime your DNS cache with manually entered records, you can prime it with the following records to restore service at your site:
netbsd.org. 86400 IN NS ns.netbsd.org. netbsd.org. 86400 IN NS ns.netbsd.de. ns.netbsd.org. 86400 IN A 199.233.217.200 ns.netbsd.org. 86400 IN AAAA 2001:470:a085:999::53
(Presumably if you are reading this you have some DNS records in the netbsd.org zone cached, but they will eventually expire and you may need some others.)
Hrishikesh Goyal worked on the project "Implement Ext4fs support in ReadOnly mode". He tackled two features of an ext4fs implementation: extents and HTree DIR read/write support. His work was committed into the NetBSD source tree in multiple commits.
charles cui worked on adding tests for better "POSIX Test Suite Compliance". This involved porting the Open Posix benchmark suite to NetBSD. Many of the tests showed missing features, and Charles worked with his mentors to improve the results. See his summary for details.
Leonardo Taccari worked on "Split debug symbols for pkgsrc builds". He already blogged about this in much detail on this blog.
A big thank you to Google for sponsoring the students to work on NetBSD, the students for working on the projects, and the mentors that were helping them along!
This turned into a BSD User Group event list, which makes me happy. There was nothing like that 3 or 5 or whatever years ago.
I'm trying to compile a program with clang++ and libc++ on NetBSD. Clang version is 3.9.0, and NetBSD version is 7.0.2. The compile is failing with:
$ make
clang++ -D_NETBSD_SOURCE -m64 -pthread -stdlib=libc++ -fPIC -fvisibility=hidden \
-std=c++11 -D_REENTRANT -O3 -Wall -Ibuild/include -c ./src/lib/asn1/alg_id.cpp \
-o build/obj/lib/asn1_alg_id.o
In file included from ./src/lib/asn1/alg_id.cpp:8:
In file included from build/include/botan/alg_id.h:11:
In file included from build/include/botan/asn1_obj.h:11:
In file included from build/include/botan/secmem.h:11:
In file included from build/include/botan/mem_ops.h:11:
build/include/botan/types.h:14:10: fatal error: 'cstddef' file not found
#include <cstddef>
^
1 error generated.
*** Error code 1
<cstddef> is present, but it appears to be GCC's:
$ find /usr -name cstddef
/usr/include/g++/cstddef
If I am parsing Index of pub/NetBSD/NetBSD-release-7/src/external/bsd/libc++ correctly, the library is available. When I attempt to install libc++ or libcxx:
bash-4.4$ sudo PKG_PATH="http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/`uname -m`/`uname -r`/All/" pkg_add libcxx
pkg_add: no pkg found for 'libcxx', sorry.
pkg_add: 1 package addition failed
bash-4.4$ sudo PKG_PATH="http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/`uname -m`/`uname -r`/All/" pkg_add libc++
pkg_add: no pkg found for 'libc++', sorry.
pkg_add: 1 package addition failed
Is Clang with libc++ a supported configuration on NetBSD? How do we use Clang and libc++ on NetBSD?
This week’s BSDNow has extended notes about FreeBSD and lld, the LLVM linker, plus notes on the NetBSD scheduler, OpenBSD changes, and so on. It’s very ecumenical.
For reference, here is the text that was added to the sysctl(7) manpage:
kern.sched (dynamic)
Influence the scheduling of LWPs, their priorisation and how they
are distributed on and moved between CPUs.
Third level name Type Changeable
kern.sched.cacheht_time integer yes
kern.sched.balance_period integer yes
kern.sched.average_weight integer yes
kern.sched.min_catch integer yes
kern.sched.timesoftints integer yes
kern.sched.kpreempt_pri integer yes
kern.sched.upreempt_pri integer yes
kern.sched.maxts integer yes
kern.sched.mints integer yes
kern.sched.name string no
kern.sched.rtts integer no
kern.sched.pri_min integer no
kern.sched.pri_max integer no
The variables are as follows:
kern.sched.cacheht_time (dynamic)
Cache hotness time in which a LWP is kept on one particu-
lar CPU and not moved to another CPU. This reduces the
overhead of flushing and reloading caches. Defaults to
3ms. Needs to be given in ``hz'' units, see mstohz(9).
kern.sched.balance_period (dynamic)
Interval at which the CPU queues are checked for re-bal-
ancing. Defaults to 300ms. Needs to be given in ``hz''
units, see mstohz(9).
kern.sched.average_weight (dynamic)
Can be used to influence how likely LWPs are to be
migrated from one CPU's queue of LWPs that are ready to
run to a different, idle CPU. The value gives the per-
centage for weighting the average count of migratable
threads from the past against the current number of
migratable threads. A small value gives more weight to
the past, a larger values more weight on the current sit-
uation. Defaults to 50 and must be between 0 and 100.
kern.sched.min_catch (dynamic)
Minimum count of migratable (runable) threads for catch-
ing (stealing) from another CPU. Defaults to 1 but can
be increased to decrease chance of thread migration
between CPUs.
kern.sched.timesoftints (dynamic)
Enable tracking of CPU time for soft interrupts as part
of a LWP's real execution time. Set to a non-zero value
to enable, and see ps(1) for printing CPU times.
kern.sched.kpreempt_pri (dynamic)
Minimum priority to trigger kernel preemption.
kern.sched.upreempt_pri (dynamic)
Minimum priority to trigger user preemption.
kern.sched.maxts (dynamic)
Scheduler specific maximal time quantum (in millisec-
onds). Must be set to a value larger than ``mints'' and
between 10 and ``hz'' as given by the kern.clockrate
sysctl. Provided by the M2 scheduler.
kern.sched.mints (dynamic)
Scheduler specific minimal time quantum (in millisec-
onds). Must be set to a value smaller than ``maxts'' and
between 1 and ``hz'' as given by the ``kern.clockrate''
sysctl. Provided by the M2 scheduler.
kern.sched.name (dynamic)
Scheduler name. Provided both by the M2 and the 4BSD
scheduler.
kern.sched.rtts (dynamic)
Fixed scheduler specific round-robin time quantum in mil-
liseconds. Provided both by the M2 and the 4BSD sched-
uler.
kern.sched.pri_min (dynamic)
Minimal POSIX real-time priority. See sched(3).
kern.sched.pri_max (dynamic)
Maximal POSIX real-time priority. See sched(3).
http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC1/
Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC1 tag.
There have been quite a lot of changes since 7.0. See src/doc/CHANGES-7.1 for the full list.
Please help us out by testing 7.1_RC1. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at [email protected]''
The first release candidate of NetBSD 7.1 is now available for download at:
http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC1/
Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC1 tag.
There have been quite a lot of changes since 7.0. See src/doc/CHANGES-7.1 for the full list.
Please help us out by testing 7.1_RC1. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at [email protected]
(EE) module ABI major version (20) doesn't match the server's version (10)
(EE) Failed to load module "qxl" (module requirement mismatch, 0)There’s always a rush of links after a holiday, as people sit at home and catch up on what they’ve wanted to do.
Last night, mere moments from letting me commit a new package of Test::Continuous (continuous testing for Perl), my computer acted as though it knew its replacement was on the way and didn’t care to meet it. This tiny mid-2013 11” MacBook Air made it relatively ergonomic to work from planes, buses, and anywhere else when I lived in New York and flew regularly to see someone important in Indiana, and continued to serve me well when that changed and changed again.
The next thing I was planning to do with it was write this post. Instead I rebooted into DiskWarrior and crossed my fingers.
Things get in your way, or threaten to. That’s life. But when you have slack time, you can
Having enough slack is as virtuous a cycle as insufficient slack is a vicious one.
Last year I decided to spend more time and energy improving my health. Having recently spent a few weeks deliberately not paying attention to any of that, I’m quite sure that I prefer paying attention to it, and am once again doing so.
Learning to make my health a priority required that I make other things non-priorities, notably Agile in 3 Minutes. It no longer requires that. I’ve recently invested in making the site easier for me to publish, and you may notice that it’s easier for you to browse. I didn’t have enough slack to do these things when I was writing and recording a new episode every week. Now that enough of them have been taken care of, I feel prepared to take new steps with the podcast.
Earlier this week I noticed a broken link in a comment on Refactorings for web hosting, so I took a moment to check for other broken links on this site (ikiwiki makes it easy).
Before that, I inspected and minimized the differences between “dev” (my
laptop) and “prod” (my server, where you’re reading this), updated prod
with the latest ikiwiki settings, and (because it’s all in Git)
rebased
dev from prod.
In so doing, I observed that more config differences could be easily
harmonized by adjusting some server paths to match those on my laptop.
(When Apple introduced
System Integrity Protection,
pkgsrc
on Mac OS X could no longer install under /usr, and moved to /opt.
With my
automated NetBSD package build,
I can easily build the next batch for /opt/pkg as well, retaining
/usr/pkg as a symlink for a while.
So I have.)
I’ve been running lots of these builds in the past week anyway, because a family of packages I maintain in pkgsrc had been outdated for quite a while and I finally got around to catching them up to upstream. Once they built on OS X, I committed the updates to the cross-platform package system, only to notice that at least one of them didn’t build on NetBSD. So I fixed it, ran another build, saw what else I broke, and repeated until green.
Due to another update that temporarily broke the build of TMDA, I was freshly reminded that that’s a relatively biggish liability in my server setup. I use TMDA to send mail, which is not mainly what it’s for, and I never got around to using it for what it’s for (protecting against spam with automated challenge-response), and it hasn’t been maintained for years, and is stuck needing an old version of Python.
On the plus side, running a weekly build means that when TMDA breaks more permanently, I’ll notice pretty quickly. On the minus side, when that happens, I’ll feel pressure to fix or replace it so I can (1) continue to send email like a normal person and (2) restart the weekly build like a me-person. If I can reduce the liability now, maybe I can avoid feeling that pressure later.
Investigating alternatives, I remembered that
Spamdyke,
which I already use for
delaying the SMTP greeting,
blacklisting from a
DNSBL
as well as To: addresses that only get spam anymore, and
greylisting from unknown senders,
can provide
SMTP AUTH.
So I’ll try keeping
stunnel
and replacing tmda-ofmipd with a second instance of spamdyke.
If that’s good, I’ll remove mail/tmda from
the list of packages I build every week,
then build spamdyke with OpenSSL support and try letting it handle the
TLS encryption directly.
If that’s good, I’ll remove security/stunnel from the list of packages
too, leaving me at the mercy of fewer pieces of software breaking.
Leaning more heavily on Spamdyke isn’t a clear net reduction of risk.
When a bad bug is found, it’ll impact several aspects of my mail service.
And if and when NetBSD moves from GCC to Clang, I’ll have to add
lang/gcc to my list of packages and instruct pkgsrc to use it when
building Spamdyke, or else come up with a patch to remove Spamdyke’s use
of anonymous inner functions in C.
(That could be fun. I recently
started learning C.)
I could go on, but I’m a nice person who cares about you.
All these builds pushing my soon-to-be-replaced laptop through its final paces as a development machine might have had something to do with triggering its misbehavior last night. And all this work seems like, well, a lot of work. Is there some way I could do less of it?
Yes, of course.
But given my interests and goals, it might not be a clear net improvement.
For instance, when
Tim Ottinger
drew my attention to that Test::Continuous Perl module, being a pkgsrc
developer gave me an easy way to uninstall it if I wound up not liking
it, which meant it was easy to try, which meant I tried it.
I want
conditions in my life to favor trying things.
So I’m invested in preserving and extending those conditions.
In
Gary Bernhardt’s
formulation, I’m aiming to maximize the
area under the curve.
I’m not looking to add new goals for myself for 2017. I’m not even trying to make existing things “good enough” — there are too many things, and as a recovering perfectionist I have trouble setting a reasonable bar — I’m just trying to make them “good enough enough” that I can expect small slices of time and attention to permit small improvements.
Jessica Kerr has a thoughtful side blog named True in software, true in life. Here’s something that’d qualify:
Paying down my self-debts (technical and otherwise) feels like resolving. I have, at times, felt quite out of position at managing myself. Lately I’m feeling much more in position, and much more like I can expect to continue to make small improvements to my positioning.
When you want the option to change your body’s direction, you take smaller steps, lower your center, concentrate on balance. That’s #Agile.
—Moi
My current best understanding is that a balanced life is a small-batch-size life. If that’s the case, I’m getting there.
This coming Monday, I’ll be switching to one of these weird new MacBook Pros with the row of non-clicky touchscreen “keys”. If my current computer survives till then, that’ll be one smooth step in a series of transitions. (In other news, Bekki defends her dissertation that day.)
The following Monday, I’ll be starting my next project, a mostly-remote gig pairing in Python to deliver software for a client while encouraging and supporting growth in my Pillar teammates. I’ll be in Des Moines every so often; if you’re there and/or have recommendations for me, I’d love to hear from you.
The Monday after that, we’ll pack up a few things the movers haven’t already taken away, and our time in Indiana will come to an end. We’re headed back to the New York area to live near family and friends.
For 2017, I declare my intentions to:
I hope to see and hear you along the way.
So over the holidays, I managed to get in some good quality family time and find some time to work on some Open Source stuff. I meant to work mainly on dhcpcd, but it turned out I spent most of my time working on NetBSD curses library so that Python Curses now works with it. Now, most people r...
Another area where changing of the system's configuration is the amount of Ramdom Access Memory (RAM) of a system. Usually fixed, this is determined at system start time, and then managed by the operating system's memory managent system. But esp. with today's virtualized hardware systems, even the amount of RAM assigned to a system can easily be changed. For example a VM can be assigned more RAM when needed, without even rebooting the system, leading to increased system performance without introducing swapping/paging overhead. Of course this required support from the operating system and its memory management subsystem.
For NetBSD, the UVM virtual memory system was now changed to support this via the uvm_hotplug(9) API, and a first user for this is the Xen balloon(4) driver. Quoting from the balloon(4) manpage, ``The balloon driver supports the memory ballooning operations offered in Xen environments. It allows shrinking or extending a domain's available memory by passing pages between different domains.''
The uvm_hotplug(9) manpage gives us more information on the UVM hotplug functionality: ``When the kernel is compiled with 'options UVM_HOTPLUG', memory segments are handled in a dynamic data structure (rbtree(3)) com- pared to a static array when not. This enables kernel code to add or remove information about memory segments at any point after boot - thus "hotplug".''
To answer more questions for portmasters who want to change their ports, Cherry G. Mathew has now posted a uvm_hotplug(9) port masters' FAQ. It covers questions on the background, affected files, and needed changes.
For more information on UVM, see Charles' Chuck' Cranor's PhD disertation on Design and Implementation of UVM (PDF) as well as his Usenix talk on the UVM Virtual Memory System (PS). There is also plenty of information available on Xen ballooning - check it out and share your experiences on NetBSD's port-xen mailing list!
Last anything for the year!
My brother got me some very tasty presents for Christmas (and my up-coming Birthday), namely the GIGABYTE BRIX J1900 and a Samsung EVO 750 250G. Santa also brought me 8G of Crucial memory. Putting them all together is a nice new machine to install NetBSD Xen! The key part is this is a low...
In general hardware interrupts need to be processed immediately, at least so as to acknowledge it and do some first level of processing. As I understand this is not scheduled activity. Please correct me.
So the question is how to choose a processor that would actually execute this hardware interrupt handler?
One can answer this for Linux and/or BSD systems
Now, two things remain to be seen:
Researched its usability and added ATF test, it's close to be marked for removal - it's marked as broken as it is on all ports... this call was inherited from BSD4.2 (VAX) and was never useful since the inception as it is enabling singlestepping before calling execve(2) and tracing libc calls before switching to new process image.
This library was designed for Scheduler Activation, and this feature was removed in NetBSD 5.0.
typedef struct ptrace_watchpoint {
int pw_index; /* HW Watchpoint ID (count from 0) */
lwpid_t pw_lwpid; /* LWP described */
struct mdpw pw_md; /* MD fields */
} ptrace_watchpoint_t;
For example amd64 defines MD as follows:
struct mdpw {
void *md_address;
int md_condition;
int md_length;
};
These calls are protected with the __HAVE_PTRACE_WATCHPOINTS guard.
Tested on amd64, initial support added for i386 and XEN.
.... and several critical ones not reported and fixed directly by the NetBSD team.
Credit for Christos Zoulas and K. Robert Elz for helping with the mentioned bugs.
Features in ELF, DWARF, CTF, DTrace are out of scope of the above list.
My initial goal is to copy the Linux Process Plugin and add minimal functional support for NetBSD in LLDB. It will be followed with running and passing some tests from the lldb-server test-suite.
This is a shift from the original plan about porting FreeBSD Process Plugin to NetBSD, as the FreeBSD one is lacking remote debugging support and it needs to be redone from scratch.
I'm going to fork wip/lldb-git (as for git snapshot from 16 Dec 2016) for the purpose of this task to wip/lldb-netbsd and work there.
Next steps after finishing this task are to sync up with Pavel from the LLDB team after New Year. The NetBSD Process Plugin will be used as a reference to create new Common Process Plugin shared between Linux and (Net)BSD.
How can I activate Keyboard Latin American on NetBSD? Because when I am installing I never saw the Latin American keyboard, only Spanish.
Where can I find and install an AR9271 driver for the latest NetBSD? The target machine does not have Internet access and I need to setup the WiFi dongle first.
I only found this: https://www.daemon-systems.org/man/athn.4.html
UPDATE: wpa_supplicant was already written, but I didn't see my device.
When I plug in the dongle it's shown as:
ugen0 at uhub4 port 8
ugen0: Mediatek 802.11 n WLAN, rev 2.01/00, addr 2
ifconfig shows only re0 and lo0 interfaces.
UPDATE: I saw on some Linux forums that the dongle uses an Atheros chip, but I checked in Windows and see Ralink. The ral driver is also integrated in NetBSD, but the situation doesn't change - I see no ra~ device in dmesg.boot.
First part at NetBSD and TP-Link TL-WN727N (Atheros AR9271 or Ralink RT5370)
So, I've installed NetBSD 7 and device shown again as ugen(ugein, lol).
ugen0 at uhub4 port 8
ugen0: Mediatek 802.11 n WLAN, rev 2.01/00, addr 2
Then I'm installed FreeBSD 10.2 and ugen again.
usbconfig gives me ugen4.3: <product 0x7601 vendor 0x148f> at usbus4, cfg=0 md=HOST spd=HIGH(480Mbps) pwr=ON (90ma)
So, what's next? Buying new dongle is a last thing, which I'll make.
UPD: NDIS driver not works.
At Agile Testing Days, I facilitated a workshop called “DevOps Dojo”. We role-played Dev and Ops developing and operating a production system, then figured out how to do it better together.
We wrote down our takeaways on the final “Retrospective” slide.
You’re welcome to use the workshop materials for any purpose, including your own workshop. If you do, I’d love to hear about it.
I’ve spoken at several instances of
pkgsrcCon
(including
twice
in nearby
Berlin),
but that’s more like a hackathon with some talks.
Agile Testing Days was a proper conference, with hundreds of
people and plenty of conferring.
If someone asks whether I’m an “international speaker”,
or claims I am one,
I now won’t feel terribly uncomfortable going along with it.
I met a fellow
WeDoTDD
practitioner,
Carlos Blé
of
Codesai.
(Here’s
their WeDoTDD interview
and
Pillar’s.)
Carlos and I have both
relied on Twitter
to build our careers.
Who knows, maybe we’ll give a talk together about it.
At the Tuesday morning
Lean Coffee,
I found a bug in myself (not a first).
What I expected from many previous Lean Coffees: I’d have to control myself to not say all the ideas and suggestions that come to mind.
What happened at this Lean Coffee: It was very easy to listen, because I didn’t have many ideas or suggestions, because the topics came from people who were mostly testers.
Conclusions I immediately drew:
Matt Heusser
invited me to perform in the Late Night Cabaret with Bob
(Lisa Crispin’s spouse)
and Jack
(Janet Gregory)’s).
We did the
Four Yorkshiremen sketch
as
popularized by Monty Python.
Thanks to
Troy Magennis,
Markus Gärtner,
and
Cat Swetel,
I decided to
try a new idea
and spend
a few slides
drawing attention to the existence and purpose of Agile Testing Days’
Code of Conduct.
I can’t tell yet how much good this did, but it took so little time that
I’ll keep trying it in future conference presentations and workshops.
My next gig will be remote coaching, centered around what we notice as
we’re pair programming and delivering working software.
I’ve done plenty of
coaching
and plenty of
remote work,
but not usually at the same time.
Thanks to Lean Coffee with folks like Janet and
Alex Schladebeck,
I got some good advice on being a more effective
influencer
when it takes more intention and effort to have face-to-face interactions.
As I learn about remote coaching, I expect to write things down at
Shape My Work,
a wiki about distributed Agile that
Alex Harms
and I created.
You’ll notice it has a Code of Conduct.
If it makes good sense to you, we’d love to learn what you’ve learned as
a remote Agilist.
I found Agile Testing Days to be a lovingly organized and carefully
tuned mix of coffee breaks, efficiency, flexibility, and whimsy.
The love and whimsy shone through.
I’m honored to have been part of it, and I sure as heck hope to be back
next year.
We’d be back next year anyway; we visit family in Germany every December. Someday we might choose to live near them for a while. It occurs to me that having participated in Agile Testing Days might well have been an early investment in that option, and the thought pleases me. (As does the thought of hopping on a train to participate again.)
I’m in Europe through Christmas.
I consult,
coach,
and train.
Do you know of anyone who could use a day or three of my services?
One aspect of being a tester I do identify with is being frequently
challenged to explain their discipline or justify their decisions to
people who don’t know what the work is like (and might not recognize the
impact of their not knowing).
In that regard, I wonder how helpful
Agile in 3 Minutes
is for testers.
Let’s say I could be so lucky as to have a few guest episodes about testing. Who would be the first few people you’d want to hear from? Who has a way with words and ideas, knows the work, and can speak to it — in their unique voice — to help the rest of us understand a bit better?
This may not only be of interest to the OpenDarwin folks (or rather their successors in PureDarwin) but more investigation not only on the code itself, but also the license it is released under is neccessary to learn if anything can be gained back for NetBSD.
Why "back"? As you may or may not remember, mac OS includes some parts of NetBSD (besides lots of FreeBSD, probably some OpenBSD, much other Open Source software and sure a big lot of Apple's own code).
My first job was in Operations. When I got to be a Developer, I promised myself I’d remember how to be good to Ops. I’ve sometimes succeeded. And when I’ve been effective, it’s been in part due to my firsthand knowledge of both roles.
Part of what people mean when they say DevOps is automation. Once a system or service is in operation, it becomes more important to engineer its tendencies toward staying in operation. Applying disciplines from software development can help.
These words are brought to you by a Unix server I operate. I rely on it to serve this website, those of a few friends, and a tiny podcast of some repute. Oh yeah, and my email. It has become rather important to me that these services tend to stay operational. One way I improve my chances is to simplify what’s already there.
Another way is to update my installed third-party software once a week. This introduces two pleasant tendencies: it’s much…
Updating software every week also makes two strong assumptions about safety (see Modern Agile’s “Make Safety a Prerequisite”): that I can quickly and easily…
Since I’ve been leaning hard on these assumptions, I’ve invested in making them more true.
The initial investment was to figure out how to configure pkgsrc to build a complete set of binary packages that could be installed at the same time as another complete set. My hypothesis was that then, with predictable and few side effects, I could select the “active” software set by moving a symbolic link.
It worked.
On my
PowerPC Mac mini,
the best-case upgrade scenario went from half an hour’s downtime (bring down services, uninstall old packages, install new packages, bring up services) to less than a minute (install new packages, bring down services, move symlink, bring up services, delete old packages after a while).
The worst case went from 
over an hour to 
maybe a couple of minutes.
I liked the payoff on that investment a lot. I’ve been adding incremental enhancements ever since. I used to do builds directly on the server: in place for low-risk leaf packages, as a separate full batch otherwise. It was straightforward to do, and I was happy to accept an occasional reduction in responsiveness in exchange for the results.
After
the Mac mini died,
I moved to
a hosted Virtual Private Server
that was much easier to mimic.
So I took the job offline to a local
VirtualBox running the same release and architecture of
NetBSD
(32-bit i386
to begin with,
64-bit amd64
now, both under
Xen).
The local job ran faster by some hours (I forget how many), during which the server continued devoting all its I/O and CPU bandwidth to its full-time responsibilities.
Last time I went and improved something was to fully automate the building and uploading, leaving myself a documented sequence of manual installation steps. Yesterday I extended that shell script to generate another shell script that’s uploaded along with the packages. When the upload’s done, there’s one manual step: run the install script.
If you can read these words, it works.
Applying Dev concepts to the Ops domain is one aspect. When I’m acting alone as both Dev and Ops, as in the above example, I’ve demonstrated only that one aspect.
The other, bigger half is collaboration across disciplines and roles. I find it takes some not-tremendously-useful effort to distinguish this aspect of DevOps from BDD — or from anything else that looks like healthy cross-functional teamwork. It’s the healthy cross-functional teamwork I’m after. There are lots of places to start having more of that.
If your team’s context suggests to you that DevOps would be a fine place to start, go after it! Find ways for Dev and Ops to be learning together and delivering together. That’s the whole deal.
Two weeks from today, at Agile Testing Days in Potsdam, Germany, I’m running a hands-on DevOps collaboration workshop. Can you join us? It’s not too late, and you can save 10% off the price of the conference ticket. Just provide my discount code when you register. I’d love to see you there.
According to NetBSD's wiki I can use pkg_add -uu to upgrade packages. However, when I attempt to use pkg_add -uu it results in an error.
pkg_add -uu
pkg_add: missing package name(s)
...
pkg_add -uu *
pkg_add: no pkg found for `*`, sorry
...
pkg_add -uu all
pkg_add: no pkg found for `all`, sorry
...
I've tried to parse the pkg_add man page but I can't tell what the command it to update everything.
I can't use pkg_chk because its not installed, and I can't get the package system to install it:
pkg_chk -b
pkg_chk: command not found
pkg_add pkg_chk
pkg_add: no pkg found for `pkg_chk`, sorry
What is the secret command to get the OS to update everything?
Please forgive my ignorance with this question. I only have NetBSD systems for testing software. It gets used a few times a year, and I don't know much about it otherwise.
What is the maximum number of machines that can be set up in this LAN and why? (This comes under class C network so the maximum would be 255 or less - correct me if i'm wrong)
Suresh - [email protected] sends a mail to my friend Rahul - [email protected] with these three files as separate attachments as below
-march-reports.ppt - Powerpoint file of size 256 KB.
-locations.rar - Rar archive file of size 460 KB
-me-snap.tiff - Tiff picture file of size 2970 KB.
a) What is the size of the outgoing mail including mail headers ?
b) What is the outgoing mail size if all the three files are archived as one single .rar file and sent out as one single attachment ?
c) Show the MIME based mail structure of the outgoing mail.
Many programming guides recommend to begin scripts with the #! /usr/bin/env shebang in order to to automatically locate the necessary interpreter. For example, for a Python script you would use #! /usr/bin/env python, and then the saying goes, the script would “just work” on any machine with Python installed.
The reason for this recommendation is that /usr/bin/env python will search the PATH for a program called python and execute the first one found… and that usually works fine on one’s own machine.
Unfortunately, this advice is plagued with problems and assuming it will work is wishful thinking. Let me elaborate. I’ll use Python below for illustration purposes but the following applies equally to any other interpreted language.
i) The first problem is that using #! /usr/bin/env lets you find an interpreter but not necessarily the correct interpreter. In our example above, we told the system to look for an interpreter called python… but we did not say anything about the compatible versions. Did you want Python 2.x or 3.x? Or maybe “exactly 2.7”? Or “at least 3.2”? You can’t tell right? So the the computer can’t tell either; regardless, the script will probably run with whichever version happens to be called python–which could be any thanks to the alternatives system. The danger is that, if the version is mismatched, the script will fail and the failure can manifest itself at a much later stage (e.g. a syntax error in an infrequent code path) under obscure circumstances.
ii) The second problem, assuming you ignore the version problem above because your script is compatible with all possible versions (hah), is that you may pick up an interpreter that does not have all prerequisite dependencies installed. Say your script decides to import a bunch of third-party modules: where are those modules located? Typically, the modules exist in a centralized repository that is specific to the interpreter installation (e.g. a .../lib/python2.7/site-packages/ directory that lives alongside the interpreter binary). So maybe your program found a Python 2.7 under /usr/local/bin/ but in reality you needed it to find the one in /usr/bin/ because that’s where all your Python modules are. If that happens, you’ll receive an obscure error that doesn’t properly describe the exact cause of the problem you got.
iii) The third problem, assuming your script is portable to all versions (hah again) and that you don’t need any modules (really?), is that you are assuming that the interpreter is available via a specific name. Unfortunately, the name of the interpreter can vary. For example: pkgsrc installs all python binaries with explicitly-versioned names (e.g. python2.7 and python3.0) to avoid ambiguity, and no python symlink is created by default… which means your script won’t run at all even when Python is seemingly installed.
iv) The fourth problem is that you cannot pass flags to the interpreter. The shebang line is intended to contain the name of the interpreter plus a single argument to it. Using /usr/bin/env as the interpreter name consumes the first slot and the name of the interpreter consumes the second, so there is no room to pass additional flags to the program. What happens with the rest of the arguments is platform-dependent: they may be all passed as a single string to env or they may be tokenized as individual arguments. This is not a huge deal though: one argument for flags is too restricted anyway and you can usually set up the interpreter later from within the script.
v) The fifth and worst problem is that your script is at the mercy of the user’s environment configuration. If the user has a “misconfigured” PATH, your script will mysteriously fail at run time in ways that you cannot expect and in ways that may be very difficult to troubleshoot later on. I quote “misconfigured” because the problem here is very subtle. For example: I do have a shell configuration that I carry across many different machines and various operating systems; such configuration has complex logic to determine a sane PATH regardless of the system I’m in… but this, in turn, means that the PATH can end up containing more than one version of the same program. This is fine for interactive shell use, but it’s not OK for any program to assume that my PATH will match their expectations.
vi) The sixth and last problem is that a script prefixed with #! /usr/bin/env is not suitable to being installed. This is justified by all the other points illustrated above: once a program is installed on the system, it must behave deterministically no matter how it is invoked. More importantly, when you install a program, you do so under a set of assumptions gathered by a configure-like script or prespecified by a package manager. To ensure things work, the installed script must see the exact same environment that was specified at installation time. In particular, the script must point at the correct interpreter version and at the interpreter that has access to all package dependencies.
All this considered, you may still use #! /usr/bin/env for the convenience of your own throwaway scripts (those that don’t leave your machine) and also for documentation purposes and as a placeholder for a better default.
For anything else, here are some possible alternatives to using this harmful shebang:
Patch up the scripts during the “build” of your software to point to the specific chosen interpreter based on a setting the user provided at configure time or one that you detected automatically. Yes, this means you need make or similar for a simple script, but these are the realities of the environment they’ll run under…
Rely on the packaging system do the patching, which is pretty much what pkgsrc does automatically (and I suppose pretty much any other packaging system out there).
Just don’t assume that the magic #! /usr/bin/env foo is sufficient or even correct for the final installed program.
Bonus chatter: There is a myth that the original shebang prefix was #! / so that the kernel could look for it as a 32-bit magic cookie at the beginning of an executable file. I actually believed this myth for a long time… until today, as a couple of readers pointed me at The #! magic, details about the shebang/hash-bang mechanism on various Unix flavours with interesting background that contradicts this.
I’ve been running a public web server since 1999, when my employer
registered schmonz.com for me as a gag gift. Last week, I
learned from Twitterbrausen
that in German, “Schmonz” means something akin to “bullshit”. That’s not
what my employer had meant by it; I consider nonetheless that my
incessant blogging has acquired a fine new patina of significance.
As I recall, when I was first looking for web server software, there was not a wide variety to choose from. Apache was popular and featureful, a safe default choice. As a novice programmer, I was very much taken with the idea of building dynamic sites, and Apache offered many ways to go about that. Done deal.
In the intervening years, my server machine has changed several times, from Macintosh IIci to Mini-ITX box to Mac Mini to Xen Virtual private server. (I’m particularly fond of the present arrangement wherein hardware is someone else’s problem and I continue to have root access.) No matter the system architecture, the OS has always been NetBSD, which remains unobtrusively thrilling, and the web server has always been Apache, which has gradually become more noisome.
Between my own sites and those of friends I’ve hosted, I’ve needed many times to adapt my Apache configuration to accommodate changes in external modules (such as mod_php), to interfaces (such as PHP via FastCGI instead), and within Apache itself (such as basic access control). Each time I forcibly revisited my config, I found myself revisiting my discomfort with its complexity. I never felt sure that I understood exactly, in its entirety, what my Apache installation would and wouldn’t do. And as a result of years of entanglement and unclarity, I never saw a way to give my users full administrative control over their own sites.
I’ve been imagining moving off Apache for a while. But it always seemed like a project, so I never did anything about it. I can’t usually afford to start on something unless I know I’m going to be able to stop soon, and I won’t usually want to stop unless I know how I can easily start next time. That leaves me needing a sequence of small-enough steps in my desired direction. Or, more precisely, two expectations: that at least one such sequence exists, and that I’ll be able to discover one as I go.
Conveniently, I’ve had plenty of professional practice at incremental problem-solving, enough to identify my first few steps and start making progress. Here’s the rest of the sequence, naming the refactorings I’ve found along the way.
I wanted to see what I’d learn by persuading one site to become its own
self-contained thing running its own Apache instance. I picked a
relatively basic site, told the
system Apache to reverse-proxy that virtual host,
added just enough configuration to
start a site-specific Apache on localhost,
verified that as far as I could discern the site
worked equally well, and cut over to the new configuration.
Inserting a proxy usually means, at the very least, server logs start
reporting requests coming from the proxy’s IP rather than the browser’s.
For this to be a refactoring, the system Apache needed to send an
X-Forwarded-For header (it automatically does), and the site-specific Apache needed
to know to look for it (by enabling the bundled
mod_remoteip).
Manually starting an instance of a service usually means the system
won’t automatically know how to do the same next time it boots up. For
this to be a refactoring, I needed to
add an entry to the site owner’s crontab.
To validate that the site would continue to be served by its
own Apache as well as it’d been served the old way, I rebooted the
system. The site stayed up.
Good, because there were 17 more sites to go. Each of them would also be
listening on its own non-standard port on localhost. To identify them
at a glance in netstat, I added the port to /etc/services. Now I had
a pattern worth repeating.
Some sites were more complex than others (PHP, language negotiation, other wrinkles), but I didn’t need to invent their configurations from scratch, merely uncover the tiny portions of the existing giant config that were relevant and copy them over.
Near the end, I couldn’t start new Apache instances without increasing
some kernel IPC parameters (kern.ipc.msgmni from 40 to 80,
kern.ipc.semmni from 10 to 20). This felt like a small backward step.
I hoped to be able to undo it later.
It also might have felt like a small step backward to suddenly have lots more instances of Apache. But it was a large step forward in my understanding.
En route to that understanding, I was fairly sure I’d reduced the system
Apache to a single responsibility: being a reverse HTTP proxy. To
validate that it was no longer serving any other purpose, I turned off
most LoadModule directives — even the typical and enabled-by-default
ones — leaving only those that prevented Apache from running when I
tried turning them off.
I’d been hoping to replace Apache with bozohttpd. Now that I had
small, explicit per-site configurations, I could try converting one. The
site worked, but the logs were missing lots of basic information. I
still think this is where I want to go, but since it’s not a
refactoring, I can’t go there yet.
I tried
converting the same site from Apache to lighttpd,
which is a
little more featureful than bozohttpd. The site worked, and with
mod_extforward
enabled, its server logs were indistinguishable from
Apache’s. I gzipped the now-retired Apache config to prevent it from
being used by mistake while keeping it for reference, updated the
site’s crontab entry to start Lighttpd instead of Apache, and
rebooted. Bingo!
I converted a bunch more sites. After doing a few, I figured out how to extract shared configuration. Simpler sites have extremely short config files (just a few lines). More complex sites only define what’s unusual about them.
With a few Apache-powered sites left to convert, I was pretty sure none
of them was using PHP. To test this hypothesis, I stopped the php-fpm
service. After a week, with nothing broken, I uninstalled it.
With only a few Apache-powered sites remaining, could I return kernel IPC parameters to their default values? Yes, all the Lighttpd and Apache sites ran just fine that way.
Getting married is the opposite of a refactoring. There’s no internal change, but many callers have new expectations.
I expected three sites to be relatively tricky to convert:
Once they were converted, there were zero remaining Apache-powered sites.
A single Apache instance remained: the system one that was nothing but a reverse proxy to a bunch of Lighttpd instances.
Had I known that’d be its only job, I’d have chosen software designed for the purpose. I knew that now, and chose Pound. On a non-standard port, I figured out how to express a few sites’ worth of reverse proxying in Pound’s configuration language, continued until I’d translated everything in the Apache config, stopped Apache, and started Pound.
Not a single Apache instance remained. To my knowledge, all sites were operating as normal. After a week, I uninstalled Apache, deleted its corresponding Unix user and group, and gzipped all its config files for reference.
Apache had been serving multiple roles. I brought the number down to zero, then got rid of it. To do that, I…
For human site visitors, all of these steps were genuine refactorings. (Atypical and automated visitors might notice the HTTP header reporting different server software.) For site owners, most of these steps were also genuine refactorings. (In a couple cases, using the shared Lighttpd config required changing the names of log files by a small nonzero amount.)
I replaced one big application with two small ones. Better. Still, could be more better.
The replacement virtual-host multiplexer (Pound) feels simple, good, and
necessary, in the sense that nothing like it is included with the OS.
The replacement web server (Lighttpd) feels simpler and better, by far
— I understand what it’s doing, my users finally have full
administrative control over their own sites, and unlike Apache, this
configuration doesn’t require extra system resources — but NetBSD
does include a web server, the one I experimented with in Step 4. If
bozohttpd did a few more things, then “Replace Lighttpd with
Bozohttpd” would be a refactoring, one that could be followed
immediately by “Remove Dependency (on Lighttpd)”.
I’ve been
practicing C.
In some kind of cosmic coincidence, next week I’ll be joining a project
that’s being developed primarily in C. Hacking on bozohttpd will be
good practice. Here’s the incremental sequence of features awaiting my
next increment of time and attention, perhaps on tomorrow’s
transatlantic flight:
syslog or stderr)X-Forwarded-For header whose contents we’ll use as the true client
source address (for logs, access control decisions, etc.)Since I believe I’ll be able to stop, I’ll be able to start. It might not be terribly long before I have more progress to share.
I'm running NetBSD in a virtual machine. Documentation and explanations on how to use pkgsrc are scarce. Let's say I want to install vim for NetBSD. What would I type? Do I need a URL? Do I need a specific version? Do I need to set up a directory for building the source of vim?
Here are some notes on installing and running NetBSD/evbarm on the AllWinner A20 powered CubieBoard2. I bought this board a few weeks ago for its SATA capabilities, despite the fact that there are now cheaper boards with more powerful CPUs.
Required steps for creating a bootable micro SD card are detailed on the NetBSD Wiki, and a NetBSD installation is required to run mkubootimage.
I used an USB to TTL serial cable to connect to the board and create user accounts. Do not be afraid of serial, as it has in fact only advantages: there is no need to connect an USB keyboard nor an HDMI display, and it also brings back nice memories.
Connecting using cu (from my OpenBSD machine) :
cu -s 115200 -l /dev/cuaU0
Device name might be different when using cu on other operating systems.
Adding a regular user in the wheel group :
useradd -m -G wheel username
Adding a password to the newly created user and changing default shell to ksh :
passwd username
chpass -s /bin/ksh username
Installing and configuring pkgin :
export PKG_PATH=http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/earmv7hf/7.0/All
pkg_add pkgin
echo $PKG_PATH > /usr/pkg/etc/pkgin/repositories.conf
pkgin update
Finally, here is a dmesg for reference purposes :
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 7.0.1 (CUBIEBOARD.201605221355Z)
total memory = 1024 MB
avail memory = 1008 MB
sysctl_createv: sysctl_create(machine_arch) returned 17
kern.module.path=/stand/evbarm/7.0/modules
timecounter: Timecounters tick every 10.000 msec
mainbus0 (root)
cpu0 at mainbus0 core 0: 912 MHz Cortex-A7 r0p4 (Cortex V7A core)
cpu0: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu0: isar: [0]=0x2101110 [1]=0x13112111 [2]=0x21232041 [3]=0x11112131, [4]=0x10011142, [5]=0
cpu0: mmfr: [0]=0x10101105 [1]=0x40000000 [2]=0x1240000 [3]=0x2102211
cpu0: pfr: [0]=0x1131 [1]=0x11011
cpu0: 32KB/32B 2-way L1 VIPT Instruction cache
cpu0: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu0: 256KB/64B 8-way write-through L2 PIPT Unified cache
vfp0 at cpu0: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
vfp0: mvfr: [0]=0x10110222 [1]=0x11111111
cpu1 at mainbus0 core 1
armperiph0 at mainbus0
armgic0 at armperiph0: Generic Interrupt Controller, 160 sources (151 valid)
armgic0: 32 Priorities, 128 SPIs, 7 PPIs, 16 SGIs
armgtmr0 at armperiph0: ARMv7 Generic 64-bit Timer (24000 kHz)
armgtmr0: interrupting on irq 27
timecounter: Timecounter "armgtmr0" frequency 24000000 Hz quality 500
awinio0 at mainbus0: A20 (0x1651)
awingpio0 at awinio0
awindma0 at awinio0: DMA
awindma0: interrupting on irq 59
awincnt0 at awinio0
timecounter: Timecounter "CNT64" frequency 24000000 Hz quality 900
com0 at awinio0 port 0: ns16550a, working fifo
com0: console
awindebe0 at awinio0 port 0: Display Engine Backend (BE0)
awintcon0 at awinio0 port 0: LCD/TV timing controller (TCON0)
awinhdmi0 at awinio0: HDMI 1.3
awinwdt0 at awinio0: default period is 10 seconds
awinrtc0 at awinio0: RTC
awinusb0 at awinio0 port 0
awinusb0: no restrict gpio found
ohci0 at awinusb0: OHCI USB controller
ohci0: OHCI version 1.0
usb0 at ohci0: USB revision 1.0
ohci0: interrupting on irq 96
ehci0 at awinusb0: EHCI USB controller
ehci0: EHCI version 1.0
ehci0: companion controller, 1 port each: ohci0
usb1 at ehci0: USB revision 2.0
ehci0: interrupting on irq 71
awinusb1 at awinio0 port 1
awinusb1: no restrict gpio found
ohci1 at awinusb1: OHCI USB controller
ohci1: OHCI version 1.0
usb2 at ohci1: USB revision 1.0
ohci1: interrupting on irq 97
ehci1 at awinusb1: EHCI USB controller
ehci1: EHCI version 1.0
ehci1: companion controller, 1 port each: ohci1
usb3 at ehci1: USB revision 2.0
ehci1: interrupting on irq 72
motg0 at awinio0: OTG
motg0: interrupting at irq 70
motg0: no restrict gpio found
motg0: Dynamic FIFO sizing detected, assuming 16Kbytes of FIFO RAM
usb4 at motg0: USB revision 2.0
awinmmc0 at awinio0 port 0: SD3.0 (DMA)
awinmmc0: interrupting at irq 64
ahcisata0 at awinio0: AHCI SATA controller
ahcisata0: interrupting on irq 88
ahcisata0: ignoring broken port multiplier support
ahcisata0: AHCI revision 1.10, 1 port, 32 slots, CAP 0x6f24ff80<CCCS,PSC,SSC,PMD,SAM,ISS=0x2=Gen2,SCLO,SAL,SALP,SSS,SSNTF,SNCQ>
atabus0 at ahcisata0 channel 0
awiniic0 at awinio0 port 0: Marvell TWSI controller
awiniic0: interrupting on irq 39
iic0 at awiniic0: I2C bus
awge0 at awinio0: Gigabit Ethernet Controller
awge0: interrupting on irq 117
awge0: Ethernet address: 02:0a:09:03:27:08
rlphy0 at awge0 phy 1: RTL8201L 10/100 media interface, rev. 1
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
awinac0 at awinio0: CODEC
audio0 at awinac0: full duplex, playback, capture, mmap, independent
awinhdmiaudio0 at awinio0: HDMI 1.3
audio1 at awinhdmiaudio0: half duplex, playback, mmap
awinnand0 at awinio0
awinir0 at awinio0 port 0: IR
awinir0: interrupting on irq 37
cir0 at awinir0
gpio0 at awingpio0: 18 pins
gpio1 at awingpio0: 25 pins
gpio2 at awingpio0: 28 pins
gpio3 at awingpio0: 12 pins
gpio4 at awingpio0: 12 pins
gpio5 at awingpio0: 28 pins
gpio6 at awingpio0: 22 pins
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
cpu1: 912 MHz Cortex-A7 r0p4 (Cortex V7A core)
cpu1: DC enabled IC enabled WB disabled EABT branch prediction enabled
cpu1: isar: [0]=0x2101110 [1]=0x13112111 [2]=0x21232041 [3]=0x11112131, [4]=0x10011142, [5]=0
cpu1: mmfr: [0]=0x10101105 [1]=0x40000000 [2]=0x1240000 [3]=0x2102211
cpu1: pfr: [0]=0x1131 [1]=0x11011
cpu1: 32KB/32B 2-way L1 VIPT Instruction cache
cpu1: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache
cpu1: 256KB/64B 8-way write-through L2 PIPT Unified cache
vfp1 at cpu1: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals
vfp1: mvfr: [0]=0x10110222 [1]=0x11111111
sdmmc0 at awinmmc0
uhub0 at usb0: Allwinner OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
uhub1 at usb1: Allwinner EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 1 port with 1 removable, self powered
uhub2 at usb2: Allwinner OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 1 port with 1 removable, self powered
uhub3 at usb3: Allwinner EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 1 port with 1 removable, self powered
uhub4 at usb4: Mentor Graphics MOTG root hub, class 9/0, rev 1.00/1.00, addr 1
uhub4: 1 port with 1 removable, self powered
ld0 at sdmmc0: <0x02:0x544d:SA08G:0x14:0x12436e27:0x0e6>
ld0: 7388 MB, 3752 cyl, 64 head, 63 sec, 512 bytes/sect x 15130624 sectors
ld0: 4-bit width, bus clock 50.000 MHz
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
I’ve been itching to go wireless on my office desk for sometime. The final wires to eradicate are from my Mac into a USB hub connected to two hard discs for backups. Years ago I had an Apple Time Capsule. The Time Capsule is an Airport Wi-Fi basestation with a hard disc for Macs to back up to using the Time Machine backup software. It was pretty solid kit for a couple of years. Under the hood, it runs NetBSD and as an aside, I have had a few beers with the guy who ported the operating system. The power supply decided to give up – a very common fault apparently.
I will clean the cables up. I promise.
When I was on my travels and living in two places, I had hard discs in both locations. The Mac supports multiple discs for backups and I encrypted the backups in case the discs were stolen. But now I’m in one home, I want to be able to move around the house with the Mac but still backup without having to go to the office. We are a two Mac house, so we need something more convenient.
I already have a base station and I don’t really want to shell out loads of money for an Apple one. There are several options to setup a Time Capsule equivalent.
Enter the Raspberry Pi. I have a Raspberry Pi 3 and within minutes one can install the Netatalk software. This has been available for years on Linux and implements the Apple file sharing protocols really well. With an external drive added, I was able to get a Time Machine backup working using this article.
I could not use my existing backup drive as is. Linux will read and write Mac OS drives, but there is a bit of too-ing and fro-ing so it is best to start with a fresh native Linux filesystem. Even if you can get it to work with the Mac OS drive, it will not be able to use a Time Machine backup from a drive previously directly connected.
I’ve been using this setup for the last couple of weeks. I have not had to do a serious restore yet and I should caveat that I still have a hard drive I use directly into the machine just in case. The first rule of backups – a file doesn’t exist unless there are three copies on different physical media.
(The Raspberry Pi is setup to be MiniDLNA server. It will stream media to Xbox’s and other media players.)
I installed sudo on NetBSD 7.0 using pkg. I copied /usr/pkg/etc/sudoers to /etc/sudoers because the docs say /etc/sudoers and possibly /etc/sudoers.local is used. I uncommented the line wheel ALL=(ALL) ALL. I then added myself to the wheel group. I verified I am in wheel with groups. I then logged off and then back on.
When I attempt to run sudo <command>, I get the standard:
jwalton is not in the sudoers file. This incident will be reported
What is wrong with my sudo installation, and how can I fix it?
I am interested to use NetBSD as the operating system on my server. I have not used a system where security updates are performed by source, but have read enough in the guide to feel comfortable trying it. However, I do not know how long this operation is likely to take.
Given a fairly modest server of say 1 processor core and 0.5 to 1.0 GB RAM, how long might it be expected to take to build the userland and kernel of an x86_64 system, following the directions of Chapter 33. Updating an existing system from sources in the guide?
Also, how much local disk space does this operation require? I have not seen mention of that in the guide.
I just install NetBSD in from ISO image inside VirtualBox.
Official source ISO media contains only MD5/SHA512 sums.
Similarly pkg_add binary packag repository contains only SHA512 sums.
Both FTP and HTTP distribution channels is unsecured and check sums file transfer may be altered.
Is digital signature used by NetBSD project?
How enforce checks for package integrity when pkg_add is used?
I confess I am late to the game: the Go programming language came out in 2009 and I had not had the chance to go all in for a real project until two weeks ago. Here is a summary of my experience. Spoiler alert: I’m truly pleased.
What I set out to build is a read-only caching file system to try to solve the problems I presented in my previous analysis of large builds on SSHFS. The reasons I chose Go are simple: I had to write a low-level system component and, in theory, Go excels at this; I did not want to use plain C; Go had the necessary bindings (for FUSE and SQLite); and, heck, I just wanted to try it out!
It only took me a little over two days to get a fully-functional implementation of my file system, and this is without having ever written a FUSE-based file system before nor any non-toy Go code. That said, I had previously written a kernel file system in 2005 and that helped navigating the whole endeavor and tuning the resulting performance.
Where is the code you ask? Nowhere yet unfortunately; I hope to make it available once it is ready.
The summary of what follows is simple: after two weeks, I’m in love with Go.
I remember languages people dismiss Go back in 2009 on the grounds that it was a simplistic language without novel concepts… but there lies its beauty: Go lets you get things done quickly and safely with few performance penalties. I feel I can write much more robust code with Go than with any other comparable language.
Disclaimer: Keep in mind that I’m still a newbie. I’m sure that some of the items below are naïve, incomplete or plain incorrect. If so, please let me know!
Enforced coding style: You may or may not like Go’s coding style (I personally would change a few things), but the fact that it is chosen for you and that gofmt exists beats any personal preferences you may have. No more thinking about tabs vs. spaces; no more thinking about brace placement; no more thinking about the look of your code. It. Just. Does. Not. Matter. Focus on your code’s logic and let the machine format it in a consistent manner across the whole Go ecosystem.
Explicit error handling: Yes, the if err != nil { return err } pattern gets old very quickly, but having to explicitly handle errors shows you how hard it is to write robust code.
You’d say that C is similar in this regard, but not really. There are two key differences: the first is that Go functions can return more than one result, which makes it more difficult to ignore the error code if you want to use the actual result; and the second is the defer keyword, which helps avoid leaks in error paths. I personally prefer C++’s ability to implement the RAII pattern, but defer is a good compromise.
Do I miss exceptions? Not really. While exceptions let you keep your main code path clean of error handling, they also relegate error handling to a secondary place. This is OKish if the language has provisions to ensure that exceptions are handled at some point like Java’s throws definition, but that’s not how the majority of languages work. Just think of how easy is for exceptions leak all the way through the program’s entry point in C++ or Python.
Strong typing, even for native types: The fact that the equivalent of C’s typedef generates types that can’t be implicitly mixed, or the fact that numeric types are not automatically promoted to other types, is a good way to prevent subtle bugs. Having to explicitly convert among types forces you to think about the consequences of doing so: Integer overflow? Check. Sign issues? Check. Precision issues? Check.
Builtin profiling with pprof: The pprof profiling tool was invaluable to diagnose performance and memory consumption deficiencies in my caching file system and the libraries I used. What’s best is that pprof was also incredibly easy to plug into my program.
Honestly, I’m in awe to see that this tooling is open-source because I’ve gotten very accustomed to this particular way of debugging at Google.
The standard library: While Go by itself is a decent language, it becomes excellent because of its extensive standard library. The existence of tooling to implement pretty much anything you want wins over any deficiencies you may think the language has. The same applies to Java, by the way.
Simple package management: Getting started with a piece of code is easy, and pulling in additional dependencies is trivial. I actually do not like the underpinnings of this “modern” trend of language-specific package management systems, but I must admit that it has been a pleasure to just add new imports, call go get, and have everything up and running in a matter of seconds.
Build speed: This item is not tied to my recent experience with Go but is worth mentioning. About two years ago, I built the Go language on my underpowered NetBSD VM. I was expecting horrendously long build times like those of GCC or Clang, but was pleasantly surprised to see that the Go compiler, along with its extensive standard library, compiled in a few minutes.
GOPATH: Sorry but having to modify the environment to get a program to run in the default case is ridiculous in this day and age: things should “just work” and Go is different. I have mitigated my concerns about this by writing a Makefile that sets things up automatically so I do not have to worry about customizing my environment on a project basis.
Package management: As we saw in the good parts, Go’s package management gets the job done but its views on the world are too simplistic. On the one hand, the development environment assumes that you will always want to use the HEAD versions of your dependencies; and on the other hand, the default is to merge the dependencies with your personal code in the same workspace.
I’ve managed to work around the latter by splitting my dependencies in a separate directory and using GOPATH to find them. The excellent article So you want to write a packaging system is a great resource on this topic, and as you read through it, you will realize that it’s geared towards revamping Go’s package manager.
The simplistic log module: It is good that the standard library provides a logging module, but it seems way too simple for anything other than trivial logging. Fortunately, the external glog module provides increased functionality such as level-based logging and persistent logging. Therefore, you may say that being able to choose between something simple and something more complex is good, right?!
Unfortunately, that’s not the case and this is the reason I pinpoint this particular module from the standard library. The real benefits of using glog come when the whole stack uses the same logging infrastructure and principles. Because glog is not the standard module, some Go packages will use it and others will not. As a result, you do not get the full benefits of using glog in your project because most libraries you depend on will not do the same.
Mutability by default (aka no const keyword): Before you say “hey, const is currently not present, but if we find the need for it later on, we’ll add it!”, that’s… not great. The problem with adding a const keyword post-facto is that this is the wrong keyword to add: state should be immutable by default, and what the language should have is a mutable keyword to clearly mark those variables that hold multiple values throughout their lifetime.
The reason I point out this specific language feature is because I’ve mentally gotten very accustomed to separating constant vs. mutable variables in code. Reading code that adheres to this strict separation is easy to follow: you can quickly map the immutable state in your mind and then focus on the mutable pieces of the code to understand what’s happening. Without these clues, you need extra effort.
Short identifiers: A lot of Go code, including the snippets in the documentation, is plagued by extremely short identifiers. Those are used as local variables, function arguments, method receivers, and even structure members. Combined with the heavy use of interfaces, it is hard to read the code: What’s a c? And an r? And an s.wg? Yes, there is bad code all around, but when the official language documentation seems to recommend this approach, you can expect that others will follow the apparent recommendation in all cases.
Remember that code is written to be read many more times than it is written. Fully spelled-out identifiers help significantly.
Maximum line length: Even though Go has a predefined coding style, there is no pre-specified maximum line length. While it is great to want to abolish a custom that remains from the punch-card days, keeping code narrow has its benefits: think of two or three side-by-side editors in a small laptop screen.
This would be a non-issue if editors were able to wrap long code lines in a way that made sense—just like word processors do with text—but current editors are terrible at this. So, personally, I will stick to the 80 character limit; I have given 100 a try per more modern recommendations… but when I opened the code in my laptop and couldn’t see two files at once without wrapping, I knew I had to go back to 80.
To conclude, let’s talk about editors. I’ve been a long-time Vim and Emacs user and now decided to give another editor a try. Heresy!
In fact, some time last year, I started playing with Atom after all the hype around this editor. It really is full-featured, but it also is overwhelming and fragile: overwhelming because there are knobs everywhere and fragile because I had never had an editor throw errors at me routinely during “normal” operation. (I’m sure I can blame some plugin I installed, but still.)
Soon after, Visual Studio Code (VSCode) came out: an open source editor from Microsoft no less. I installed VSCode just to take a look and I found much sought simplicity: just look at the way to configure the editor, which is by adding personal or project-specific overrides to an empty JSON file. I felt I’d enjoy this editor more than Atom though I did not have a use for it at the time: most of my development still happens over SSH so VSCode would not fit the bill.
This experiment with Go gave me the chance to use VSCode, particularly because I was writing the project primarily for OS X (my desktop) and because I wanted an IDE-like experience. As it turns out, VSCode has a pretty good Go plugin. The reason I wanted IDE integration is because writing code in an unknown language with an unknown standard library is a task that truly benefits from autocompletion and inline syntax validation. I wouldn’t have been able to prototype my file system as quickly as I did with a bare editor.
I know, I know, I could have set up Emacs to do the same thing, but I just wanted to give VSCode a ride. And, mind you, I just took a look at what’s involved to set Emacs up for Go and didn’t find it pleasing; way too much manual work involved.
Do you Go already?
So I use this great cheat sheet in order to use letsencrypt free Certificate authority on my own servers, but while this small doc is very straightforward it doesn’t explain much about nginx‘s configuration. So I’ll drop my own right here so your journey through TLS is even simpler:
1 | $ cat /usr/pkg/etc/nginx/nginx.conf |
A very basic proxy_pass location would be:
1 | $ cat /usr/pkg/etc/nginx/sites/example.com |
For an even more hardened configuration, you might want to checkout 2*yo‘s own configuration.
I recently added a fantastic graphing tool named facette to pkgsrc.
Facette knows how to pull data sources from various backends, and among them, the famous collectd.
In this article, we will see how to setup both on NetBSD but keep in mind it should also work for any platform supported by pkgsrc.
First up, collectd installation. It can be done either with pkgin (binary installation) or pkgsrc (source installation):
pkgin1 | $ sudo pkgin in collectd collectd-rrdtool |
pkgsrc1 | $ cd /usr/pkgsrc/sysutils/collectd |
Tune up a minimal collectd configuration
1 | Hostname "myname" |
Enable and start collectd
1 | # echo "collectd=YES" >> /etc/rc.conf |
Wait a couple of minutes for collectd to actually collect some datas, you should see them appear in /var/db/collectd/rrd.
As I write these lines, facette is not yet available as a binary package, it will probably be available in the next pkgsrc release, so we’ll have to install it using pkgsrc
1 | $ cd /usr/pkgsrc/sysutils/facette |
That’s right, no configuration, I’ve setup the package so it is working out-of-the-box with a basic collectd installation.
If everything went well, you should be admiring a facette console by connecting with your web browser to the port 12003
I have read a shitload of overcomplicated setups to bring up a postfix / dspam SMTP + antispam server, and finally came to a much lighter and simpler configuration by basically reading documentation and real life examples.
Note this is suitable for a personnal and basic environment, no database, no virtual setup. Basic stuff.
The target system is NetBSD but this short doc should apply to pretty much any UNIX / Linux.
On dspam‘s side, I added the following parameters:
1 | # really postfix |
On postfix‘s main.cf side:
1 | # don't overwhelm dspam, only one message at a time |
Warning, I used regexp: instead of pcre: because that’s what NetBSD base’s postfix supports.
The dspam_filter_access pipes the message to dspam‘s socket by matching everything:
1 | $ cat /etc/postfix/dspam_filter_access |
The only remaining piece is to declare the dspam service in postfix‘s master.cf file:
1 | dspam unix - n n - 10 pipe |
The final delivery method is up to you, but I chose procmail, mostly because I have written my rules a while ago and am too lazy to adapt them to sieve :)
1 | mailbox_command = /usr/pkg/bin/procmail |
Sources:
Not so long ago, I wrote about using pkgsrc on Debian GNU/Linux, and assumed you’d start an installed service using rc.d. When I setup the new iMil.net server, I decided to give a try to kvm as it is easier to maintain, has good performances (sometimes better than Xen), nice administration tools, plus NetBSD now has a good VirtIO driver but no PVHVM support yet.
The first thing I do when setting up a Debian Jessie server is getting rid of systemd, whose philosophy and quality don’t match my personnal taste; but in that case, I wanted to use libvirtd so I could manage my virtual machines with virt-manager, and as a matter of fact, libvirtd has a hard dependency on systemd. There was no escape this time, I had to learn and use it.
Once nginx installed through pkgsrc, I wrote a unit file:
1 | $ cat /etc/systemd/system/nginx.service |
Then enabled the nginx service:
1 | $ sudo systemctl enable nginx |
And finally started it:
1 | $ sudo systemctl start nginx |
Witness everything went as expected:
1 | $ sudo systemctl status nginx -l |
Note that this does not prevent from using nginx‘s -t or -s flags.