Savannah Internals

This page describes inner workings of the GNU Savannah platform (web frontend + backend processes).

Some terminology

'GNU Savannah' (or just 'Savannah') is the name of the entire platform, encompassing the various hosts (see SavannahArchitecture), services (see SavannahServices), people (see SavannahTeam) and other odds and ends.
'Savane' (French for 'savannah') is the code name for the source code which runs many parts of the savannah platform (most notably: the web frontend at http://savannah.gnu.org).

While usually these are distinguished, they are sometimes used interchangeably, and should be appropriately understood by the context.

'groups' - in the savane code, projects are referred to as "groups" (and in a few cases, these are conceptually groups rather than programming projects, e.g., the Free Software Free Society Portuguese and the GNU Spanish Translation Team). The database table is named groups, and several php files are named groupXXX.php while referring to projects. group_id=N in urls refer to the project internal ID.
Savannah Administrators are volunteers with special access permissions on the savannah frontend website (see 'becoming super-user' below), and administrator ssh access to the savannah hosts. See SavannahTeam and List of current administrators.
Project Administrators (in the context of this document) are savannah users who have administrative access to their own project. As an example, these people are project administrators for GNU AWK: https://savannah.gnu.org/project/memberlist.php?group=gawk.

Savannah source code

The source code for the php frontend and most of the perl/shell backend scripts is available at: http://git.savannah.gnu.org/cgit/administration/savane.git. N.B. That's the administration project/group, and the git repository named savane within it.

To experiment with running a local copy of Savane see RunningSavaneLocally and SavaneInABox.

Warning about repositories: There are several other repositories/clones/forks which look like the savane repositories (e.g., savane-cleanup, another savane.git ). The repository listed above is the only official one. All code running on the savannah hosts comes from the administration project, using several version control systems there: git, cvs, subversion; each of which holds different stuff.

Savannah hosts

Please see SavannahArchitecture and SavannahServices rather than repeating information here.

workflow - overview

savannah-internals-2015.dia

Users submit a new project. Savannah administrators approve the project (see ApprovingSubmission).
Project administrators of existing project can change project settings (e.g., add a mailing list, switch from cvs to git, etc.).
The PHP code on frontend updates the MySQL database, and optionally runs one-time project creation scripts.
On vcs and download, cron jobs regularly query the database for updates, and make the necessary changes (e.g., create a new git repository for a new project).
The cvsweb repositories hold the html files which are the source for a project's web pages. When these are updated, a special hook calls a cgi script on www.gnu.org to request a website update.
A cron job on www.gnu.org regularly checks for update requests, and cvs updates the requested repositories.
A cron job on internal regularly query the database for mailing list updates. It then creates the mailing lists on lists.gnu.org. This is done because lists.gnu.org is managed by the FSF, not Savannah.
Publishing a new tarball release for GNU packages is performed by an automated process, explained in the GNU Maintainer's guide. It is not managed by Savannah administrators, and only mentioned here for completeness.
GNU Bug Tracker (http://debbugs.gnu.org) is based on Debian's Bug Tracking System. It is not managed by Savannah administrators. Packages hosted on Savannah, either gnu or nongnu, can request to use the bug tracker. More information: http://debbugs.gnu.org/Using.html.
For nongnu projects, project members with write access can upload new files to be available for download as 'releases' (e.g., http://download.savannah.gnu.org/releases/freetype/). While GNU projects can also store files this way, it is very strongly recommended that official GNU packages use ftp.gnu.org via the automated ftp uploads (see item 8, above). For details, see download section in SavannahServices. See also UserAuthentication.
Project members with write access can push code to the repositories on vcs using ssh. See 'vcs' section in SavannahServices for configuration details. See CvsGettingStarted, Svn, UsingHg, UsingBzr, UsingGit for usage information, and UserAuthentication for ssh authentication details.

User registration (frontend)

Anyone with a valid email account can register on GNU Savannah.

Public URL: https://savannah.gnu.org/account/register.php
Source code: /frontend/php/account/register.php
The register.php form creates a new record in the database user table, with STATUS='P', and sends the user a confirmation URL (e.g., https://savannah.nongnu.org/account/verify.php?confirm_hash=a1b2...).
The confirmation URL leads to /frontend/php/account/verify.php, which updates the user's record in user table to STATUS='A'.

Warning: the email that Savannah sends will often be considered spam by big email providers, or outright rejected.

Project submission (frontend)

Any savannah user can submit a submit a new project request. This procedure is for nongnu projects only (for gnu projects see here: http://www.gnu.org/help/evaluation.html).

Public URL: https://savannah.gnu.org/register/
Source code: /frontend/php/register/index.php (and see all files in /frontend/php/register/ ).
After form submission, The php confirmation.php file in register/:
- updates the database record in the groups table, with STATUS='P'.
- sends a notification email to [email protected].
- Creates a new task in the task tracker (FIXME: how?)
- FIXME: the source code mentions something about cookbook and recipes database tables?
The project is in pending state until approved by Savannah administrators.

Project approval, creation (frontend)

Savannah admins approve and create projects after careful and manual review. Project creation is the same for GNU projects (the manual review for GNU projects is done by the GNU evaluation volunteers, not savannah admins).

Public URL: https://savannah.nongnu.org/siteadmin/groupedit.php?group_id=N (only available to savannah admins in superuser mode).
The URL is display as 'group administration' link when viewing a pending project ticket.
Source code: /frontend/php/siteadmin/groupedit.php.
In this page the following changes can be made:
- system name (i.e. 'short name') - the name will be used in URLs, as a unix group name for access control, and as directory names in various locations. Must be unique.
- status: pending,active,deleted,maintenance,incomplete.
- license
- visibility (public/private)
These changes are written to the groups table.
Two special commands are available in this page as well:
- The green "V" icon: calls groupedit.php again with cgi parameters updatefast=1 and status=A, thus marking the project as active in the groups table, and in savannah.
- The configure (screwdriver and wrench) icon: calls /frontend/php/siteadmin/triggercreation.php, which calls the necessary one-time project creation scripts.
The triggercreation.php script does the following:
- Sets project defaults (e.g., having a homepage, having 'downloads', using cvs repository, using task,support,bugs trackers, using news, etc.). This are fields such as use_homepage=1 in the groups table.
  The items set by default are hard-coded in triggercreation.php. (this is one reason that by default, all new projects on savannah start with a cvs repository). This settings can later be changed by the project administrators.
- Sends an email to the project administrator (based on the savannah user who submitted/registered the project).
FIXME: triggercreation.php mentions "site-specific triggers executed"
- which ones are these and where are they executed?

Project administration - general information (frontend)

Project admins can modify various items for their projects using the savannah web interface.

Name, Short/Long descriptions
- public URL: https://savannah.nongnu.org/project/admin/editgroupinfo.php?group=N
- source code: /frontend/php/project/admin/editgroupinfo.php.
- The script modifies the group table.
Project members (additional savannah users with write-access):
- public URL: https://savannah.nongnu.org/project/admin/useradmin.php?group=N
- source code: /frontend/php/project/admin/useradmin.php
- modifications are stored in the user_group table (a many-to-many relation table between user and groups tables).
- uses code from /frontend/php/include/member.php.
Project squads
- FIXME: a per-project list of multiple savannah users?
- public URL: https://savannah.nongnu.org/project/admin/squadadmin.php?group=N
- Source code: /frontend/php/project/admin/squadadmin.php.
Permissions
- per-user permissions for website features (e.g., tracker management)
- Public URL: https://savannah.nongnu.org/project/admin/userperms.php?group=N
- Source code: /frontend/php/project/admin/userperms.php.
Jobs
- "jobs" are postings which appear in the Contributors Wanted page.
- Public URL: https://savannah.nongnu.org/people/createjob.php?group=N
- Source code: /frontend/php/people/createjob.php.
Select Features
- See "feature selection" section below.
Configure Mailing lists
- See "mailing lists" section below.
Configure Cookbook
- FIXME: document/explain cookbooks (or remove them).
Set notifications:
- Select savannah users which will be notified upon tracker activity (e.g., new task, new support ticket, new bug report, etc.).
- Public URL: https://savannah.nongnu.org/project/admin/editgroupnotifications.php?group=N
- Source code: /frontend/php/project/admin/editgroupnotification.php.
- FIXME: document/explain trackers

Project administration - feature selection (frontend)

Project admins can enable/disable features for their projects, using the savannah web interface.

features in this context are the facilities used by the project, such as git, cvs, homepage, downloads, news, bugs/support/patches/tasks trackers, mailing lists, etc.
Each feature has a corresponding field in the groups table (e.g., use_git, use_homepage, use_downloads) which is set to 1 or 0.
Some features are associated with a url. The urls for these features are initialized to the savannah defaults at project creation, but can be later changed by the project administrators. Examples:
1. If use_homepage=1, the url_homepage field defaults to http://www.gnu.org/software/PROJECT (for gnu projects) or http://www.nongnu.org/PROJECT (for nongnu projects). Project administrators can change this, although GNU packages should not do so.
2. If use_git=1, the url_git field defaults to git/?group=PROJECT (which resolves to https://savannah.gnu.org/git/?group=PROJECT, a page containing instructions about accessing git on savannah for this project). Project administrators can change this to any other URL, although they should not do so, especially not GNU packages. (The purpose of Savannah is to host source code.)
Public URL: https://savannah.gnu.org/project/admin/editgroupfeatures.php?group=PROJECT (only available to project admins).
Source code: /frontend/php/project/admin/editgroupfeatures.php.
When updating features (selecting or deselecting features), the changes are saved to the database, in the groups table. No other changes are performed.
These database changes will be picked up by the cron jobs running on other VMs (see below). This is the reasons some features take time to propagate (e.g., when a project administrator turns on git, the git repository is not created immediately). It would be better to do things synchronously, or at least very quickly.

Project administration - mailing lists (frontend)

Project admins can add/configure/delete mailing lists for their projects using the savannah web interface.

Mailing lists require special handling: the mailing list server (lists.gnu.org, using GNU Mailman) is administered by the FSF, not Savannah.
Special configuration on lists.gnu.org allows remote execution of shell scripts from frontend to automate list creation/deletion. More info: ListServer.
Public URL: https://savannah.gnu.org/mail/admin/?group=PROJECT (only available to project admins).
Source code: /frontend/php/mail/admin/index.php.
The php code updates the mail_group_list table (no other direct changes are done by the script). New mailing lists are added as new records in the tables. Change requests to mailing lists are done by changing the status field (see source code for possible values).
A cron job on internal reads the changes in the mail_group_list and executes the necessary changes on lists.gnu.org. See "cron jobs on internal" section below.

MySQL Database

See SavannahDatabaseSchema and SavannahDatabaseExamples.

Cron job on vcs

Stored in vcs:/etc/cron.d/sv:

# remake user list for outgoing mail.
*/10 * * * *    root    nice -n 11 sv_aliases --cron && sed -e '/# Savannah include start/,/# Savannah include end/s/\([^:]\+\): .*/\1: [email protected]/' /etc/email-addresses -e 's,This is /etc/email-addresses,DO NOT EDIT - GENERATED FROM email-addresses.,' > /etc/email-addresses_SENDER

# remake list of git repositories in each project.
*/15 * * * *    root    nice -n 11 /usr/src/infra/git/refresh-project-list.sh && /usr/src/infra/git/sv_cgit.pl

# remake CVSROOT/{commit,log}/info (for cvs commit email?).
31 * * * *  root    nice -n 11 /usr/src/infra/cvs/generate_log_accum.pl

# remake list of groups for the vcs.
35,05 * * * *   root    nice -n 11 sv_groups --cron --only-cvs --only-homepage --only-svn --only-git --only-hg --only-bzr

/backend/mail/sv_aliases.in (installed in vcs:/usr/local/bin/sv_aliases) - This script prepares an alias-type file for for exim4 mail server, used when sending commit updates to mailing lists.
- The script queries the MySQL database, extracting user names and emails for all savannah users. Additionally, emails for 'squads' (per-project mailing groups) are queried. The results are saved in vcs:/etc/email-addresses.
- vcs:/etc/email-addresses is symlinked to the standard vcs:/etc/exim4/email-addresses file. Example:
```
  agn: [email protected]
  karl: [email protected]
```
- The sed command in the cronjob file replaces each user's email with [email protected], and saves the results in vcs:/etc/email-addresses_SENDER. Example:
```
  agn: [email protected]
  karl: [email protected]
```
- FIXME: where is /etc/email-addresses_SENDER mentioned/used? in an exim4 config file?
refresh-project-list.sh - this script updates the vcs:/srv/git/project-list file which is used by gitweb CGI application.
- FIXME: file is vcs:/usr/src/infra/git/refresh-project-list.sh, and is not in the savane repository. Add it?
- gitweb configuration file is vcs:/etc/gitweb.conf.
- Implemetation detail: the script does not re-generate the project-list from scratch. Instead, it scans /srv/git for new git repositories, and only adds them to the file if they aren't already listed. This allows manually adding information to the project-list file, ensuring it will not be overwritten. The manually added information is the owner name/email for some repositories. Example:
```
  $ cat /srv/git/project-list
  ...
  administration.git Savannah+Hackers
  anubis.git [email protected]
  autoconf.git [email protected]
  automake.git [email protected]
  autostrap.git Sylvain+Beucler
  coreutils.git Jim+Meyering
  ...
```
  These names (e.g. Jim Meyering) will appear in the owner field when viewing the gitweb page of the project (example: http://git.savannah.gnu.org/gitweb/?p=coreutils.git).
sv_cgit.pl - this script updates the vcs:/srv/git/cgitrepos file which is used by the cgit CGI application.
- FIXME: file is vcs:/usr/src/infra/git/sv_cgit.pl and is not in the savane repository. Add it?
- cgit configuration file is vcs:/etc/cgitrc.
- Implementation detail: the script reads the /srv/git/project-list file (generated by refresh-project-list.sh, above), and for each mentioned project, writes an entry in cgitrepos. Example:
```
  $ cat /srv/git/cgitrepos
  ...
  repo.group=GNU Core Utilities
  repo.url=coreutils.git
  repo.path=/srv/git/coreutils.git
  repo.desc=GNU coreutils
  repo.readme=README.html
  repo.owner=Jim Meyering
  ...
```

generate_log_accum.pl - This script updates the CVS hooks in savannah repositories.

FIXME: file is vcs:/usr/src/infra/cvs/generate_log_accum.pl and is not in the savane repository. Add it?
The script queries the database (groups table) for project using cvs repositories, and projects using homepages (which use the cvsweb repositories).
For each cvs repository, it creates/updates the <project>/CVSROOT/{loginfo,commitinfo} files.

Example of commitinfo for source-code (not web) CVS repository:

  $ cat /sources/pspp/CVSROOT/commitinfo
  #<savane>
  DEFAULT /usr/local/bin/commit_prep.pl -T pspp_70 %p
  #</savane>
  ALL $CVSROOT/CVSROOT/no-commits %{s}

Example of loginfo for source-code (not web) CVS repository:

  $ cat /sources/pspp/CVSROOT/loginfo
  #<savane>
  DEFAULT /usr/local/bin/log_accum.pl --mail-to [email protected] --send-diff --file-prefix pspp_70 --config /etc/log_accum-sources.config %p %{sVv}
  #</savane>

Example of loginfo for web CVS repository:

  $ cat /srv/cvs/web/datamash/CVSROOT/loginfo
  #<savane>
  ALL echo 'Triggering webpages update...'; cat > /dev/null; curl http://www.gnu.org/new-savannah-project/new.py -s -F type=gnu -F project=`basename %r`
  #</savane>

sv_groups - This script creates new repositories for projects in the savannah database
- reminder: when a new project is approved/created on savannah, or when a project administrator selects a new repository feature the frontend php code only updates the database. this script picks up the changes and executes them.
- file is /backend/accounts/sv_groups.in.
- The script queries the groups table, and creates the needed repositories.
  Example: For project XXX, if groups.use_git=1, it look for vcs:/srv/git/XXX.git. If it doesn't exist, create it with git init and additional savannah-specific configurations (e.g. permissions, hooks, xattrs).
- Repository initialization and configuration is delegated to the relevant perl module. See Git.pm, Cvs.pm, Bzr.pm, Svn.pm, Hg.pm, Download.pm.
- Two types of CVS repositories can be created: cvs (Using CvsMakeArea in CVS.pm) - source code repository; webcvs (Using WebCvMakeAreaSavannahGNU in CVS.pm) - webcvs repository (content will be available on www.gnu.org/www.nongnu.org).
- The script also contains legacy code to create unix groups on the local machine (using /usr/sbin/groupadd). This part is not used any more, since groups and users are delegated to nsswitch/libnss-mysql (see UserAuthentication).
- FIXME: What as "Area Attic" (mentioned in the code)?
- The script takes parameteres determining which items to create (e.g. --only-git).
- On vcs host, the git/hg/bzr/cvs/cvsweb repositories are created.
- On download host, the download (and legacy 'arch') are created.

Cron jobs on download

stored in download:/etc/cron.d/sv:

*/30 * * * *    root    sv_groups --cron --only-download --only-arch

sv_groups - This script creates new repositories for projects in the savannah database. See description for sv_groups in the above section.

Cron jobs on frontend

frontend:/etc/cron.d/sv_export [non-functional] - contains the following commands:

  # XML data export
  02 * * * *      root    sv_export --xml-path /var/cache/savane

  # Remove old data export
  30 1 * * *      root    sv_export_cleaner --xml-path /var/cache/savane

  # Get list of Git repositories
  3,33 * * * *    root    scp vcs.sv.gnu.org:/srv/git/cgitrepos /etc/savane

sv_export is an old feature which exports project data into XML.
FIXME: remove this?

frontend:/etc/cron.daily/sv_list_groups [unfinished] - an attempt to publish an authoritative list of active projects hosted on GNU Savannah. The script contains the following code:

  mysql savane -B -e 'SELECT group_type.type_id, group_type.name,
                             unix_group_name, group_name
                      FROM groups, group_type
                      WHERE groups.type = group_type.type_id
                            AND groups.status="A"
                      ORDER BY type;' \
                      > /var/www/overlay/cooperation/groups.tsv

FIXME: complete this nice feature, and expose it with apache configuration. Also possibly with JSON data, and more fields.

Cron jobs on internal

stored in internal:/etc/cron.d/sv:

# Assign uidNumber and gidNumber inthe database (and rebuild cache for
# libnss-mysql-bg). 7211 was the latest group created by the old
# sv_groups cron - maybe one day we'll move everything to the 70000
# range. Then create homedirs and authorized_keys files.
*/5 * * * *   root   sv_assign_uid_gid --min-gidNumber=77212


# New list creation
*/5 * * * *     root    sv_mailman --cron

# Database cleaning
*/25 * * * *    root    sv_cleaner --cron
48 12 2 * *     root    sv_cleaner --cron --big-cleanup

# Reminder
4 4 * * *       root    sv_reminder --cron
25 18 * * *     root    sv_reminder --cron

sv_assign_uid_gid - creates unique user ids and group ids for users/projects in the savannah system.
- reminder: across all savannah hosts, unix user/group ids are managed with nsswitch/libnss-mysql (see UserAuthentication for details).
- Source code: /backend/accounts/sv_assign_uid_gid.in.
- When a new user (or project) is created on savannah, the php code on frontend does not assign uid/gid.
- This script picks up new users (SQL: user.uidNumber is NULL) and new projects (SQL: groups.gidNumber is NULL) and assigns them ids.
sv_mailman - queries the mail_group_list table, and creates/updates the mailing lists on lists.gnu.org.
- Source code: /backend/mail/sv_mailman.in.
- This script is tightly-coupled with the php code in /frontend/php/mail/admin/index.php.
- The index.php code sets the mail_group_list.status field to the requested new status, and the sv_mailman script executes the changes and updates the database.
- sv_mailman calls internal:/usr/sbin/newlist. The newlist script is a stub which delegates the call to lists.gnu.org using SSH. See MailSystem for more details.
  The gist of internal:/usr/sbin/newlist is:
```
  #!/bin/bash
  list_full_name=$1
  admin_mail=$2
  password=$3
  cat <<EOF | ssh -T -i ~/.ssh/id_dsa.lists.gnu.org [email protected]
  COMMAND=newlist
  LIST_FULL_NAME=$list_full_name
  ADMIN_MAIL=$admin_mail
  PASSWORD=$password
  EOF
```
sv_cleaner - cleans up left-overs from interrupted php code?
- source code: /backend/misc/sv_cleaner.in.
- FIXME: investigate this script.

sv_reminder - sends email reminders to users.

source code: /backend/misc/sv_reminder.in.

The help message says:

    This script will send reminder to users in some specific cases. 
    Case 1: an item is about to start or finish, all people from the item 
    notification list should be warned. 
    Case 2: an user decided to receive a batch of open item assigned to
    him each x days, minimum priority being > 5
    Case 3: a project admin decided that members assigned to some item of 
    should receive a batch each x days, minimum priority > 5
    (Case 4: a project admin decided that members assigned to some item
    whose statut/resolution havent changed for some period should receive
    a batch each x days)

FIXME: investigate this script.

www.gnu.org update mechanism

Mailing list configuration

Additional functionality

super-user and impersonate on frontend

frontend configuration files

(files which are not in the administration (or any version control) repositories)