Continuous integration/Jenkins
Jenkins is a Java tool used to handle recurring tasks such as running tests or building packages. Our primary install is at https://integration.wikimedia.org/ci/.
The tool is permanently connected to our review tool (Gerrit) and can be made to react on changes submitted to Gerrit. A typical example, is running MediaWiki unit tests whenever a change is submitted to the mediawiki/core.git
repository.
The main repository for the layout of Jenkins itself is integration/jenkins.git
(installed in /var/lib/jenkins
on contint1001
.
The configuration of individual jobs is abstracted via Jenkins job builder. The jobs are triggered with Zuul (which abstracts Gerrit's events).
You can add new Jenkins slaves.
Services
These services run alongside on the Jenkins server as a result of certain jobs that publish results outside the Jenkins realm for other uses:
Nightly snapshots
They are being generated at 3am UTC via the Jenkins job "Nightly - MediaWiki core". The ant target is nightly-mediawiki-core
and is really straightforward : it get the latest version of master and uses git-archive
to generate a zip file. It is then copied under /org/mediawiki/integration/nightly
hierarchy which is publicly available via https://integration.wikimedia.org/nightly/.
The latest snapshot is always available at https://integration.wikimedia.org/nightly/mediawiki/core/mediawiki-latest.zip (it is a symbolic link to the latest snapshot).
Wikipedia Android alpha app builds
The mobile apps team uses Jenkins to build its alpha release of the Wikipedia app, available at http://android-builds.wmflabs.org/. Each time a change to /apps/android/wikipedia is merged in Gerrit, an apps-android-wikipedia-publish job is triggered in Jenkins. Upon successful completion, the resulting APK and a JSON file containing the build timestamp are preserved as build outputs.
Every 15 minutes, a script invoked by a cron job running on the android-builder
instance in Wikimedia Cloud VPS checks for a new APK in the job outputs, and publishes it at the above web site if found.
For more info, see Wikimedia_Apps/Team/Android/App_hacking/Alpha_build_server.
Installation
This page is outdated. |
Automatic installation
curl https://raw.github.com/valhallasw/wikimedia-mkjenkins/master/mkjenkins.sh | bash
To make the install go faster, it helps to have a mediawiki-core checkout in ~/src/mediawiki-core - if this repository exists, it will make a local clone. If it doesn't, it will download from gerrit instead (slow!).
Manual installation
git clone https://gerrit.wikimedia.org/r/integration/jenkins.git ~/.jenkins
~/.jenkins
is the default jenkins configuration directory
- WM-specific configuration patch I -
ln -s $HOME/.jenkins /var/lib/jenkins
- because some jobs assume jenkins is installed in
/var/lib/jenkins
- because some jobs assume jenkins is installed in
- Download jenkins and place it in ~/.jenkins
- install the following plugins (download into
~/.jenkins/plugins
): - Download jenkins-job-builder and its configuration ‒ see Continuous_integration/Jenkins job builder for more information. You don't need a password when you install Jenkins locally.
- WM-specific configuration patch II - Patch the JBB configuration that depends on Zuul (see the mkjenkins script for a diff)
- If you already have a checkout of mediawiki-core:
git clone --mirror -l -- your_existing_checkout /var/lib/jenkins/git/mw-core-bare
. Otherwise,git clone --mirror -- https://gerrit.wikimedia.org/r/mediawiki/core.git /var/lib/jenkins/git/mw-core-bare
- Start Jenkins:
cd ~/.jenkins && java -jar jenkins.war&
- When Jenkins is running, install the JBB jobs:
rm -f $HOME/.cache/jenkins_jobs/jenkins_jobs_cache.yml && jenkins-jobs --conf jenkins_jobs.ini update config/
.
Issue?
Hung beta code/db update
This deadlock seems to happen more often than not following or during a database update that is taking a while to complete.
- Take deployment-deploy01 offline in Jenkins https://integration.wikimedia.org/ci/computer/deployment-deploy01/markOffline
- Kill any jenkins jobs running on deployment-deploy01 via Jenkins UI
- Kill all pending jobs in the Jenkins queue that are "waiting on executors"
- Disconnect deployment-deploy01 https://integration.wikimedia.org/ci/computer/deployment-deploy01/disconnect
- Bring deployment-deploy01 back online (button labeled "Bring this node back online")
- Launch slave agent (there's a button that says this)
- Check agent log to see that it connected https://integration.wikimedia.org/ci/computer/deployment-deploy01/log
Sometimes you have to do this whole dance several times before Jenkins realizes that the there are a bunch of executors that it can use.
Alternate method:
- Go to https://integration.wikimedia.org/ci/manage
- Go to "Configure System"
- Search page for "Enable Gearman"
- Un-check the checkbox
- Save
- Wait 30s
- Check the "Enable Gearman" checkbox
- Save
This second method may interrupt communication between running Jenkins jobs and Zuul but it seems to work even when the offline/online method fails to clear the deadlock.
Alternate alternate method:
- Login to https://integration.wikimedia.org/ci
- Hit the red
[x]
to cancel one pendingbeta-scap-eqiad
job - That's it!
It seems that this is some conflict between the Jenkins native scheduler and the Gearman scheduler. Cancelling the build seems to fix the problem. Subsequent builds will deploy the same code to production.
Restart
Zuul should not be restarted. Zuul preserves the queue and continues after the restart.
Via web interface
Apply the self-serve Jenkins repair!
With a safeRestart
any currently running jobs will block a restart until they are canceled. Any long running jobs should be killed. Check for jobs on the main jenkins dashboard, cancel any long-running jobs there. Bonus points: make a note of the patches for which you have canceled jobs on the zuul dashboard, comment "recheck" for any patches in the test queue that you have aborted.
- Head to https://integration.wikimedia.org/ci/safeRestart
- Login with your labs account being part of the 'wmf' LDAP group
- press "Yes"
- in #wikimedia-operations connect: "!log restarting stuck Jenkins".
Shell
On contint1001.wikimedia.org either the legacy:
sudo /etc/init.d/jenkins restart
Or the new:
sudo service jenkins restart
OOM Issues
Troubleshooting
Whenever Jenkins appears to be stuck or facing high CPU usage, you will want to look at the Java threads: https://integration.wikimedia.org/ci/threadDump
This is the way to do it from the CLI
jstack -l -F <pid of jenkins>
Last time this happened (2017-05-20) a restart of Jenkins "fixed" the problem, but we were unable to troubleshoot without a stacktrace from jstack
Sometimes, changes in other repositories may cause your builds to fail. You can check the Shared Build Failure board to see if any existing issues are similar to your build failure; if there aren’t any, and you’re reasonably certain that your build failures are unrelated to your own changes, you can create a new task.
Debugging
Start Jenkins with Java option:
-Dhudson.plugins.git.GitSCM.verbose="true"
Text thread dump: https://integration.wikimedia.org/ci/monitoring?part=threadsDump