Deprecating a pkgdown site served via GitHub Pages

Patrick Schratz, Kirill Müller


Websites for R packages built with {pkgdown} have become a de-facto standard over the last few years. Many R packages build their site during Continuous Integration (CI) runs, pushing the assets to the special gh-pages branch (even though now any branch can be used to deploy a website).

Photo by Paweł Czerwiński


Sometimes it happens that repositories are transferred to a new user/organization or the package is renamed. While GitHub takes care of redirecting repository URLs, the pkgdown URLs (https://<username>.github.io/<rpackage>) are not redirected. Since some users might have bookmarked certain URLs or the URLs appear in their browsing history, it would be great to have these links not returning a 404 from one day to another.

This blog post proposes several ways to handle this gracefully:

  • Redirection
  • Deprecation with CSS
  • Deprecation via bulk edit

All options hinge on the observation that users and organizations can create a user or organization site that will be the source for https://<username>.github.io/<package> after the renaming. The user site will also serve robots.txt that advises crawlers to avoid deprecated contents.

User or organization site

In GitHub, users can create a user repository <username>/<username>.github.io. This repo will be served automatically as a web page on https://<username>.github.io/ . In this repo, a directory can be created which corresponds to the respective GitHub Pages site of the original repo. Example: The rpackage/ directory in the <username>/<username>.github.io repository corresponds to https://<username>.github.io/rpackage. If both <username>/<rpackage> and <username>/<username>.github.io/<rpackage> exist, the former takes precedence. This means that you can prepare everything in your user repository <username>/<username>.github.io and it will work right away after you rename your package repository. The following has worked for https://krlmlr.github.io/fledge/, which has moved to https://cynkra.github.io/fledge/:

  • Create repository <username>/<username>.github.io
  • In <username>/<username>.github.io create directory <rpackage>
  • Populate the <rpackage> directory using one of the methods described below
  • Push to GitHub
  • Rename repository

All of this works the same way for organizations. The munch package was previously located at https://cynkra.github.io/SwissCommunes/ The original pages, with a warning, are defined at cynkra/cynkra.github.io.

Redirection

Basic idea: set up an HTML redirect from https://<username-old>.github.io/<package> to https://<username-new>.github.io/<package>.

To achieve this, create an index.html in <username>/<username>.github.io/rpackage with the following contents:

<meta http-equiv="refresh" content="0; url=<url to redirect to>" />

However, some redirection practices like this one are considered bad practice (“Use of meta refresh is discouraged by the World Wide Web Consortium (W3C).”)[^1]. Also, users might find it sketchy to see some redirection happening shortly after they visited a site. Last, the redirection shown above only works for the top-level domain. Level 2 or level 3 links like <url>/level1/level2 will not work and return a 404.

Deprecation via CSS

A better way to deprecate a pkgdown/GitHub Pages site is to serve a static version of the last state before the package was moved and add information to the user that the site has moved.

An easy way to achive this is to include a little CSS snippet. The following will add a colored line before the page-header div in the pkgdown site.

.page-header:after {
  content: "You are viewing an outdated page which is not going to be updated anymore. Please go to <https:/new-url.com> for the latest version.";
  font-size: 12px;
  font-style: italic;
  color: #f03333;
}

Deprecation information in the header via CSS


Place this code in the pkgdown/ directory of your package and it will be automatically picked up when the site is built next time:

  • In your package, add the CSS snippet from above to pkgdown/extra.css (CSS name can be different) in the repository/R package which should be deprecated
  • Call pkgdown::build_site() one last time
  • Copy the contents of docs/ to <username>/<username>.github.io/<packagename>

Unfortunately, the :after operator does not allow hyperlinks, so the new URL will not be clickable.

Deprecation via bulk edit

For the URL to be clickable, the HTML files must be edited. The find, xargs and sed utilities help automating this.

pkgdown uses the Bootstrap framework, which has alerts that serve the purpose. They look best just before the closing </header> element. The following command line adds an alert to each HTML page, in this case advertising https://cynkra.github.io/munch as the target URL. It must be run in the rpackage directory of <username>/<username>.github.io:

find -name "*.html" |
  xargs sed -i -r 's#(^.*[<]/header[>])#<div class="alert alert-warning" role="alert"><strong>Warning!</strong> This content has moved to <a href="https://cynkra.github.io/munch">https://cynkra.github.io/munch</a>.</div>\n\1#'

This assumes GNU sed. MacOS users will need to use gsed, or -i.bak instead of -i and deal with the leftover *.bak files.

Deprecation information in the header via editing HTML


Always advertising the new root works well enough, because it is very likely that the structure of the site will eventually change after the repository rename.

Web crawlers

It is a good idea to make the deprecated contents invisible to web crawlers. Add a file robots.txt to the root of <username>/<username>.github.io. The following contents forbids crawling the /SwissCommunes/ directory which contains the old snapshot with pointers to the new location:

User-agent: *
Disallow: /SwissCommunes/




gfortran support for R on macOS

Patrick Schratz


For a long time, gfortran support on macOS could be achieved by installing the homebrew cask gfortran via brew cask install gfortran.

As of 2021, both the brew cask command and the cask gfortran are deprecated. Users who have installed this cask already won’t notice as things will continue to work as normal. Only new users who want to install gfortran this way will get the message that the cask is “not available”. The cask was removed in December 2020 and merged into the gcc formula (which can be installed via brew install gcc). Now, one could go to https://github.com/fxcoudert/gfortran-for-macOS/releases and install the respective .dmg file manually. However, this is not a long-term approach and usually one would like to do this via brew, the most popular package manager for macOS.

Unfortunately, this change did not result in a smooth experience for R users who want to compile packages from source that require a functional gfortran compiler. This requirement does not occur very often as most users install R package binaries on macOS. These don’t require a working gfortran installation.

However, in some cases when calling install.packages(), a working gfortran installation is needed. And if type = "source" is used it needs to be there.

The issue after the integration of gfortran into the gcc formula is that the official R binary installer for macOS expects the gfortran installation at /usr/local/gfortran. This was fulfilled by the old gfortran cask but is not by the new gcc integration. Hence, trying to install the “cluster” package via install.packages("cluster", type = "source") fails and complains about not being able to find gfortran:

* installing *source* package ‘cluster’ ...
** package ‘cluster’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c clara.c -o clara.o
gfortran -mmacosx-version-min=10.13 -fno-optimize-sibling-calls  -fPIC  -Wall -g -O2  -c daisy.f -o daisy.o
make: gfortran: No such file or directory
make: *** [/Library/Frameworks/R.framework/Resources/etc/Makeconf:196: daisy.o] Error 127
ERROR: compilation failed for package ‘cluster’

There was a discussion about these changes in the homebrew PR but the comments mentioning potential upcoming issues were not really heard. Also some workarounds posted in the thread do not work.

So how does one now install gfortran on macOS these days?

It is likely that the workaround presented below will not be needed at some point in the future as it will probably be fixed in the R installer at some point (hopefully). In the meantime, the following helps:

  1. Create a file ~/.R/Makevars (if it does not exist yet)

  2. Add the following to ~/.R/Makevars

    FC      = usr/local/opt/gcc/bin/gfortran
    F77     = /usr/local/opt/gcc/bin/gfortran
    FLIBS   = -L/usr/local/opt/gcc/lib
    
  3. Restart R

  4. Test the changes by calling install.packages("cluster", type = "source")

The output should look like this

* installing *source* package ‘cluster’ ...
** package ‘cluster’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c clara.c -o clara.o
/usr/local/opt/gcc/bin/gfortran -fno-optimize-sibling-calls  -fPIC  -Wall -g -O2  -c daisy.f -o daisy.o
/usr/local/opt/gcc/bin/gfortran -fno-optimize-sibling-calls  -fPIC  -Wall -g -O2  -c dysta.f -o dysta.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c fanny.c -o fanny.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c init.c -o init.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c mona.c -o mona.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c pam.c -o pam.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c sildist.c -o sildist.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c spannel.c -o spannel.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c twins.c -o twins.o
clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o cluster.so clara.o daisy.o dysta.o fanny.o init.o mona.o pam.o sildist.o spannel.o twins.o -L/usr/local/opt/gcc/lib -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: object file (daisy.o) was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: object file (dysta.o) was built for newer macOS version (11.2) than being linked (11.0)
installing to /Users/pjs/Library/R/4.0/library/00LOCK-cluster/00new/cluster/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (cluster)

Caution: After using this approach for some days, I have seen issues with certain packages (e.g. hsdar). It is unclear to me if the issues trace back to the packages or the new way of using gfortran. You might want to re-think using the approach proposed here and eventually install the linked standalone gfortran binary shown earlier in this post binary manually.

Notes

  • I am not sure about the ld: warning: object file (dysta.o) was built for newer macOS version (11.2) than being linked (11.0) warning but it does not seem to have a practical impact.
  • This approach was tested with R 4.0.4, macOS 11.2.3 in March 2021
  • If you still have the old gfortran cask installed, you may want to switch to the new approach as the cask is no longer being updated. Hence you will run a very outdated gfortran as some point without noticing. You can remove the old cask with brew remove – cask gfortran.




Seasonal Adjustment of Multiple Series

Christoph Sax


seasonal is an easy-to-use and full-featured R-interface to X-13ARIMA-SEATS, the seasonal adjustment software developed by the United States Census Bureau. The latest CRAN version of seasonal makes it much easier to adjust multiple time series.

Photo by Meriç Dağlı


seasonal depends on the x13binary package to access pre-built binaries of X-13ARIMA-SEATS on all platforms and does not require any manual installation. To install both packages:

install.packages("seasonal")

seas is the core function of the seasonal package. By default, seas calls the automatic procedures of X-13ARIMA-SEATS to perform a seasonal adjustment that works well in most circumstances:

seas(AirPassengers)

For a more detailed introduction, read our article in the Journal of Statistical Software.

Multiple Series Adjusmtent

In the latest CRAN version 1.8, it is now possible to seasonally adjust multiple series in a single call to seas(). This is done by using the built-in batch mode of X-13. It removes the need for loops or lapply() in such cases, and finally brings one missing feature of X-13 to seasonal – the composite spec.

Multiple adjustments can be performed by supplying multiple time series as an "mts" object:

library(seasonal)
m <- seas(cbind(fdeaths, mdeaths), x11 = "")
final(m)

This will perform two seasonal adjustments, one for fdeaths and one for mdeaths. X-13 spec-argument combinations can be applied in the usual way, such as x11 = "". Note that if entered that way, they will apply to both series. The vignette on multiple adjustments describes how to specify options for individual series.

Backend

X-13 ships with a batch mode that allows multiple adjustments in a single call to X-13. This is now the default in seasonal (multimode = "x13"). Alternatively, X-13 can be called for each series (multimode = "R"). The results should be usually the same, but switching to multimode = "R" may be useful for debugging:

seas(cbind(fdeaths, mdeaths), multimode = "x13")
seas(cbind(fdeaths, mdeaths), multimode = "R")

In general, multimode = "x13" is faster. The following comparison on a MacBook Pro shows a modest speed gain, but bigger differences have been observed on other systems:

many <- rep(list(fdeaths), 100)
system.time(seas(many, multimode = "x13"))
#   user  system elapsed
#  9.415   0.653  10.079
system.time(seas(many, multimode = "R"))
#   user  system elapsed
# 11.130   1.039  12.324

composite spec

Support for the X-13 batch mode makes it finally possible to use the composite spec – the one feature of X-13 that was missing in seasonal. Sometimes, one has to decide whether seasonal adjustment should be performed on a granular level or on an aggregated level. The composite spec helps you to analyze the problem and to compare the direct and the indirect adjustment.

The composite argument is a list with an X-13 specification that is applied on the aggregated series. Specification works identical as for other series in seas(), including the application of the defaults. If you provide an empty list, the usual defaults of seas() are used. A minimal composite call looks like this:

seas(
  cbind(mdeaths, fdeaths),
  composite = list(),
  series.comptype = "add"
)

You can verify that the composite refers to the total of mdeaths and fdeaths by running:

seas(ldeaths)

where ldeaths is the sum of mdeaths and fdeaths.

Acknowledgement

Many thanks to Severin Thöni and Matthias Bannert, for demonstrating the benefits of the X-13 batch mode. Also to the ETH KOF, for partially funding this development.









More posts

Deprecating a pkgdown site served via GitHub Pages

Patrick Schratz, Kirill Müller

gfortran support for R on macOS

Patrick Schratz

Seasonal Adjustment of Multiple Series

Christoph Sax

Dynamic build matrix in GitHub Actions

Kirill Müller

Setting up a load-balanced Jitsi Meet instance

Patrick Schratz

DevOps Expert (f/m/d, 60-100%)

cynkra team

Maintaining multiple identities with Git

Kirill Müller

Relational data models in R

Angel D'az, Kirill Müller

tempdisagg: converting quarterly time series to daily

Christoph Sax

tsbox 0.2: supporting additional time series classes

Christoph Sax

DevOps System Engineer (40-60%)

cynkra team

Introducing dm: easy juggling of tables and relations

Balthasar Sager

tsbox 0.1: class-agnostic time series

Christoph Sax

Data Scientist/Engineer (40-100%)

cynkra team

Time series of the world, unite!

Christoph Sax

Done “Establishing DBI”!?

Kirill Müller


Other blogs

R-bloggers