The Devil is in the Data on Lucid Manager

Visualise and Analyse Bathymetric Survey Data with R

Peter Prevos — Sun, 14 Dec 2025 00:00:00 +0000

The first step in the value chain for water utilities is extracting water from natural sources, such as rivers, groundwater, oceans, or reservoirs. Water managers and the public alike are interested in knowing how much water a reservoir currently holds. This is not an easy problem to solve, as it requires some analysis to move from survey to volume measurement. This article shows how to visualise and analyse bathymetric survey data using the R language and data from the Prettyboy Reservoir in Maryland, USA.

The Prettyboy reservoir

While searching the web for a publicly available bathymetric survey, I stumbled across the Maryland Geological Survey (MGS) website. This organisation investigates the geology and water resources of the American state of Maryland. The Prettyboy Reservoir helps provide water for nearly two million residents in Baltimore City and parts of five neighbouring Maryland counties.

The MGS publishes bathymetric data for their reservoirs, including the Prettyboy reservoir in the county of Baltimore.

View Larger Map

Pretty Boy reservoir in Maryland.

Extract, Transfer and Load

The Maryland Geological Survey website hosts the raw data as a zipped CSV file.

The file has a double header, the first row contains the field names and the second row the units. The data has three columns:

Easting	Northing	Depth
UTM-NAD83-Meters	UTM-NAD83-Meters	Feet
352606.8	4386507	9.7
352606.8	4386507	8.8
352606.5	4386507	11.4

Prettyboy reservoir bathymetry data structure.

The Easting and Northing are in the Universal Transverse Mercator projection. As a demonstration of unit confusion, the reservoir's depth in feet. As usual in bathymetry surveys, depth is measured from the datum as a positive number. The zero datum is the reservoir's Full Supply Level.

The script below extracts, transforms and loads (ETL) the data. The file contains duplicates and a zero point at each of the four corners, which are removed from the data. This leaves us a clean file to work from.

  # Bathymetry visualisation and analysis

  # libraries
  library(tidyverse)
  library(RColorBrewer)

  rm(list = ls())

  # Data source
  # http://www.mgs.md.gov/publications/data_pages/reservoir_bathymetry.html

  if (!file.exists("data/PrettyBoy1998.dat")) {
    download.file("http://www.mgs.md.gov/output/reservoir_bathy/PrettyBoy1998.zip",
                  destfile = "data/prettyboy-1998.zip")
    unzip("data/prettyboy-1998.zip", exdir = "data")
    file.remove("data/prettyboy-1998.zip")
  }

  # Read data
  rawdata <- read.csv("data/PrettyBoy1998.dat", skip = 1)
  fsl <- 145 # Full Supply Level

  # Data cleaning
  prettyboy <- rawdata |>
    rename(easting = 1, northing = 2, depth_ft = 3) |>
    filter(!duplicated(rawdata) & !(easting %in% range(easting))) |> 
    mutate(depth_m = depth_ft * 0.304)

Visualise the Data

This visualisation shows the lines that the survey vessel sailed to capture the sonar soundings. The edge of the reservoir was measured with a terrestrial survey and has a depth of zero.

The code defines a colour palette for this data using green and blue. The colours combine two palettes from the green and blue palettes so that only zero values are green and positive values are blue.

  # Basic Visualisation
  bathymetry_colours <- c(rev(brewer.pal(3, "Greens"))[-2:-3], 
                          brewer.pal(9, "Blues")[-1:-3])

  ggplot(prettyboy) + 
    aes(easting, northing, colour = depth_m) + 
    geom_point(size = .1) + 
    scale_x_continuous(labels = scales::label_comma()) + 
    scale_y_continuous(labels = scales::label_comma()) + 
    coord_equal() + labs(colour = "Depth [m]") + 
    scale_colour_gradientn(colors = bathymetry_colours) +
    labs(title = "Prettyboy Reservoir Bathymetry (1988)",
         caption = "Source: Maryland Geological Survey",
         x = "Easting (UTM NAD83)", y = "Northing (UTM NAD83)") + 
    theme_minimal(base_size = 8)

Prettyboy Reservoir bathymetry.

Mapping the data

The coordinates are provided in the North American UTM / NAD83 projection. To map this survey data onto a map tile, we need to convert these to longitude and latitudes for the WGS84 spheroid.

The SF (Simple Features) package converts the coordinates for use in Google maps. To achieve this we need the CRS (Coordinate Reference System) with number 26918, as Maryland is in zone 18N: EPSG:26918 NAD83 / UTM zone 18N. The second CRS (4326) is the system used by Google maps: EPSG:4326 WGS 84.

We can now plot the data on a Google Maps tile using the ggmap package. This functionality requires a Google API key, which I am not sharing here. You can also use this data in other spatial systems, such as the interactive tooling in the Leaflet package.

  # Spatial mapping
  library(sf)

  # Convert prettyboy tibble to an sf object
  prettyboy_sf <- st_as_sf(prettyboy,
                           coords = c("easting", "northing"),
                           crs = 26918)   # NAD83 / UTM zone 18N

  # Transform to latitude/longitude (WGS84)
  prettyboy_wgs <- st_transform(prettyboy_sf, crs = 4326)

  # Convert back to tibble with lat/lon columns
  coords <- st_coordinates(prettyboy_wgs)

  prettyboy_wgs <- cbind(prettyboy_wgs, coords) |>
    rename(lon = X, lat = Y) |>
    st_drop_geometry()

  library(ggmap)
  ## Register Google Maps API
  api <- readLines("data/google-maps.api")
  register_google(key = api)
  prettyboy_map <- get_map(c(mean(prettyboy_wgs$lon), 
                             mean(prettyboy_wgs$lat)),
                           zoom = 14, color = "bw")

  ggmap(prettyboy_map, extent = "device") + 
    geom_point(data = as_tibble(prettyboy_wgs), 
               aes(lon, lat, col = depth_m), size = 0.1) +
    scale_colour_gradientn(colors = bathymetry_colours, name = "Depth [m]") + 
    labs(title = "Prettyboy Reservoir Bathymetry (1988)",
         caption = "Source: Maryland Geological Survey") + 
    theme_void(base_size = 10)

Prettyboy bathymetry on Google map tile.

Analyse Bathymetric Survey: Full Supply Volume

Two methods are available to estimate the total volume of a reservoir using bathymetric data. The analogue method uses contour lines to draw cross-sections on graph paper at regular intervals. Then use a planimeter or the shoelace formula to measure the surface area of each section and calculate the total volume by multiplying the surface area by the spacing between sections. I used this method more than thirty years ago when working on dredging projects. But technology has moved on.

The second method is more numeric, and there are two methods: interpolate a grid or build a Triangular Irregular Network.

The bathymetry provides data only for the lines on the map, as visualised above. To estimate the full supply analytically, we can overlay the reservoir with a grid and interpolate the depth for each cell. We can then estimate the Full Supply Volume by summing the product of each cell area and depth for each grid item. When we have a grid with n elements, and x and y indicate the element size and d_i the depth of an element, the volume is calculated with:

$$V = \sum_{i=1}^{n} {x \times y \times d_i}$$

The akima package provides functionality for interpolation of irregularly and regularly spaced data, making it ideal for our problem. First, we set the spacing to 10 meters. The smaller the interval, the more accurate the estimation. But small spacing comes at a cost, as the interpolation will take some time to finalise.

Once we have a grid with a depth for each location, we can estimate the Full Supply Volume by multiplying each location's depth by its grid area and summing across all grids.

This analysis provides a volume of 71.50 × 10⁶ m³, which is close to the 19 billion US gallons (71.92 × 10⁶ m³) reported by the Maryland Geological Survey.

  # Extrapolate gridded version
  library(akima)

  # Grid size [m]
  dx <- 10
  dy <- 10

  # Regular grid coordinates
  x_seq <- seq(min(prettyboy$easting), max(prettyboy$easting), by = dx)
  y_seq <- seq(min(prettyboy$northing), max(prettyboy$northing), by = dy)

  # Interpolate scattered (x, y, depth) -> regular grid
  # no extrapolation outside convex hull
  interp_res <- with(prettyboy,
                     interp(x = easting,
                            y = northing,
                            z = depth_m,
                            xo = x_seq,
                            yo = y_seq,
                            duplicate = "strip",
                            linear = TRUE,
                            extrap = FALSE))

  # Volume
  prettyboy_grid <- interp2xyz(interp_res, data.frame = TRUE) |>
    as_tibble() |>
    rename(easting = x, northing = y, depth_m = z)

  grid_volume_m3 <- sum(prettyboy_grid$depth_m * dx * dy, na.rm = TRUE)
  grid_volume_m3 / 100^3

We can use the interpolated values to create a nice map of the reservoir with contour lines and cross-sections. The cross sections are taken at four random points in the grid.

  # Visualise Cross sections
set.seed(1234)
northing_sections <- sample(prettyboy_grid$northing, 4)
prettyboy_sections <- filter(prettyboy_grid, northing %in% northing_sections)
max_depth <- round(max(prettyboy_sections$depth_m, na.rm = TRUE))

map <- ggplot(filter(prettyboy_grid)) + 
  aes(easting, northing) +
  geom_raster(aes(fill = depth_m), interpolate = TRUE) +
  geom_contour(aes(z = depth_m), colour = "white", alpha = 0.4) +
  geom_hline(yintercept = northing_sections, col = "orange") + 
  coord_equal() +
  scale_x_continuous(labels = scales::label_comma()) + 
  scale_y_continuous(labels = scales::label_comma()) + 
  scale_fill_gradientn(
    colours = c(NA, bathymetry_colours[-1]),
    na.value = "transparent",
    guide = guide_colourbar(reverse = TRUE),
    name = "Depth [m]") +
  theme_minimal(base_size = 28) +
  labs(title = "Prettyboy Reservoir",
       caption = "Source: Maryland Geological Survey",
       x = "Easting [m]", y = "Northing [m]") + 
  theme(legend.position = "bottom", 
        legend.key.width = unit(2, 'cm'))

sections <- ggplot(prettyboy_sections) + 
    aes(easting, (fsl - depth_m)) + 
    geom_area(fill = bathymetry_colours[1], alpha = 0.5) +  
    scale_x_continuous(labels = scales::label_comma()) + 
    facet_wrap(~rev(northing), ncol = 1) +
    coord_cartesian(ylim = c(fsl - 1.1 * max(prettyboy_sections$depth_m, na.rm = TRUE), fsl)) +
    theme_minimal(base_size = 28) +
    labs(x = "Easting [m]", y = "Elevation [m]")

library(gridExtra)
png(filename = "prettyboy-sections.png", width = 1920, height = 940, units = "px")
grid.arrange(map, sections, ncol = 2)
dev.off()

Prettyboy Reservoir cross sections.

The second method for calculating the total volume is to use a Triangulated Irregular Network (TIN). A TIN is a representation of a surface consisting of triangular facets. The main difference with the interpolated values is that this model is irregular, so each triangle has a different projected area.

The geometry package implements Delaunay triangulation, where each row contains the indices of three points forming a triangle. In the next step, we use the map_dfr() function to create a data frame with the index number of each triangle, the three coordinates and the associated depths.

To measure the volume, we can use the shoelace formula to calculate the area of each triangle:

$$A_i = \frac{1}{2}| x_{i_1} (y_{1_2} - y_{i_3}) + x_{i_2} (y_{1_3} - y_{i_1}) + x_{i_3} (y_{1_1} - y_{i_2})|$$

The volume of the element is the area multiplied by the average depth:

$$V = \sum{A_i \frac{\sum d_i}{3}}$$

This method provides the same volume but is a bit faster to process because we are not creating new data.

  # Triangulated Irregular Network
  library(geometry) 
  library(purrr)

  # 2D coordinates
  coords2d <- as.matrix(prettyboy[, c("easting", "northing")])

  # Delaunay triangulation
  tri <- delaunayn(coords2d)

  # Convert the triangles into a long data frame for ggplot
  tin_df <- map_dfr(seq_len(nrow(tri)), function(i) {
    idx <- tri[i, ]
    data.frame(
      tri_id   = i,
      easting  = prettyboy$easting[idx],
      northing = prettyboy$northing[idx],
      depth_m  = prettyboy$depth_m[idx])})

  triangle_area_shoelace <- function(x, y) {
    abs(x[1] * (y[2] - y[3]) +
        x[2] * (y[3] - y[1]) +
        x[3] * (y[1] - y[2])) / 2}

  tin_volume <- tin_df |>
    dplyr::group_by(tri_id) |>
    dplyr::summarise(
             area       = triangle_area_shoelace(easting, northing),
             mean_depth = mean(depth_m),
             volume     = area * mean_depth,
             .groups    = "drop")

  tin_volume_m3 <- sum(tin_volume$volume)
  tin_volume_m3 / 1e6

Generate Capacity Charts

Last but not least, we need a chart that converts the measured reservoir level to a volume. The easiest method to develop this curve is to use the grid. The TIN becomes mathematically problematic when not all three triangle points are underwater.

While bathymetry measures depth from the top down, with higher numbers indicating greater depth, reservoir levels are expressed in elevation. I could not find the official elevation of the Full Supply Level. Google Maps provides an elevation of 145 meters for the dam, so let's use this.

The measured values closely matched a cubic regression, so we can calculate the volume of the reservoir using the level with:

$$V(\ell) =-3.463 \times 10^{9} + 8.883 \times 10^{7}\,\ell - 7.640 \times 10^{5}\,\ell^{2} + 2.203 \times 10^{3}\,\ell^{3}$$

  # Capacity Curve
  # levels to evaluate
  levels <- seq(fsl + 15 - round(max(prettyboy$depth_m) / 10) * 10, fsl, by = 1)

  grid_capacity <- tibble(level = levels) |>
    mutate(volume_m3 = map_dbl(level, \(L) {
      bed <- fsl - prettyboy_grid$depth_m
      depth_L <- pmax(0, L - bed)
      sum(depth_L * dx * dy, na.rm = TRUE)}))

  # Cubic regression
  cubic <- lm(volume_m3 ~ poly(level, 3, raw = TRUE), data = grid_capacity)

  grid_capacity$cubic = predict(cubic)
    
  ggplot(grid_capacity) + 
    aes(level, volume_m3 / 1E3) + 
    geom_point(col = "grey") + 
    geom_line(aes(level, cubic / 1E3)) + 
    geom_hline(yintercept = grid_volume_m3 / 1E3, linetype = 6, col = "red") + 
    scale_y_continuous(labels = scales::label_comma()) + 
    theme_minimal() + 
    labs(title = "Prettyboy Reservoir Capacity Curve",
         x = "Water level", y = expression(Volume~km^3))

Prettyboy Capacity Curve.

A Brief Guide to Providing Insights as a Service

Peter Prevos — Sat, 21 Jun 2025 00:00:00 +0000

Analysing data requires a high level of technical skill, which Drew Conway summarised in his often-cited Venn diagram. Following this model, data scientists, individually or as a team, require competencies in computer science, statistics, and knowledge of the domain they are analysing. There is, however, more to providing value from data than these three technical skills. This article introduces the concept of Insights as a Service (IaaS), a framework to assist data science teams with maximising the value they deliver to their internal or external customers.

With the ongoing improvement of AI capabilities, the technical skills of data professionals are becoming increasingly automated. This development requires data engineers, analysts, and scientists to embrace the elusive softer skills and use insights from marketing and the humanities. The insights as a Service Framework enables data professionals to plan and execute their projects, creating greater value for their clients.

This model incorporates lessons I learned while managing a data team for over ten years, as well as my research into applying Service-Dominant Logic.

The content of this article was recently discussed in the Value Driven Data Science podcast by Genevieve Hayes.

Introducing the Insights as a Service Framework

Technology managers regularly apply the acronym "PPT" (People, Process, and Technology) as a mnemonic. The golden triangle of technology management dates back to the early 1960s when Harold Leavitt developed his Diamond Model. Bruce Schneier modified and popularised the tool in the 1990s.

From a service delivery perspective, we need to add some more dimensions. The Services Marketing Mix, first developed by Bernard Booms and Mary Bitner in 1981, employs seven dimensions as a model for delivering services. A marketing mix is not a model that guarantees success. You need to use it as a heuristic device to think more deeply about the service your team provides.

These two models conceptually overlap, which leads to the IaaS framework. Insights as a Service recognises the importance of the core elements of PPT and adds elements from the services marketing mix. This combination leads to six dimensions: Product, Price, Promotion, People, Process, and Platform (technology is renamed for a lovely alliteration of six Ps).

Six Ps of delivering Insights as a Service.

Insights as a Service (IaaS) is in most cases used exclusively in a B2B (Business-to-Business) context. In this scenario a commercial service provider generates insights for a client using a cloud-based platform. This article reinterprets the IaaS model to include contexts where an internal data team provides electronic services to colleges (B2E, Business-to-Employee).

The Insights as a Service Framework

The remainder of this article briefly discusses the significance of each of these framework dimensions and their practical relevance.

Product

The data product, which is, in most cases, a report, either static or interactive, is the vehicle that delivers value to the end-user. We can use an analogy from architecture by following the Vitruvian rules for buildings and translating them to our context:

Sound: Is the analysis mathematically and statistically correct?
Useful: Can the client generate insights from it?
Aesthetic: Is the data product well-designed and, as such, easy to understand?

This previous article about strategic data science discusses this model of good data science in more depth, which is also explained in detail in my book on the topic.

Principles of Strategic Data Science

Principles of Strategic Data Science helps you join the dots between mathematics, programming, and business analysis. With a unique approach that bridges the gap between mathematics and computer science, this book takes you through the entire data science pipeline.

Packt Publishing

The value of a service product, in simplest terms, is the difference between the benefits and the cost. The product provides the benefits of the value equation, visualised below. While the product delivers value to the customer, there is also a cost, which is the next dimension in the framework.

The value equation, balancing benefits and cost.

Price

Whether you are an internal data science team or a commercial service provider that assists other organisations with creating value from data, all development has a price. However, the monetary cost of a data product is only one dimension of the total Price equation.

Firstly, consumers of services incur a time cost. The time cost of a service is the amount of time a consumer (in our case, the user of a data product) spends using the tool. The time we spend using a service is an opportunity cost because we could be doing something else.

The time cost also extends to the development phase as the input from end users is essential. A data professional relies on the detailed, contextual knowledge of the end-user to create insights.

An extension of the time cost to use a service is the mental cost. How much mental effort does an end-user need to use to understand the output of your data product? The cognitive cost of consuming insights can be minimised in two ways. Firstly, the design of the report needs to be aesthetic. Not in the sense of beatification, but in making the end product easy to understand.

The mental cost to users of a report depends a lot on their data literacy. The advanced specialisation of data science has created a knowledge gap between subject-matter experts who use the report and the advanced statistical methods used by analysts. Reducing this knowledge gap requires the service provider to educate their customers, which falls under the next dimension of the IaaS framework.

Promotion

When working in a B2B environment, the necessity for promotion is easily understood. However, promotion is also essential for internal teams working in B2E.

A data science service provider is often invisible to the consumer of its services. Being invisible is a good thing because it means our data products are working flawlessly, but this great uptime also comes at a cost, as your services might be taken for granted. The most effective way to combat invisibility is promotion.

For teams providing services to external entities (B2B), promotion is an essential business activity. However, this also applies to internal data teams. In my workplace, we ensure that our colleagues have regular visibility of the tools we develop through intranet articles and other internal communication means. These promotional activities ensure that management understands the relevance of the team and improves data literacy across the organisation.

To promote data science skills in my industry, I wrote a textbook for water professionals who want to move away from their spreadsheets, the content of which I also deliver as a course. One objective of this course is to equip water professionals with the skills to communicate effectively with data scientists. Increasing data literacy among those who work with data scientists is essential to ensure the full value of our services is realised.

Data Science for Water Utilities

Data Science for Water Utilities published by CRC Press is an applied, practical guide that shows water professionals how to use data science to solve urban water management problems using the R language for statistical computing.

Routledge Amazon

Another important aspect of providing valuable data services is that your client knows the team. Don't hide your experts behind the veil of technology. This thought segways nicely onto the next dimension of the framework.

People

Enough has been said and written about the technical requirements of being a data scientist. The Conway Venn diagram is a widely cited model of the technical skills required for data science. In this model, a data scientist is somebody who understands the subject matter under consideration in mathematical terms and writes computer code to solve problems.

The Data Science Venn Diagram.

Many alternatives have been proposed that incorporate soft skills into this diagram, and the Insights as a Service Framework also does so. Especially with the steady advancement of AI developments, non-technical skills will become increasingly crucial for a data scientist to thrive in this industry.

The importance of domain knowledge cannot be underestimated. The time price of a service suggests that the client needs to invest time in explaining the context of the data problem to the developer. However, the development team also needs to have sufficient domain knowledge to advise the client. I have been in many situations where an end-user knows what problem to solve but is unable to articulate what they want clearly. It is then the task of the service provider to understand their customer and provide them with what they need.

The well-worn "faster horses" marketing cliché also applies to providing Insights as a service. A data service provider needs to be able to provide the client with a vision (in this analogy, a car) that solves or prevents problems.

Process

The process of service delivery relates to your pipeline from ideation to production. Each organisation will have a governance model to decide which projects are executed and how they are implemented.

From a service perspective, your process needs to be seamless and easy to understand while also minimising the time the client has to invest in development and testing. Service Blueprinting is a commonly used technique to map a service. The primary difference between this approach and standard process maps is that a service blueprint also considers the customer's perspective. A good service blueprint adds equal weight to the inputs of both the service provider and the service beneficiary.

Another important consideration in the service provision process is the method for measuring service quality. Service Quality goes beyond customer satisfaction and seeks to provide deep insight into which aspects of a service need improving. At a basic level, it consists of four dimensions:

Reliability: The ability to provide the service(s) accurately and dependably.
Assurance: The ability to project trust and confidence.
Tangibles: The physical or digital proof of the service provider's status.
Empathy: The ability to provide empathetic attention to customers.
Responsiveness: The ability to provide timely services.

Platform

Last but not least is the technology platform that you use for developing the data product. Data scientists use a complex web of applications to transform raw data into the final product. The data consumption platform is the interface between this technology stack and the end-user.

When choosing a data consumption platform, two factors are essential to consider: complexity and degrees of freedom.

Firstly, complexity relates to the level of coding required to create data products, ranging from no-code to high-code. No-code means that everything can be done with a mouse and a few keystrokes. High code implies that the developer needs to use scripting languages. For example, the R Shiny platform provides interactive web applications using R and Python. This package is a high-code solution that requires a substantial amount of training to utilise effectively. KNIME is a no-code solution with a node-based graphical interface to build analytical

The less coding is involved, the more democratised access to the tool is, as more people can use it. A no or low-code platform can provide a self-service interface for subject-matter experts to analyse data without needing data science skills. Low-code platforms, such as spreadsheets also have disadvantages, most importantly that they are often not reproducible and hard to reverse-engineer when updates are required.

The second dimension is the degree of freedom available to the end user. Tools such as Power BI offer a high degree of freedom, whereas a report in MS Word provides little to no freedom for the user. While Power BI or Tableau provide a 'create-your-own-adventure' interface, other systems are more limited and provide

The choice of platform should align with the end-user's needs. A no-code solution with a high level of freedom is not always best.

Conclusion

This article summarises the Insights-as-a-Service Framework. I am currently working on a book on the topic that will dive much deeper into the model, including practical examples.

Follow or connect with me on LinkedIn if you'd like to stay informed about my progress and learn how this concept develops.

Project Euler 35: Circular Primes below One Million

Peter Prevos — Thu, 12 Jun 2025 00:00:00 +0000

Euler Problem 35 takes us back to the world of prime numbers, in particular circular primes. The Online Encyclopedia of Integers (A068652) lists the first 46 decimal numbers for which every cyclic permutation is a prime.

Project Euler 35 Definition

The number 197 is a decimal circular prime because all rotations of the digits: 197, 971, and 719, are themselves prime.

There are thirteen such primes below 100: 2, 3, 5, 7, 11, 13, 17, 31, 37, 71, 73, 79, and 97.

How many circular primes are there below one million?

Proposed Solution in R

This code reuses the function that strips all digits for a number, presented in Euler Problem 30. The solution for this problem also use the is_prime function to test whether a number is a prime, developed for Euler Problem 7.

The following auxiliary functions help to solve the problem:

prime_sieve: Generate primes
is_prime: Checks whether a number is prime
rotate_left: Rotates the a number

The is_circular_prime function first removes any numbers greater than 9 that contains either a 0, 2, 4, 5, 6 or 8, as at least one rotation will be divisible by either 2 or 5, which drastically reduces the search space.

The main program tests each of the 78,498 primes below one million and counts the number of circular ones.

  prime_sieve <- function(x) {
    if (x == 1) return(NULL)
    if (x == 2) return(n)
    l <- 2:x
    i <- 1
    p <- 2
    while (p^2 <= x) {
      l <- l[l == p | l %% p!= 0]
      i <- i + 1
      p <- l[i]
    }
    return(l)
  }

  is_prime <- function(x) {
    if (x <= 1) return(FALSE)
    if (x == 2 | x == 3) return(TRUE)
    primes <- prime_sieve(ceiling(sqrt(x)))
    return(all(x %% primes != 0))
  }

  x <- 101

  rotate_left <- function(x) {
    if (x < 10) return(x)
    n <- floor(log10(x)) + 1
    first_digit <- x %/% 10^(n - 1)
    rest <- x %% 10^(n - 1)
    rotated <- rest * 10 + first_digit
    return(rotated)
  }

  is_circular_prime <- function(x) {
    if (x > 9 & any(grepl("[024568]", as.character(x)))) return(FALSE)
    n <- floor(log10(x)) + 1
    p <- vector(length = n)
    for (r in 1:n) {
      p[r] <- is_prime(x)
      if (!p[r] ) return(FALSE)
      x <- rotate_left(x)
    }
    return(TRUE)
  }

  primes <- prime_sieve(1E6)

  system.time({
    circular_primes <- primes[sapply(primes, is_circular_prime)]
    print(length(circular_primes))
  })

Mapping the Ancient World: A Digital Odyssey through Ptolemy's Geography

Peter Prevos — Fri, 17 Nov 2023 00:00:00 +0000

In the chronicles of cartographic history, few works have captivated the imagination, such as Claudius Ptolemy's "Geography". This monumental second-century creation is the oldest fully preserved theoretical treatise on cartography, written almost 2000 years ago. This article discusses Mapping Ptolemy's Geography with ggplot2.

His cartography is not just a collection of maps of the known world in Hellenic times. Ptolemy also provides latitudes and longitudes with fantastic accuracy. Interestingly, he wrote that the best way to store geographic information is as a collection of numbers because copying maps leads to distortions over time. Thus, he created one of the world's first geographic information systems. His book contains the locations of thousands of towns, rivers, mountains and other points of interest.

Unfortunately, no freely available database of these coordinates exists in electronic format, but there are some textual versions, such as the example below.

Extract from Ptolemy's Geography (Nobbe edition, 1843).

I thus set forth to scrape these sources and create a Ptolemaic geographic database. The code and data for this project are available on GitHub.

Obtaining the Data

The process begins with the scraping of Ptolemy’s text from ToposText, which is a treasure trove of classical literature. This website contains an almost complete transcription of Ptolemy's tome. I saved the webpage to disk to prevent hitting the ToposText server too often. The source data is available on the Github repo in RDS format.

The website provides an accurate transcription of a large portion of Ptolemy's work. Some typographical quirks and curious coordinates were standardised using regular expressions.

  library(stringr)
  library(dplyr)
  library(ggplot2)

  # library(rvest)
  # Scrape the website and save it to disk
  #
  # url <- "https://topostext.org/work/209"
  #
  # raw_html <- read_html(url) %>%
  #     html_nodes("p") %>%
  #     html_text()
  # saveRDS(raw_html, "topostext.org-work-209.rds")

  raw <- readRDS("topostext.org-work-209.rds")

  # Fix issues and typos
  raw <- str_replace_all(raw, "\\. equator", "0°0") # Equator
  raw <- str_replace_all(raw, "0°0 and", "0°0'")
  raw <- str_replace_all(raw, "(\\d+.\\d+') S ", "-\\1") # Southern hemisphere
  raw <- str_replace_all(raw, "(\\d) c", "\\1'")
  raw <- str_replace_all(raw, "(\\d) d", "\\1' \\.")
  raw <- str_replace_all(raw, "(\\d) e", "\\1' \\.")
  raw <- str_replace_all(raw, "5°00' . 3°00", "45°15") # Estimated lat
  raw <- str_replace_all(raw, "4°00' . 3°00", "43°00") # Estimated lat
  raw <- str_replace_all(raw, "(\\d+)'(\\d+)", "\\1°\\2")
  raw <- str_replace_all(raw, "(Elateia . 51°38' )", "\\1. 38°15'")
  raw <- str_replace_all(raw, "(Kambyses)'", "\\1")
  raw <- str_replace_all(raw, "(Libison)'", "\\1")

The script then loops through all paragraphs that include coordinates and extracts the place names, longitude and latitude in degrees and minutes.

  # Detect place names and coordinates
  locs <- str_split(raw[str_detect(raw, "°|º")], "'")

  p_regex <- "\\b[A-Za-z ,:\\-]+\\b"
  c_regex <- "(-?[0-9]{1,3}[°|º][0-9]{1,3})"

  place <- vector()
  lon_raw <- vector()
  lat_raw <- vector()
  i <- 1

  for (j in seq_along(locs)) {
    t <- unlist(locs[j])
    n <- ceiling(sum(str_detect(t,  "°|º")) / 2)
    k <- i + (n - 1)
    place[i:k] <- str_extract(t, p_regex)[(1:n) * 2 + - 1]
    lon_txt <- unlist(str_extract_all(t[(1:n) * 2 + - 1], c_regex))
    lat_txt <- unlist(str_extract_all(t[(1:n) * 2], c_regex))
    if (length(lat_txt) == 0) lat_txt <- NA
    if (length(lon_txt) != n) stop(j)
    lon_raw[i:k] <- lon_txt
    lat_raw[i:k] <- lat_txt
    i <- i + n
    j <- j + 1
  }

The next phase creates a table of coordinates by converting the coordinates into numeric values and cleaning the place names. The degrees and minutes of longitude and latitude are stored in separate variables. Regular expressions split the geographic coordinates (e.g. 14°20' . 54°30') into four numbers (14, 20, 54, 30).

  # Generate data frame
  extract_nums <- function(x, regex) {
    as.numeric(unlist(str_extract_all(x, regex)))
  }

  d_regex <- "(-?[0-9]+)(?=°|º)"
  m_regex <- "(?<=°|º)([0-9]+)"

  locations <- tibble(place, lon_raw, lat_raw) %>%
    mutate(lon = extract_nums(lon_raw, d_regex),
           lon_min = extract_nums(lon_raw, m_regex),
           lat = extract_nums(lat_raw, d_regex),
           lat_min = extract_nums(lat_raw, m_regex),
           place = str_squish(place),
           place = str_to_sentence(place)) %>%
    select(placename = place, 4:7)

The table of place names and their coordinates is available on the GitHub repository.

Mapping Ptolemy's Geography

The visualisation was realised through the ggplot2 mapping geometry. Ptolemy's ancient world is reborn; its cities and landmarks emerge from the mists of time on this digital canvas.

This map uses the Conic projection, similar to one of the projections used by Ptolemy himself. This projection distorts the southern part of Africa slightly.

The longitudes are corrected to better match them with modern coordinates. Ptolemy used the Canary Islands as the zero meridian of his coordinate system, which is 18 degrees west of the Greenwich meridian. Research also shows that Ptolemy's longitudes are stretched by a factor of 1.428.

Ptolemy's work might seem full of mistakes from our modern perspective, but in fact, every world map we have ever produced is essentially wrong. Turning a globe into a two-dimensional object always results in distortions, no matter how hard you try.

Mapping Ptolemaic coordinates on a modern map.

# Visualise
world <- map_data("world")

ggplot() +
    geom_map(data = world, map = world,
             aes(long, lat, map_id = region),
            fill = "darkgreen", colour = "lightgrey", linewidth = .1) + 
    geom_point(data = locations,
               aes(x = (lon + lon_min / 60 - 18) / 1.428,
                   y = (lat + lat_min / 60)),
               col = "orange", alpha = 0.7, size = .1) +
    scale_x_continuous(limits = c(-15, 70)) +
    scale_y_continuous(limits = c(-10, 60)) +
    coord_map(projection = "conic", lat0 = 37) +
    labs(title = "Ptolemaeus, Geography (II-VI)",
         subtitle = "Longitude correction: 0.7(lon - 18)",
         caption = "Source: topostext.org/work/209",
         x = NULL, y = NULL) + 
    theme_minimal(base_size = 8) +
    theme(panel.background = element_rect(fill = "lightblue", colour = NA))

Ptolemy knew the Earth was spherical – a fact established by Greek philosophers well before his time. In his Geographic, Ptolemy provides a cornucopia of coordinates, illustrating the known world from Britain to Southeast Asia and from the northern reaches of Scandinavia to below the equator. But their knowledge only covered a small part of the whole globe.

The Greeks blended mythology, philosophy, and early scientific speculation to theorise what lies beyond the world they mapped. They imagined a vast, unending ocean encircling the world. Beyond Oceanus, some thought, might be other worlds or unknown lands. The concept of "antipodes" was also bandied about - these are places where people's feet are directly opposite to ours. Though the existence of such lands or peoples was often debated or outright denied.

Analysing the Customer Experience

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Summarising data through descriptive statistics such as averages, percentiles or variances does not add new knowledge but merely summarises what you already know. This chapter of Data Science for Water Utilities introduces basic inferential statistics, a set of methods for finding patterns in data to create new knowledge. Technical professionals often focus on the tangible aspects of services and measure performance in physical units. Technical data does, however, only tell part of the story of the customer experience. Analysing the customer experience requires different techniques to analyse data compared to physical processes. This chapter discusses how to use surveys to learn how customers experience services and demonstrates some techniques to explore this type of data using the R language. The learning objectives for this chapter are:

Understand the principles of measuring the customer experience with surveys
Assess the reliability of surveys with a correlation matrix
Assess the validity of surveys using factor analysis

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Measuring Mental States

Suppose we like to know how somebody feels or what they think about something, such as the quality of service their water utility provides. In that case, we need to measure their mental state. There is currently no direct method to measure mental states, so we use indirect methods such as customer surveys. The basic idea is the way somebody completes a survey is causally related to their state of mind.

This chapter analysis a scale for consumer involvement where customers answered the following question bank of ten items:

Personal Involvement Inventory semantic-differential scale (Zaichowsky 1994).

Reliability and Validity

The basic principle of measuring a mental state using a survey is that we have a group of questions that relate to the same underlying mental construct. We need to evaluate reliability and validity to assess whether the survey items measure the underlying latent variable.

The reliability of a survey relates to the extent to which the answers to the survey items are internally consistent. The internal reliability of a survey can be informally assessed by reviewing the extent to which each of the items in the survey correlates with each other. The psych package provides more advanced methods, such as Cronbach Alpha, to assess reliability.

Correlation matrix for the Personal Involvement Index.

The validity of a survey is broadly the extent to which it measures the underlying mental states. One of the methods to assess validity is factor analysis.

This previous article about using factor analysis to measure consumer involvement summarises how to undertake exploratory factor analysis.

Analysing the Customer Experience Screencast

Chapter eight of Data Science for Water Utilities explains the theory of measuring the customer experience and how to implement it in R in more detail. This screencast runs through the code in the customer experience chapter.

Analysing the Customer Experience screencast.

The data and code used in this chapter are available on GitHub:

Additional Resources

Surveys are a useful tool to analyse what customers think about tap water. Still, it can be better to move outside the formalities of the survey and use the customer's own words. Sentiment analysis using tweets about tap water can provide illuminating insights.

My other book Customer Experience Management for Water Utilities by IWA publishing delves deeper into using marketing theory for urban water supplies.

Customer Experience Management for Water Utilities: Marketing Urban Water Supply

Practical framework for water utilities to become more focused on their customers following Service-Dominant Logic.

IWA Publishing

Other Chapters

Previous Chapter: Managing Dirty Data

Next Chapter: Basic Linear Regression.

Feel free to contact me if you have any comments, suggestions or questions about this book.

Basic Linear Regression

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Regression analysis is one of the most common methods to investigate relationships between variables. Understanding basic linear regression is the first step toward predictive analysis and machine learning. This chapter of Data Science for Water Utilities explores possible linear relationships between the responses in the customer survey and uses these results to explain the theory and practice of building and assessing linear models. The learning objectives for this chapter are:

Understand the principles of linear regression
Perform a linear regression of the customer survey data
Assess the significance of a linear regression

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Principles of Linear Regression

The purpose of a regression model is to predict one variable by measuring one or more other variables through a linear relationship. For example, predict water consumption based on the forecast temperature or investigate how customer complaints relate to the pressure level.

Principles of linear regression.

The task of linear regression analysis is to find the line that minimises the difference between the observed values $(y)$ and the predicted values $(\hat{y})$.

Linear Regressions in R

The lm() function is the regression workhorse that provides detailed output to assess the hypothesised relationship stored in a list.

Plotting the output of the linear model provides a detailed graphical assessment of the model.

The first plot reviews the residuals versus the fitter (predicted) values. Ideally, the red line overlaps perfectly with the x-axis.
The QQ-Residuals Plot tests whether the distribution of the residuals is normal, which is the case when the observations are all on the diagonal line.
The scale location plot tests for the assumption of homoscedasticity, which is the case when the red line is horizontal.
Lastly, the leverage plot tests the data for outliers. The numbers relate to the rows with outliers. Leverage is the extent to which coefficients would change of these values are removed.

Linear regression graphical review.

Basic Linear Regression Screencast

Chapter nine of Data Science for Water Utilities explains the theory of linear regression and how to implement it in R in more detail. This screencast runs through the code in the basic linear regression chapter.

Basic Linear Regression Screencast.

The data and code used in this chapter are available on GitHub:

Additional Resources

Chapter 13 discusses multiple linear regression.

Other Chapters

Previos Chapter: Analysing the Customer Experience

Next Chapter: Clustering customers to define segments

Feel free to contact me if you have any comments, suggestions or questions about this book.

Basics of the R Language

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

The second chapter of Data Science for Water Utilities introduces the basic principles of the R language and RStudio to analyse and visualise data. Writing code is not only about using the correct syntax and the appropriate functions; this chapter also covers some issues regarding styling your code, making it easy to understand. This chapter concludes with a case study to measure water flow in an open channel using the Kindsvater-Carter formula. The learning objectives for this chapter are:

Install R and RStudio and identify the different parts of the RStudio screen.
Understand the principles of writing code to analyse data.
Apply R code to solve a simple water problem.

Data Science for Water Utilities

Data Science for Water Utilities published by CRC Press is an applied, practical guide that shows water professionals how to use data science to solve urban water management problems using the R language for statistical computing.

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Installing R and Rstudio

The best way to use R is through an Integrated Development Environment (IDE). This type of software helps you to write and manage code. An IDE typically comprises a source code editor, automation tools, and functionality to simplify crafting and running code. Of course, you can use R without the IDE, but it will be less user-friendly.

Several IDEs are available to help you write R code. RStudio by Posit is the most popular option.

This software is also an open-source project, with free and paid versions for companies that want to use advanced features and support services. RStudio can also work with other languages, such as SQL and Python. Follow these steps to install the required software:

Go to the R Project website: cran.r-project.org
Download the base version for your operating system and install the software
Go to the download page on the RStudio website: posit.co
Download the installer for the free desktop version and install the software

Alternatively, you can sign up for a free and fully featured account to access RStudio’s cloud version (posit.cloud). This service gives you full access to R and RStudio in your browser without installing software. The free version provides enough hours of computing time to work through this book. You’ll have to pay for a subscription or install the desktop version if you need more time.

The RStudio IDE interface.

Other R IDEs

R Tools for Visual Studio: Microsoft’s development environment.
Rattle is a popular graphical user interface for data mining in the R programming language.
Emacs Speaks Statistics: For advanced computer users.

Principles of Writing Code

Writing code has a steep learning curve and can be both frustrating and rewarding at the same time. Here are some principles that will help you write good code that is easy to understand:

Use descriptive variable names. Don’t, for example, use d, but pipe-diameter or another descriptive name.
Add plenty of whitespace in your code.
Use comments starting with a hashtag: # (comments are not evaluated)
Follow a coding style guide, for example, the Tidyverse Style Guide.

The R Learning Curve can be steep and full of surprises.

R is Meme Proof

The basic arithmetic in the R language follows the BODMAS rules (Brackets, Order, Division, Multiplication, Addition, and Subtraction). R is thus meme-proof and can solve the silly mathematical ‘challenges’ spread on social media.

R is meme-proof and can solve arithmetic challenges.

Channel Flow Case Study

Determining the flow in an open channel is usually achieved by measuring the water depth through a section with a known shape. Forcing the water through a given shape over a sharp weir creates a boundary condition that allows us to calculate the flow. A mathematical relationship determines the volume of water that passes through the channel. This case study uses a rectangular weir. A simplified version of the Kindsvater-Carter formula:

$$Q = \frac{2}{3} C_d \sqrt{2g} \\ bh^{\frac{3}{2}}$$

$Q$: Flow rate ($m^3/s$)
$C_d$ : Discharge coefficient
$g$: Gravitational acceleration (9.81 $m/s^2$ )
$b$: Measured width of the notch [$m$]
$h$: Upstream head above crest level [$m$]

Rectangular weir.

Questions:

What is the flow in $m^3/s$ when $h=100$
What is the average flow when the level above the weir is 150, 136 and 75mm?

Cheat Sheet

To help you remember the various functions discussed in the first five chapters of the book, a cheat sheet is available.

Data Science for Water Utilities Cheat Sheet.

Basics of the R Language Screencast

Chapter two of Data Science for Water Utilities explains the principles of writing code to analyse data in more detail. This screencast runs through the code in this chapter.

Basics of the R Language.

The data and code used in this chapter are available on GitHub:

Additional Resources

To help you remember the various functions discussed in the first five chapters of the book, a cheat sheet is available.

Project Euler provides an excellent way to practice solving basic mathematical methods in the R language.

Other Chapters

Previous Chapter: Introduction to Data Science for Water Utilities

Next Chapter: Loading and Exploring Water Quality Data

Feel free to contact me if you have any comments, suggestions or questions about this book.

Clustering Customers to Define Segments

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

The ideal form of customer service is personal attention, where the needs of each individual are met. Unfortunately, this level of service is more often than not impossible or too costly to achieve. Service providers, therefore, segment their customers into groups with similar characteristics. Cluster analysis to segment customers is a commonly used technique, which analyses and divides an unlabeled dataset into groups of observations with similar properties. This chapter of Data Science for Water Utilities shows how to detect patterns and define segments in customer data. The learning objectives for this chapter are:

Understand the principles of customer segmentation
Apply and interpret hierarchical cluster analysis
Apply and interpret k-means clustering

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Principles of Customer Segmentation

The ideal situation for customer-centric services is that each customer receives individual attention. For large organisations, giving each customer individual attention is very costly and treating everybody the same is not very good either. Customer segmentation helps service providers group customers into segments with similar needs.

Demographic: Age, gender, income, education, ethnicity.
Behavioural: Purchasing habits, spending habits, water consumption.
Psychographic: Interests, lifestyle, motivations, and water-related priorities.
Geographic: Town, postal code, water system.

Hierarchical Cluster Analysis

This example contains data from ten hypothetical customers (A–J). The first data dimension in the test data is the average annual water consumption, and the second is the size of the land on which the house resides. The clusters should be easily visible by viewing the image.

Cluster analysis example.

Hierarchical cluster analysis is a deterministic method to find the relevant clusters. This method reviews all possible combinations of data points and can thus be problematic when analysing large amounts of data. This tree diagram shows how all the points in the chart relate to each other.

Hierarchical clustering example.

k-means clustering

The k-means method uses a stochastic approach, which means that the outcome is not always the same when some clusters are in doubt. But, this method can digest much larger data sets than the hierarchical method. Another difference with the first method is that in k-means, you need to specify the number of clusters before the analysis starts.

The elbow method visualises the Within-Clusters Sum of Squares for the number of clusters. The location where the graph has the smallest angle is most likely the ideal number of clusters.

k-Means cluster analysis elbow method.

Interpreting cluster analysis

Clustering analysis methods are a form of unsupervised machine learning. The computer detects patterns in the data but cannot relate them to meaning. The results of a cluster analysis require human interpretation to link it to the context of the data.

In this simple example, we could name the two clusters of households with and those without a garden.

In reality, cluster analysis occurs with many more variables, as in these simplified examples. The book provides a small case study that also uses categorical data.

Cluster Analysis to Segment Customers Screencast

Chapter ten of Data Science for Water Utilities explains the principles of cluster analysis for customer segmentation in more detail. This screencast demonstrates how to undertake cluster analysis to segment customers using the code explained in the book.

Clustering Customers to Define Segments.

The data and code used in this chapter are available on GitHub:

Additional Resources

My book Customer Experience Management for Water Utilities by IWA publishing delves deeper into using marketing theory for urban water supplies.

Customer Experience Management for Water Utilities: Marketing Urban Water Supply

Practical framework for water utilities to become more focused on their customers following Service-Dominant Logic.

IWA Publishing

Other Chapters

Previous Chapter: Basic Linear Regression

Next Chapter: Working with Dates and Times

Feel free to contact me if you have any comments, suggestions or questions about this book.

Descriptive Statistics in Water Quality

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

The first step in telling data stories is summarising the available data. Descriptive statistics are tools that use a single number to describe a distribution. Analysts calculate averages, medians, percentiles and other statistical summaries to represent a data set's tendency, spread, positions and shape. Statisticians have defined five types of descriptive statistics: frequency, central tendency, position, dispersion and shape. This chapter of Data Science for Water Utilities shows how to calculate descriptive statistics using synthetic water quality data from an imaginary city. The learning objectives for this chapter are:

Summarise water quality data using various descriptive statistics
Evaluate compliance of water quality data with relevant regulations.
Perform grouped data analysis

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Descriptive Statistics

We have already seen measures of frequency in Chapter 3, which count the number of times each observation occurs in a data sample. The remaining four descriptive statistics are:

Central tendency summarises a distribution with a single number
Position describes how an observation relates to the others
Dispersion describes how far the data deviates from the central tendency
Shape summarises the shape of a distribution

Measure of Central Tendency

A central tendency summarises a distribution with a single number, such as the mean, median or mode.

R has the mean() and median() functions, but no basic function to calculate the mode is available but can easily be added. Water quality data tends to be heavily skewed, so care needs to be taken when using the mean to describe a sample of water quality samples.

Measures of central tendency.

Measures of Position

A measure of position is a number’s relative position within the sample. Examples of measures of position are quartiles, deciles, and percentiles, generically known as a quantile. A percentile is a quantile multiplied by 100. Percentiles are a standard method to describe water quality data. For example, if we state that the 95th percentile of turbidity was 4 NTU, 95% of results were lower than or equal to 4 NTU, allowing the occasional spike. An earlier article on the website describes how R undertakes percentile calculations in more detail.

Measures of Dispersion

Dispersion is the extent to which the measurements are spread. Several measures are available, such as the range and Inter-Quartile-Range, variance and standard deviation. The base R package has built-in functions for each of these measures. Please note that R uses Bessel’s Correction to calculate variances to correct the bias in the estimated population variance.

Measures of Shape

Lastly, we also need to know something about the shape of the distribution. The skewness of a distribution measure the eccentricity of the distribution curve. The kurtosis measures the ‘fatness’ of the tails of the distribution. A high skewness and kurtosis are, therefore, a sign of outliers in the data.

Measures of shape (skewness and kurtosis)

The moments and e1071 packages provide functions to calculate skewness and kurtosis. These packages use slightly different approaches, so the results can differ. Refer to the book or the package documentation for details.

Grouping Data

Most data sets contain observations from various groups of data. In this case, we can analyse data by analyte, suburb or sample point. The group_by() function in the dplyr package creates a grouped data frame which means you can calculate descriptive statistics by group. The example below shows a subset of the water quality data, and the two tables on the right are grouped by measure (analyte). The summarise function is the workhorse that lets us analyse the data frame for each group.

Grouped data frames (tibbles) in Tidyverse.

Descriptive Statistics in R Screencast

Chapter four of Data Science for Water Utilities explains descriptive statistics and how to calculate them in R in more detail. This screencast below reviews the code for this chapter.

Descriptive Statistics in Water Quality screencast.

The data and code used in this chapter are available on GitHub:

Additional Resources

To help you remember the various functions discussed in the first five chapters of the book, a cheat sheet is available.

Other Chapters

Previous Chapter: Loading and Exploring Data

Next Chapter: Visualing Water Quality Data with ggplot2.

Feel free to contact me if you have any comments, suggestions or questions about this book.

Detecting Outliers and Anomalies in Time Series Data

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Reporting all available data has little value because it overwhelms the organisation with information that is not actionable. A more productive approach to processing the data deluge is to show those processes where something interesting has occurred. A range of techniques is available to detect anomalies in the data. Reporting outliers and anomalies focuses the organisation's attention by raising questions and motivating action. This chapter of Data Science for Water Utilities discusses finding the most exciting points in your data to create actionable reports. This chapter also shows how to streamline code with functions and develop a leak detection tool to use with digital metering data. The learning objectives for this session are:

Apply statistical methods to detect outliers
Find anomalies in a time series
Develop R functions to streamline your code
Write a function to detect leaks from digital metering data

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Anomaly Detection Methods

Several methods are available to detect anomalies:

Visual inspection of graphs (such as boxplots, see chapter 6)
Static thresholds, such as physical constraints or a regulatory limit
Statistical criteria (such as Z-scores and Median Absolute Deviation)
Domain knowledge

Visually detecting outliers with a boxplot of chlorine data.

Leak Detection

Domain knowledge also helps to detect anomalies. For example, reservoir levels cannot extend much above the Full Supply Level, and a pH probe cannot measure values of zero and so on. Domain knowledge helps to define typical situations.

Detecting leaks downstream of a meter is easy when we can make some assumptions about the consumption pattern. For any residential connection, we can safely assume there is no flow for at least a few hours daily. This assumption can easily be transformed into code.

Visually detecting outliers with a boxplot of chlorine data.

The data for this case study was simulated using R scripts. A previous blog post about analysing digital meter data contains more details.

Detecting Outliers and Anomalies Screencast

Chapter twelve of Data Science for Water Utilities explains the theory and application of anomaly detection in more detail. This screencast below reviews the code for this chapter.

Detecting Outliers and Anomalies Screencast.

The data and code used in this chapter are available on GitHub:

Additional Resources

The video below provides an excellent introduction to the principles of anomaly detection.

Anomaly Detection 101.

Other Chapters

Previous Chapter: Working with Dates and Times

Next Chapter: Introduction to Machine Learning

Feel free to contact me if you have any comments, suggestions or questions about this book.

Introduction to Machine Learning

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Machine learning is an approach to statistical analysis whereby a computer detects and then uses the results to predict outcomes from new data. The combination of large amounts of available data becoming cheaply available and open source machine learning algorithms is causing a revolution in many industries, including water management. For example, water utilities can apply machine learning to predict which water or sewer main is likely to fail soon. This chapter of Data Science for Water Utilities introduces the principles of machine learning and the basic modelling process. The learning objectives for this chapter are:

Understand the principles of machine learning
Apply cross-validation to linear regression
Implement a decision tree prediction

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Principles of Machine Learning

Machine learning is an area of study in computer science and statistics that involves developing algorithms that can automatically learn patterns from data and make predictions or decisions, without being explicitly programmed to do so.

We have already seen linear regression, which is a form of supervised learning. In this type of machine learning, The model is trained on a dataset that contains both input and output variables. Cluster analysis belongs to the unsupervised leanring category. In these types of problems, the computer seeks dor structure in the data without independent variable.

Family tree of machine learning.

This chapter introduces multiple linear regression and decision trees.

Cross Validation and Fitting

Fitting a machine-learning model.

Multiple Linear Regression

Concrete mixture decision tree.

Comparing models

RMSE

Decision Tree

Concrete mixture decision tree.

Comparing models

Confusion matrix

Introduction to Machine Learning Screencast

Chapter thirteen of Data Science for Water Utilities explains the theory and application of machine learning in more detail. This screencast below reviews the code for this chapter.

Introduction to Machine Learning Screencast.

The data and code used in this chapter are available on GitHub:

Additional Resources

Other Chapters

Previous Chapter: Detecting Outliers and Anomalies

Feel free to contact me if you have any comments, suggestions or questions about this book.

Introduction to R for Utilities

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Data science is the activity of creating value from data. While the mechanics of developing data products by writing code is an essential skill, data science as an activity needs to be strategic to deliver business value. This chapter develops a framework for a strategic approach to data science by creating useful, sound and aesthetic data products. Usefulness relates to a data product's ability to achieve business outcomes. A sound data product is developed using valid and reliable models, is reproducible, and, when using data about people, is ethical. Lastly, data products need to be aesthetic, not to beautify but to ensure that the outcomes of analyses are easily understood so that benefits can be realised. The book Data Science for Water Utilities introduces using R for water utilities through realistic case studies. The first chapter introduces the principles of data science and how they relate to water management.

Data Science for Water Utilities

Routledge Amazon

What is Data Science?

Data science involves analyzing data in a systematic way to create value for businesses. This field combines mathematics, computer science, and domain knowledge. Some professionals in the industry believe that individuals with expertise in all three areas are rare and often referred to as "data science unicorns."

As part of my Data Science Unicorn Breeding Program, I have written a book aimed at water professionals who already possess mathematical skills and domain knowledge. I want to create the elusive data science unicorns by teaching them computer science.

Data science competencies (Based on Conway, 2009).

See my article about strategic data science, which develops a systematic approach to creating value from data.

Principles of Strategic Data Science

Packt Publishing

Learning how to code

This book uses the R language for Statistical Computing. This language is specifically designed for statistical analysis. Data science often uses other languages, such as Python, SQL, and Julia. The fact is that all data scientists need to be multi-lingual.

These seven steps will help you in your journey of becoming fluent in any programming language:

Understand the basics
Code by hand
Create simple programs
Practice
Ask for help
Build projects
Help others

This book introduces the basics of coding with R. Coding by hand is essential to thoroughly learning the language. Although generative AI, such as GPT or GitHub CoPilot, can write code, it will not teach you how to write code. Generative AI can assist you with finding solutions, but make sure you reverse-engineer the output so you understand the code.

R for Water Utilities

The following twelve chapters in this book introduce the R language by developing case studies:

Water quality data: Chapters 3–6 and 12
Customer survey: Chapters 7–10
Digital metering: Chapters 11–12
Concrete curing: Chapter 13

Additional Resources

The water industry produces a lot of resources on how to implement smart water networks.

The Smart Water Networks Forum (SWAN) is the leading, global voice for the smart water sector. They have detailed resources on how to implement a smart water network.
Fee Digital Water Book: A Strategic Digital Transformation for the Water Industry.

Other Chapters

Next Chapter: Basics of the R Language

Feel free to contact me if you have any comments, suggestions or questions about this book.

Loading and Exploring Water Quality Data from Spreadsheets

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

The first action in any analysis project is to obtain and load data. This data is often available in a database, the internet, or a file on your computer. One of the essential skills is thus to load this data into memory and view its content. This chapter introduces how to read CSV files and spreadsheets into R and explore their content using the Tidyverse suite of libraries. This chapter of Data Science for Water Utilities also presents a case study about water quality using synthetic data from an imaginary water supply network. This chapter has the following learning objectives:

Download and install R packages
Load and describe CSV files and spreadsheet
Explore the content of rectangular data (data frames)

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

The Tidyverse

The R language has a lot of capabilities out of the box. But its strongest feature is that it can be easily extended. Thousands of packages are available for almost every imaginable specialised task. Packages are available in public repositories, such as CRAN (The Comprehensive R Archive Network).

The Tidyverse is a collection of packages that makes working with R a bit easier. The remainder of this book will rely heavily on the Tidyverse way of analysing data.

A package provides additional functions for specialised tasks. You install packages with the install.packages("package-name") function. To use the functions in a package, you must evaluate library(package-name).

Loading Data into R

Using specialised packages, R can read almost every common data format, such as REST APIs, SQL databases and GIS shape files.

The Tidyverse readr package reads rectangular data from text files, such as the standard CSV format. Of course, R can also do this with base functions. Still, the Tidyverse version is faster and better at recognising data types. The Tidyverse readxl package reads data from Excel files.

Water Quality Case Study

This case study uses synthetic data from an imaginary water system named Gormsey. This data contains four analytes:

Escherichia coli (E. coli): A harmful bacterium.
Total chlorine [mg/l]. Chlorine is dosed to prevent bacteria from multiplying.
THM: Trihalomethanes. A byproduct of disinfection with chlorine that can cause a negative health impact.
Turbidity [NTU]: A measure of cloudiness.

The data is available in the GitHub repository for this book.

Variable Types and Classes

The previous chapter introduced scalar and vector variables. However, most data used in practice is stored in a rectangular format as a table. In the R language, a data frame is a rectangular data set where each column is a variable and each row an observation. The Tidyverse data frame is a Tibble, but these names are used interchangeably.

Scalar, vector and data frame variables in R.

Exploring Water Quality Data Screencast

Chapter three of Data Science for Water Utilities explains loading and exploring CSV and Excel files in more detail. This screencast below reviews the code for this chapter.

Loading and exploring spreadsheet data in R.

The data and code used in this chapter are available on GitHub:

Additional Resources

To help you remember the various functions discussed in the first five chapters of the book, a cheat sheet is available.

Other Chapters

Previous Chapter: Basics of the R Language

Next Chapter: Descriptive Statistics in Water Quality

Feel free to contact me if you have any comments, suggestions or questions about this book.

Managing and Cleaning Dirty Data

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Data is almost always produced to manage operations and rarely collected to analyse it in the future. The available data is thus never in an ideal format and needs to be converted to be suitable for analysis. We need, at minimum, a clear data structure, the correct variable types, and readable variables names. Preparing data for analysis, sometimes called data munging or wrangling, is an essential part of the data science workflow. This chapter of Data Science for Water Utilities introduces some techniques to clean data with R and the Tidyverse to create reproducible code. This chapter introduces a case study about customer perceptions about tap water. The learning objectives for this session are:

Use the dplyr package to transform data.
Apply the principles of tidy data.
Develop a script to automate data cleaning.

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Case Study 2: Understanding the Customer Experience

The data for this case study is a survey to measure the customer experience of the customers of a water utility. This data is generated with online survey software and contains information about:

Consumer Involvement
Customer contact frequency
Level of financial hardship
Service Quality

Data Cleaning

Data cleaning, or data jujitsu as some call it, consists of three steps:

Load and explore the data
Convert the data structure
Remove invalid data

A basic principle of good data science is never to manually clean data. Using a script to clean your data means that it is reproducible for other data sets with the same structure and issues and you can always go back to the raw data in case something went wrong.

Tidy Data

The Tidyverse uses the principles of Tidy Data, which means that:

Each variable forms a column
Each observation forms a row
Each type of observational unit forms a table

The data on the left is untidy because the variable is spread over multiple columns. We can fix this by pivoting the data so that the locations are no longer column names, but they become data.

Untidy and tidy version of laboratory data.

Managing Dirty Data Screencast

Chapter seven of Data Science for Water Utilities explains using the dplyr package to clean data in more detail. This screencast below reviews the code for this chapter.

Managing Dirty Data Screencast.

The data and code used in this chapter are available on GitHub:

Additional Resources

Other Chapters

Previous Chapter: Sharing Results of Data Analysis

Next Chapter: Analysing the Customer Experience.

Feel free to contact me if you have any comments, suggestions or questions about this book.

Sharing the Results of Data Analysis with R Markdown

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Data science aims to create value from data by creating useful, sound and aesthetic data products. Analysing data is rewarding, but creating value requires you to communicate the results. Analysing data in RStudio is fun but hard to share with anyone who does not understand the language. Of course, you could copy and paste results into a document, but that is neither efficient nor reproducible. This chapter explains how to communicate the fruits of your labour with colleagues and other interested parties by generating reports that combine text and analysis through literate programming. This chapter of Data Science for Water Utilities a workflow for data science products and how to create data products using the R Markdown method. The learning objectives for this chapter are:

Implement the workflow for data science projects
Apply the principles of reproducible and replicable research
Use basic R Markdown to create a Powerpoint presentation from data

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Data Science Workflow

The data science workflow starts with preparation and data cleaning, which can take most of the time in some projects. All projects should begin with a concise problem statement to prevent open-ended data dredging. Programmatic data cleaning is the topic of the next chapter.

Once the data has been tidied, we can try to understand the data through exploratory analysis using descriptive statistics and visualisation. That insight allows us to model and reflect, an iterative process known as the 'data vortex'.

Once we have created the model that best answers the problem statement, we can communicate these results to the consumers of our data products. One method is through literate programming with tools such as R Markdown or Org Mode.

Data science workflow.

Literate Programming Reproducible Analysis

Literate programming means integrating the text, the data and the code. This approach ensures that the analysis is reproducible, which means you can repeat the same process with new data in the same way.

A data science project consists of at least three sets of code:

Cleaning code to get from raw data to a tidy state
Analytical code to analyse the data
Presentation code to create summaries, tables and figures.

Reproducible research pipeline.

With literate programming, you can combine text and code in one script. The most popular method with R is to use a Markdown file. This is the WYSIWYM (What You See is What You Mean) approach instead of the more popular WYSIWYG (What You See Is What You Get) approach in word processors.

What You See Is What You Mean (WYSIWYM)

The left part of the screen shows what it looks like when I write a book, and the right part shows the end result when exported as a PDF file.

The screencast below introduces the principles of this approach.

Sharing the Results of Data Analysis Screencast

Chapter six of Data Science for Water Utilities explains using literate programming with R Markdown in more detail. This screencast below reviews the code for this chapter.

Sharing the Results of Data Analysis with RMarkdown.

The data and code used in this chapter are available on GitHub:

Additional Resources

The consumer-involvement.Rmd in the case-studies folder of the GitHub repository shows how to write a journal article, including references using a BibTeX file. The concept of Consumer Involvement is explained in Chapter 8.

The R Markdown: The Definitive Guide by Yihui Xie and others explains the how to use literate programming in R in great detail.

Other Chapters

Previous Chapter: Visualising Data with ggplot2

Next Chapter: Managing Dirty Data

Feel free to contact me if you have any comments, suggestions or questions about this book.

Visualising Data with ggplot2

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Visualisation is essential for data scientists who want to explore and understand data and communicate their results. Scientists have studied how the mind perceives graphics in great depth and devised a comprehensive body of knowledge that helps us create sound, useful and aesthetic visualisations. The first part of this chapter of Data Science for Water Utilities introduces some principles of best practices in data visualisation. The second part presents the ggplot2 library and the principles of the Grammar of Graphics to create basic visualisations. The learning objectives for this chapter are:

Evaluate data visualisations using basic principles.
Apply the principles of the Grammar of Graphics.
Visualise water quality data with the ggplot2 package.

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Principles of Visualisation

Data visualisation needs to tell a story, which means that when somebody looks at the image, they need to draw a conclusion. A visualisation should not be a case of "create your own adventure". Achieving this objective requires simplicity and graphic austerity.

The data-to-pixel ratio is a rule of thumb that helps to focus your visualisation on the story. This ratio expresses the number of pixels in the image (excluding the neutral background) versus those representing data.

The example below illustrates the idea. The image on the left might be interesting, but the background is distracting, and the colours don't express data. The version on the right is a lot clearer on the story. We can immediately see which imaginary suburb has the best taste scores for tap water.

Data to Pixel Ratio.

Grammar of Graphics and ggplot2

The ggplot2 package is part of the Tidyverse. It implements the Grammar of Graphics, a systematic approach to visualising data using layers. The various functions in ggplot2 are chained with a + sign, forming the individual layers, as shown below. You can combine as many layers as you like.

Grammar of Graphics and ggplot2 layers.

Using Colour in Visualisations

Most graphs overuse colour to visualise data. Colours should only be used to comply with a style guide or to visualise data.

The Color Brewer website provides a great interface to explore various colour palates.

Since writing this book, the latest R version has received an upgraded colour palette. This article explains the new built-in colour palettes.

Emil Hvitfeld has gathered a large collection of colour palettes in R.

Visualising Data with ggplot2 Screencast

Chapter five of Data Science for Water Utilities explains visualising data with ggplot2 in more detail. This screencast below reviews the code for this chapter.

To help you remember the various functions discussed in the first five chapters of the book, a cheat sheet is available.

Visualising Water Quality Data with ggplot2.

The data and code used in this chapter are available on GitHub:

Additional Resources

To help you remember the various functions discussed in the first five chapters of the book, a cheat sheet is available.

Addendum

The width parameter in ggplot2 geometries that use lines has been deprecated and is replaced with linewidth.

Other Sources

Other Chapters

Previous Chapter: Descriptive Statistics

Next Chapter: Sharing the Results

Feel free to contact me if you have any comments, suggestions or questions about this book.

Working with Dates and Times in R

Peter Prevos — Wed, 10 May 2023 00:00:00 +0000

Most data collected in operational processes are time series. These are data sets where each measurement is associated with the time of the measurement or observation. This chapter introduces a study which looks at synthetic data collected from water meters from individual houses. Meter reads collected at a high frequency provide valuable insights into water consumption, such as leak detection. This chapter of Data Science for Water Utilities explains how the R language manages dates and times and uses this knowledge to explore time series data. The learning objectives for this chapter are:

Understand the principles of how the R language processes date and time variables.
Apply the functionality of the Lubridate package to manipulate dates and times.
Use date and time variables in calculations and visualisations.

Data Science for Water Utilities

Routledge Amazon

The data and code used in this chapter are available on GitHub:

Date and Time Variables

All date and time variables in R use integer numbers starting at the Unix Epoch (1 January 1970). Spreadsheets tend to use 31 December 1899 as their datum.

The basic time string uses the ISO 8601 convention, i.e. "2023-06-15 18:23:45 AEST".

Lubridate Package

Working with dates and times in base R can be challenging. The lubridate package in the Tidyverse lubricates dates and times with some convenient functions.

Digital Metering Leak Detection

The case study for this chapter uses synthetic digital metering data. The data for this case study was simulated using R scripts.

The article on analysing digital metering data explains some of the principles and creates a diurnal curve.

Example of a diurnal curve for domestic water consumption.

Working with Dates and Times in R Screencast

Chapter eleven of Data Science for Water Utilities explains working with dates and times and leak detection in more detail. This screencast below reviews the code for this chapter.

Working with Dates and Times in R.

The data and code used in this chapter are available on GitHub:

Additional Resources

Addendum

The code on page 150 uses the summarise() function to calculate daily flows. This function has been deprecated for this particular case. While summarise() requires that each argument returns a single value, and mutate() requires that each argument returns the same number of rows as the input, reframe() is a more general workhorse with no requirements on the number of rows returned per group.

Other Chapters

Previous Chapter: Clustering Customers to Define Segments

Next Chapter: Detecting Anomalies and Outliers in Water Data.

Feel free to contact me if you have any comments, suggestions or questions about this book.

Monte Carlo Cost Estimates: Engineers Throwing Dice

Peter Prevos — Mon, 23 Aug 2021 00:00:00 +0000

Estimating the cost of a complex project is not a trivial task. Traditional cost estimates are full of assumptions about the future state of the market and the final deliverable. Monte Carlo cost estimates are a tool for better understanding your project's risks and enabling better cost control. Monte Carlo simulations are a technique to control your “known unknowns”.

Albert Einstein famously said that “God does not play dice”. While this might or might not be the case, engineers play dice and embrace the stochastic nature of reality to predict the future. Monte Carlo Simulations are a method for estimating the cost of your project across thousands of parallel universes to assess financial risk.

While I have a pet hate against using matrices to manage risk, Monte Carlo simulations are an analytical method for dealing with uncertainty and risk. This article explains the principles of Monte Carlo cost estimates and how to implement them in the R language for statistical computing.

You can download the code from GitHub in the case-studies folder.

The principles of cost estimates

The basic principle of cost estimation is deceptively simple. To estimate the cost for each item in a project, multiply the quantity of work $Q_i$ times the rate $R_i$ you will pay for each unit of work. Sum the cost of each item, and you have the total project cost $P$ for a project with $j$ items:

$$P = \sum_{i=1}^j Q_i R_i$$

The reality is, unfortunately, more inconsistent than this equation can express. Determining the correct quantity and rate for each item is a fine art requiring knowledge of engineering and economics.

Cost estimates reflect the state of knowledge at the time of developing them. Many aspects of the project have yet to be discovered and might only surface once we start digging. As the late Donald Rumsfeld philosophically said:

… there are known knowns; there are things we know we know. We also know there are known unknowns; that is, we know there are things we do not know. But there are also unknown unknowns — the ones we don’t know, we don’t know.

Donald Rumsfeld.

Deterministic

The basic method is helpful, but it assumes we know the amount of work or the unit rate accurately. Both parameters are subject to uncertainty and risk because the project has yet to be realised. Only at the end of the project do we know precisely how much it costs.

In deterministic cost estimation, this uncertainty is accounted for by adding a contingency to the estimate. This contingency is simply a percentage of the total. The less information about the project (usually at the early stages of development), the higher the rate.

In the planning phases, 20% or even higher might be suitable, while uncertainty can be much lower after tender award. This uncertainty covers any risk that quantities or rates are higher, as well as unforeseen activities that were not costed. The table below shows a typical deterministic cost estimate.

Item	Quantity	Unit	Rate	Cost
Materials	12,000	$m^1$	75	$900,000
Excavation	2,200	$m^3$	15	$33,000
Pipe laying	12,000	$m^1$	13	$156,000
SUBTOTAL				$1,089,000
Contingency			20%	$217,000
TOTAL				$1,306,000

Example of a simple pipeline project estimate.

A deterministic method is a blunt tool. Uncertainties are not estimated; they are based on rules of thumb. In this example, we cannot know which part of the project contributes most to the uncertainty or the likelihood of staying within budget. Is the project manager simply inflating the budget to reduce the risk of spending more? What is the probability that the project will cost more than the estimated amount? We cannot answer these questions with the deterministic method.

Probabilistic

A better understanding of the financial risk in a project leads to better decisions. Probabilistic cost estimation methods assess the uncertainty of each item rather than applying a percentage to the total. Breaking the uncertainty into smaller chunks allows project managers to better understand the financial risk in their projects and reduces the risk of significant estimating errors.

The quantity of work and the rate we pay vary with local conditions, external factors, market rates, and other conditions. We thus need to introduce an error rate $\epsilon_i$ for each item to account for the uncertainty in the estimate. Uncertainty relates to the inaccuracy of each cost item due to a lack of information. Thus, our formula for a project with $j$ items and $k$ events cost now becomes:

$$P = \sum_{i=1}^j (Q_i R_i)+ \epsilon_i$$

The most basic way to implement this concept is to assign a contingency to each item in the estimate and nominate the risk events in something like this:

Item	Quantity	Unit	Rate	Cost	Uncertainty
Materials	12,000	$m^1$	75	$900,000	$9,000
Excavation	2,200	$m^3$	15	$33,000	$16,500
Pipe laying	12,000	$m^1$	13	$156,000	$30,000
SUBTOTAL				$1,089,000	$55,500
TOTAL				$1,144,500

Example of a simple pipeline project estimate.

Note that we now call it 'uncertainty' rather than 'contingency'. This terminology is more than a semantic difference, as we have estimated our uncertainty at the item level in the Work-Breakdown-Structure.

This approach provides additional intelligence about where the project's risks lie, which can help the team allocate efforts where they are most needed. The fact that the estimate is lower than before is less interesting than what we can gather from the details. If this were my project, I would gather more information about the events with the highest relative uncertainty to reduce risk.

A probabilistic cost estimate produces higher accuracy because the error rate for some items will be more controllable than for others. In the traditional method, the amount of contingency is anyone's best guess. Estimates often hide contingencies in cost items to reduce the risk of requesting more budget. While this approach is pragmatic, looking at each item is more transparent.

Monte Carlo Cost Estimate

The probabilistic approach in the table above is an acceptable way to assign uncertainties and risks. Still, it does not give us insight into how likely this estimate will eventuate.

Experience shows that the final project outcome rarely matches the estimate. It is, after all, an estimate, not a prediction. In other words, if you do the same project thousands of times in parallel universes, you will have thousands of different outcomes. This science-fiction concept is the basis for Monte Carlo simulations. With some statistics, we can recreate this hypothetical situation by repeating the same estimate thousands of times and analysing the distribution of possible outcomes.

Monte Carlo simulations model reality using large volumes of randomised numbers drawn from known distributions. They can be used for any situation where deterministic methods are too complex, such as modelling contact centre call traffic, or in this case, a cost estimate.

To add more information to the estimate, we assign each item a low, likely and high cost. The low cost ($a$) is rarely achieved and is your ‘bargain-basement’ price. The likely cost ($c$) is what you would typically use in your estimate. Finally, the high cost ($b$) is what you will pay when everything you can think of goes wrong. Statistically, each cost item now has a triangular probability distribution, visualised below.

Triangular probability density function.

Our cost estimate would now look something like this:

Item	Low	Medium	High
Materials	900000	909000	950000
Excavation	33000	49500	50000
Pipe laying	156000	186000	200000

For each item, we can calculate the average or any other percentile. The average cost of an item is $\frac{a+b+c}{3}$. The expected cost $x$ with likelihood $p$ is defined by the quantiles for each item, given by:

$$x_p = \begin{cases} a + \sqrt{(b-a)(c-a)p} & 0 \leq p \leq p_c \\ b - \sqrt{(b-a)(b-c)(1-p)} & p_c < p \leq 1 \end{cases}$$

Where $p_c$ is the probability of the likely estimate ($c$): $p_c=\frac{c-a}{b-a}$, and $a \leq c$ and $b \geq c$.

Triangular cumulative distribution function.

While the triangular distribution is the most common method for assessing uncertainty, other distributions are possible; the principles remain the same. Be mindful, however, that most probability distributions range from minus to plus infinity, which is not a realistic assumption for cost estimates.

If your cost estimate contains only one item, you are done, as you can calculate the project cost by plugging in a percentile $p$ into the formulas. The final project cost will be lower than this number at probability $p$. The chosen percentile depends on your risk appetite. A percentile of .95 provides much more confidence that the actual cost will be lower than the budget.

But when the estimate includes multiple items, doing so analytically will be too complex for mere mortals. This situation is where the Monte Carlo technique comes in. The Monte Carlo method simulates reality by calculating thousands of possible outcomes. The result is not a singular number but a distribution of $n$ possible outcomes.

The outcome of a Monte Carlo simulation is a cost estimate for thousands of parallel universes. If you were to undertake the project $n$ times, the distribution of the final cost is expected to resemble the simulation outcome.

Monte Carlo Simulation in R

Several R libraries can calculate triangular distributions. The triangle package provides a set of functions to work with triangular distributions. The rtriangle() function provides a vector of random quantiles for the chosen distribution, as shown below:

  # Monte Carlo Simulation in R
  library(triangle)

  # Triangular distribution
  hist(rtriangle(n = 10000, a = 12000, b = 15000, c = 14000), 
       breaks = 100, 
       main = "Triangular distribution simulation")

When plotting the histogram of these results, the triangular shape becomes evident, albeit a bit rough around the edges. Notice that a Monte Carlo simulation is only ever an estimate. The higher the number of simulations (in this diagram, 10,000), the higher the resolution of the result. To test this proposition, change the n parameter in the rtriangle() function.

Simulated triangular distribution.

To add the probability distribution of all your cost items, store all simulations in a matrix with n columns (number of simulations) and j (number of cost items) rows. Then, in the next step, you add all the simulated values and create a histogram or calculate percentiles.

The example below reads a CSV file with the same content as the table above. It then creates a results matrix, runs the simulations and calculates and visualises the results.

The set.seed(1234) function fixes the pseudo-random number generator to always use the same 'random' numbers. This way we always have the same result instead of introducing stochastic noise. If you need true randomness, then use a specialised random generator package that obtains numbers form the random.org website.

  ## Read Data
  estimate <- read.csv("data/cost-estimate.csv")

  ## Simulation settings
  n <- 10000 ## Simulations
  j <- nrow(estimate)
  mc_sims <- matrix(ncol = n,
                    nrow = j)

  ## Simulation
  set.seed(1234)
  for (i in 1:j){
    mc_sims[i,] <- rtriangle(n = n,
                             a = estimate$Low[i],
                             b = estimate$High[i],
                             c = estimate$Medium[i])}

  ## Determine estimates and 95th percentile
  mc_results <- colSums(mc_sims)
  p95 <- quantile(mc_results, 0.95)

  ## Visualise
  hist(mc_results, breaks = 100)
  abline(v = p95, col = "red", lwd = 2)

A Monte Carlo simulation outcome is thus never one number but a vector of numbers from which you can analyse. The example below uses the 95^th percentile as the budget figure.

Adding triangular distributions will create a new distribution specific to this project. The mc_results vector holds the estimated possible total cost.

To set a budget figure, you need to nominate a percentile. The 50^th percentile has an equal chance of being higher or lower than the actual. You should choose the 95^th percentile or whatever percentile matches your risk appetite to have more certainty that a sufficient budget is available.

Monte Carlo cost estimate and 95^th percentile.

Monte Carlo cost estimates are not only for projects. You can also undertake these calculations at the program level and assess the level of financial risk you hold within the program.

Monte Carlo cost estimates are a powerful tool for better understanding your project. They are, however, like any modelling, subject to the GiGo principle (Garbage-in-Garbage-out). Therefore, determining the low, likely and high costs will require domain knowledge. While these three estimates have their level of uncertainty, they will provide better insights than relying only on a monolithic amount for contingency.

Probabilistic cost estimates do not guarantee that your project will remain under budget, but they help you achieve this elusive goal.

Visualise Org-Roam Networks With igraph and R

Peter Prevos — Sat, 14 Aug 2021 00:00:00 +0000

The Emacs package Org-Roam provides a powerful tool to take notes following the idea of the Zettelkasten method. You can write notes with all the power that Emacs provides while linking your thoughts to each other and with your bibliography. This article shows how to analyse and visualise Org-Roam knowledge networks with the iGraph package and the R language.

The image below shows the current state of my personal knowledge network as visualised by iGraph. The Org-Roam community is working hard to create a visual user interface for Org-Roam with network diagrams. While this user interface is a great tool to explore your network, Emacs users tend to prefer writing code instead of clicking buttons. This article adds to this concept by showing how to apply some mathematical analysis and delve deep into your Zettelkasten. The beauty of iGraph is that it is available in R, Python, Mathematica and C/C++.

My personal Org-roam notes network.

You can download the code in this article from GitHub.

Loading the Database

Org-Roam stores information about the notes in your system in an SQLite database. This database holds information about the nodes (registered files and headings) and links. The DBI package enables connecting to these databases.

The first lines open the connection. You will have to change the path of the database location to your preferences for this to work. The two queries extract the unique ID and the title for all nodes. The second query extracts the links between nodes. The Org-Roam database also stores external links, which are excluded from this analysis.

The next part cleans the titles by removing quotation marks and joins the tables. I could have done this in the query, but I know Tidyverse better than SQL.

  ## Visualise Org-Roam databse

  ## Connect to database and extract nodes and links
  library(DBI)
  roam <- dbConnect(RSQLite::SQLite(), "~/.config/emacs/org-roam.db")
  nodes <- dbGetQuery(roam, "SELECT id, title FROM nodes")
  links <- dbGetQuery(roam, "SELECT source, dest FROM links WHERE type = '\"id\"'")
  dbDisconnect(roam)

  ## Clean node names and create network table
  library(tidyverse)
  nodes <- nodes %>% 
    mutate(title = str_remove_all(title, "\""))

  network <- left_join(links, nodes, by = c("source" = "id")) %>% 
    left_join(nodes, by = c("dest" ="id")) %>% 
    select(from = title.x, to = title.y)

The result of this code is a data frame with two names of the nodes at each vertex in the network. Any unconnected nodes are ignored due to the left join. The next step is to convert this data to iGraph format and start visualising and analysing.

Create iGraph network

The iGraph package uses a specific data format. The graph.data.frame() function converts a data frame to an iGraph network. The simplify() function removes any multiple links and self-references.

  ## Create network diagram
  library(igraph)
  g <- graph.data.frame(network)
  g <- simplify(g)

Analyse the Network

The iGraph package has an extensive library of functions to analyse networks. In this section, we look at a measure for centrality and clustering.

Centrality

Not all nodes in the network are of equal importance. Centrality is a concept in graph theory that describes the importance of a node in a network. The most straightforward method of calculating the centrality of a node is to count the number of vertices (connections) that it has with other nodes. The example below shows a simple undirected graph and the degree of each node.

Example of the degree centrality of an undirected graph.

In a directed graph, like with Org-Roam, we can calculate the in-degree and out-degree, the number of vertices point to the node, or the number of vertices pointing from the node (the number of backlinks and the number of links). The degree() function of the iGraph package calculates degree centrality. The code below generates a table with the top ten most connected nodes.

  ## Centrality
  centrality <- tibble(Node = names(degree(g)),
                       Degree = degree(g, mode = "total"),
                       Links = degree(g, mode = "out"),
                       Backlinks = degree(g, mode = "in")) 

  centrality %>% 
    top_n(n = 10) %>% 
    arrange(desc(Degree))

Communities

While some nodes might be more connected than others, it would also be good to know which nodes are closest to each other. In other words, can we find clusters of nodes (communities) to find meta-structure in the network? A community is a sub-network with densely connected nodes, but less so with nodes outside the community. Expressed probabilistically, two random nodes are more likely to be associated when they form part of the same community than when they don't. Thus, community detection increases the parsimony of the network by identifying those groups of nodes that are most closely related to each other.

Using the example network, two communities can be visually distinguished: nodes 1–4 and nodes 5–7. Texts A, B and C belong to the first community, while texts C and D belong to the second community. Text C (nodes 2 and 5) spans the two communities of discourse. This solution is valid because each node has more connections to nodes within its own community than nodes outside it.

Example of a network with two communities.

The examples' communities are easily detected visually, but community detection becomes more difficult as the network grows. Community detection is a mathematical process to cluster nodes into cohesive sub-networks.

Several algorithms for community detection have been developed based on a range of mathematical principles. However, the numerical validation of community structure has not yet been satisfactorily solved, and no agreed single method exists to assess the quality of communities. In my research into knowledge networks, the Spinglass algorithm provided the most interpretable results.

  ## Community-detection
  c <- cluster_spinglass(g)

Note that clustering algorithms only work with complete graph, which means that all nodes are connected to each other by one or more steps. If you have multiple clusters and one or more single nodes, you can decompose the graph into its connected components and analyse each of these.

  g.comp <- decompose(g)
  largest <- which.max(sapply(g.comp, diameter))
  g <- g.comp[[largest]]
  c <- cluster_spinglass(g)

This algorithm results in a named vector that assigns a number of each community to each node. Making sense of community structure requires some human interpretation because the algorithm does not analyse the texts, only the connections between them. Please note that clustering algorithms are computationally intensive and can take a while.

We can assign a name to each community by using the node's name with the most backlinks within each community. The code below creates a table of the community memberships and joins it with the centrality table. The last part assigns a name to each community by selecting the node's name with the most backlinks, counts the number of members and arranges the table by community size.

  communities <- tibble(Node = c$names,
                        Community = c$membership) %>% 
    left_join(centrality) %>% 
    group_by(Community) %>% 
    summarise(Name = Node[which.max(Backlinks)],
              Nodes = n()) %>% 
    arrange(desc(Nodes))

Visualise Org-Roam

Now that we have analysed the graph, it is time to visualise the results. iGraph has extensive plotting functionality that gives full control over every aspect of the visualisation. To create the graph shown at the start of this article, use the following code:

  ## Visualise graph
  par(mar = c(0, 0, 0, 0))
  plot(g,
       layout = layout.fruchterman.reingold,
       mark.border = NA,
       vertex.color = c$membership,
       vertex.label = NA,
       vertex.size = sqrt(degree(g, mode = "in") + 1),
       vertex.frame.color = NA,
       edge.arrow.size = 0,
       edge.color = "lightgrey")

The layout of graphs in this example follows the Fruchterman-Reingold method. This method tries to position the nodes so that the ones with the highest centrality are in the centre. Several other methods are available to compute an optimal layout.

The colour of each node relates to its community. The node's size is the square root of the number of backlinks (indegree) plus one so that larger nodes don't overpower the picture and small nodes vanish.

The remainder of the options improve the readability of the graph. This image is aesthetically pleasing, but it is not very informative. Adding text to the nodes would make it unreadable.

Subnetworks

When a network is fairly large it might be more informative to visualise only a section. The code below determines the number of the relevant node. The V() function holds the properties of al vertices (the nodes) in the network. The make_ego_graph() subsets a network from a given node (ego) and the order. This function creates a list of networks as you can enter more than one node.

  ## Subnetwork
  node <- which(names(V(g)) == "Topology")
  g1 <- make_ego_graph(g, order= 2, nodes = node)
  c1 <- cluster_walktrap(g1[[1]])

  plot(g1[[1]], layout = layout.kamada.kawai,
       mark.border = NA,
       vertex.size = sqrt(degree(g1[[1]])) + 2,
       vertex.color = c1$membership,
       vertex.frame.color = NA,
       vertex.label.family = "Sanserif",
       vertex.label.color = "black",
       edge.arrow.size = .3)

Subnetwork example.

Interactive visualisation

The networkD3 package can create interactive graphs, as implemented in this code snippet. The first step converts the iGraph object to a D3 object. Then, the htmlwidgets package lets you save this object as an interactive HTML file.

The code below adds the number of backlinks to the node table. Then, it defines the legend using the community names described above.

  ## Interactive visualisation
  library(networkD3)
  library(htmlwidgets)

  ## Convert to object suitable for networkD3
  d3 <- igraph_to_networkD3(g, group = communities$Name[membership(c)])
  d3$nodes$backlinks <- degree(g, mode = "in")

  ## Create force directed network plot
  p <- forceNetwork(Links = d3$links, Nodes = d3$nodes, 
                    Source = "source", Target = "target", 
                    NodeID = "name", Group = "group", 
                    Nodesize = "backlinks",
                    zoom = TRUE, legend = TRUE)

  saveWidget(p, file=paste0( getwd(), "/org-roam.html"))

Further Analysis

Graph theory is a broad field with lots of approaches to analyse graphs. This article only scathes the surface of what you can find out about your personal knowledge garden. Graph theory is a vital tool in many applications, including the fight against the Coronavirus.

Graph theory lectures by Anthony Bosman.

Generate H-Fractals using ggplot

Peter Prevos — Fri, 04 Jun 2021 00:00:00 +0000

A fractal is a geometric figure in which the same pattern repeats infinitely at an ever-smaller scale. One of the most simple examples is the H-Fractal. You construct a H-Fractal by starting with the line segments that resemble the capital letter H. You then iteratively place smaller H’s centred at the top and bottom of each free line segment. This article shows how to generate H-Fractals with the R language.

The H-Fractal or H-Tree is a self-similar fractal with ever-smaller line segments. This fractal is a Paeno Curve, which means that it eventually will fill a whole area and becomes a 2-dimensional object. The H-Fractal is a canopy fractal, a type of graph where a line segment is split into smaller lines at the end. Fractal canopies are very useful in creating models of plants.

The H-Fractal principle has practical applications in electronics. For example, microstrip antennae are printed on circuit boards and resemble the tree structure in this fractal. In addition, the splitting of fractal canopies is similar to the structure of bronchial tubes in the human body. A US Patent from 2006 lists a H-Fractal as the geometry for a microwave antenna.

H-Fractal with five iterations.

Method

The algorithm starts with a horizontal line on the interval $[-1,1]$. Each iteration adds new shortened lines to the ends of the existing lines. The first iteration results in a shape that resembles the letter H. After that, the direction of the lines switches at each iteration from horizontal to vertical.

If the number of iterations of the fractal is $p$, then each iteration adds $2^p$ lines to the tree, and the total number of lines in the fractal is $2^{p+1} + 1$.

The new lines are shortened by a factor, which is set at $\frac{1}{\sqrt{2}}$. If you set it any higher, the lines will overlap—a lower value results in a more sparse design.

Generate H-Fractals with ggplot

The code below starts with setting the initial variables and defines the first line in a data frame. The data frame holds the start and end coordinates and the iteration number.

  library(ggplot2)
  ## H-Fractal Parameters
  p <- 8 ## Number of iterations
  a <- 1 / sqrt(2) ## Shortening parameter
  
  ## Define horizontal starting line
  h_fractal <- data.frame(x1 = -1,
                          y1 = 0,
                          x2 = 1,
                          y2 = 0,
                          m = 0)

The loop defines the iterations and adds a new row to the data frame for each line. First, the direction flips by setting the dx and dy values to either 0 or 1. The second part defines the row numbers for the previously defined lines and the new lines. The code then calculates the coordinates for the new line segments.

  # Generate coordinates (each line on a row)
  for (m in 1:p) {
    ## Flip direction
    dx <- (m - 1) %% 2
    dy <- m %% 2
    # Previous and current line numbers
    i <- (2^(m - 1)):(2^m - 1)
    j <- (2^m):(2^(m + 1) - 1)
    ## Calculate coordinates
    h_fractal[j, "x1"] <- c(h_fractal[i, "x1"] - dx * a^m, h_fractal[i, "x2"] - dx * a^m)
    h_fractal[j, "x2"] <- c(h_fractal[i, "x1"] + dx * a^m, h_fractal[i, "x2"] + dx * a^m)
    h_fractal[j, "y1"] <- c(h_fractal[i, "y1"] - dy * a^m, h_fractal[i, "y2"] - dy * a^m)
    h_fractal[j, "y2"] <- c(h_fractal[i, "y1"] + dy * a^m, h_fractal[i, "y2"] + dy * a^m)
    ## Set line property
    h_fractal[j, "m"] <- m
  }

Lastly, The fractal is visualised with ggplot and the line segment geometry. The colour of the line slowly fades by setting it to the iteration number.

    ## Visualise
  ggplot(h_fractal) + 
    aes(x = x1, y = y1, xend = x2, yend = y2,
        col = m) +
    geom_segment(size = .5) +
    scale_size(guide = NULL) +
    scale_colour_gradientn(colours = gray.colors(p), guide = NULL) + 
    theme_void()

H-Fractal with twelve iterations.

Variations on the Theme

Besides changing the number of iterations and the shortening factor, you can also play with line thickness, colours, and the lines’ direction. This Shiny application lets you play with all variables in this algorithm to generate

Tantalising Games in R: Fun with Statistical Computing

Peter Prevos — Sat, 26 Dec 2020 00:00:00 +0000

Computers exist to solve pressing problems to improve the human condition. Although our world is full of urgent issues to address, the human need for play is a robust internal motivator. This general statement also applies to the R Project for Statistical Computing. Analysing data analysis is a serious business without much room for fun. This specialised computer language is not designed for games in R.

Hacking culture is one of the main reasons that computer science has evolved so fast. One of the many definitions of hacking is to use technology for a purpose it was not designed for, and thus writing games in the R language is an exciting challenge.

Not many R developers have written games. R has a games package in CRAN, but this library is about the mathematics of game theory. This article explains the R language’s possibilities and limitations for writing games, plus links to actual games others and myself have developed.

Action Games

Fully functioning action or arcade games are not easy to create in the R language because it has no interrupt function to process input from a keyboard or joystick while the battle continues. R can process keyboard input and mouse clicks on a graph, but the game waits for input. This makes it impossible to have moving elements on a screen.

The other impediment in R to write arcade games is the lack of support for sound. The beepr package by Rasmus Bååth plays a sound after a process has completed. The beep function in this package can play eleven different sounds. However, R can only do one thing at a time, so paying a sound stops any other gameplay.

The Gravedigger game from the 1980s book Creepy Computer Games is a pseudo arcade game that was initially developed for the venerable ZX81, my first computer. In this game, the antagonist only moves once after the player moved. It is thus as much a strategy game as it is an action game.

MattSen has interpreted of the classic ZX81 game 3D Monster Maze in the R language. His version of this game is a cross between an action and an adventure game.

Although R is unsuitable for arcade games, I have managed to simulate the classic game of Pong. This code simulates a player that follows the path of the ball. Since I wrote this rudimentary Pong version, two people have developed exciting functionality to make arcade games possible.

Carsten Croonenbroek has developed the classic Snake game. This package uses tcltk to obtain user interaction via callback functions (almost as flexible as a real system-level message pump). Perhaps we can generalise the code in this package to create a joystick function in other games.

Stephen Kozak wrote a library that uses a Windows executable to constantly check for keyboard input. The KeyboardCheckR provides the QueryKeyboard() function. This package is not on CRAN, and you will need to use devtools to install it.

Strategy Games

Developing software to play strategic games has been an objective of artificial intelligence research from the early days of computing. Deep Blue defeating Gary Kasparov in 1997 was one of the significant milestones in the history of computing.

More recently, Google’s AlphaGo beat the European Go champion, Fan Hui. Deep Blue and AlphaGo used quite different approaches. Whereas Deep Blue used traditional brute force, AlphaGo uses deep learning. Strategic games are ideal for the R language because of its specialised functionalities in statistical computing and machine learning.

You can play the famous combinatorial number-placement puzzle with the Sudoku package. This library generates, plays, and solves Sudoku puzzles. The playSudoku() graphical version depends on tkrplot, if you are not on Windows. The sudokuALT package provides an alternative. This package presents more tools, such as plot and print methods.

The rchess package by Joshua Kunst provides some functionality to play chess, but it is not a fully playable game. This package can plot beautiful chess boards, move pieces and detect check. However, this package is not able to generate moves by the computer and does not include a playable interface.

Tic Tac Toe, or Noughts and Crosses is one of the most accessible strategic board games. The minimax algorithm can easily play a perfect game, which means that the computer can never beat itself. The Tic Tac Toe game in R uses the locator function to manage the human player. This function reads the position of the graphics cursor when the user presses the left mouse button.

Roland Krasser developed a library to play Codebreaker, inspired by the classic Mastermind game and classic 1980s design. He also developed Pixelpuzzle, both games are available on CRAN. MattSen has also developed a version of this game, called Bulls and Crows.

Pixelpuzzle by Roland Krasser.

Adventure Games

A text adventure is a form of interactive fiction where a player can freely explore a virtual world described in prose. These games were trendy in the 1980s. The basic principle is that the computer shows a text that describes the current situation. The player enters a text command, usually a simple verb-noun combination. The computer processes the command and displays the new status. This interchange continues until the game reaches a defined state. Either the player dies, or the player has achieved the game’s objectives.

Most games are set in typical Dungeons and Dragons settings with wizards, dragons and other usual characters. The lines below are a short extract from Colossal Cave, by Will Crowther and Don Woods, the very first text adventure written in the mid-1970s:

You are standing at the end of a road before a small brick building. 
Around you is a forest. A small stream flows out of the building and down a gully.
> go south 
You are in a valley in the forest beside a stream tumbling along a rocky bed.
> enter stream 
Your feet are now wet.

The BetaBit package by Smart Poland provides an educational console game where the user has to crack codes. This game comes close to the look-and-feel of traditional text adventures but is not quite the same.

This website has two adventure games. The Tenliner Cave Adventure is a programming challenge to cram as much functionality as possible into ten lines of BASIC code. This version uses the minimal amount of R code to recreate this masterpiece of programming.

My Secret of Landusia game provides a fully-functioning set of functions to write classic adventure games. The code can be easily expanded or modified to create new stories.

Games in R: Having fun with the R Language

R is not an optimal language for writing computer entertainment. Multipurpose languages such as Python or Java are much more suited to meet the expectations of gamers.

The many limitations don’t hold me back from writing computer games in the R language because working with limitations create a significant challenge. The Devil is in the Data irregularly publishes code to have fun with R. Subscribe to the blog to receive updates or follow the Rgames repository on Github.

If you know of any other games written in R, then please contact me. I would love to create a complete compendium of how to have fun in this quirky computing language.

Tenliner Cave Adventure in R: Miniature Text Adventure

Peter Prevos — Wed, 19 Aug 2020 00:00:00 +0000

Computing has come a long way in the past 75 years. I learned how to write code in the 1980s on my tiny 1 kilobyte ZX81 home computer. With only 1 kilobyte of memory, capabilities were limited, but this machine set me off on an adventure writing code in the BASIC language. An active subculture of computer enthusiasts still develops new code and keeps these old machines alive. I recently stumbled across the fantastic Tenliner Cave Adventure (2016) by Einar Saukas from Brazil. This tiny ZX81 program provides a fully functional classical text adventure using some cunning techniques. This article contains a port of this game to the R language to explore its inner workings.

Interactive Fiction

Text adventures, or interactive fiction, were trendy in the 1980s. The basic premise is that the computer shows a text and the player responds with a text command. The computer parses the command, calculates the new state of play and progresses the game with a new bit of prose. Last year I wrote an article about writing interactive fiction using the R language with a translation of Secret of Landusia. The Tenliner Cave Adventure provides a minimalist approach by squashing the basic principles of a text adventure into 1024 bytes of memory. The backstory of this Saukas' mini-adventure is a typical fantasy setting:

As a young warrior apprentice, you have been chosen by the village elders to seek out the evil menace that lurks in some nearby caves. Once found, use any means at your disposal to defeat it… Good luck on your quest!

ZX81 Screenshot of the Tenliner Cave Adventure.

Tenliner Cave Adventure in R

The Tenliner Cave Adventure is a jewel of efficient programming. The game is simple, but it has all typical elements of a text adventure. As I no longer own a ZX81, I could not resist translating this code to R to be able to play and reverse-engineer the game.

The game recognises the following commands:

north, south, east, west (move)
inventory
look
look <object>
get <object>
open <object>
kill <object>

Most advanced games understand abbreviations and synonyms, but in this simple version, you have to provide precise commands. Can you rid the village of the evil menace?

  ## Tenliner Cace Adventure by Einar Saukas
  ## Ported from ZX81 BASIC to R by Peter Prevos

  ## Helper functions to keep syntax close to ZX81
  val <- function(p) as.numeric(p)
  x <- function(p, q = p) substr(X, p, q)

  ## Ported ZX81 code
  X <- "100cannot doyou walk opened   closed   a sword  a key    nothing  a chest  a dragon a corpse taken    you died you won. "
  repeat {
      cat(paste("You are in a", substr("cavepit halllake", val(x(1)) * 4 - 3, val(x(1)) * 4), "\n"))
      u <- tolower(readline(prompt = "What would you like to do? "))
      m <- 2 * (u == "north") * (x(1) < "3") -
        2 * (u == "south") * (x(1) > "2") +
        (x(1) == "2" & u == "west") -
        (x(1) == "3" & u == "east")
      a <- (3 * val(x(2)) + 2 * (x(3) == "2")) * (x(1) == 2 & u == "look chest") +
        (11 + (x(3) == "2")) * (x(1) == "3" & u == "kill dragon") +
        (m != 0) +
        (5 + val(x(1))) * (u == "look") +
        (6 - val(x(3))) * (u == "inventory") +
        (6 - (x(3) == "0")) * (x(1) == 4 & u == "look corpse") +
        10 * (x(1, 3) == "400" & u == "get key") +
        2 * (x(1, 3) == "201" & u == "open chest") +
        10 * (x(1, 3) == "211" & u == "get sword")
      X <- paste0(val(x(1)) + m, val(x(2)) + (a == 2), val(x(3)) + (a == 10), substring(X, 4))
      cat(paste(substr(X, a * 9 + 4, a * 9 + 12), "\n"))
      if (a >= 11) break
  }

This port of the Ten-liner Text Adventure stay as close as possible to the original code by creating two helper functions, val() and x(). In ZX81 basic, you can subset a string with parenthesis, which is mimicked in this code with the x() function. The val() function in ZX81 BASIC converts a string to a value. The rest of the code is almost literally the same as the ZX81 version.

The workings of the game are ingenious and minimise memory usage on the ZX81 to a minimum. All numeric values are stored in the X string to save memory, which also contains all game responses. The first number is the room number the player is in, the second number records whether the chest is closed or open and the third number records the objects held by the player (the inventory). The responses in the X variable are all nine characters long.

In ZX81 BASIC, all lines have numbers to control sequencing. This version uses a repeat{} loop that keeps running until the game ends.

The game opens with displaying the room the player is in. Next, the player is asked to provide a response.

The variable m determines the movement of the player depending on their direction. The result is a number between -2 and 2, which indicates the difference between the current and new room. The code uses a complex boolean expression to work this out.

The variable a determines the outcome after the command if it is not a change in direction. This is the most sophisticated line in the game. In the original listing, it took three lines to determine the new game state due to line-length restrictions. In this code, I concatenated it into one line in line with the games minimalism philosophy. This line combines the input of the player with the current game state to determine the new situation.

In the next step, the code changes the state of the game by updating the three numbers in X. The game displays a response, depending on the content of a.

It takes much more space to describe how this game works than the size of the code itself.

Have fun!

Storytelling with Data: Visualising the Receding Sea Ice Sheets

Peter Prevos — Fri, 14 Aug 2020 00:00:00 +0000

Good data science has to be aesthetic to ensure that the person who consumes your data product draws the right conclusion. Storytelling with data is a craft where mathematics and aesthetics meet. For data science to provide value, it has to be sound, useful and aesthetic. The aesthetics of data science refers to the way the results are communicated. This article discusses an informative but messy graph about the extent of global sea ice. The second part describes how to improve it with the ggplot2 library.

Many data scientists discuss at length which computing language they should learn. I claim that English is the most essential language an analyst can learn (or whatever other language yous peak) and the language of art. A good visualisation is a piece of data art that is composed to achieve a purpose. Whenever somebody looks at your visualisation, you want them to reach the same conclusion as you and they should be able to do so without having to dissect the information.

I subscribe to both the Data is Beautiful and Data is Ugly subreddits. While beauty might be in the eye of the beholder, aesthetic data science has to follow some rules. I subscribe to the concept that you need to maximise the data-pixel-ratio of your graphs. In other words, each pixel in your chart should ideally form part of the story. No superfluous elements, only use colour if it is part of the story. Also, the type of chart is quite important, as each geometry tells a different story. Lets put this to practice with some ggplot code.

This post has been updated based on feedback from Clark Richards as the original used the wrong data set.

Storytelling with data case study: Receding sea ice sheets

A little while ago, one of the posts on the Data is Beautiful subreddit showed a disturbing graph about the changes in the extent of global sea ice from 1978 till recently. The chart is created by an anonymous citizen data scientist who calls him or herself Wipneus, which is Dutch for a snub nose and also a character in an old Dutch comic.

This graph contains all the necessary information to come to the conclusion that the total surface area of global sea ice in the has reduced significantly over the past forty years. This conclusion does, however, require quite a bit of squinting and close examination of the graph.

Global Sea Ice Area 1978–2020.

This graph is a typical multivariate time series where the colour of each line indicates a category. While this approach might work fine for one or two lines, the cacophony of colours makes it hard to distinguish which line belongs to which year. This graph would be impossible to interpret for the eight per cent of men who are colour blind. Also, the story this graph tells is confusing. While the story seems to be that sea ice sheets are melting the past forty years, the chart shows the seasonal variations.

Storytelling with data and ggplot

To redesign this graph, we first need to define what the story is we are telling with this data. The original chart shows that the area of the global sea ice is receding over the past forty years. This means that our dependent variable (y-axis) is the surface area, and the independent variable (x-axis) is the year.

The original graph shows the month as the independent variable. The sea ice sheets grows and shrink with the seasons, as the sinusoidal shape in the chart indicates. While this is an interesting pattern, it is not the story we want to tell with this data. To show the influence over time, the year should be the independent variable and perhaps a colour for each month. To prevent a palette with twelve colours, it is probably better to use a grid for each month. The graph below is my proposed improved version.

The ribbon indicates the maximum and minimum extent of ice for each month over the years. For many months, the maximum amount of ice in recent years is about the same as the minimum amount at the start of this time series. The dark blue line is the linear regression of the mean sea ice area, which clearly indicates the downward trend.

Global Sea Ice Area 1978-2020 for each month of the year.

Good storytelling with data requires that you first define the conclusion you want the consumer of your data product to draw. The next step is to identify the dependent and independent variables and perhaps some grouping as well. Only after you can clearly define these aspects can you choose the best geometry. There are many chart choosers available on the web that help you with this choice.

The code below loads the data from the website and transforms it into a tidy format, calculates the minimum, mean and maximum sea ice area and visualises the results.

  library(dplyr)
  library(forcats)
  library(readr)
  library(ggplot2)
  download.file("https://sites.google.com/site/arctischepinguin/home/sea-ice-extent-area/data/nsidc_global_nt_final_and_nrt.txt.gz", destfile = "global-sea-ice.gz")
  system("gunzip global-sea-ice.gz")
  sea_ice <- read_csv("global-sea-ice", skip = 21)

  ## Original plot
  ggplot(sea_ice, aes(yearday, area, col = format(date, "%Y"))) +
    geom_line() +
    scale_color_discrete(name = "") +
    labs(title = "Global Sea Ice Area",
         subtitle = "from NSIDC NASA Team sea ice concentration data",
         y = expression(Sea~Ice~Area~10^6~km^2))

  ## Ribbon version
  sea_ice_min_max <- sea_ice %>%
    mutate(month = fct_relevel(as.factor(format(date, "%b")), month.abb),
           year = as.numeric(format(date, "%Y"))) %>%
    group_by(year, month) %>%
    summarise(min_area = min(area, na.rm = TRUE),
              mean_area = mean(area, na.rm = TRUE),    
              max_area = max(area, na.rm = TRUE))

  ggplot(sea_ice_min_max, aes(x = year, ymin = min_area, ymax = max_area)) +
    geom_ribbon(fill = "dodgerblue", alpha = .5) +
    scale_y_continuous(breaks = seq(12, 24, 2)) +
    geom_smooth(aes(y = mean_area), se = FALSE, method = "lm") + 
    facet_wrap(~month) +
    labs(title = "Global Sea Ice Area",
         subtitle = "from NSIDC NASA Team sea ice concentration data\nMinimum and maximum sea ice area and linear regression over the mean",
         y = expression(Sea~Ice~Area~10^6~km^2)) +
    theme_light(base_size = 8) +
    theme(panel.background = element_rect(fill = "#f5f5dc"))
  ggsave("../../static/images/data-science/ice-data-beautiful.png", width = 6, height = 4)

Strategic Data Science

If you like to know more about this topic and how to use data science to create business value, perhaps you like to read my book Principles of Strategic Data Science.

Principles of Strategic Data Science

Packt Publishing

Cheesecake Diagrams: Pie Charts with a Different Flavour

Peter Prevos — Wed, 05 Aug 2020 00:00:00 +0000

Part of my job at a regional water utility involves visualising operational data. We manage water and sewerage services for a large number of small and medium-sized towns in regional Victoria (Australia). Traditionally, performance reports consist of extensive tables filled with numbers with a line for each city. To make this data easier to consume, I developed the cheesecake diagram to spatially visualise performance data. A cheesecake diagram is just like a pie chart, but different.

The code for this article is available on GitHub:

Reporting performance spatially

Displaying geographic performance requires geometric objects at appropriate locations. The geographic bubble chart visualises a single quantitative variable through the size of the circles, plus a second qualitative variable through the colour.

The performance data is randomised for the purpose of this example. Performance is measured using four aggregated parameters. My paper on visualising water quality performance describes the algorithm in detail.

The size of the bubbles in the diagram below communicates the consumed volume of water. Due to the orders of magnitude difference in town size, the area is transformed with the square root. The colour of the bubble communicates a random level of performance.

In most traffic-light reporting systems, the colour for negative performance is red and excellent performance is green. This combination is, however, not useful for the 8% of men who struggle to see the difference between red and green. The RColorBrewer package provides several diverging colourblind-safe colour palates.

Visualising a single qualitative value with a bubble chart.

The first part of the code creates mock performance data. The geocode function of the ggmap package provides the latitude and longitude for each location. The consumption data is taken from the 2019 Coliban Water annual report. The data has a row for each location, the total consumption, four performance variables and the average over these four.

The background map comes from Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under ODbL. The second part of the code projects the point geometry on the map and sets the scales and labels. The toner version of the Stamen maps is ideal for this visualisation because of its sparse background.

The next section shows how to split these bubbles into quadrants to separately visualise each variable.

  ## Cheesecake diagram
  library(tidyverse)
  library(ggmap)
  library(RColorBrewer)

  ## Register Google Maps API
  api <- readLines("case-studies/google-maps.api")
  register_google(key = api)

  ## Create mock performance data
  ## Towns with water treatment plants
  towns <- c("Bendigo", "Boort", "Bridgewater", "Castlemaine", "Cohuna", "Echuca", "Elmore", "Goornong", "Gunower", "Heathcote", "Korong Vale", "Kyneton", "Laanecoorie", "Leitchville", "Lockington", "Pyramid Hill", "Rochester", "Serpentine", "Trentham")
  t <- length(towns)

  ## Volume produced
  ## https://www.coliban.com.au/files/2019-10/FINAL_CW_AnnualReport2019_200919pm.pdf p. 24
  consumption <- c(11682, 138, 141, 2064, 610, 3017, 106, 44, 48, 243, 117, 862, 106, 161, 55, 84, 857, 17, 94)

  set.seed(1969)
  performance <- tibble(Town = towns) %>%
      bind_cols(geocode(paste(towns, "Victoria, Australia"))) %>%
      mutate(Consumption = consumption,
             Treatment = sample(0:100, t),
             Network = sample(0:100, t),
             Regulation = sample(0:100, t),
             Perception = sample(0:100, t),
             Performance = round((Treatment + Network + Regulation + Perception) / 4))

  ## Get map
  bbox <- make_bbox(range(performance$lon), range(performance$lat), f = .1)
  map <- get_stamenmap(bbox, maptype = "toner-hybrid")

  ## Single variable
  ggmap(map, extent = "device") + 
      geom_point(data = performance,
                 aes(lon, lat, size = sqrt(Consumption), col = Performance),
                 alpha = 0.9) +
      scale_size_area(max_size = 24, guide = "none") +
      scale_color_gradientn(colors = brewer.pal(7, "RdYlBu")) +
      labs(title = "System Performance",
           subtitle = "Simulated data") +
      theme_void(base_size = 8)

Introducing the Cheesecake Diagram

If we want to report more than one variable per location, the circle needs to he divided into two or more sectors. While this might sound like a pie chart, it is not. A pie chart visualises information through the size of the sectors in the diagram. The colours of the sectors communicate the data categories. Visualisation experts generally discourage using pie charts because they are not easy to interpret, but cheesecake diagrams are different.

A cheesecake diagram visualises information through the size of the circle and the colour of the sectors. The size of the sector is the same for each. A cheesecake diagram is a type of pie chart, but with a different flavour. The cheesecake diagram should not be used for more than four slices to ensure it remains readable.

The Scatterpie package provides functionality to plot pie charts on a map, but it does not allow you to link the colours to the aesthetics. The code below uses the powerful ggforce package to construct the cheesecakes from four circle sectors. To create the sectors, the performance data frame is pivoted, and each sector is defined by its starting and ending angle, being 90 degrees ($\pi/2$ radians). Because the circle now contains information, it is not sized to the level of consumption to keep the sectors clearly visible.

Adding the legend is a bit finicky. The annotation_custom() function allows you to plot two ggplot graphs on top of each other. In this case we need to transform the coordinate systems to match as the map is in latitude /longitude and the legend .

Cheesecake diagram with randomised variables.

  ## Cheesecake diagram
  library(ggforce)
  library(gridExtra)

  ## Convert data
  cheesecake = pivot_longer(performance, -1:-4,
                            names_to = "Aspect",
                            values_to = "Performance") %>%
    filter(Aspect != "Performance") %>%
    mutate(start = rep(seq(0, 2 * pi, length.out = 5)[-5], t),
           end = rep(seq(0, 2 * pi, length.out = 5)[-1], t))

  ## Legend
  cheesecake_legend <- tibble(x0 = c(.1, .1, -.1, -.1),
                              y0 = c(.1, -.1, -.1, .1),
                              start = seq(0, 2 * pi, length.out = 5)[-5],
                              end = seq(0, 2 * pi, length.out = 5)[-1],
                              dimension = unique(cheesecake$Aspect))

  l <- ggplot(cheesecake_legend) +
    aes(x0 = x0, y0 = y0, r0 = 0, r = 1, start = start, end = end) +
    geom_arc_bar(col = NA, fill = "lightgrey") +
    geom_text(aes(x0 * 6, y0 * 6,
                  label = unique(cheesecake$Aspect)), size = 2) + 
    theme_void() +
    coord_equal()

  ## Visualise
  m <- ggmap(map, extent = "device",
             base_layer = ggplot(data = cheesecake,
                                 aes(x0 = lon,
                                     y0 = lat,
                                     r0 = 0,
                                     r = .05,
                                     start = start,
                                     end = end,
                                     fill = Performance))) +
    geom_arc_bar(col = "darkgrey", size = .1) +
    scale_size_area(max_size = 12, guide = FALSE) +
    scale_fill_gradientn(colors = brewer.pal(7, "RdYlBu")) +
    labs(title = "System Performance",
         subtitle = "Simulated data")  +
    theme_void(base_size = 4)

  m + coord_cartesian() + coord_equal() +
    annotation_custom(grob = ggplotGrob(l),
                      xmin = bbox[3] - .3, xmax = bbox[3],
                      ymin = bbox[4] - .3, ymax = bbox[4])

Data Science for Water Utilities

Routledge Amazon

Export WordPress to Hugo Markdown or Org Mode with R

Peter Prevos — Sat, 18 Jul 2020 00:00:00 +0000

I started my first website in 1996 with hand-written HTML. That became a bit of a chore, so for about fifteen years, WordPress became my friend. WordPress has been great to me, but it is slowly becoming a pain to keep updating plugins, security issues, slow performance and the annoying block editor. I am also always looking for additional activities I can do with Emacs. Hugo takes a lot of the pain of managing site away as you can focus on the content and Emacs provides me with powerful editing functionality.

I recently returned to a static website using Hugo. This article explains how to export a WordPress blog to Hugo and customise it with R code. The only reason I used R is because it is the only programming language I know well enough.

You will also need to install the mighty Pandoc software to convert the content to Org mode and the WP All Export WordPress plugin to export your website to a CSV file.

Convert the content to Markdown or Org Mode

The first step is to export the WordPress posts database to a CSV file. Several plugins are available that help you with this task. I have used the WP All Export plugin to export the data. You need to download the ZIP file and install this plugin manually in your WordPress setup. Follow the steps in the All Export plugin and create a CSV file from your posts with at least these fields:

Title
Slug
Date
Content
Categories
Tags

Alternatively, you can link directly to the WordPress database and extract the data with the RMySQL package.

The content files for Hugo are either Markdown or Org Mode. I prefer to use Org Mode as it provides me with access to the extensive functionality that Emacs has to offer, including writing and evaluating R code.

The Content field in the exported CSV file contains HTML code of the article. The code below reads the CSV file and saves each content field as an HTML file, using the post's slug as the filename. The mighty Pandoc software converts this file to Org mode. Any draft posts or pages in the export file will have NA as the file name and are as such skipped.

Now that we have some content, we need to add the Org mode front matter so that Hugo can build a site. The last part of the code generates the front matter for each post, prepends it to the exported Org mode file and cleans some entries.

Copy the code below and save it as wp2org.R. You need to change the filename in the line that starts with file to the name of your export file. The script also creates two subdirectories to store the HTML and Org files.

You run this code with Rscript wp2org.R from the same directory where the CSV file is stored. The result will be a collection of Org mode files.

This new site will not be perfect just yet. To show the images, you need to download your wp-content folder and move it to the static/images folder in Hugo.

The internal links in your blogs will be hard-coded, which means that you need to configure Hugo to ensure your slugs stay the same.

There will be other bits and pieces that might not have adequately converted, so do check your pages.

All you have to do now is to add a theme to your website, and your blog is fully converted. The Hugo website has a great Quick Start page that will get you going.

You can create new posts and edit your content with your favourite text editor. I use Org mode in Emacs to develop this website.

Summary

In summary, you need to take the following steps:

Install pandoc software and WP All Export WordPress plugin.
Download your website as a CSV file with the WordPress plugin.
Copy the R script in a file called wp2org.R and save it in the same location as the CSV file.
Open your console and move to the folder with the script and CSV file
Run Rscript wp2org.R
Review the Org mode files and clean-up any issues

Script

  ## Export WP to Hugo

  ## Read exported WP content
  library(tibble)
  library(readr)
  library(dplyr)
  library(stringr)

  ## Replace the filename with the exported file
  posts <- read.csv("filename", skipNul = TRUE)

  ## Create subdirectories
  if (!dir.exists("tmp")) dir.create("tmp")
  if (!dir.exists("org")) dir.create("org")

  ## Read posts
  for (i in 1:nrow(posts)) {
      ## Save content as temporary html file
      filename <- paste0(posts$Slug[i], ".html")
      writeLines(posts$Content[i], paste0("tmp/", filename))
      ## Convert to Org mode with Pandoc
      pandoc <- paste0("pandoc -o ", paste0("org/", posts$Slug[i],
                                            ".org ", paste0("tmp/", filename)))
      system(pandoc)
  }

  ## Create front matter for all posts
  fm <- tibble(title = paste("#+title:", posts$Title),
               date = paste0("#+date: [", as.POSIXct(posts$Date, origin = "1970-01-01"), "]"),
               lastmod = paste0("#+lastmod: [", Sys.Date(), "]"),
               categories = paste("#+categories[]:", str_replace_all(posts$Categories, " ", "-")),
               tags = paste("#+tags[]:", str_replace_all(posts$Tags, " ", "-")),
               draft = "#+draft: true") %>%
      mutate(categories = str_replace_all(categories, "\\|", " "),
             tags = str_replace_all(tags, "\\|", " "))

  ## Load Hugo files an prepend front matter
  for (f in 1:nrow(posts)) {
      filename <- paste0("org/", posts$Slug[f], ".org")
      post <- c(paste(fm[f, ]), "", readLines(filename))
      ## Repoint images
      post <- str_replace_all(post, paste0("http.*wp-content"), "/images")
      ## Cleanup LaTeX
      post <- str_replace_all(post, "\\$latex ", "$")
      ## Remove remaining Wordpress artefacts
      post <- str_remove_all(post, ':::|\\{.wp.*|.*\\"\\}')
      ## Write to disk
      writeLines(post, filename)
  }

GEDCOM Reader for the R Language: Analysing Family History

Peter Prevos — Sat, 20 Jul 2019 21:15:37 +0000

Understanding who you are is strongly related to understanding your family history. Discovering ancestors is now a popular hobby, as many archives are available on the internet. The GEDCOM format provides a standardised way to store information about ancestors. This article shows how to develop a GEDCOM reader using the R language.

The GEDCOM format

The GEDCOM format is not an ideal way to store information, but it has become the de-facto standard for family history. This format includes metadata and two sets of data. The file contains a list of the individuals, and it lists the families to which they belong.

The basic principle is that each line has a level, indicated by the first digit. At level zero, we find metadata and the individuals and their family. At level one, we see the various types of data, such as births, deaths and marriages. The deeper levels provide the data for these events.

Heiner Eichmann maintains a website that explains the format and provides some examples of files to help you understand the syntax.

The GEDCOM format is not only old in the way it stores data, but it is also limited in the types of human relationships. These files only store genetic relationships between people and assume that these relationships are marriages between a wife and a husband. Human relationships are, however, a lot more complicated than the genetic relationships between children and their parents, grandparents and ancestors.

These issues aside, all genealogical software can export a file to GEDCOM. The next section shows how to create a basic GEDCOM reader using the stringr, tibble and dplyr packages from the Tidyverse.

GEDCOM Reader

The read.gedcom() function takes a GEDCOM file as input and delivers a data frame (tibble) with basic information:

ID
Full name
Gender
Birthdate and place
Father
Mother
Death date and place

This code only can be easily expanded to include further fields by adding lines in the while-loops and including the fields in the data frame.

The first lines read the file and setup the data frame. The extract() function extracts an individual's ID from a line in the file. The for loop runs through each line of the GEDCOM file. When the start of a new individual is detected, the GEDCOM reader collects the relevant information.

Births and christenings are considered equal to simplify the data. In older data, we often only know one or the other. The function looks for the start of a family. It extracts the husband and wife and assigns these as parents to each of the children. The last section cleans the data and returns the result.

  ## Read GEDCOM file for individuals
  library(stringr)
  library(lubridate)
  library(dplyr)

  read_gedcom_ind <- function(gedcom_loc) {
    gedcom <- str_squish(readLines(gedcom_loc))
    idv <- sum(grepl("^0.*INDI$", gedcom))
    fam <- sum(grepl("^0.*FAM$", gedcom))
    cat(paste("Individuals: ", idv, "\n"))
    cat(paste("Families: ", fam, "\n"))
    family <- data.frame(id = NA,
                         Full_Name = NA,
                         Gender = NA,
                         Birth_Date = NA,
                         Birth_Place = NA,
                         Father_id = NA,
                         Mother_id = NA,
                         Death_Date = NA,
                         Death_Place = NA)
    ## Extract data
    extract <- function(line, type) {
      str_trim(str_sub(line, str_locate(line, type)[2] + 1))
    }
    id <- 0
    for (l in 1:length(gedcom)) {
      if (str_detect(gedcom[l], "^0") & str_detect(gedcom[l], "INDI$")) {
        id <- id + 1
        family[id, "id"] <- unlist(str_split(gedcom[l], "@"))[2]
        l <- l + 1
        while(!str_detect(gedcom[l], "^0")) {
          if (grepl("NAME", gedcom[l]))
            family[id, "Full_Name"] <- extract(gedcom[l], "NAME")
          if (grepl("SEX", gedcom[l]))
            family[id, "Gender"] <- extract(gedcom[l], "SEX")
          l <- l + 1
          if (grepl("BIRT|CHR", gedcom[l])) {
            l <- l + 1
            while (!str_detect(gedcom[l], "^1")) {
              if (grepl("DATE", gedcom[l]))
                family[id, "Birth_Date"] <- extract(gedcom[l], "DATE")
              if (grepl("PLAC", gedcom[l]))
                family[id, "Birth_Place"] <- extract(gedcom[l], "PLAC")
              l <- l + 1
            }
          }
          if (grepl("DEAT|BURI", gedcom[l])) {
            l <- l + 1
            while (!str_detect(gedcom[l], "^1")) {
              if (grepl("DATE", gedcom[l]))
                family[id, "Death_Date"] <- extract(gedcom[l], "DATE")
              if (grepl("PLAC", gedcom[l]))
                family[id, "Death_Place"] <- extract(gedcom[l], "PLAC")
              l <- l + 1
            }
          }
        }
      }
      if (str_detect(gedcom[l], "^0") & str_detect(gedcom[l], "FAM")) {
        l <- l + 1
        while(!str_detect(gedcom[l], "^0")) {
          if (grepl("HUSB", gedcom[l]))
            husband <- unlist(str_split(gedcom[l], "@"))[2]
          if (grepl("WIFE", gedcom[l]))
            wife <- unlist(str_split(gedcom[l], "@"))[2]
          if (grepl("CHIL", gedcom[l])) {
            child <- which(family$id == unlist(str_split(gedcom[l], "@"))[2])
            family[child, "Father_id"] <- husband
            family[child, "Mother_id"] <- wife
          }
          l <- l + 1
        }
      }
    }
    family %>%
      mutate(Full_Name = gsub("/", "", str_trim(Full_Name)),
             Birth_Date = as.Date(family$Birth_Date, format = "%d %b %Y"),
             Death_Date = as.Date(family$Death_Date, format = "%d %b %Y"))
  }

Analysing the data

There are many websites with GEDCOM files of family histories of famous and not so famous people. The Famous GEDCOMs website has a few useful examples to test the GEDCOM reader.

Once the data is in a data frame, you can analyse it any way you please. The code below downloads a file with the presidents of the US, with their ancestors and descendants. The alive() function filters people who are alive at a certain date. For people without birth date, it sets a maximum age of 100 years.

The histogram shows the distribution of ages at time of death of all the people in the presidents file.

These are just some random examples of how to analyse family history data with this GEDCOM reader. The next article will explain how to plot a population pyramid using this data. A future article will discuss how to visualise the structure of family history.

Birth years of people of in the Presidents file.

  ## Basic family history statistics
  library(ggplot2)

  presidents <- read_gedcom_ind("https://webtreeprint.com/tp_downloader.php?path=famous_gedcoms/pres.ged")

  mutate(presidents, Year = year(Birth_Date)) %>%
      ggplot(aes(Year)) +
      geom_histogram(binwidth = 10, fill = "#6A6A9D", col = "white") +
      labs(title = "Birth years in the presidents file")

  ggsave("us-presidents-birth-year.png")

Project Euler 144: Laser Beams and Elliptical Billiards

Peter Prevos — Thu, 16 May 2019 00:00:00 +0000

Playing pool at a square table can be hard enough for most people. But what about testing your skills on an oval table? Project Euler 144 investigates multiple reflections of a laser beam. This problem is mathematically equivalent to bouncing billiard balls in an elliptical pool table, demonstrated by Ales Bellow in the video below. This article provides a solution to the Euler problem and simulates playing pool in an elliptical table.

Alex Bellos, Elliptical Pool Table - Numberphile.

Project Euler 144 Definition

In laser physics, a “white cell” is a mirror system that acts as a delay line for the laser beam. The beam enters the cell, bounces around on the mirrors, and eventually works its way back out.

The specific white cell we will be considering is an ellipse with the equation $4x^2 + y^2= 100$. The section corresponding to $-0.01 \leq \times \leq +0.01$ at the top is missing, allowing the light to enter and exit through the hole.

White cell.

The light beam in this problem starts at the point (0.0, 10.1) just outside the white cell, and the beam first impacts the mirror at (1.4, -9.6). Each time the laser beam hits the surface of the ellipse, it follows the usual law of reflection “angle of incidence equals the angle of reflection.”

Both the incident and reflected beams make the same angle with the normal line at the point of incidence. In the figure on the left, the red line shows the first two points of contact between the laser beam and the wall of the white cell; the blue line indicates the line tangent to the ellipse at the point of incidence of the first bounce.

The slope $m$ of the tangent line at any point $(x,y)$ of the given ellipse is $m = -4x/y$. The normal line is perpendicular to this tangent line at the point of incidence.

How many times does the beam hit the internal surface of the white cell before exiting?

Proposed Solution

The first step was to rewrite the equation to use functions to generalise the problem. The general Cartesian equation for an ellipse is:

$$\frac{x^2}{a^2} + \frac{y^2}{b^2} = 1, a < b$$

The length of the axes for this problem is $a =5$ and $b = 10$. While the Project Euler description gives the formula for the tangent slope to the ellipse, I have generalised the code to reuse it for the elliptical billiards table. The slope of the tangent to an ellipse at point $(x,y)$ is:

$$m=-\frac{b^2x}{a^2y}$$

This first code snippet defines functions to draw an ellipse and calculate the bouncing angle. The last part of the code bounces the laser inside the cell until it exits through the top.

The formula to find the intersection between a line and an ellipse has two possible solutions, one of which is the same as the original point.

  plot_ellipse <- function(a, b, colour = NA, line = "black") {
      plot.new()
      plot.window(xlim = c(-a, a), ylim = c(-b, b), asp = 1)
      par(mar = rep(0,4))
      x <- seq(-a, a, length = 200)
      y <- sqrt(b^2 - (b^2 / a^2) * x^2)
      lines(x, y, col = line)
      lines(x, -y, col = line)
      polygon(x, y, col = colour, border = NA)
      polygon(x, -y, col = colour, border = NA)
  }

  bounce <- function(coords) {
      x <- coords$x
      y <- coords$y
      ## Tangent to ellipse
      t <- -(b^2 / a^2) * (x[2] / y[2])
      ## Deflection on sloping mirror y = mx + c
      dydx <- diff(y) / diff(x)
      m <- tan(pi - atan(dydx) + 2 * atan(t))
      c <- y[2] - m * x[2]
      ## Determine intersection point
      ## Source: http://www.ambrsoft.com/TrigoCalc/Circles2/Ellipse/EllipseLine.htm
      x[1] <- x[2]
      y[1] <- y[2]
      x2 <- (-a^2 * m * c + c(-1, 1) * (a * b * sqrt(a^2 * m^2 + b^2 - c^2))) /
            (a^2 * m^2 + b^2)
      x[2] <- ifelse(round(x[1] / x2[1], 6) == 1, x2[2], x2[1])
      y[2] <- m * x[2] + c
      return(data.frame(x, y))
  }

  # Initial conditions
  a <- 5
  b <- 10
  x1 <- 0
  y1 <- 10.1
  x2 <- 1.4
  y2 <- -9.6
  answer <- 0
  plot_ellipse(a, b)
  points(c(0,0), c(-c, c), pch = 19)

  ## Bounce laser breams
  laser <- data.frame(x = c(x1, x2), y = c(y1, y2))
  while((laser$x[2] < -0.01 | laser$x[2] > 0.01) | laser$y[2] < 0) { ## Escape?
      lines(laser$x, laser$y, col = "red", lwd = .5)
      laser <- bounce(laser)
      answer <- answer + 1
  }
  print(answer)

Multiple reflections of a laser beam.

Elliptical Pool Table

We can use the solution to Euler problem 144 to play billiards on an elliptical billiards table. To close the article, we return to the elliptical pool table demonstrated by Alex Bellos. This code draws the pool table to the dimensions mentioned in the video. We know that the table has an eccentricity of $e = 0.43$ and a long axis of $a = 130$ cm. The code defines the short axis ($b$) and the distance of the focal points from the centre.

The code selects a random starting point and angle of the shot. The code first determines whether the line passes through the pocket. If this is not the case, the algorithm then finds the place where the ball hits and keeps bouncing it until it falls into the pocket, or the ball bounces 100 times.

Elliptical pool table simulation.

Elliptical billiard tables have four possible outcomes. Any ball the pass through a focal point will fall into the pocket, ending the simulation. Any ball that passes outside the focal points will bounce around, and the combined trajectories form an ellipse. When the ball moves between the foci, the result is a hyperbola. Lastly, some unique circumstances result in a regular polygon.

If simulations are not enough for you, then head over to the Instructables website to find out how you can construct an elliptical billiards table. There is even a patent for an elliptical pocket billiard table, with the pockets at the edge.

  e <- 0.43
  a <- 130
  b <- a * sqrt((1 + e) * (1 - e)) # a > b
  f <- sqrt(a^2 - b^2)

  plot_ellipse(a, b, "darkgreen", NA)
  points(-f, 0, pch = 19, cex = 2)
  points(f, 0, pch = 19, col = "grey")

  ## Simulate random shot
  angle <- runif(1, 0, 2 * pi)
  x1 <- runif(1, -a, a)
  ymax <- sqrt(b^2 - (b^2 / a^2) * x1^2)
  y1 <- runif(1, -ymax, ymax)

  ## First shot
  m <- tan(angle)
  c <- y1 - m * x1
  x2 <- (-a^2 * m * c + c(-1, 1) * (a * b * sqrt(a^2 * m^2 + b^2 - c^2))) / (a^2 * m^2 + b^2)
  y2 <- m * x2 + c
  x2 <- x2[which(((x2 - x1) < 0) == (cos(angle) < 0))]
  y2 <- y2[which(((y2 - y1) < 0) == (sin(angle) < 0))]
  shot <- (data.frame(x = c(x1, x2), y = c(y1, y2)))

  ## Bounce ball
  for (i in 1:100){
      dydx <- diff(shot$y) / diff(shot$x)
      if (all.equal(dydx, (shot$y[1] - 0) / (shot$x[1] - -f)) == TRUE) {
          shot[2, ] <- c(-f, 0)
      }
      lines(shot$x, shot$y, col = "yellow", lwd = 1)
      if (shot[2,2] == 0) break
      shot <- bounce(shot)
  }
  points(x1, y1, pch = 19, col = "blue", cex = 1.8)

The Secret of Landusia: A Text Adventure in the R Language

Peter Prevos — Sun, 12 May 2019 00:00:00 +0000

In the 1980s and early 1990s, I spent a lot of time playing and writing text adventure games on my Atari 8-bit computer. Text adventures, or interactive fiction, are games where the player reads a story on the screen. The player can change the outcomes of the story by issuing text instructions to the computer. These lines are an extract from Colossal Cave, by Will Crowther and Don Woods, the very first text adventure written in the mid-1970s:

You are standing at the end of a road before a small brick building. Around you is a forest. A small stream flows out of the building and down a gully.
> go south
You are in a valley in the forest beside a stream tumbling along a rocky bed.
> enter stream
Your feet are now wet.

The basic premise of a text adventure is that the player finds herself in a simulated world. Unlike contemporary simulations, this virtual world is created only with text. This might seem a strange idea to modern games, but simulated worlds through text have existed for thousands of years. A text adventure enhances the traditional book by giving the player the freedom to act within the narrative. The objective of these games is to solve problems to reach a specified goal.

This article describes how to write a text adventure in the R language. This article replicates a game and the techniques described by Hal Renko and Sam Edwards in the Dutch book Adventures! (1986).

Cover of the 1986 book Adventures!

Spoiler Alert!

The sections below describe the detailed workings and the narrative of the game. If you don't like spoilers, then perhaps you should download the code and play The Secret of Landusia before reading further.

The Secret of Landusia Text Adventure

This section explains the detailed workings of the Landusia game. The text adventure consists of four files:

adventure.R: The code.
actions.md: Markdown file with descriptions of rooms and actions.
map.csv: Matrix of the relationship between rooms.
objects.csv: Matrix of the status of objects and actors.

The Game Loop

The computer displays the current state of the game with a short piece of prose. The player then enters a command in natural language, which changes the state of the game. When the state of the player or the game reaches a certain point, the player has either won or lost the game. The game loop processes the commands of the player and determines the new state of the game. The state of the game consists of:

Player: The player can have many attributes, such as health, objects they are carrying and their location.
Map: Logic of the game localities and a description of each area.
Objects: Are either placed in a room or are with the player.
Actors: Can act independently.

Text adventure flowchart.

Initialisation

The initialisation defines the vocabulary through a vector of verbs and a vector of directions. The code reads the map, objects and prose into memory and sets the starting variables for the player. The initialisation includes two auxiliary functions that the player cannot call directly.

The prose function takes a key phrase and finds the location of the heading for that key in the actions variable. The following lines of code print the text up to the next heading indicator. The prose function describes rooms, actions, actors and so on. Using a markdown file simplifies writing the text so it is not hidden amongst code. The prose function reads keywords after double hashtags. Other headings are used for convenience.

The inventory function lists all objects that the player carries. This function is a separate verb in most adventure games, but in this game, it is integrated with other verbs.

The Player

The program simulates the player with a set of parameters, such as their location, level of health and whatever other aspect is relevant. A player can, for example, only carry a certain number of objects, can only endure two blows by a sword, and so on.

In the Landusia game, the player has a level of health and maximum carrying capacity. The room variable keeps track of where the player is located.

The Map

The game consists of 28 'rooms'. The map is stored in a matrix which indicates the relationships between the rooms. Each direction can have one or no exit. The map can also dynamically change during the game. Specific directions can, for example, only become available after the player opens a door. A rockfall could close an entrance, and so on. The player can move around the map by issuing directional commands. These commands are the four cardinal compass directions, and up and down.

In Landusia, the first nine rooms are a forest. The blacksmith lives south of the forest. Clovar's castle lies to the south-east and the temple north-east of the forest. The connection between room 14 and 15 opens after the player uses the rope in room 14.

Secret of Landusia map.

Objects

The descriptions of the different rooms of the game can contain objects. These objects are fixed in that their description is hard-coded into the game. A text adventure can also have movable objects. Each object can have its own properties, such as weight, and location. The location of the object can either be a room or the player (room zero). These properties are stored in another matrix.

The Landusia game has four objects: bandage, sword, flute and rope. These objects are scattered around the map and have a certain weight. The capacity variable limits what the layer can carry simultaneously.

Landusia castle battlements.

Actors

Actors are like objects, with the difference that they are active in the game. As the game progresses, actors take independent actions. Actors can move around the map and respond to the actions of the player. They can have properties, such as health and strength. Actors can be either friends or enemies that create a dynamic narrative.

Landusia has four actors which are stored in the same matrix a the objects. Each actor has a name, location, health and status. The actors function controls their actions.

Wizard

The wizard appears in the living quarters. His only task is to give the player the flute, once they manage to reach him. After this one action, the wizard vanishes by changing his location to room 100.

Crow

When the player is in the forest, a random variable determine whether they are swooped by a crow. Another random variable determines whether the player was hurt. If the crow manages to hit the player, the code reduces the health variable.

Crow attacking the player.

Dragon

The dragon sequence depends on the room the player is in. In room 17, the dragon has a 50% chance of hitting, while in room 18, the dragon is always sucessful. The player needs to kill the dragon with the sword (see below). The success of these fights depends on a random variable.

Blacksmith

The blacksmith has the most complex handler of them all. He starts lying wounded on the floor. If the player does not help the blacksmith by using the bandage, he will die. If the player uses the bandage, the blacksmith is healed and move to the workshop to forge a sword for the player. The status variable controls the evolution of the blacksmith in the game.

Command interpreter

The command interpreter asks the player for instructions, which the program then interprets. The state of the game (player, map, objects and actors) is changed, depending on the input. If the state of the game reaches a particular defined state, the game will end either successfully or miserably.

The basic principle of all command translators in interactive fiction is that they analyse the input using regex-type approaches. The interpreter compares the player's instructions with a vocabulary and strips verbs and nouns.

For example, "take lamp" calls the take function and passes it the parameter lamp. The object lamp will then be moved from its current location to the player. The game could also check if the player still has enough carrying capacity and so on.

If the player enters a command that the interpreter cannot parse, then a negative response follows. The better the text adventure, the higher the likelihood that the game understands the command. Traditional text adventures use elaborate command interpreters to maximise the level of realism of the simulated world. This text adventure uses a minimised vocabulary and each of the verbs has its own function.

The dragon of Landusia.

Look

The look function takes a room number as input and displays the prose related to the room. The next section lists the available passages and objects in the room. The final part of the look function displays any objects that the player is carrying.

Take and Put

These two functions allow a player to take an object or place it on the ground. The weight of the object is added or subtracted to the carrying capacity. When the object is not present in the room or not carried by the player, the code responds appropriately.

These functions cannot respond to objects mentioned in the room descriptions or actors, such as trees or the dragon. This can be easily remedied by creating two types of objects, moveable and immovable.

Wait

The wait function does nothing. Waiting might be useful to let other actors do their thing. The wait function has a dummy parameter because there is none, but the main loop assigns one to each verb. This verb is needed to give the blacksmith enough time to forge the sword.

Use

The use function starts by checking whether the player actually has the object in their possession. The remainder of the function contains instructions for each of the four objects. The bandage will either heal yourself or the blacksmith. The sword is not very useful in this context. The flute activates the magical teleportation by changing the room variable. Finally, using the rope changes the map by opening the connection between rooms 14 and 15. This action also changes the prose so that the room descriptions are enhanced.

Kill

This last function is conditional upon having the sword. If the player has no sword, then their fists are the weapon of choice. If an opponent is present in the same room as the player, they have a 60% chance of hitting the opponent. The sword has a higher strength than a fist fight. The actor function controls the return hits.

Expanding the Game

This game is modular in structure so that it should be easy to enhance this game or to write a completely different game. They key to writing a good adventure is simplicity. It is easy to get carried away with writing complex command structure that the player has to guess are available.

Feel free to leave a message below if you find bugs or like to enhance the prose or code.

A lot has changed in computing since the heyday of text adventures. Although these games provided an illusion of freedom to act in a simulated world, the player is always limited by what the writer intended them to do.

Perhaps natural language processing, complex Markov chains and text-generating deep learning methods can generate the ultimate text adventure. Imagine a text adventure where the computer understands the basic structure but freely develops the prose and interactions.

Analyse Site Structure networks with R and igraph

Peter Prevos — Tue, 09 Apr 2019 00:00:00 +0000

One of the essential parts of Search Engine Optimisation (SEO) is a logical site structure. Site structure relates to how the pages of a website link to each other. A well-designed structure of internal links improves the user's experience. It also helps web crawlers to find their way through the website. This article shows how to use the igraph software to analyse site structure.

The Lucid Manager website has been around for ten years now, and the content has grown organically over time, which is a reflection of my changing interests and evolving career. Although I use categories and tags to organise the articles, internal linking between posts is a bit haphazard. This post shows how I extracted the relevant data from the Wordpress database and visualised the internal links with the iGraph package. This analysis shows that my website needs a bit of rework to improve its jungle of internal links to provide you and search engines a better experience.

Site Structure

A website needs to have a logical structure to prevent it from being a loose collecting of articles. Internal links on a website organise the available information through taxonomy, and these links provide context to the text by referring to related pages. The taxonomy consists of menus, breadcrumbs, categories, tags and other structural elements. The body of the text of each article contains the contextual which provide additional or related information to the reader.

Wordpress automatically adds links to categories and tags and the creates an ordered structure. Body text links are created organically and can therefore quickly turn into a chaotic jungle of relationships. Wordpress has plugins that help with site structure, but not of these tools provide a complete overview of the structure in the internal links.

The taxonomy provides an automated structure of the website by categorising articles. The internal linking structure is organic and needs some further consideration. This analysis focuses on the organic links within the body of the text by visualising and analysing the internal linking structure.

Extracting Wordpress Data

The Lucid Manager used to be created with Wordpress. This system stores the body text of all articles in the wp_posts table in the database. To analyse site structure, we need the following fields:

post_name (slug)
post_content (full text)
post_type (page, post or other types)
post_status (use only published posts)

The post type is in most instances either a page or a post, but can also be attachments and so on. The post name (slug) links posts together, and the post content contains the full text of the article in HTML.

Several methods are available to extract the required data from the Wordpress database. The easiest way is to use a plugin such as WP All Export. You can drag-and-drop the required fields or run a SQL query. The second method is to log in to the phpMyAdmin on your cPanel and run the relevant queries.

The third method involves using the RMySQL package in R to directly download data from the site. You will need to create a new user with limited access rights and open the database to your IP address.

The first step reads the data from the database and selects all published posts. The credentials are read from the dbconnect.R file, so I don't publish sensitive data on this website. The table names will be different if you use a Wordpress network. Thanks to 'phaskat' on the Wordpress StackExchange for helping with the queries. Please note that when using a Wordpress network, you will need to replace the wp prefix in the table name with the relevant string.

## Load data (CSV exported from wp_posts table in MySQL database)
library(tidyverse)
library(tidytext)
library(igraph)
library(RMySQL)
library(RColorBrewer)

## Download data using RmySQL
source("dbconnect.R")
## lucidmanager <- dbConnect(MySQL(),
##                          user = "username",
##                          password = "Password1",
##                          dbname = "databe name",
##                          host = "URL")

posts.query <- "SELECT p.post_name, p.post_content, t.name
                FROM wp_posts p
                JOIN wp_term_relationships tr ON (tr.object_id = p.ID)
                JOIN wp_term_taxonomy tt ON (tt.term_taxonomy_id = tr.term_taxonomy_id )
                JOIN wp_terms t ON (t.term_id = tt.term_id)
                WHERE p.post_type='post' AND p.post_status='publish' 
                      AND tt.taxonomy = 'category'"

pages.query <- "SELECT post_name, post_content
                FROM wp_posts
                WHERE post_type='page' AND post_status='publish'"

posts.data <- dbSendQuery(lucidmanager, posts.query)
posts <- fetch(posts.data, n = -1)

pages.data <- dbSendQuery(lucidmanager, pages.query)
pages <- fetch(pages.data, n = -1)

dbDisconnect(lucidmanager)

pages$name = "Page"
content <- rbind(posts, pages)
as_tibble(content)

The result of this code is a data frame with all posts and pages:

post_name: The slug of the post or page.
post_content: Content of the page (HTML file without headers).
name: Post category.

I currently no longer use Wordpress but have moved everything to the Hugo static website generator, using Emacs Org Mode to write content.

Converting the data to a network

The second step uses the tidytext package to convert the texts of the articles into tokens. In this case, a token is any set of characters between spaces. In the default setting, the tokenisation function also splits words at dashes, which is not helpful because we want to preserve the hyperlinks as one token.

The code uses regular expressions to filter all internal links (detect lucidmanager.org) and retain the slug. The slug is the text after the website name. The post_name field in the database contains the slug for the post itself. We thus identify a link from one internal page to another.

These steps result in a table that shows how all slugs link to each other. In network analysis, this is an adjacency list as each line represents a relationship in the network. The category variable identifies the post category, which is helpful to colour the resulting graph. Note that any post without outgoing links will not appear in this list.

This code also works with any data frame that contains the slug and content of the web pages as fields. The category variable is optional. Replace lucidmanager.org with the URL of the website you are analysing.

  links <- content %>%
    unnest_tokens(word, post_content,
                  token = stringr::str_split, pattern = " ") %>%
    filter(grepl(".+lucidmanager.org/", word)) %>%
    mutate(link = gsub(".+lucidmanager.org/", "", word),
           link = gsub("/.+", "", link)) %>%
    filter(link != "tag" & link !=  "wp-content" & link != "author") %>%
    select(category = name, post_name, link)

The resulting adjacency list will look something like this:

category	post_name	link
RStats	analyse-site-structure	export-wordpress-to-hugo
RStats	export-wordpress-to-hugo	analyse-site-structure

Extract from adjacency list with two pages referring each other.

Analyse Site Structure with iGraph

Network analysis is the perfect tool to analyse site structure because each post on the website is a node and a link between two articles is a graph edge (arrow).

The Lucid Manager currently discusses strategic and fun data analysis, but in the past, I wrote articles about water utility marketing and critical perspectives on management theories. The questions I like to answer with this analysis are:

Are all pages connected to each other in one or more steps?
What is the most linked page?
Which pages link to themselves?
Are there duplicated links within one post?

The iGraph software transforms the link table to a network. Any posts and pages without any links (solitary pages) are added separately. Each post is also given a colour related to its category, and pages are a separate colour.

This analysis shows that the internal link structure contains several sub networks, which are groups of pages that are only linked to each other. This graph will help in to identify more linking opportunities to dissolve sub networks.

The degree function in iGraph determines the number of adjacent edges of each graph. The degree function can specify incoming and outgoing links. My article about service quality in water utilities is the most linked post on this site. In network analysis, this is the node with the highest degree of linking.

The loops in the diagram indicate self-referencing articles. The which_loop function identifies which arrows have the same start and end. The E function shows the list of edges, which shows that four pages on my website refer to themselves.

Lastly, the which_multiple function identifies any duplicated edges in the graph. Within this context, these are pages that contain the same link more than once. The Lucid Manager website had seventeen instances of duplicated links when I first analysed its structure.

This first phase gives me plenty of homework to improve the internal linking structure of this website.

Visualising website structure. (click to enlarge)

  solitary <- c(posts$post_name, pages$post_name)[!(c(posts$post_name, pages$post_name) %in%
                                                    unique(c(links$post_name, links$link)))]
  network <- select(links, post_name, link) %>%
    graph_from_data_frame(directed = TRUE) + 
    vertices(solitary)

  colour <- tibble(post_name = V(network)$name) %>%
    left_join(content) %>%
    mutate(name = factor(name),
           colour = brewer.pal(length(unique(content$name)), "Set2")[name]) %>%
    select(-post_content)

  V(network)$color <- as.vector(colour$colour)

  par(mar = rep(0, 4))
  plot(network,
       layout = layout.fruchterman.reingold,
       vertex.label.cex = .7,
       vertex.size = degree(network),
       vertex.label.color = "black",
       vertex.frame.color = NA,
       edge.arrow.size = .2,
       edge.color = "darkgray")

  ## Diagnose network
  which.max(degree(network, mode = "in"))

  E(network)[which_loop(network)]

  E(network)[which_multiple(network)]

Future improvements

This analysis is a great step to start systematically analysing organic website structure. Several other techniques help to understand the structure of a website further. Community detection and the inclusion of tags and categories in the data can provide additional context. SEO experts might find this concept useful. Perhaps somebody else can develop a Wordpress plugin to provide this visualisation.

Decode Lyrics in Pop Music with the Songsim algorithm

Peter Prevos — Sat, 23 Mar 2019 03:14:17 +0000

Music is an inherently mathematical form of art. Ancient Greek mathematician Pythagoras was the first to describe the logic of the scales that form melody and harmony. Numbers can also represent the rhythm of the music. Even the lyrics have a mathematical structure. Poets structure syllables and repeat words to create pleasing sounding prose. This article shows how to decode lyrics from pop songs and visualise them using the Songsim method to analyse their metre.

Decode Lyrics using the Songsim algorithm

Data visualiser, pop music appreciator and machine learner Colin Morris has extensively analysed the repetitiveness of song lyrics. Colin demonstrated that lyrics are becoming more repetitive since the early days of pop music. The most repetitive song is Around the World by Daft Punk, which should not be a surprise since the artist repeats the same phrase 144 times. Bohemian Rhapsody by Queen has some of the least repetitive lyrics in popular music.

The TedX presentation (see below) by Colin Morris shows how he visualises the repetitiveness of song lyrics with what he calls the Songsim algorithm. As a result, the more points in the graph, the more often the composer repeated a word

Visualisation of the lyrics of Daft Punk's 'Around the World' and Queen's 'Bohemian Rhapsody'.

The visual language of song lyrics

Morris decided to use a self-similarity matrix, which biologists use to visualise DNA sequences, to decode lyrics. In this method, the individual words of the song are the names of the columns and the names of the rows in a matrix. For every point in the song where the row name equals the column name, shows a dot. By definition, the diagonal of every similarity matrix is filled. The timeline of the song thus runs along the diagonal from top left to bottom right.

Patterns away from the diagonal represent two different points in time that have the same words. The more of these patterns we see, the more repetitive a song is. Let's demonstrate this with the first words ever recorded by Thomas Edison in 1877.

Original Edison 1877 tin foil recording.

Mary had a little lamb, whose fleece was white as snow. And everywhere that Mary went, the lamb was sure to go.

The similarity matrix below visualises the two first sentences of the famous nursery rhyme. It shows where the words "Mary", "lamb" and "was" are repeated once.

Self-similarity matrix for Mary had a Little Lamb by Thomas Edison.

The snowflake diagrams are a visual language to decode lyrics. The verses are the gutters with only diagonal lines. A verse is not very repetitive besides some stop words. The verse repeats through the song. Many songs have a bridge that contrasts with the rest of the song. As a result, the bridge is in most songs a unique pattern with self-similarity.

The diagram below visualises the lyrics of one of the most famous pop songs ever, Waterloo by Abba. The first 30 words are the opening verse, which shows little repetition, other than stop words such as and the pronoun I. After that we see diagonal lines appearing that represent the repetitive use of the song title. Towards the end of the song, we see the bridge, which is like a little snowflake within the diagram.

Decoding lyrics: Waterloo by Abba.

The next section shows how to implement this approach with ggplot, scraping pop song lyrics from the azlyrics.com website.

Pop Music is Stuck on Repeat | Colin Morris | TEDxPenn

Implementing Songsim with ggplot

The code below visualises song lyrics or poetry as suggested by Colin Morris. The code uses four libraries. I use the tidyverse series of libraries because it makes life very easy. The tidytext library uses the tidyverse principles to analyse text. The old reshape2 library helps to transform a matrix, and lastly, rvest helps to scrape song lyrics from the azlyrics website.

The first function scrapes song lyrics from the azlyrics website using the artist and song as input. The first three lines clean the artist and song variables. This code removes any character that is not a number or a letter, converts to lowercase and lastly removes the definite article in the artist name. These two fields are then concatenated to create the URL, which the function prints. The remainder of the code scrapes the lyrics from the website or trips on an error 404 when it cannot find the song/artist combination.

The second function implements the Morris method to visualise the lyrics. The code extracts single words from the text and places them in a data frame (tibble). This data frame is subsequently converted to a boolean matrix that contains the visualisation.

The code looks at each word and places the value TRUE where reappears in the song. Each of the vectors is then concatenated to a matrix. Lastly, ggplot visualises the matrix is visualised as a raster.

What does your favourite song look like a snowflake diagram?

  ## Decoding lyrics
  library(tidyverse)
  library(tidytext)
  library(reshape2)
  library(rvest)

  get_lyrics <- function(artist, song) {
      artist <- gsub("[^A-Za-z0-9]+", "", tolower(artist))
      song <- gsub("[^A-Za-z0-9]+", "", tolower(song))
      artist <- gsub("^the", "", artist)
      url = paste("http://azlyrics.com/lyrics/", 
                  artist, "/", song, ".html", sep = "")
      print(url)

      azlyrics <- read_html(url)
      lyrics <- html_nodes(azlyrics, "div")
      lyrics <- html_text(lyrics[23])
      gsub("\r|\n", " ", lyrics)
  }

  plot_snowflake <- function(artist, song){

      lyrics <- get_lyrics(artist, song)
      lyrics <- data_frame(line = lyrics) %>%
          filter(line != "")

      words <- lyrics  %>%
          unnest_tokens(word, line) 
      words_matrix <- lapply(1:nrow(words),
                             function(w){
                                 as.character(words[w, 1]) == words
                             }
                             ) %>%
          do.call(cbind, .)
      rownames(words_matrix) <- 1:nrow(words)
      colnames(words_matrix) <- 1:nrow(words)
    
      melt(words_matrix, varnames = c("x",  "y")) %>%
          ggplot(aes(x, -y, fill = value)) +
          geom_raster() +
          scale_fill_manual(values = c("white", "dodgerblue4"), guide = FALSE) +
          theme_void() +     
          ggtitle(artist, subtitle = song)
  }


  plot_snowflake("Abba", "Waterloo")
  ggsave("Abba-Waterloo.png")

  artist = "Thomas Edison"
  song <- "Mary Had a Little Lamb"
  lyrics <- "Mary had a little lamb, whose fleece was white as snow. And everywhere that Mary went, the lamb was sure to go."

  library(gridExtra)
  png("DaftPunk-Queen.png", width = 1024, height = 768)
  l1 <- plot_snowflake("Daft Punk", "Around the world")
  l2 <- plot_snowflake("Queen", "Bohemian Rhapsody")
  grid.arrange(l1, l2, ncol = 2)
  dev.off()
  getwd()

  artist <- "Frank Zappa"
  song <- "Titties Beer"

Strategic Data Science: Create Value from Data Big and Small

Peter Prevos — Sun, 24 Feb 2019 00:00:00 +0000

Data science is without a doubt the most popular business fad of the past decade. The wonderful promises of machine learning blind many managers, so they forget about deploying these new approaches strategically. To create value from data, managers need to use a strategic approach to data science. This article provides a framework for data science, following my book Principles of Strategic Data Science, published by Packt.

Principles of Strategic Data Science

Packt Publishing

What is Data Science?

The manager to understand the importance of data was Frederick Taylor. He was a pioneer in using data to manage a business. He used data because he wanted to lower the influence of opinions and rules-of-thumb in favour of a scientific approach to management. The term data science emerged in the middle of the last century when electronic computation first became a topic of study. In those days, the discipline was literally a science of storing and manipulating data.

The current definition has drifted away from this initial academic activity to a business activity. The current data science hype can be traced back to an article in the 2012 edition of Harvard Business Review. Davenport and Patil proclaimed data scientist to be the sexiest job of the twenty-first century. Subsequently, in the wake of this article, the number of data science searches in Google increased rapidly.

Organisations have for a long time used data to improve the lives of their customers, shareholders or society overall. Management gurus promoted concepts such as the data-driven organisation, evidence-based management, business intelligence and Six Sigma to help businesses realise the benefits of their data. Data science is an evolution of these methods enabled by the data revolution.

The Data Revolution

Recent developments in information technology have significantly improved what we can do with data, resulting in what we now know as data science. Firstly, most business processes are managed electronically, which has exponentially increased the amount of available data. Developments in communication, such as the Internet of Things and personal mobile devices, have significantly reduced the price of collecting data.

Secondly, the computing capabilities on the average office worker's desk outstrip the capabilities of the supercomputers of the past. Not only is it cheaper to collect vast amounts of electronic data, but processing these enormous volumes has also come within reach of the average office worker.

Lastly, developments in applied mathematics and open source licensing have accelerated our capabilities in analysing this data. These new technologies allow us to discover patterns that were previously invisible. Data science is available to anyone wanting to learn because most tools to examine data are freely available on the internet with a helpful community sharing knowledge on how to use them.

These three developments enabled an evolution from traditional business analysis to data science. Subsequently, data science has become the strategic and systematic approach to analysing data to achieve organisational objectives using electronic computing. This definition is agnostic of the promises of machine learning and leverages the three developments mentioned above. Data science as such is the next evolution in business analysis that maximises the value we can extract from data.

Data Science Strategy Competencies

The craft of data science combines three different competencies. To illustrate this idea, data scientist Drew Conway visualised these core competencies of data science in a Venn diagram.

Data science competencies (Based on Conway, 2009).

Firstly and most importantly, data science requires domain knowledge. Any analysis needs to be grounded in the reality it seeks to improve. Subject-matter expertise is a necessary condition to make sense of an investigation. Professional expertise in most areas uses mathematics to understand and improve outcomes. New mathematical tools expand the traditional approaches to develop a deeper understanding of the domain under consideration. Lastly, computer science is the competency that binds the available data with mathematics. Writing computer code to extract, transform and analyse data to create information and stimulate knowledge is an essential skill for any data scientist.

Good Data Science

In order to create value with data, we need to know how to create or recognise good data science. The second chapter of the book uses three principles originally introduced two thousand years ago by the Roman architect and engineer Vitruvius. He wrote that buildings need to be useful, sound and aesthetic. These requirements are also ideally suited to define best-practice in data science.

The Vitruvian triangle for data science.

Usefulness

For data science to be useful, it needs to contribute to the objectives of an organisation positively. It is in this sense that data science is an applied science and not an academic pursuit. The famous Data-Information-Knowledge pyramid visualises the process of creating value from data.

Useful data science meaningfully improves our reality through data. Data is a representation of either a social or physical reality. Any data source is ever only a sample of the fullness and complexity of the real world. Information is data imbued with context. The raw data collected from reality needs to be summarised, visualised and analysed so that managers understand the reality of their business. This information increases knowledge about a business process, which is subsequently used to improve the reality from which the data was collected. This feedback loop visualises the essence of analysing data in businesses. Data science is a seductive activity because it is reasonably straightforward to create impressive visualisations with sophisticated algorithms. If data products don't improve or enlighten the current situation, they are in essence useless.

Reality - Data - Information - Knowledge pyramid.

Soundness

Data science needs to be sound to ensure that the outcomes are valid and reliable. The validity and reliability of data are where the science meets the traditional approaches to analysing data. Validity is the extent to which the data represents the reality it describes. The reliability of data relates to the accuracy of the measurement. These two concepts depend on the type of data under consideration. Measuring physical processes is less complicated than the social aspects of society. Validity and reliability are in essence a sophisticated way of expressing the well-known Garbage-In-Garbage-Out principle.

Reliability and validity of data and analysis.

The soundness of data science also relates to the reproducibility of the analysis to ensure that other professionals can review the outcomes. Reproducibility prevents that the data and the process by which it was transformed and analysed become a black-box where we have no reason to trust the results. Data science also needs to be sound concerning the governance of the workflow. All data sources need to be curated by relevant subject matter experts to ensure their validity and reliability. Data experts provide that the data is available to those who need it.

Aesthetics

Lastly, data science needs to be aesthetic to ensure that any visualisation or report is easy to understand by the consumer of the analysis. This requirement is not about beautification through infographics. Aesthetic data products minimise the risk or making wrong decisions because the information is presented without room for misinterpretation. Any visualisation needs to focus on telling a story with the data. This story can be a comparison, a prediction, a trend or whatever else is relevant to the problem.

Example of low and high data-to-pixel ratio.

One of the essential principles of aesthetic data science is the data-to-pixel ratio. This principle means that we need to maximise the ratio between all the pixels on a screen and those pixels that present information. Good data visualisation practices austerity to ensure that the people that consume the information understand the story that needs to be told.

Strategic Data Science

The data science continuum is a strategic journey for organisations that seek to maximise value from data. As an organisation moves along the continuum, increased complexity is the payoff for increased value. This continuum looks like hierarchy, but all phases are equally important. The last stages cannot exist without the previous ones. Therefore, implementing machine learning without a solid grasp of data collection will result in useless outcomes.

Data science strategy continuum.

Collecting data requires considerations on what to collect, how to collect it and at what frequency. To collect meaningful data requires a good understanding of the relationship between reality and data. There is no such thing as raw data as all information relies on assumptions and practical limitations.

Type of Value from Data

Describing the data is the first step in extracting value.
Descriptive statistics are the core of most business reporting and are an essential first step in analysing the data.
Diagnostics or analysis is the core activity of most professions. Each subject area uses specialised methods to create new information from data.
Predictive analysis seems to be the holy grail for many managers. A prediction is not a perfect description of the future but it provides the distribution of possible futures. Managers can use this information to change the present to construct their desired future.
Prescriptive analysis uses the knowledge created in the previous phases to automatically run a business process and even decide on future courses of action.

Any organisation starting with data science should follow the five phases in this process and not jump ahead to try to bypass the seemingly less valuable stages.

The Data-Driven Organisation

Implementing a data science strategy is more than a matter of establishing a specialised team and solve complex problems. Creating a data-driven organisation that maximises the value of data requires a whole-of-business approach that involves people with the right attitude and skills, appropriate systems and robust processes.

A data science team combines the three competencies described in the Conway Venn diagram. People that have skills in all three of these areas are rare, and the industry calls them unicorns. There is no need for recruiters to start hunting unicorns because these three areas of expertise can also exist within a team. Possibly more important than the technical skills are the social skills of a data scientist. Not only need they create useful, sound and aesthetic data science, they also need to convince the consumers of their work of its value.

Data Literacy

One of the problems of creating value with data is ensuring that the results are implemented in the organisation. A starting point to achieve this is to ensure that the users of data products have a relevant level of data literacy. Developing data literacy among consumers of data science is probably one of the greatest challenges. The required level of data literacy depends on the type of position and the role of the data consumer within the organisation.

Data scientists use an extensive range of tools and are often opportunistic in their choice of software. Spreadsheets are not very suitable to create good data science. Data science requires coding skills and the Python and R languages are powerful tools to solve complex problems. After the data specialists have developed the best way to analyse data, they need to communicate these to their customers. Many specific products exist to communicate data to users with interactive dashboards and many other dynamic systems.

The final part of this book on data science strategy delves into the ethics of data science. From the fact that something can be done, we certainly cannot conclude that it should be done. Just like any other profession that impacts humans, data scientists need ethical guidelines to ensure that their activities cause no harm. The book provides some basic guidelines that can assist data scientists to assess the ethical merits of their projects.

Summary

In summary, data science is about more than using the latest algorithms to analyse data. Good data science has to be useful in that it positively influences the objectives of the organisation. Furthermore, good data science has to be sound through a methodical approach.

My books Principles of Strategic Data Science and Data Science for Water Utilities provide detailed insights into these topics.

Principles of Strategic Data Science

Packt Publishing

Data Science for Water Utilities

Routledge Amazon

Factor Analysis in R: Measuring Consumer Involvement

Peter Prevos — Sat, 19 Jan 2019 00:00:00 +0000

The first step for anyone who wants to promote or sell something is to understand the psychology of potential customers. Getting into consumers' minds is often problematic because measuring psychological traits is complex. Consumer involvement is a measure of people's attitudes towards a product or service. This article introduces the concept of consumer involvement. An example using data from tap water consumers illustrates the theory. This article analyses data collected from these consumers using factor analysis in R, with the psych package.

The most common method to measure psychological traits is to ask people several questions. Analysing this data is complicated because it is difficult to determine how the survey responses relate to the software of the mind. While the answers given by survey respondents are the directly measured variables, we like to know the hidden (latent) states in their minds. Factor Analysis is a technique that identifies latent variables within a data set, such as a customer survey.

The basic principle of measuring consumer attitudes is that their state of mind causes them to respond in a certain way. Factor analysis reverses this causality by analysing patterns in responses that indicate the consumer's state of mind. Using a computing analogy, factor analysis is a technique for reverse-engineering source code by analysing the input and output.

The data and code for this article are available on GitHub in the case-studies folder:

What is Consumer Involvement?

Involvement is a marketing metric that describes the relevance of a product or service in somebody's life. Judy Zaichkowsky defines consumer involvement formally as “a person's perceived relevance of the object based on inherent needs, values, and interests”. People who own a car will most likely be highly involved with purchasing and driving the vehicle due to the money involved and the social role it plays in developing their public self. Consumers will most likely have a much lower level of involvement with the instant coffee they drink than with their clothes.

Managerial Relevance

The level of consumer involvement depends on a complex array of factors. These factors relate to psychology, situational factors, and the service provider's marketing mix. The lowest level of involvement is considered a state of inertia which occurs when people habitually purchase a product without comparing alternatives.

From a managerial point of view, involvement is crucial because it is causally related to willingness to pay and perceptions of quality. Consumers with a higher level of involvement are willing to pay more for a service and have a more favourable perception of quality. Understanding involvement in the context of urban water supply is also crucial because sustainably managing water as a common pool resource requires the active involvement of all users.

Cult products have the highest possible level of involvement as customers are fully devoted to a particular product or brand. Commercial organisations use this knowledge to their advantage by maximising consumer involvement through branding and advertising. This strategy is used effectively by the bottled water industry. Manufacturers focus on enhancing the emotional aspects of their products rather than on improving the cognitive elements. Water utilities tend to use a reversed strategy and emphasise the cognitive aspects of tap water, the pipes, plants and pumps rather than trying to create an emotional relationship with their consumers.

Measuring Consumer Involvement

For my dissertation about customer service in water utilities, I measured the level of involvement that consumers have with tap water using the Personal Involvement Index (PII).

Customer Experience Management for Water Utilities: Marketing Urban Water Supply

Practical framework for water utilities to become more focused on their customers following Service-Dominant Logic.

IWA Publishing

Asking consumers directly about their level of involvement would not yield a stable answer, as each respondent would interpret the question differently. The best way to measure psychological states or psychometrics is to ask a series of questions linguistically related to the topic of interest.

The most cited method for measuring consumer involvement is the Personal Involvement Index, developed by Judy Zaichowsky. This index is a two-dimensional scale consisting of the following:

Cognitive involvement (importance, relevance, meaning, value, and need)
Affective involvement (involvement, fascination, appeal, excitement, and interest).

Personal Involvement Inventory measurement model.

The survey instrument consists of ten Semantic Differential items. A Semantic Differential is a rating scale designed to measure the meaning of objects, events, or concepts. The researcher translates the concept, such as involvement, into a list of synonyms and their associated antonyms.

In the involvement survey, participants position their views along a continuum, such as Worthless-Valuable or Boring-Interesting. The level of involvement is the sum of all answers, ranging from 10 to 70.

Personal Involvement Inventory semantic-differential scale (Zaichowsky 1994).

Obtaining the Data

The data form part of a survey of tap water customers.

  ## Consumer Involvement
  library(tidyverse)
  library(psych)

  # Clean data
  customers <- read_csv("data/customer_survey.csv")[-1, ] %>%
    type_convert() %>%
    filter(is.na(term)) %>%
    select(c(1, 21:51, -33)) %>% 
    rename(customer_id = 1)

The scale uses reverse polarity as some of the items are from low to high (boring – interesting) and others from high to low (important – unimportant), so we need to correct this by reversing the scores for six items. The data also has missing values, so we only use the complete cases.

# Correct polarity

pii <- select(customers, customer_id, starts_with("p")) %>%
  mutate(p01 = 8 - p01,
         p02 = 8 - p02,
         p07 = 8 - p07,
         p08 = 8 - p08,
         p09 = 8 - p09,
         p10 = 8 - p10)

# Remove missing values

pii <- pii[complete.cases(pii), ]

Exploratory Analysis

This data set contains additional information, and the code selects only variable names that start with "p" (for the Personal Involvement Inventory). Before we analyse data, we remove customers who provided the same answers to all items or did not respond to all questions. These responses are most likely invalid, which leaves 757 rows of data.

A boxplot is a convenient way to visualise responses to multiple survey items in a single chart. This plot immediately shows an interesting pattern in the answers. Responses to the first five items were generally higher than those for the last five. This result indicates a distinction between cognitive and affective involvement.

    # Visualise PII

    pii_long <- pii %>%
      pivot_longer(-customer_id, names_to = "Item", values_to = "Response")

    ggplot(pii_long) +
      aes(Item, Response) +
      geom_boxplot(fill = "#f7941d") +
      theme_bw(base_size = 12) + 
      labs(title = "Personal Involvement Index",
           subtitle = paste0("Tap Water Consumers USA and Australia (n = ",
                            nrow(pii), ")"))

Responses to Personal Involvement Index by tap water consumers.

The next step in the exploratory analysis is to investigate how these factors correlate. The correlation plot below shows that all items strongly correlate with each other. As shown in the boxplots above, the first five and the last five items correlate more strongly. This plot suggests that the two dimensions of the involvement index correlate. The following section shows how to use factor analysis in R to check the significance of these correlation patterns.

  # Visualise correlation matrix
  c_matrix <- cor(pii[, -1])

  library(ggcorrplot)
  
  ggcorrplot(c_matrix, outline.col = "white") +
    labs(title = "Personal Inventory Index",
         subtitle = "Correlation Matrix")

Correlation matrix for the Personal Involvement Index.

Factor Analysis in R

Researchers often confuse Factor Analysis with Principal Component Analysis. The outcomes are very similar when applied to the same data set. Both methods are similar but serve different purposes. Principal Component Analysis is a data-reduction technique that reduces the number of variables in a problem. The specific purpose of Factor Analysis is to uncover latent variables. The mathematical principles for both methods are similar but not identical, and they should not be confused.

One of the most crucial decisions in factor analysis is to decide how to rotate the factors. There are two types: orthogonal and oblique. In simple terms, orthogonal rotations reduce the correlation between dimensions, whereas oblique rotations allow dimensions to relate to each other. Because of the strong correlations in the correlation plot and the fact that both dimensions measure involvement, this analysis uses oblique rotation. The visualisation below shows how each item and the two dimensions relate to one another.

  library(psych)
  pii_fa <- fa(pii[, -1], nfactors = 2, rotate = "oblimin", fm = "ml")
  fa.diagram(pii_fa, main = NULL, marg = c(0, 8, 0, 0))

Factor analysis in R with Psych package.

Consumer Involvement Scores

This analysis suggests that the items for the PII measure an underlying construct, which we can call consumer involvement. To work out the scores for this measure, we can sum the items for each respondent.

  # Calculating the PII

  pii_scores <- pii %>%
    mutate(cognitive = p01 + p02 + p03 + p04 + p05,
           affective = p06 + p07 + p08 + p09 + p10) %>%
    select(customer_id, cognitive, affective) %>%
    pivot_longer(cols = -customer_id)

  ggplot(pii_scores) +
    aes(value) +
    geom_histogram(fill = "dodgerblue 4", binwidth = 1) +
    facet_wrap(~name) +
    theme_minimal(base_size = 12) +
    labs(title = "Consumer Involvement with Tap Water",
         subtitle = "Personal Involvement Index",
         x = "Score", y = NULL)

Personal Involvement Index for tap water.

This simple factor analysis in R shows the basic principle of analysing psychometric data. The psych package has many more specialised tools to dig deeper into the information. This article has not assessed this construct's validity or evaluated the factors' reliability. That may be for a future article.

Chapter 8 in Data Science for Water Utilities explains the principles of analysing survey data to understand the customer experience in more detail.

Data Science for Water Utilities

Routledge Amazon

5½ Reasons to Ditch Spreadsheets for Data Science

Peter Prevos — Tue, 11 Dec 2018 00:00:00 +0000

When I studied civil engineering some decades ago, we solved all our computing problems by writing code. Writing in BASIC or PASCAL, I could quickly perform fundamental engineering analysis, such as reinforced concrete beams, with my home-brew software library. Soon after I started my career, spreadsheets became widely available, and I fully embraced this fantastic business tool, first Lotus 123 and later grudgingly moved to MS Excel. This article describes why you should ditch spreadsheets for data science and start writing code in R or Python instead. Computer code is Poetry, spreadsheets are chaos.

Screendump of Atari BASIC program to estimate concrete reinforcement surface area.

Spreadsheets were excellent in those early days because data, code, visualisations and tabular output are all stored in one convenient file. Creating graphs with computer code was a bit of a nightmare in those days, so spreadsheets were a minor miracle. The next twenty years, I must have created thousands of spreadsheets of varying complexity. I even developed a ‘jungle' of interlinked spreadsheets to manage progress reporting.

In the pioneering days of spreadsheets, they provided enormous convenience for engineers and other professionals to quickly develop analytical tools. But after using this tool for a few years, cracks started to appear.

Meet the inventor of the spreadsheet | Dan Bricklin

Spreadsheets for Data Science are Chaos

Throughout my career, I had many nightmarish experiences trying to reverse engineer spreadsheets, even the ones I wrote myself. The combination of data, code and output that I loved at the start of my career had reached its limits. Spreadsheets use incomprehensible names for variables (AZ346, XC89 and so on) and the formulas are impossible to read because all code is cramped on one line with deeply nested logic. The multiple parentheses make Excel formulas are even harder to read than LISP expressions.

Furthermore, spreadsheets hide the formulas behind the results, which renders spreadsheets notoriously hard to understand. My love affair with the spreadsheet came to an end when I started writing my dissertation about customer service for water utilities. Excel was incapable of helping my need for solving complex mathematical problems. A colleague suggested I look into this new thing called 'Data Science' and this advice changed my career.

My focus is to implement strategic data science to help organisations to create value from data. One of the ways to achieve this goal is to ditch the spreadsheet and start writing code instead.

Example of a reverse-engineering a spreadsheet.

Code is Poetry

I decided to learn how to write code in the R language for statistical analysis. The R language is like a Swiss army chainsaw for engineers with capabilities that far exceed anything a spreadsheet can do. Writing in code, such as R or Python, is like writing an instruction manual on how to analyse data. Anyone who understands the language will be able to know how you derived your conclusions. Modern data science languages can generate print-quality visualisations and can output results in many formats, including a spreadsheet. In my job as a data science manager for a water utility, I use R code for everything.

The awesome power of being able to easily combine large data sets, visualise data and undertake complex analysis. Now that I have rediscovered the poetry of writing computer code, I advocate learning to use R or perhaps Python and ditch the spreadsheet. On my this website, I share examples of creating value and having fun with the R language. Computer code is poetry in the sense that you can read it like a poem. Computer code consists of a sequential logic that is easy to follow. A spreadsheet is a chaotic jumble of formulas in a geometrical pattern. Each spreadsheet has a different pattern. The only purpose I still have for spreadsheets is as an interface for small data sets.

5½ Reasons to Ditch the Spreadsheet

If you are still using spreadsheets, or you are trying to convince a colleague to ditch this tool, here are 5½ reasons to start using code to analyse data:

Good analysis is reproducible and can be peer-reviewed. Spreadsheets are hard to understand because of non-sequential references. Computer code is like an instruction book that can be read step-by-step.
Spreadsheet variables are hard to understand (e.g. ZX81:ZX99). In computer code, you give them meaningful names (e.g. sales[81:99]).
Best practice in data management is to separate data, code and output. In spreadsheets, it is not immediately clear which cell is the result of another cell and which ones are raw data. Computer code separates the data from the code and the output.
You can only share spreadsheet output with people who have access to the relevant software package. Computer code can produce output in multiple formats, such as HTML, PD or even Excel, including interactive dashboards.
Functionality in spreadsheets is limited to what is made available by Microsoft. The R and Python languages are extendable and have extensive libraries to solve complex problems.

The bonus reason to ditch the spreadsheet is that the best data science software, such as R and Python, is Open Source and freely available on the web. No license fees and it comes with terrific community support.

Feel free to leave a comment if you like to defend the spreadsheet. If you have additional reasons to ditch this venerable but largely obsolete tool. Subscribe to this monthly blog if you are interested in using the R language for practical data science and some fun.

Principles of Strategic Data Science

Packt Publishing

Discourse Network Analysis: Undertaking Literature Reviews in R

Peter Prevos — Thu, 15 Nov 2018 00:00:00 +0000

Literature reviews are the cornerstone of science. Keeping abreast of developments within any given field of enquiry has become increasingly difficult given the enormous amounts of new research. Databases and search technology have made finding relevant literature easy but, keeping a coherent overview of the discourse within a field of enquiry is an ever more encompassing task.

Scholars have proposed many approaches to analysing literature, which can be placed along a continuum from traditional narrative methods to systematic analytic syntheses of text using machine learning. Traditional reviews are biased because they rely entirely on the interpretation of the researcher. Analytical approaches follow a process that is more like scientific experimentation. These systematic methods are reproducible in the way literature is searched and collated but still rely on subjective interpretation.

Machine learning provides new methods to analyse large swaths of text. Although these methods sound exciting, these methods are incapable of providing insight. Machine learning cannot interpret a text; it can only summarise and structure a corpus. Machine learning still requires human interpretation to make sense of the information.

This article introduces a mixed-method technique for reviewing literature, combining qualitative and quantitative methods. I used this method to analyse literature published by the International Water Association as part of my dissertation into water utility marketing. Detailed information about the methodology is available through FigShare.

A literature review with RQDA

The purpose of this review was to ascertain the relevance of marketing theory to the discourse of literature in water management. This analysis uses a sample of 244 journal abstracts, each of which was coded with the RQDA library. This library provides functionality for qualitative data analysis. RQDA provides a graphical user interface to mark sections of text and assign them to a code, as shown below.

Marking topics in an abstract with RQDA.

You can load a corpus of text into RQDA and mark each of the texts with a series of codes. The texts and the codes are stored in an SQLite database, which can be easily queried for further analysis.

I used a marketing dictionary to assess the abstracts from journals published by the International Water Association from the perspective of marketing. This phase resulted in a database with 244 abstracts and their associated coding.

Discourse Network Analysis

Once all abstracts are coded, we can start analysing the internal structure of the IWA literature. First, let's have a look at the occurrence of the topics identified for the corpus of abstracts.

The first lines in this snippet call the tidyverse and RQDA libraries and open the abstracts database. The getCodingTable function provides a data frame with each of the marked topics and their location. This function allows us to visualise the occurrence of the topics in the literature.

library(tidyverse)
library(RQDA)
## Open project
openProject("IWA_Abstracts.rqda", updateGUI = TRUE)

## Visualise codes
getCodingTable() %>%
    group_by(codename) %>%
    count() %>%
    arrange(n) %>%
    ungroup() %>%
    mutate(codename = factor(codename, levels = codename)) %>%
    ggplot(aes(codename, n)) +
        geom_col() +
        coord_flip() +
        xlab("Code name") + ylab("Occurence")

Frequencies of topics in IWA literature.

This bar chart tells us that the literature is preoccupied with asset management and the quality of the product (water) or the service (customer perception). This insight is interesting, but not very enlightening information. We can use discourse network analysis to find a deeper structure in the literature.

Discourse Network Structure

We can view each abstract with two or more topics as a network where each topic is connected. The example below shows four abstracts with two or more codes and their internal networks.

Examples of complete networks for four abstracts.

The union of these four networks forms a more extensive network that allows us to analyse the structure of the corpus of literature, shown below.

Union of networks and community detection.

We can create a network of topics with the igraph package. The first step is to create a Document-Term-Matrix. This matrix counts how often a topic occurs within each abstract. From this matrix, we can create a graph by transforming it into an Adjacency Matrix. This matrix describes the graph which can be visualised and analysed. For more detailed information about this method, refer to my dissertation.

library(igraph)
library(reshape2)
dtm <- getCodingTable()[,c(5, 4)] %>%
    mutate(freq = 1) %>%
    acast(filename ~ codename, sum)
adj <- crossprod(dtm)
g <- graph.adjacency(adj, weighted = T, mode = "undirected")
g <- simplify(g)
## Network Graphs
V(g)$name <- gsub(" ", "\n", V(g)$name)
par(mar = rep(0, 4))
plot(g,
     layout = layout.fruchterman.reingold,
     vertex.label.cex = 1,
     vertex.size = degree(g),
     vertex.label.color = "black",
     vertex.frame.color = "white",
     vertex.color = "Dodgerblue",
     edge.width = E(g)$weight * 1,
     edge.color = "darkgray")

The network of topics in IWA literature.

In this graph, each node is a topic in the literature, and each edge implies that a topic is used in the same abstract. This graph uses the Fruchterman-Reingold algorithm to position each of the nodes, with the most connected topic in the centre.

The last step is to identify the structure of this graph using community detection. A community is a group of nodes that are more connected with each other than with nodes outside the community.

set.seed(123)
comms <- spinglass.community(g, spins = 100)
par(mar = rep(0, 4))
plot(comms, g,
     layout = layout.fruchterman.reingold,
     vertex.label.cex = .7,
     vertex.size = degree(g),
     vertex.label.color = "black",
     vertex.frame.color = NA,
     edge.color = "black",
     vertex.label.family = "sanserif",
     mark.border = NA)

Community detection in IWA literature.

We have now succeeded to convert a corpus of 244 journal abstracts to a parsimonious overview of four communities of topics. This analysis resulted in greater insight into how marketing theory applies to water management, which was used to structure a book about water utility marketing.

Customer Experience Management for Water Utilities: Marketing Urban Water Supply

Practical framework for water utilities to become more focused on their customers following Service-Dominant Logic.

IWA Publishing

Celebrate Halloween with Creepy Computer Games in R

Peter Prevos — Sun, 28 Oct 2018 00:00:00 +0000

Halloween is upon us once more and who ever said that data science can't be scare. This article translates the Gravedigger game from the 1983 Creepy Computer Games book to celebrate Halloween. This article is part of my series on gaming with the R language.

In the 1980s I spent my time writing code on my 8-bit ZX81 and Atari computers. I learnt everything I know about programming from copying and modifying printed code listings from books with computer games. The games in these books are mostly simple text-based games. The authors gave them enticing names, often imaginatively illustrated to visualise the virtual world they represent. A line and a dot become a game of tennis, and a computer that was able to play Tic Tac Toe seemed like your machine had come alive.

Let's step into the creepy world of computer games as imagined by Usborne Publishing.

Reynold, Colin and McCaig, Rob, Creepy Computer Games (Usborne, London).

Creepy Computer Games - Gravedigger

Alan Ramsey's Gravedigger is a typical example of the games of the early days of home computing. You can download the original book for free from the publisher's Google drive. The Gravedigger listing starts on page 10. The lyrical description of the game provides the instructions:

It's dark and windy—not the kind of night to be lost in a graveyard, but that's where you are. You have until midnight to find your way out. Skeletons lurk in the shadows waiting to scare you to death should you come to close. You can dig holes to help keep them away but digging is tiring work and you cannot manage more than five in one game. You have to be careful not to fall down the holes you have dug. Grave stones (marked +) and the walls of the graveyard (marked :) block your way. The holes you digs are marked O, you are * and the skeletons are X. See if you can escape.

Gravedigger screenshot (Emacs ESS).

Creepy Alien

Roland Krasser has developed vastly improved version of the code and published the Creepy Alien game on CRAN.

  install.packages("creepyalien")
  creepyalien::creepyalien()

Creepy Alien by Roland Kasser.

The Gravedigger code

I translated the BASIC code as close to the original as possible. I changed some of the variable names because, in BASIC, single variables and vectors can have the same name and names of character vectors end in a $ character.

The game is quite tricky, and I have only managed to escape the graveyard once. It looks like the likelihood of success very much depends on the random distribution of the graves. Perhaps we need some machine learning to optimise strategy.

I leave it up to you to deconstruct the program and safely work your way through the graveyard.

Happy Halloween!

indicate different versions for various dialects of the BASIC language.

Partial page of the Gravedigger game in BASIC. The markings

## Creepy Computer Games
## Reynold, Colin and McCaig, Rob, Creepy Computer Games (Usborne, London).
## https://archive.org/details/Creepy_Computer_Games_1983_Usborne_Publishing/
## Gravedigger by Alan Ramsey

## Initiate board
A <- matrix(ncol = 20, nrow = 10)
A[, ] <- " "

## Starting variables
W <- 0 # Move number
X <- 5 # Remaining holes
death <- 0 # Game over?

## Initiate pieces
Y <- "*"
B <- "+"
C <- "O" 
D <- ":"
E <- "X"
Z <- " "

## Draw board
## Add borders
A[c(1, 10), ] <- D
A[, 1] <- D
A[1:8, 20] <- D
## Add graves
for (i in 1:20){
    A[floor(runif(1) * 7 + 2), floor(runif(1) * 15 + 3)] <- B
}

## Starting positions
## Player
M <- 2
N <- 2
A[N, M] <- Y
## Skeletons
S <- c(4, 19, 3, 19, 2, 19)

## Game play
repeat{    
    ## Position skeletons
    for (J in seq(1, 5, by = 2)) {
        A[S[J], S[J + 1]] <- E
    }
    W <- W + 1 ## Move counter
    if (W > 60) {
        print("The clock's struck midnight")
        print("Aghhhhh!!!!")
        break
    }
    ## Print board
    v <- paste(as.vector(t(A)), collapse = "")        
    for (i in 1:10)
        print(substr(v, (i - 1) * 20 + 1, (i - 1) * 20 + 20))
    ## Enter move
    A1 <- toupper(readline(paste0("Enter move ", W, " (You can go N, S, E or W): ")))
    ## Move player
    T <- N
    U <- M
    if (A1 == "N") {
        T <- N - 1
    } else if (A1 == "E") {
        U <- M + 1
    } else if (A1 == "S") {
        T <- N + 1
    } else if (A1 == "W") {
        U <- M - 1
    }
    ## Collision detection
    if (A[T, U] == D | A[T, U] == B) { # Edge or grave
        print("That way's blocked")
    } else if (A[T, U] == C) { # Hole
        print("You've fallen into one of your own holes")
        break
    } else if (A[T, U] == E) { # Skeleton
        death <- 1
    } else if (T == 9 & U == 20) { # Escaped
        print("You're free!")
        print(paste0("Your performance rating is ",
                    floor((60 - W) / 60 * (96 + X)), "%"))
        break
    } else if (death == 1) {
        print("Urk! You've been scared to death by a skeleton.")
        break
    } else if (A[T, U] == Z) { # Player can move
        ## Move player and dig hole
        A [N, M] <- Z
        if (X != 0) {
            B1 <- toupper(readline("Would you like to dig a hole (Y or N): "))
            if (B1 == "Y") {
                X <- X - 1
                A[N, M] <- C
            }
        }
        N <- T
        M <- U
        A[T, U] <- Y
        ## Move skeletons
        for (J in seq(1, 5, by = 2)) {
            ## Store skeleton position in temp variable
            P <- S[J]
            Q <- S[J + 1]
            if (any(c(A[P + 1, Q], A[P - 1 , Q], A[P, Q - 1], A[P, Q + 1]) == Y)) {
                death <- 1
            } else
            {
                ## Move skeletons
                if (A1 == "S" & A[P + 1, Q] == Z){
                    S[J] <- S[J] + 1 # Follow player
                    A[P, Q] <- Z
                } else if (A1 == "N" & A[P - 1, Q] == Z){
                    S[J] <- S[J] - 1 # Follow player
                    A[P, Q] <- Z
                } else if (A1 == "E" & A[P, Q - 1] == Z & M < Q){
                    S[J + 1] <- S[J + 1] - 1 # Move towards player
                    A[P, Q] <- Z
                } else if (A1 == "E" & A[P, Q + 1] == Z & M > Q) {
                    S[J + 1] <- S[J + 1] + 1 # Reverse direction
                    A[P, Q] <- Z
                }
            }
        }
    }
}

Mapping and Geocoding with ggmap and the Google API

Peter Prevos — Wed, 10 Oct 2018 00:00:00 +0000

In the old days, obtaining latitude and longitude required a physical survey and lots of mathematics. Google Maps has made this task much easier. This article describes how to use the Google Maps API with the ggmap library. Geocoding with ggmap is a convenient way to create meaningful spatial visualisations.

The ggmap package

Several R libraries exist to visualise spatial information. ggmap by David Kahle which aligns with the ggplot philosophy of layered visualisations.

The geocode function from the ggmap package extracts longitude and latitude from Google maps, based on a location query. The example below shows the result of a simple geocoding call for the White House and Uluru. The geocode function essentially constructs a URL to obtain the data. However, when you run this code you see an error:

  library(ggmap)
  geocode(c("White House", "Uluru"))

  Error: Google now requires an API key.
         See ?register_google for details.

In the middle of 2018, Google tightened access to the database, which means you need to register an API for it to work. This article explains how to use the latest version of ggmap and a Google account to continue using this function.

The Google API

Before we can start geocoding, we need to obtain an API key from Google. Go to the registration page, and follow the instructions (select all mapping options). The geocoding API is a free service, but you nevertheless need to associate a credit card with the account. Please note that the Google Maps API is not a free service. There is a free allowance of 40,000 calls to the geocoding API per month, and beyond that calls are $0.005 each.

Geocoding with ggmap

You will need to ensure that you have the latest version of ggmap installed on your system. The current version on CRAN is 3.0.

The code snippet below shows a minimum-working-example of how you can map coordinates using ggplot. The register_google function stores the API key. I have stored the key itself in a private text file. The getOption("ggmap") function summarises the Google credentials to check how you are connected.

The geocode function converts the request into a URL and captures the output into a data frame. The plot shows the places I have lived, projected orthogonally on the globe. The mapproj library provides the orthogonal projection.

api <- readLines("google.api") # Text file with the API key
register_google(key = api)
getOption("ggmap")

locations <- c("Hoensbroek", "Johannesburg", "Barrow-in-Furness",                
               "Hong Kong", "Singapore", "Tangail", "Maastricht", "Bendigo") %>%
     geocode()

library(ggplot2)
library(mapproj)

world <- map_data("world")
ggplot() +
     geom_polygon(data = world,  aes(long, lat, group = group),
                  fill = "grey") +
     geom_point(data = locations, aes(lon, lat),
                colour = "red", size = 5) + 
     coord_map("ortho", orientation = c(30, 80, 0)) +
     theme_void()

Places where I have worked and lived.

Flat Earth Mathematics with examples in the R Language

Peter Prevos — Sun, 02 Sep 2018 00:00:00 +0000

One of the craziest fake news trends is the flat earth conspiracy. Some of you might ask whether my trip actually around the world or am I travelling across a flat disk? This article discusses flat earth mathematics, and how to convert and visualise map projections in R. This article discusses flat earth mathematics, using code I published earlier in articles about pacific island hopping and creating air travel maps.

The Flat Earth

YouTube contains thousands of videos from people that claim the earth is flat. These people pontificate that science as we know it is a "globalist conspiracy". The claims about a flat earth are hilarious and entertaining. People who believe our planet is flat even conduct simple experiments and engage in naive flat earth mathematics. Needless to say that these experiments are not very convincing, but I respect their entrepreneurial attitude.

Adherents to the idea that the world is flat often propose Gleason's Map as their version of the correct representation of our planet. Every printed map of the earth is flat, but somehow this one seems to have a special status.

The Gleason map is not a map of a flat earth but a polar azimuthal equidistant projection of the globe. Even though the map itself states that it is a projection of the world, flat earth believers nevertheless see it as a literal representation of the world they live on.

Section of Gleason's map.

Their belief is based on an appeal to common sense as the world appears to be flat when we look at the horizon. The second ingredient is a deep distrust in science, often inspired by religious motives.

This article shows two different ways to look at the earth and show that the spherical model better fits the reality of my trip around the world.

Projecting the spherical earth on a flat surface is a complex task. Maps require compromise as it is impossible to truthfully draw the surface of the globe on a piece of paper without distortion. The video below provides a practical introduction to map projections and shows how maps are stretched to display them on a flat surface.

Why all world maps are wrong.

The Azimuthal equidistant method is popularised in the flag of the United Nations. Antarctica is in this projection displayed as a ring around the world because the south pole projects to infinity. Flat earth evangelists believe that the South Pole a ring of ice that prevents us from proceeding beyond the disc. The edge of this disc cannot be shown because this method projects the south pole at an infinite distance from the centre of the map. We can recreate Gleason's map with ggplot, which incorporates the mapproj package to show maps in various projections.

  ## Flat Earth Mathematics
  library(ggplot2)

  world <- map_data("world")

  ggplot(world) +
    geom_path(aes(x = long, y = lat, group = group), linewidth = .2) +
    theme(axis.text = element_blank(),
          axis.ticks = element_blank()) +
    labs(x = "", y = "") +
    coord_map("azequidistant", orientation = c(90, 0, 270))

The coord_map function allows you to change projections. The orientation argument defines the centre point of the projection. The most common location is the North Pole (90, 0). The third value gives the clockwise rotation in degrees. Gleason's map uses a rotation of 270 degrees.

Polar azimuthal equidistant projection with ggmap.

My Round-the-World Itinerary

In September 2018 I embarked on a trip around the world to speak at the World Water Congress in Tokyo. I also visited family and friends in the Netherlands and did some tourism in San Francisco. One of the workshops I speak at is about Fake News in the water industry.

My trip started in Australia, onward to Japan, from where I go to the Netherlands. The last two legs will bring me back to Australia via San Francisco. The itinerary is stored in a data frame. The ggmap package geocodes the longitude and latitude of each of the locations on my trip. You will need a Google API to enable the geocoding function. As the earth is a sphere, an intermediate point needs to be added for trips that pass the dateline, as I explained in my article about flight maps.

The geosphere package helps to estimate the total travel distance. The travel distance is approximately 38,995 km, slightly less than a trip around the equator. This distance is the great circle distance, which is the shortest distance between two points on a sphere. The flight paths on this map are curved because of the inevitable distortions when projecting a sphere on a flat surface.

  ## Round the World Itinerary
  library(ggmap)
  library(dplyr)

  api <- readLines("google.api") # Text file with the secret API key
  register_google(key = api)
  airports <- c("Melbourne", "Tokyo", "Amsterdam", "San Francisco")
  itinerary <- geocode(airports)
  itinerary <- rbind(itinerary, itinerary[1, ]) %>%
    mutate(location = c(airports, airports[1]))

  ## Split travel past dateline
  dl <- which(diff(itinerary$lon) > 180)
  dr <- ifelse(itinerary$lon[dl] < 0, -180, 180)
  dateline <- tibble(lon = c(dr, -dr),
                     lat = rep(mean(itinerary$lat[dl:(dl + 1)]), 2),
                     location = "dateline")
  itinerary <- rbind(itinerary[1:dl, ], dateline,
                     itinerary[(dl + 1):nrow(itinerary), ])
  itinerary

  ## Visualise
  worldmap +
    geom_point(data = itinerary, aes(lon, lat), colour = "red", size = 4) +
    geom_path(data = itinerary, aes(lon, lat), colour = "red", size = 1) +
    coord_map("azequidistant", orientation = c(90, 0, 270))

  ## Great Circle Distance
  library(geosphere)
  sapply(1:(nrow(itinerary) - 1), function(l)
    distVincentyEllipsoid(itinerary[l, 1:2], itinerary[(l + 1), 1:2]) / 1000) %>%
    sum()

Round the world trip in polar Azimuthal equidistant projection.

Flat Earth Mathematics

If the Gleason map were an actual map of the flat earth, then the flight paths on the map would show as straight lines.

The mapproj package contains the mapproject function that calculates the projected coordinates based on longitude and latitude. The output of this function is a grid with limits from $-\pi$ to $\pi$. The first part of the code converts the longitude and latitude from the world data frame to projected coordinates.

A line from the lon/lat 0,0 to the north pole has a projected distance of $\pi/2$, which in the spherical world is $\frac{\pi}{2} 6378.137 = 10018.75$ km. We need to multiply the Euclidean distances with the radius of the Earth to derive the Gleason map coordinates.

This last code snippet converts the world map to flat earth coordinates. It calculates the Euclidean distance between the points on the itinerary and multiplies this with the Earth's diameter.

The best way to show that Gleason's map is not a map of a flat earth is best demonstrated with a flight between Australia and Chili because the distortions are huge in the Southern hemisphere. This last code snippet shows why the Gleason map is not a map of a flat earth. On this map, the shortest distance between Sydney and Santiago de Chili is about 25,000 km, more than twice the real value. The actual travel time is about 14 hours, which implies that passenger jets break the sound barrier. This problem exists for all journeys along the lines of latitude in the Southern hemisphere.

This map looks like the first one, but the coordinate system is now Euclidean instead of polar (longitudes and latitudes), as indicated by the square grid. On a projected map, the shortest distance is a curved line, parallel to Antarctica, which is how ships and aeroplanes move between these cities.

Sydney to Santiago de Chili on a flat earth map.

This video shows a unique insight into the navigational issues for intercontinental flights in the Southern hemisphere.

Aerolineas Argentinas Southern Transpolar Flight from Sydney to Buenos Aires.

  library(mapproj)
  flatearth.coords <- mapproject(world$long, world$lat,
                          "azequidistant", orientation = c(90, 0, 270))
  r <- 6378.137
  flatearth.coords <- mutate(world,
                  x = flatearth.coords$x * r,
                  y = flatearth.coords$y * r) %>%
      select(x, y, group, order, region, subregion)

  flatearth <- ggplot(flatearth.coords) +
      geom_path(aes(x, y, group = group), size = .2) +
      theme(axis.text = element_blank(),
            axis.ticks = element_blank()) +
      labs(x = "", y = "")

  ## Australia - South America
  airports <- tibble(city = c("Sydney", "Santiago de Chili"))
  itinerary <- geocode(airports$city) %>%
      mutate(location = airports$city)
  itinerary
  coords <- mapproject(itinerary$lon, itinerary$lat, "azequidistant",
                       orientation = c(90, 0, 270))
  coords <- tibble(x = coords$x * r, y = coords$y * r)
  sum(sqrt(diff(coords$x)^2 + diff(coords$y)^2))

  flatearth + 
      geom_point(data = coords, aes(x, y), colour = "red", size = 4) +
      geom_path(data = coords, aes(x, y), colour = "red", size = 1)

Conclusion

This article proves that the Gleason map is not a representation of a flat earth. Aeroplanes would have to break the sound barrier to fly these distances in the time it takes to travel. Whichever way you project the globe on a flat map will lead to inevitable distortions. The Gleason map itself mentions that it is a projection and not a flat earth. However, these facts will not dissuade people from believing in a flat earth. I am after all an engineer and thus part of the globalist science conspiracy.

Call Centre Workforce Planning Using Erlang C in R language

Peter Prevos — Thu, 19 Jul 2018 00:00:00 +0000

We all hate the experience of calling a service provider and being placed on hold for a very long time. Organisations that take their level of service seriously plan their call centres. Good planning ensures that waiting times for customers are within acceptable limits. Having said this, making people wait for something can, in some instances, increase perceived value. This article shows how to implement call centre workforce planning with the R language.

Contact centre workforce planning involves an exciting mix of rational production management and the soft skills of providing service with a smile. The mathematics of managing a call centre ensures customers have a great experience by limiting waiting times.

The Grade of Service can numerically express call centre performance. This KPI is the percentage of calls answered within a specific time. For example, 90% of calls are responded to within 30 seconds. This Grade of Service depends on the following:

The volume of calls made to the centre.
The number of available agents
The time it takes to service a customer.

Working in a call centre can be chaotic, so we need some maths to understand it. The Erlang C formula describes the relationship between the Grade of Service and various variables to bring some order to the chaos.

Call centre workforce planning is a complex activity that is a perfect problem to solve in R code. This article explains using the Erlang C formula in the R language to manage a contact centre. The presented code calculates the number of agents needed to meet a required Grade of Service. This approach is extended to a Monte Carlo situation to understand the real world's stochastic nature better.

The Erlang C Formula

The Erlang C formula describes the probability that a customer needs to queue instead of being immediately serviced $(P_w)$. This formula is closely related to the Poisson distribution, which describes queues such as traffic lights or hospital waiting lists.

$$P_w = \frac{\frac{A^N}{N!}\frac{N}{N-A}}{\Big( \sum_{i=0}^{N-1} \frac{A^i}{i!} \Big)+\frac{A^N}{N!}\frac{N}{N-A}}$$

The intensity of traffic $A$ is the number of calls per hour multiplied by the average duration of a call. Traffic intensity is measured in dimensionless Erlang units, which expresses the time it would take to manage all calls if they arrived sequentially. The intensity measures the effort that needs to be undertaken in an hour. In reality, calls arrive at random times during the hour, which is where the Poisson distribution comes in. The waiting time is also influenced by the number of available operators $N$.

Both common sense and the formula show that increasing the number of calls or decreasing the number of operators increases the likelihood that customers are placed in a queue. The more staff, the higher the service level, but how many people do you need to achieve your desired grade of service efficiently?

The Grade of Service $S$ is a function of the outcome of the Erlang C formula ($P_w$), the number of agents ($N$), the call intensity ($A$), call duration ($\lambda$) and lastly the target answering time ($t$).

The Erlang C formula can be reworked to provide that answer. I sourced this solution from callcenterhelper.com. We now have a toolset for call centre planning which we can implement in the R language.

$$S = 1 - \Large[ P_w e^ {-[(N-A](t/ \lambda)]} \large]$$

Erlang C in R

The Erlang C formula contains some factorials and powers, which become problematic when dealing with significant call volumes or many agents. The Multiple Precision Arithmetic library enables working with large integer factorials, but there is no need to wield such strong computing powers. To make life easier, the Erlang C formula includes the Erlang B formula, the inverse of which can be calculated using a small loop.

This implementation is very similar to an unpublished R package by Patrick Hubers, enhanced with work from callcenterhelper.com. This code contains four functions:

intensity: Determines intensity in Erlangs based on the rate of calls per interval, the total call handling time and the interval time in minutes. All functions default to an interval time of sixty minutes. Erlangs are a dimensionless unit as a measure of the offered load.
erlang_c: Processes the Erlang C formula using the number of agents and the variables determining intensity (call rate and call handling time). The output is the probability that a customer has to wait for service.
service_level: Calculates the service level. The inputs are the same as above, plus the period for the Grade of Service in seconds.
resource: Seeks the number of agents needed to meet a Grade of Service. This function starts with the minimum number of agents (the intensity plus one agent) and keeps searching until it finds the number of agents that achieve the desired Grade of Service.

Erlang C Code

  ## Call Centre Workforce Planning Using Erlang C in R language
  
  ## Erlang-C Functions
  intensity <- function(rate, duration, interval = 60) {
      (rate / interval) * duration
  }

  erlang_c <- function(agents, rate, duration, interval = 60) {
      int <- intensity(rate, duration, interval)
      erlang_b_inv <- 1
      for (i in 1:agents) {
          erlang_b_inv <- 1 + erlang_b_inv * i / int
      }
      erlang_b <- 1 / erlang_b_inv
      agents * erlang_b / (agents - int * (1 - erlang_b))
  }

  service_level <- function(agents, rate, duration, target, interval = 60) {
      pw <- erlang_c(agents, rate, duration, interval)
      int <- intensity(rate, duration, interval)
      1 - (pw * exp(-(agents - int) * (target / duration)))
  }

  resource <- function(rate, duration, target, gos_target, interval = 60) {
      agents <- round(intensity(rate, duration, interval) + 1)
      gos <- service_level(agents, rate, duration, target, interval)
      while (gos < gos_target * (gos_target > 1) / 100) {
          agents <- agents + 1
          gos <- service_level(agents, rate, duration, target, interval)
      }
      return(c(agents, gos))
  }

Call centre workforce planning using R

For example, you manage a call centre with 100 calls every 30 minutes and an average handling time of 3 minutes. These numbers result in an intensity of intensity(100, 3, 30) 10 Erlangs or ten calls hours of traffic per hour. This result means we need at least ten agents, assuming all calls arrive neatly after each other.

The reality of a contact centre is chaotic, as calls arrive randomly, so there is a high probability that an operator will be busy if we only use ten agents. You can enter this data into the Erlang C in R function to find the chance a customer needs to wait. With ten agents, the likelihood is 100%. With 11 agents, it is 68%. The sapply function helps to view the impact of adding more customer service agents.

  erlang_c(11, 100, 3, 30)
  [1] 0.6821182

  sapply(10:20, erlang_c, rate = 100, duration = 3, interval = 30)
  [1] 1.000000000 0.682118205 0.449388224 0.285270453 0.174131934 0.102042367
  [7] 0.057340331 0.030876110 0.015928277 0.007873558 0.003731126

The likelihood of a customer having to wait is the main target for any service that involves queueing. We need to set a target service level or Grade of Service. The service_level function calculates the Grade of Service using the previous parameters plus the target in seconds. Let's make some assumptions:

Required Grade of Service 80%
Target answer time: 20 seconds

Using the function, we can quickly see that with 11 agents and 100 calls per 30 minutes, we only have a Grade of Service of 39%, much less than the required 80%. Using sapply(), we can see that we need just over 13 available agents.

Grade of Service for a call centre with 100 calls per 30 minutes and 3-minute contacts.

The resource() function ties everything together and seeks the number of agents needed to meet the required grade of service. We need at least 14 available agents to answer 80% of calls within 20 seconds, which gives us an actual predicted Grade of Service of 88%.

  ## Example
  service_level(agents = 11, rate = 100, duration = 3, target = 20 / 60, interval = 30)

  service <- data.frame(agents = 10:20,
                        GoS = sapply(10:20, service_level, rate = 100, duration = 3, target = 20 / 60, interval = 30))

  library(ggplot2)
  ggplot(service, aes(agents, GoS)) +
    geom_line() +
    scale_x_continuous(breaks = 10:20) + 
    scale_y_continuous(labels = scales::percent) +
    geom_hline(yintercept = 0.8, col = "blue") +
    theme_bw(base_size = 10) + 
    labs(title = "Call Centre Simulation",
         subtitle = "Erlang-C Formula")
  ggsave("call-centre-example.png", width = 6, height = 4)

  resource(rate = 100, duration = 3, target = 20/60, gos_target = 80, interval= 30)

Call Centre Workforce Planning with Erlang C Monte Carlo Simulation

I used the Erlang C model to recommend staffing levels in a contact centre some years ago. This experience taught me that the mathematical model is the first step towards call centre workforce planning. Several other metrics can be built on the Erlang C model, such as agents' average occupancy and handling time.

Like all mathematical models, the Erlang C formula is an idealised version of reality. Agents are not always available; they need breaks, toilet stops and might even go on leave. Employers call this loss of labour shrinkage, a somewhat negative term to describe something positive for the employee. The Erlang C model provides the number of 'bums on seats'.

Like every model, the Erlang C formula is not a perfect representation of reality. The formula tends to overestimate the required resources because it assumes that people will stay on hold indefinitely, while the queue will automatically shorten as people lose patience.

The number of employees needed to provide this capacity depends on the working conditions at the call centre. For example, if employees are only available to take calls 70% of their contracted time, you will need $1/0.7=1.4$ staff members for each live agent to meet the Grade of Service.

Another problem is the stochastic nature of call volumes and handling times. The Erlang C model requires a manager to estimate call volume and handling time (intensity) as a static variable, while in reality, it is stochastic and subject to variation. Time series analysis can help to predict call volumes, but every prediction has a degree of uncertainty. We can manage this uncertainty by using a Monte Carlo simulation.

All the functions listed above can be rewritten to provide a vector of possible answers based on the average call volume and duration and their standard deviation. This simulation assumes a normal distribution for both call volume and the length of each call. The outcome of this simulation is a distribution of service levels.

Monte Carlo Simulation

A Monte Carlo simulation is a method to simulate reality by defining a probability distribution of the events in the model.

For example, a call centre receives, on average, 100 calls per half hour with a standard deviation of 10 calls. The average time to manage a call, including wrap-up time after the call, is 180 seconds, with a standard deviation of 20 seconds. The centre needs to answer 80% of calls within 20 seconds. What is the likelihood of achieving this level of service?

The average intensity of this scenario is 10 Erlangs. Using the resource formula suggests that we need 14 agents to meet the Grade of Service. Simulating the intensity of the scenario 1000 times indicates that the intensity is between 5.4 and 15.9 Erlangs (see code at the end of this article).

  resource(100, 180, 20, 80, 30)
  [1] 14.0000000 0.88835

  intensity_mc(100, 10, 180, 20) %>% summary()
  
  Min. 1st Qu. Median Mean 3rd Qu. Max.
  5.480 8.975 9.939 10.025 10.993 15.932

The next step is to simulate the expected service level for this scenario. The plot visualises the outcome of the Monte Carlo simulation and shows that in 95% of situations, the Grade of Service is more than 77% and half the time, it is more than 94%.

  service_level_mc(15, 100, 10, 180, 20, 20, 30, sims = 1000) %>%
  + quantile(c(.05, .5, .95))
  5% 50% 95%
  0.7261052 0.9427592 0.9914338

This article shows that Erlang C in R helps managers plan call centre workforce. We may need a Shiny application to develop a tool to manage the complexity of these functions. I would love to hear from people with practical experience in managing call centres in how they analyse data.

Simulated service levels using Erlang C and Monte Carlo simulation.

Monte Carlo Simulation Code

  ## Monte Carlo Simulation
  
  library(tidyverse)

  intensity_mc <- function(rate_m, rate_sd, duration_m, duration_sd, interval = 60, sims = 1000) {
    (rnorm(sims, rate_m, rate_sd) / (60 * interval)) * rnorm(sims, duration_m, duration_sd)
  }

  intensity_mc(100, 10, 180, 20, interval = 30) %>%
    summary()

  erlang_c_mc <- function(agents, rate_m, rate_sd, duration_m, duration_sd, interval = 60) {
    int <- intensity_mc(rate_m, rate_sd, duration_m, duration_sd, interval)
    erlang_b_inv <- 1
    for (i in 1:agents) {
      erlang_b_inv <- 1 + erlang_b_inv * i / int
    }
    erlang_b <- 1 / erlang_b_inv
    agents * erlang_b / (agents - int * (1 - erlang_b))
  }

  service_level_mc <- function(agents, rate_m, rate_sd, duration_m, duration_sd, target, interval = 60, sims = 1000) {
    pw <- erlang_c_mc(agents, rate_m, rate_sd, duration_m, duration_sd, interval)
    int <- intensity_mc(rate_m, rate_sd, duration_m, duration_sd, interval, sims)
    1 - (pw * exp(-(agents - int) * (target / rnorm(sims, duration_m, duration_sd))))
  }

  tibble(ServiceLevel = service_level_mc(agents = 14,
                                         rate_m = 100,
                                         rate_sd = 10,
                                         duration_m = 180,
                                         duration_sd = 20,
                                         target = 20,
                                         interval = 30,
                                         sims = 1000)) %>%
    ggplot(aes(ServiceLevel)) +
    geom_histogram(binwidth = 0.05, fill = "#008da1") +
    scale_x_continuous(labels = scales::percent) + 
    theme_bw(base_size = 10) +
    labs(title = "Call Centre Monte Carlo Simulation")

Project Euler 33: Digit Cancelling Fractions

Peter Prevos — Mon, 09 Jul 2018 00:00:00 +0000

Project Euler 33 takes us back to the world of fractions from our primary school days. This article provides a solution for the Euler problem and discusses how to solve this problem using Farey sequences and how to visualise Ford Circles with the ggplot2 package.

Project Euler 33 Definition

The fraction 49/98 is curious, as an inexperienced mathematician in attempting to simplify it may incorrectly believe that 49/98 = 4/8, which is correct, is obtained by cancelling the nines. We shall consider fractions like 30/50 = 3/5, to be trivial examples.

There are precisely four nontrivial examples of this type of fraction, less than one in value, and containing two digits in the numerator and denominator. If the product of these four fractions is given in its lowest common terms, find the value of the denominator.

Proposed Solution in R

To solve this problem, we create a pseudo-Farey sequence by generating all different fractions with two decimals in the numerator and denominator. The loop generates all combinations of denominators and numerators, excluding the trivial ones (multiples of 10 and 11). This solution converts the numbers to strings, strips any common duplicates, and tests the condition. The code concatenates vectors, which is not good practice. However, the loop is so short it does not matter much.

  ## Project Euler 33: Digit Cancelling Fractions
  num <- vector()
  den <- vector()
  for (a in 11:99) {
      for (b in (a + 1):99) {
          trivial <- (a %% 10 == 0 | b && 10 == 0 | a %% 11 == 0 | b %% 11 == 0)
          if (!trivial) {
              i <- as.numeric(unlist(strsplit(as.character(a), "")))
              j <- as.numeric(unlist(strsplit(as.character(b), "")))
              digs <- c(i, j)
              dup <- digs[duplicated(digs)]
              digs <- digs[which(digs != dup)]
              if (length(digs) == 2 & a/b == digs[1]/digs[2]) {
                  num <- c(num, a)
                  den <- c(den, b)
                  }
          }
      }
  }
  paste(num, den, sep = "/")
  answer <- prod(den) / prod(num)
  print(answer)

Farey Sequences and Ford Circles

Next step is to generalise Euler problem 33 and write a function to generate Farey Sequences and visualise them using Ford Circles. This Numberphile video below explains fractions and Farey sequences. A Farey sequence contains all fractions between 0 and 1 with a maximum denominator. More formally, a Farey sequence of order n is the sequence of completely reduced fractions between 0 and 1. These fractions have denominators less than or equal to n, arranged in order of increasing size. For example, the Farey Sequence with order 3 is:

$$F_3 = \Big\{ \frac{0}{1},\frac{1}{3},\frac{1}{2},\frac{2}{3},\frac{1}{1}\Big\}$$

Ford Circles are a fractal-esque way to visualise Farey sequences, but before we get to this, first solve Euler problem 33.

Funny Fractions and Ford Circles by Numberphile.

The farey function generates a data table with the numerators (p) and denominators (q) of a Farey sequence. The function builds a list of all possible fractions for the solution space, excluding those with one as a Greatest Common Denominator, as defined by the gcd() function.

  ## Farey sequence function
  farey <- function(n) {
      fseq <- list()
      fseq[[1]] <- c(0, 1)
      i <- 2
      gcd <- function(a, b) { # Euclid's method
          if (a == 0) return(b)
          if (b == 0) return(a)
          gcd(b, a %% b)
      }
      for (q in 2:n) {
          for (p in 1:(q - 1)){
              if (gcd(p, q) == 1) {
                  fseq[[i]] <- c(p, q)
                  i <- i + 1
                  }
          }
      }
      fseq[[i]] <- c(1, 1)
      fseq <- as.data.frame(do.call(rbind, fseq))
      names(fseq) <- c("p", "q")
      fseq <- fseq[order(fseq$p / fseq$q), ]
      return(fseq)
  }

Standard ggplot2 cannot draw circles where the radius of the circles is related to the coordinate system. I used a circle function sourced from StackOverflow. This function is called in a for-loop to build the circles on top of the empty canvas.

Farey Sequence and Ford Circles (n = 20).

  library(ggplot2)
  lm_palette <- c("#008da1", "#005395", "#262e43", "#3b2758", "#865596", "#f26230")

  ford_circles <- farey(20) %>%
    mutate(x = p / q,
           y = 1 / (2* q^2),
           r = y,
           c = lm_palette[(q - 1)%%6 + 1])

  g_circle <- function(r, x, y, color = NA, fill = "black", ...) {
    x <- x + r * cos(seq(0, pi, length.out = 100))
    ymax <- y + r * sin(seq(0, pi, length.out = 100))
    ymin <- y + r * sin(seq(0, -pi, length.out = 100))
    annotate("ribbon", x = x, ymin = ymin, ymax = ymax,
             color = color, fill = fill, ...)
  }

  p <- ggplot(ford_circles, aes(x, y))
  for (i in 1:nrow(ford_circles)) {
    p <- p + g_circle(ford_circles$r[i], ford_circles$x[i], ford_circles$y[i],
                      fill = ford_circles$c[i])
  }

  p +
    xlim(c(0, 1)) +
    coord_fixed() +
    theme_void()

  ggsave("images/problem-033.png", width = 6, height = 4)

Analyse Definitions of Marketing with Rvest and Tidytext

Peter Prevos — Thu, 31 May 2018 00:00:00 +0000

I am preparing to facilitate another session of the marketing course for the La Trobe University MBA. The first lecture delves into the definition of marketing. Like most other social phenomena, marketing is tough to define. Definitions of social constructs often rely on the perspective taken by the person or group writing the definition. As such, definitions also change over time. While a few decades ago, definitions of marketing revolved around sales and advertising, contemporary definitions are more holistic and reference creating value.

Heidi Cohen wrote a blog post where she collated 72 definitions of marketing. So rather than arguing over which definition is the best, why not use all definitions simultaneously? This article attempts to define a new definition of marketing, using a data science approach. We can use the R language to scrape the 72 definitions from Heidi's website and attempt text analysis to extract the essence of marketing from this data set.

I have mentioned in a previous post about qualitative data science that automated text analysis is not always a useful method to extract meaning from a text. I decided to delve a little deeper into automated text analysis to see if we find out anything useful about marketing using the rvest and tidytext packages.

Scraping Definitions of Marketing with Rvest

Web scraping is a technique to download data from websites where this data is not available as a clean data source. Web scraping starts with downloading the HTML code from the website and the filtering the wanted text from this file. The rvest package simplifies this process.

The code for this article uses a pipe (%>%) with three rvest commands. The first step is to download the wanted html code from the web using the read_html function. The output of this function is piped to the html_nodes function, which does all the hard work. In this case, the code picks all lines of text that are embedded in an ordered list. You can use the SelectorGadget to target the text you like to scrape. The last scraping step cleans the text by piping the output of the previous commands to the html_text function.

The result of the scraping process is converted to a Tibble, which is a type of data frame used in the Tidyverse. The definition number is added to the data, and the Tibble is converted to the format required by the Tidytext package. The resulting data frame is much longer than the 72 definitions because there are other lists on the page. Unfortunately, I could not find a way to filter only the 72 definitions.

  library(tidyverse)
  library(rvest)
  library(tidytext)
  library(wordcloud)
  library(RColorBrewer)
  lm_palette <- c("#f7881f", "#55ace1",  "#5f6c36")

  ## Scrape definitions from website
  definitions <- read_html("https://heidicohen.com/marketing-definition/") %>%
      html_nodes("ol li") %>%
      html_text() %>%
      as_tibble() %>%
      mutate(No = 1:nrow(.)) %>%
      select(No, Definition = value)

Tidying the Text

The Tidytext package extends the tidy data philosophy to a text. In this approach to text analysis, a corpus consists of a data frame where each word is a separate item. The code snippet below takes the first 72 rows, and the unnest_tokens function extracts each word from the 72 definitions. This function can also extract ngrams and other word groups from the text. The Tidytext package is an extremely versatile piece of software which goes far beyond the scope of this article. Julia Silge and David Robinson have written a book about text mining using this package, which provides a very clear introduction to the craft of analysing text.

The last section of the pipe removes the trailing "s" from each word to convert plurals into single words. The mutate function in the Tidyverse creates or recreates a new variable in a data frame.

  ## bag of words
  def_words <- definitions[1:72, ] %>%
      unnest_tokens(word, Definition) %>%
      mutate(word = gsub("s$", "", word))

This section creates a data frame with two variables. The No variable indicates the definition number (1–72) and the word variable is a word within the definition. The order of the words is preserved in the row name. To check the data frame you can run unique(def_words$No[which(def_words$word = "marketing")])=. This line finds all definition numbers with the word "marketing", wich results, as expected, in the number 1 to 72.

Using Rvest and Tidytext to define marketing

We can now proceed to analyse the definitions scraped from the website with Rvest and cleaned with Tidytext. The first step is to create a word cloud, which is a popular way to visualise word frequencies. This code creates a data frame for each unique word, excluding the word marketing itself, and uses the wordcloud package to visualise the fifty most common words.

  word_freq <- def_words %>%
      anti_join(stop_words) %>%
      count(word) %>%
      filter(!(word %in%
               c("marketing", "vice", "president", "chief", "executive", "’")))

  pdf("marketingcloud.pdf")
  word_freq %>%
      with(wordcloud(word, n, max.words = 50, rot.per = .5,
                     colors = rev(lm_palette)))
  dev.off()

Wordcloud of 72 definitios of marketing.

While a word cloud is indeed a pretty way to visualise the bag of words in a text, it is not the most useful way to get the reader to understand the data. The words are jumbled, and the reader needs to search for meaning. A better way to visualise word frequencies is a bar chart. This code takes the data frame created in the previous snippet, determines the top ten occurring words. The mutate statement reorders to factor levels so that the words are plotted in order.

    word_freq %>%
        top_n(10) %>%
        mutate(word = reorder(word, n)) %>%
        ggplot(aes(word, n)) + geom_col(fill = lm_palette[2]) +
        coord_flip() +
        theme(text = element_text(size = 15))
    ggsave("marketing-words.png", width = 6, height = 6)

Top ten words in marketing definitions.

A first look at the word cloud and bar chart suggests that marketing is about customers and products and services. Marketing is a process that includes branding and communication. This definition is simplistic but functional.

Topic Modeling using Tidytext

Word frequencies are a weak method to analyse text because it interprets each word as a solitary unit. Topic modelling is a more advanced method that examines the relationships between words, i.e. the distance between them. The first step is to create a Document-Term Matrix, which is a matrix that indicates how often a word appears in a text. As each of the 72 texts are very short, I decided to treat the collection of definitions as one text about marketing. The cast_dtm() function converts the data frame to a Document-Term Matrix.

The following pipe determines the top words in the topics. Just like k-means clustering, the analyst needs to choose the number of topics before analysing the text. In this case, I have opted for four topics. The code determines the contribution of each word to the four topics and selects the five most common words in each topic. The faceted bar chart shows each of the words in the four topics.

  ## Topic modeling
  library(topicmodels)

  marketing_dtm <- word_freq %>%
      mutate(doc = 1) %>%
      cast_dtm(doc, word, n)

  marketing_lda <- LDA(marketing_dtm, k = 4) %>%
      tidy(matrix = "beta") %>%
      group_by(topic) %>%
      top_n(5, beta) %>%
      ungroup() %>%
      arrange(topic, -beta)

  marketing_lda %>%
      mutate(term = reorder(term, beta)) %>%
      ggplot(aes(term, beta, fill = factor(topic))) +
      geom_col(show.legend = FALSE) +
         facet_wrap(~topic, scales = "free") +
         coord_flip() +
         theme(text = element_text(size = 20))

Topic modelling 72 definitions of marketing.

This example also does not tell me much more about what marketing is, other than giving a slightly more sophisticated version of the word frequency charts. This chart shows me that marketing is about customers that enjoy a service and a product. Perhaps the original definitions are not distinctive enough to be separated from each other.

This article is only a weak summary of the great work by Julia Silge who co-authored the Tidytext package. The video below provides a comprehensive introduction to topic modelling.

Topic modelling with R andtidy data principles.

What have we learnt?

This excursion into text analysis using rvest and Tidytext shows that data science can help us to make some sense out of an unread text. If I did not know what this page was about, then perhaps this analysis would enlighten me. This kind of analysis can assist us in wading through to large amounts of text to select the ones we want to read. I am still not convinced that this type of analysis will provide any knowledge beyond what can be obtained from actually reading and engaging with a text.

Although I am a data scientist and want to maximise the use of code in analysing data, I am very much in favour of developing human intelligence before we worry about the artificial kind.

Qualitative Data Science: Using RQDA to analyse interviews

Peter Prevos — Thu, 03 May 2018 00:00:00 +0000

Qualitative data science sounds like a contradiction in terms. Data scientists generally solve problems using numerical solutions. Even the analysis of a text is reduced to a numerical problem using Markov chains, topic analysis, sentiment analysis and other mathematical tools.

Scientists and professionals consider numerical methods to be the gold standard of analysis. There is, however, a price to pay when relying on numbers alone. Numerical analysis reduces the complexity of the social world. When analysing people, numbers present an illusion of precision and accuracy. Giving primacy to quantitative research in the social sciences comes at a high price. The dynamics of reality are reduced to statistics, losing the narrative of the people that the research aims to understand.

Being both an engineer and a social scientist, I acknowledge the importance of both numerical and qualitative methods. My dissertation used a mixed-method approach to review the relationship between employee behaviour and customer perception in water utilities. This article introduces some aspects of qualitative data science with an example from my dissertation.

In this article, I show how I analysed data from interviews using both quantitative and qualitative methods and demonstrate why qualitative data science is better to understand text than numerical methods.

Qualitative Data Science

The often celebrated artificial intelligence of machine learning is impressive but does not come close to human intelligence and ability to understand the world. Many data scientists are working on automated text analysis to solve this issue (the topicmodels package is an example of such an attempt). These efforts are impressive, but even the smartest text analysis algorithm is not able to derive meaning from text. To fully embrace all aspects of data science we need to be able to methodically undertake qualitative data analysis.

The capabilities of R in numerical analysis are impressive, but it can also assist with Qualitative Data Analysis (QDA). Huang Ronggui from Hong Kong developed the RQDA package to analyse texts in R. RQDA assists with qualitative data analysis using a GUI front-end to analyse collections texts.

For my dissertation about water utility marketing, I interviewed seven people from various organisations. The purpose of these interviews was to learn about the value proposition for water utilities. The data consists of the transcripts of six interviews which I manually coded using RQDA. For reasons of agreed anonymity, I cannot provide the raw data file of the interviews through GitHub.

Numerical Text Analysis

Word clouds are a popular method for exploratory analysis of texts. The wordcloud is created with the text mining and wordcloud packages. The transcribed interviews are converted to a text corpus (the native format for the tm package) and whitespace, punctuation etc is removed. This code snippet opens the RQDA file and extracts the texts from the database. RQDA stores all text in an SQLite database and the package provides a query command to extract data.

library(tidyverse)
library(RQDA)
library(tm)
library(wordcloud)
library(igraph)

openProject("stakeholders.rqda")
interviews <- data.frame(name = RQDAQuery("SELECT name FROM source"),
                         text = RQDAQuery("SELECT file FROM source"))
interviews <- Corpus(VectorSource(interviews$file))

interviews <-  tm_map(interviews, stripWhitespace)
interviews <-  tm_map(interviews, content_transformer(tolower))
interviews <-  tm_map(interviews, removeWords, stopwords("english"))
interviews <-  tm_map(interviews, removePunctuation)
interviews <-  tm_map(interviews, removeNumbers)
interviews <-  tm_map(interviews, removeWords, c("interviewer", "interviewee"))

# Word cloud
set.seed(1969)
wordcloud(interviews, min.freq = 10, max.words = 50, rot.per=0.35, 
          colors = brewer.pal(8, "Blues")[-1:-5])

Word cloud of interview transcripts.

This word cloud makes it clear that the interviews are about water businesses and customers, which is a pretty obvious statement. The interviews are also about the opinion of the interviewees (think). While the word cloud is aesthetically pleasing and provides a quick snapshot of the content of the texts, they cannot inform us about their meaning.

Topic modelling is a more advanced method to extract information from the text by assessing the proximity of words to each other. The topic modelling package provides functions to perform this analysis. I am not an expert in this field and simply followed basic steps using default settings with four topics.

dtm <- DocumentTermMatrix(interviews)
dtm <- removeSparseTerms(dtm, 0.99)
ldaOut <- LDA(dtm, k = 4)
terms(ldaOut,6)

This code converts the corpus created earlier into a Document-Term Matrix, which is a matrix of words and documents (the interviews) and the frequency at which each of these words occurs. The LDA function applies a Latent Dietrich Allocation to the matrix to define the topics. The variable k defines the number of anticipated topics. An LDA is similar to clustering in multivariate data. The final output is a table with six words for each topic.

Topic 1	Topic 2	Topic 3	Topic 4
water	water	customers	water
think	think	water	think
actually	inaudible	customer	companies
customer	people	think	yeah
businesses	service	business	issues
customers	businesses	service	don't

Topic modelling of interviews.

This table does not tell me much at all about what was discussed in the interviews. Perhaps it is the frequent use of the word "water" or "think"—I did ask people their opinion about water-related issues. To make this analysis more meaningful, I could perhaps manually remove the words water, yeah, and so on. This introduces a bias in the analysis and reduces the reliability of the topic analysis because I would be interfering with the text.

Numerical text analysis sees a text as a bag of words instead of a set of meaningful words. It seems that any automated text mining needs a lot of manual cleaning to derive anything meaningful. This excursion shows that automated text analysis is not a sure-fire way to analyse the meaning of a collection of words. After a lot of trial and error to try to make this work, I decided to go back to my roots of qualitative analysis using RQDA as my tool.

Qualitative Data Science Using RQDA

To use RQDA for qualitative data science, you first need to manually analyse each text and assign codes (topics) to parts of the text. The image below shows a question and answer and how it was coded. All marked text is blue, and the codes are shown between markers. Coding a text is an iterative process that aims to extract meaning from a text. The advantage of this method compared to numerical analysis is that the researcher injects meaning into the analysis. The disadvantage is that the analysis will always be biased, which in the social sciences is unavoidable. My list of topics was based on words that appear in a marketing dictionary so that I analysed the interviews from that perspective.

Example of text coded with RQDA.

My first step was to look at the occurrence of codes (themes) in each of the interviews.

## Load and transform data
codings <- getCodingTable()[,4:5]
categories <- RQDAQuery("SELECT filecat.name AS category, source.name AS filename 
                         FROM treefile, filecat, source 
                         WHERE treefile.catid=filecat.catid AND treefile.fid=source.id AND treefile.status=1")
codings <- merge(codings, categories, all.y = TRUE)
head(codings)

## Open coding
reorder_size <- function(x) {
    factor(x, levels = names(sort(table(x))))
}
ggplot(codings, aes(reorder_size(codename), fill=category)) + geom_bar(stat="count") + 
    facet_grid(~filename) + coord_flip() + 
    theme(legend.position="bottom", legend.title=element_blank()) + 
    ylab("Code frequency in interviews") + xlab("Code")

The code uses an internal RQDA function getCodingTable to obtain the primary data. The RQDAQuery function provides more flexibility and enables more complex queries of the data. You can view the structure of the RQDA database using the RQDATables function.

The occurrence of themes from six interviews.

This bar chart helps to explore the topics that interviewees discussed, but it does not help to understand how these topic relate to each other. This method provides better insights into the text than the 'bag of words' approach. Qualitative data analysis provides meaning to a text that no algorithm can discern. Although this meaning is ultimately subjective, proper methodology clarified any biases.

RQDA provides a facility to assign each code to a code category. A network diagram is a good tool to visualise this structure. The igraph package provides extensive functionality to analyse graphs.

Qualitative data analysis can create value from a text by interpreting it from a given perspective. This article is not even an introduction to the science and art of qualitative data science. I hope it invites you to explore RQDA and similar tools.

If you are interested in finding out more about this method, then feel free to read chapter three of my dissertation on customer service in water utilities.

The network of interview topics.

  edges <- RQDAQuery("SELECT codecat.name, freecode.name FROM codecat, freecode, treecode WHERE codecat.catid=treecode.catid AND freecode.id=treecode.cid")

  g <- graph_from_edgelist(as.matrix(edges), directed = FALSE) %>%
    simplify()
  V(g)$name <- gsub(" ", "\n", V(g)$name)

  c <- spinglass.community(g)
  par(mar=rep(0,4))
  set.seed(666)
  plot(c, g, 
       vertex.size = 10,
       vertex.color = NA,
       vertex.frame.color = NA,
       layout = layout.drl)

Tap Water Sentiment Analysis using Twitter and Tidytext

Peter Prevos — Thu, 29 Mar 2018 00:00:00 +0000

In developed countries, tap water is safe to drink and available cheaply. Even though high-quality drinking water is almost freely available, the consumption of bottled water is increasing every year. Bottled water companies use sophisticated marketing strategies, while water utilities are mostly passive public service providers. Australian marketing expert Russell Howcroft even called water utilities "lazy marketers". Can we use a tap water sentiment analysis to learn how people feel about tap water? What can we learn about the reasons behind this loss of trust in the municipal water supply?

Gruen Transfer (2009).

This tap water sentiment analysis estimates people's attitudes towards tap water by analysing tweets. This article explains how to examine tweets about tap water using the R language and the Tidytext package.

Sentiment Analysis

Sentiment indicates the perception a group of people has about a particular topic. Traditionally, assessing the sentiment of a group of people requires surveys or interviews. These methods are problematic because it creates an artificial environment where the respondent often answers to meet the perceived expectations of the study or customers exaggerate to get their point across. Using sentiment analysis of ego documents written by consumers can overcome these problems. Ego-documents, i.e. forms of personal writing, are a more direct way to find out what consumers think, but they are challenging to obtain and analyse.

With the advent of social media, access to ego documents has become much more straightforward, and many tools exist to collect and interpret this data. Using ego documents brings you closer to the consumer than possible with surveys or focus groups. One medium gaining popularity with market researchers is Twitter.

Tap Water Sentiment Analysis

Each tweet containing "tap water" reveals the author's attitude towards that topic. Each text expresses a sentiment about the subject it describes.

Sentiment analysis is a data science technique that extracts personal information from a text. The basic method compares a string of words with a set of words with calibrated sentiments. These calibrated sets are created by asking many people how they feel about a specific term. For example, "stink" expresses a negative sentiment, while "nice" would be a positive sentiment.

This tap water sentiment analysis consists of three steps. The first step extracts 1000 tweets containing "tap water" from Twitter. The second step cleans the data, and the third step analyses the results.

The Water Data Aggregator is a commercial application of sentiment analysis for tap water. This website sells the regional social credit score to interested parties.

Extracting tweets using the TwitteR package

The TwitteR package by Geoff Gentry makes it very easy to retrieve tweets using search criteria. You must create an API on Twitter to receive the keys and tokens. In the code below, the actual values have been removed.

Follow the instructions in this article to obtain these codes for yourself. This code snippet calls a private file to load the API codes, extracts the tweets and creates a data frame with a tweet id number and its text.

The TwitteR package is no longer availabe. A CSV file of tap water tweets is available on the GitHub repository.

  ## Tapwater tweet sentiment analysis
  library(tidyverse)
  library(tidytext)

  tapwater_tweets <- read_csv(paste0("data/tapwater-tweets-", Sys.time(), ".csv"))

When I first extracted these tweets, a tweet by CNN about tap water in Kentucky that smells like diesel was retweeted many times, so I removed all duplicate tweets from the set. Unfortunately, this left less than 300 original tweets in the corpus.

Sentiment analysis with Tidytext

Text analysis can be a powerful tool to help to analyse large amounts of text. The R language has an extensive collection of packages to help you undertake such a task. The Tidytext package extends the Tidy Data logic promoted by Hadley Wickham and his Tidyverse software collection.

Data Cleaning

The first step in cleaning the data is to create unigrams, which involve splitting the tweets into individual words that can be analysed. The first step is determining which words are most commonly used in the tap water tweets and visualising the result. The most common phrases are related to drinking and bottled water, which makes sense—also, the recent issues in Kentucky feature in this list.

Most common words in tweets about tap water.

  ## Tokenise and clean the tweets
  tidy_tweets <- tapwater_tweets %>%
    unnest_tokens(word, text)

  data(stop_words)

  tidy_tweets <- tidy_tweets %>%
    anti_join(stop_words) %>%
    filter(!word %in% c("tap", "water", "rt", "https", "t.co", "gt", "amp",
                        as.character(0:9)))
  ## Most common words
  tidy_tweets %>%
    count(word, sort = TRUE) %>%
    top_n(10) %>%
    mutate(word = reorder(word, n)) %>%
    ggplot(aes(word, n)) +
    geom_col(fill = "dodgerblue4") +
    xlab(NULL) + coord_flip() +
    theme_bw(base_size = 20) + 
    ggtitle("Most common words in tap water tweets")

Sentiment Analysis

The Tidytext package contains three lexicons of thousands of single English words (unigrams) that were manually assessed for their sentiment. The principle of sentiment analysis is to compare the words in the text with the words in the lexicon and analyse the results. For example, the statement: "This tap water tastes horrible" has a sentiment score of -3 in the AFFIN system by Finn Årup Nielsen due to the word "horrible". In this analysis, I have used the "Bing" method published by Liu et al. in 2005. This method assigns a word to either positive, negative or neutral sentiment.

This method is not foolproof, as words with the same spelling can mean different things. For example, "This tap water contains too much lead" will be assessed as a positive sentiment because the verb lead is seen as positive. The noun lead has no sentiment, as it depends on context.

The other problem with sentiment analysis is that we only see negative commentary. Very few people contact a water utility about an excellent morning shower or a wonderful glass of water. Tap water resides in the background of everyday life, and people don't have any opinion about it unless it is unavailable or does not meet their aesthetic expectations.

This tap water sentiment analysis shows that two-thirds of the words that express a sentiment were negative. The most common negative comments were "smells" and "scared". This analysis is not a positive result for water utilities. Unfortunately, most tweets were not spatially located, so that I couldn't determine the origin of the sentiment.

Tap water sentiment analysis.

  ## Sentiment analysis
  sentiment_bing <- tidy_tweets %>%
    inner_join(get_sentiments("bing"))

  sentiment_bing %>%
    summarise(Negative = sum(sentiment == "negative"), 
              positive = sum(sentiment == "positive"))

  sentiment_bing %>%
    group_by(sentiment) %>%
    count(word, sort = TRUE) %>%
    top_n(10) %>%
    ggplot(aes(word, n, fill = sentiment)) +
    geom_col(show.legend = FALSE) + 
    coord_flip() +
    facet_wrap(~sentiment, scales = "free_y") + 
    labs(title = "Contribution to sentiment", x = NULL, y = NULL)

Using Tap Water Sentiment Analysis

Sentiment analysis is an exciting exploration technique but not an absolute truth. This method cannot detect sarcasm or irony, and words don't always have the same meaning as described in the dictionary. For example, the algorithm interprets the word "lead" as positive sentiment in its function as a verb. In water, however, the chemical lead is never a positive sentiment. Interpreting tweets as bags of words is exciting but can lead to misunderstanding of the context.

The critical message for water utilities is that they need to start taking the aesthetic properties of tap water as seriously as the health parameters. A lack of trust will drive consumers to bottled water or less healthy alternatives such as soft drinks are alternative water sources.

Data Science for Water Utilities

Routledge Amazon

Analyse Digital Water Meter Data using the Tidyverse

Peter Prevos — Thu, 08 Feb 2018 00:00:00 +0000

In last week's article, I discussed how to simulate water consumption data to help develop analytics and reporting. This post describes how to analyse digital water meter data and create a diurnal consumption curve.

The code is available in the GitHub repository in the case-studies folder.

Data Source

The simulated data consists of three fields:

device_id: Unique identifier of the digital water meter.
timestamp: Time stamp in Coordinated Universal Time
count: Cumulative count of pulses.

Note that all data is recorded in UTC. The digital water meters send an hourly pulse at a random time within the hour. Each transmitter (RTU) uses a random offset to avoid network congestion. The digital meter counts each time the impeller makes a full turn, and for this analysis, we assume that this equates to a five-litre volume. The ratio between volume and count depends on the meter brand and type. The image below shows a typical data set for an RTU, including some missing data points.

Simulated water consumption with missing reads (In Australian time).

To analyse the data we need two auxiliary functions: one to slice the data we need and one to interpolate data for the times we need it. The Tidyverse heavily influences the code in this article. I prefer the Tidyverse way of doing things because it leads to elegant code that is easy to understand.

  ## Analyse digital meter data
  library(tidyverse)
  library(lubridate)
  library(magrittr)

  # Load data
  meter_reads <- read_csv("data/meter_reads.csv")
  rtu <- unique(meter_reads$device_id)

Prepare the Analysis

To construct a diurnal curve we need to slice to data to the required window and interpolate the hourly flows.

Slicing Digital Water Metering Data

Analysis is undertaken on slices of the complete data set. This function slices the available data by a vector of RTU ids and a timestamp range in AEST. This function adds a new timestamp variable in AEST. If no date range is provided, all available data for the selected RTUs is provided. The output of this function is a data frame (a Tibble in Tydiverse language).

  # Slicing meter reads
  
  slice_reads <- function(rtus, dates = range(meter_reads$timestamp)) {
    filter(meter_reads, device_id %in% rtus) %>%
      filter(timestamp >= as.POSIXct(dates[1], tz = "UTC") &
             timestamp <= as.POSIXct(dates[2], tz = "UTC")) %>%
      arrange(device_id, timestamp)
  }

  slice_reads(rtu[12], c("2050-02-12", "2050-02-13"))

Interpolation of Meter Reads

This function interpolates the cumulative counts for a series of RTUs over a vector of timestamps in AEST. The function creates a list to store the results for each RTU, interpolates the data using the approx function and then flattens the list back to a data frame. The interpolation function contains a different type of pipe because the approx for interpolation function does not take a data argument. The %$% pipe from the Magrittr package solves that problem.

The output is a data frame with DevEUI, the timestamp in AEST and the interpolated cumulative count. The image above shows the counts for two meters over two days an the graph superimposes an interpolated point over the raw data. Although the actual data consists of integer counts, interpolated values are numeric values. The decimals are retained to distinguish them from real reads.

  # Interpolation
  
  interpolate_count <- function(rtus, timestamps) {
      timestamps <- as.POSIXct(timestamps, tz = "UTC")
      results <- vector("list", length(rtus))
      for (r in seq_along(rtus)) {
          interp <- slice_reads(rtus[r]) %$%
              approx(timestamp, count, timestamps)
          results[[r]] <- data_frame(device_id = rep(rtus[r], 
                                                     length(timestamps)),
                                     timestamp = timestamps, count = interp$y)
      }
      return(do.call(rbind, results))
  }

  interpolate_count(rtu[2:3], seq.POSIXt(as.POSIXct("2050-02-01"), 
                                         by = "day", length.out = 3))

  sample <- 4:5

  slice_reads(rtu[sample], c("2050-02-01 20:00", "2050-02-02 04:00")) %>%
      ggplot(aes(timestamp, count, col = factor(device_id))) +
      geom_line() +
      geom_point() +
      geom_point(data = interpolate_count(rtu[sample], "2050-02-02 00:00"), 
                 col = "blue", size = 3) + 
      facet_wrap(~device_id, scale = "free_y") + 
      ggtitle(paste("Device", rtu[2:3])) +
      theme_bw(base_size = 10)

Interpolated digital metering data.

With these two auxiliary functions, we can start analysing the data.

Analyse Daily Consumption

Daily consumption for each connection is a critical metric in managing water resources and billing customers. The daily consumption of any water connection is defined by the difference between the cumulative counts at midnight. The interpolation function makes it easy to determine daily consumption. This function interpolates the midnight reads for each of the RTUs over the period, starting the previous day. The output of the function is a data frame that can be piped into the plotting function to visualise the data. When you group the data by date, you can also determine the total consumption over a group of services.

  # Daily consumption
  daily_consumption <- function(rtus, dates) {
    timestamps <- seq.POSIXt(as.POSIXct(min(dates), tz = "UTC") - 24 * 3600,
                             as.POSIXct(max(dates), tz = "UTC"), by = "day")
    interpolate_count(rtus, timestamps) %>%
      group_by(device_id) %>%
      mutate(consumption = c(0, diff(count)) * 5) %>%
      filter(timestamp != timestamps[1])
  }

  daily_consumption(rtu[32:33], c("2050-02-01", "2050-02-07")) %>%
    ggplot(aes(timestamp, consumption)) +
    geom_col() +
    facet_wrap(~device_id) +
    theme_bw(base_size = 10) +
    ggtitle("Daily consumption")

Daily water consumption.

Diurnal Curves

The diurnal curve is one of the most important pieces of information used in the design of water supply systems. This curve shows the usage of one or more services for each hour in the day. This curve is a reflection of human behaviour, as we use most water in the morning and the evenings.

This function slices data for a vector of RTUs over a period and then plots the average diurnal curve. The data is obtained by interpolating the cumulative counts for each whole hour in the period. The function then calculates the flow in litres per hour and visualises the minimum, mean and maximum value.

  # Diurnal curves
  
  plot_diurnal_connections <- function(rtus, dates) {
    timestamps <- seq.POSIXt(as.POSIXct(dates[1], tz = "UTC"),
                             as.POSIXct(dates[2], tz = "UTC"), by = "hour")
    interpolate_count(rtus, timestamps) %>%
      group_by(device_id) %>%
      mutate(rate = c(0, diff(count * 5)),
             hour = as.integer(format(timestamp, "%H"))) %>%
      filter(rate >= 0) %>%
      group_by(hour) %>%
      summarise(min = min(rate), mean = mean(rate), max = max(rate)) %>%
      ggplot(aes(x = hour, ymin = min, ymax = max)) +
      geom_ribbon(fill = "lightblue", alpha = 0.5) +
      geom_line(aes(x = hour, y = mean), col = "orange", linewidth = 1) +
      ggtitle("Connections Diurnal flow") +
      ylab("Flow rate [L/h]") + 
      theme_bw(base_size = 10)
  }

  plot_diurnal_connections(rtu[10:20], c("2050-02-01", "2050-03-01"))

Boxplots are also an informative way to visualise this curve. This method provides more statistical information on one page, and the ggplot function performs the statistical analysis.

Diurnal curve in UTC.

Analysing digital metering data is explained in more detail in chapter 11 of Data Science for Water Utilities.

Data Science for Water Utilities

Routledge Amazon

Simulating Water Consumption to Develop Analysis Tools

Peter Prevos — Thu, 01 Feb 2018 10:00:19 +0000

I am currently working on developing analytics for a digital water metering project. Over the next five years, we are enabling 70,000 customer water meters with digital readers and transmitters. The data is yet to be available, but we want to build reporting systems before the data is live. The R language comes to the rescue as it has magnificent capabilities to simulate data. Simulating data is a valuable technique to progress a project when data is being collected. Simulated data also helps because the analysis outcomes are known, which allows for validating the results.

The raw data that we will eventually receive from the digital customer meters has the following basic structure:

device_id: Unique device identifier.
timestamp: Date and time in (UTC) of the transmission.
count: The number of revolutions the water meter makes. Each revolution is a pulse which equates to five litres of water in a regular customer water meter.

Every device will send an hourly data burst which contains the cumulative meter read in pulse counts. The transmitters are set at a random offset from the whole hour to minimise the risk of congestion at the receivers. The timestamp for each read is set in the Coordinated Universal Time (UTC). Using this time zone prevents issues with daylight savings. All analysis will be undertaken in the Australian Eastern (Daylight) Time zone.

This article explains how we simulated test data to assist with developing reporting and analysis. The analysis of digital metering data follows in a future post.

Simulating water consumption

For simplicity, this simulation assumes a standard domestic diurnal curve (average daily usage pattern) for indoor water use. Diurnal curves are an essential piece of information in water management. The curve shows water consumption over a day, averaged over a fixed period. The example below is sourced from a journal article. This generic diurnal curve consists of 24 data points based on measured indoor water consumption, shown in the graph below.

Source: Gurung et al. (2014) Smart meters for enhanced water supply network modelling and infrastructure planning. Resources, Conservation and Recycling (90), 34-50.

This diurnal curve only includes indoor water consumption and is assumed independent of seasonal variation. This assumption is unrealistic, but the purpose of this simulation is not to accurately model water consumption but to provide a data set to validate the reporting and analyses.

Simulating water consumption in R

The first code snippet sets the parameters used in this simulation. The unique device identifiers (DevEUI) are simulated as six-digit random numbers. The timestamps vector consists of hourly date-time variables in UTC. For each transmitter, this timestamp is offset by a random time. Each transmitter is also associated with the number of people living in each house. This number is based on a Poisson distribution.

  # Simulate water consumption

  library(tidyverse)

  rm(list = ls())

  # Boundary conditions
  n <- 100 # Number of simulated meters
  d <- 100 # Number of days to simulate
  s <- as.POSIXct("2050-01-01", tz = "UTC") # Start of simulation

  set.seed(1969) # Seed random number generator for reproducibility
  rtu <- sample(1E6:2E6, n, replace = FALSE) # 6-digit id
  offset <- sample(0:3599, n, replace = TRUE) # Unique Random offset for each RTU

  # Number of occupants per connection
  occupants <- rpois(n, 1.5) + 1

  # Visualise
  as_tibble(occupants) %>%
    ggplot(aes(occupants)) + 
    geom_bar(fill = "dodgerblue2", alpha = 0.5) +
    xlab("Occupants") + 
    ylab("Connections") +
    theme_bw(base_size = 10) +   
    labs(title = "Occupants per connection")

Simulated number of occupants per connection.

The diurnal curve is based on actual data, including leaks, as the nighttime use shows a consistent flow of about one litre per hour. Therefore, the figures are rounded and reduced by one litre per hour to show a zero flow when people are usually asleep. The curve is also shifted by eleven hours because the raw data is stored in UTC.

  # Diurnal Curve
  diurnal_au <- round(c(1.36, 1.085, 0.98, 1.05, 1.58, 3.87, 9.37, 13.3, 12.1, 10.3, 8.44, 7.04, 6.11, 5.68, 5.58, 6.67, 8.32, 10.0, 9.37, 7.73, 6.59, 5.18, 3.55, 2.11)) - 1

  tdiff <- 11
  diurnal_utc <- c(diurnal_au[(tdiff + 1): 24], diurnal_au[1:tdiff])

This simulation only aims to simulate a realistic data set and not to present an accurate depiction of reality. This simulation could be enhanced by using different diurnal curves for various customer segments, including outdoor watering, temperature dependencies and so on.

Simulating Water Consumption

A leak is defined by a constant flow through the meter and the idealised diurnal curve. A weighted binomial distribution (θ = 0.1) models approximately one in ten properties with a leak. The size of the leak is derived from a random number between 10 and 50 litres per hour.

The data is stored in a matrix through a loop that cycles through each connection. The DevEUI is repeated over the simulated period (24 times the number of days). The second variable is the timestamp plus the predetermined offset for each RTU.

The meter count is defined by the cumulative sum of the diurnal flow multiplied by the number of occupants. Each point in the diurnal deviates from the model curve by ±10%. Any predetermined leakage is added to each meter read over the whole period of 100 days. The hourly volumes are summed cumulatively to simulate meter reads. The flow is divided by five as each meter revolution indicates five litres.

The following code snippet simulates the digital metering data using the above assumptions and parameters.

  # Leaking properties
  leaks <- rbinom(n, 1, prob = .1) * sample(10:50, n, replace = TRUE)
  data.frame(device_id = rtu, leak = leaks) %>%
    subset(leak > 0)

  # Digital metering data simulation
  meter_reads <- matrix(ncol = 3, nrow = 24 * n * d)
  colnames(meter_reads) <- c("device_id", "timestamp" , "count")

  for (i in 1:n) {
    r <- ((i - 1) * 24 * d + 1):(i * 24 * d)
    meter_reads[r, 1] <- rep(rtu[i], each = (24 * d))
    meter_reads[r, 2] <- seq.POSIXt(s, by = "hour", length.out = 24 * d) + offset[i]
    meter_reads[r, 3] <- round(cumsum((rep(diurnal_utc * occupants[i], d) +
                                       leaks[i]) * runif(24 * d, 0.9, 1.1)) / 5)
  }

  meter_reads <- as_tibble(meter_reads) %>%
    mutate(timestamp = as.POSIXct(timestamp, origin = "1970-01-01", tz = "UTC"))

Missing Data Points

The data transmission process is not 100% reliable, and the base station will not receive some reads. This simulation identifies reads to be removed from the data through the temporary variable removal. This simulation includes two types of failures:

Faulty RTUs (2% of RTUs with missing 95% of data)

Randomly missing data points (1% of data)

  # Set missing indicator
  meter_reads <- mutate(meter_reads, remove = 0)

  # Define faulty RTUs (2% of the fleet)
  faulty <- rtu[rbinom(n, 1, prob = 0.02) == 1]
  meter_reads$remove[meter_reads$device_id %in% faulty] <- 
    rbinom(sum(meter_reads$device_id %in% faulty), 1, prob = .5)

  # Data loss
  missing <- sample(1:(nrow(meter_reads) - 5), 0.005 * nrow(meter_reads))
  for (m in missing){
    meter_reads[m:(m + sample(1:5, 1)), "remove"] <- 1
  }

  # Remove missing reads
  meter_reads <- filter(meter_reads, remove == 0) %>%
    select(-remove)

  # Write to disk
  write_csv(meter_reads, "data/meter_reads.csv")

  ##Visualise
  filter(meter_reads, device_id %in% sample(faulty, 1)) %>%
    mutate(timestamp = as.POSIXct(format(timestamp,
                                         tz = "Australia/Melbourne"))) %>%
    filter(timestamp >= as.POSIXct("2050-02-06") &
           timestamp <= as.POSIXct("2050-02-08")) %>%
    arrange(device_id, timestamp) %>%
    ggplot(aes(x = timestamp, y = count, colour = factor(device_id))) +
    geom_line() + 
    geom_point()

The graph shows an example of the cumulative reads and some missing data points.

Simulated water consumption with missing reads.

Analysing Digital Metering Data

Data simulation is a good way to develop your analysis algorithms before you have real data. I have also used this technique while waiting for survey results during my dissertation. When the data finally arrived, I plugged it into the code and finetuned it. R has great capabilities to simulate reality to help you understand the data. The ggplot package provides excellent functionality to visualise water consumption.

In next week's article, I will outline how I used R and the Tidyverse package to develop libraries to analyse digital metering data.

This data was used as a case study in the book Data Science for Water Utilities.

Data Science for Water Utilities

Routledge Amazon

Project Euler 32: Pandigital Products

Peter Prevos — Thu, 21 Sep 2017 00:00:00 +0000

Project Euler 32 returns to pandigital numbers, which are numbers that contain one of each digit. Like so many of the Euler Problems, these numbers serve no practical purpose whatsoever. Just like palindromic products, other than some entertainment value and increasing your understanding of numbers.

You can find all pandigital numbers with digits zero to nine in sequence A050278 of the Online Encyclopedia of Integers. The Numberphile video below explains everything you ever wanted to know about pandigital numbers but were afraid to ask.

Why 381,654,729 is awesome – Numberphile.

Project Euler 32 Definition

We shall say that an n-digit number is pandigital if it makes use of all the digits 1 to n exactly once; for example, the 5-digit number, 15234, is 1 through 5 pandigital.

The product 7254 is unusual, as the identity, 39 × 186 = 7254, containing multiplicand, multiplier, and product is 1 through 9 pandigital.

Find the sum of all products whose multiplicand/multiplier/product identity can be written as a 1 through 9 pandigital.

HINT: Some products can be obtained in more than one way, so be sure to only include it once in your sum.

Proposed Solution

The pandigital.9() function tests whether a string classifies as a pandigital number. The pandigital.prod vector stores the multiplication.

The only way to solve this problem is by brute force and try all multiplications. We can limit the solution space to a manageable number. The multiplication result needs to have ten digits. For example, when the starting number has two digits, the second number should have three digits so that the total has four digits, e.g., 39 × 186 = 7254. When the first number only has one digit, the second number needs to have four digits.

  ## Project Euler 32: Pandigital Products

  pandigital.9 <- function(x) # Test if string is 9-pandigital
      (length(x)==9 & sum(duplicated(x))==0 & sum(x==0)==0)

  ## Brute force
  pandigital.prod <- vector()
  i <- 1
  for (m in 2:100) {
    if (m < 10) 
      n_start <- 1234 
    else 
      n_start <- 123
    for (n in n_start:round(10000 / m)) {
                                          # List of digits
      digs <- as.numeric(unlist(strsplit(paste0(m, n, m * n), "")))
                                          # is Pandigital?
      if (pandigital.9(digs)) {
        pandigital.prod[i] <- m * n
        i <- i + 1
        print(paste(m, "*", n, "=", m * n))
      }
    }
  }
  answer <- sum(unique(pandigital.prod))
  print(answer)

Analysing soil moisture data with the ncdf4 library

Peter Prevos — Thu, 07 Sep 2017 00:00:00 +0000

The netCDF format is popular in sciences that analyse sequential spatial data. It is a self-describing and machine-independent data format for creating, accessing and sharing array-oriented information. The netCDF format provides spatial time series such as meteorological or environmental data. This article shows how to load this data type with the ncdf4 library. This article also shows how to visualise and analyse this data format by reviewing soil moisture data published by the Australian Bureau of Meteorology.

Soil Moisture data

The Australian Bureau of Meteorology publishes hydrological data in the NetCDF format on their Australian Water Outlook. The NetCDF format provides a three-dimensional matrix of spatial data over time. You need to download the monthly time aggregation, which is approximately 1GB large.

Australian Landscape Water Balance (Bureau of Meteorology).

Reading, Extracting and Transforming with the ncdf4 library

The ncdf4 library, developed by David W. Pierce, provides the necessary functionality to manage this data. To install this R library on a non-Windows system, you must first install the NetCDF package for your relevant operating system.

The first step is to load the data, extract the relevant information and transform the data for visualisation and analysis. The data forms a complex list containing the metadata and measurements. The ncvar_get() function extracts data from the list.

The time data is stored as the number of days since 1 January 1900. The spatial coordinates are stored in decimal degrees with 0.05-decimal degree intervals. The moisture data is transformed to a three-dimensional matrix with dimensions of longitude, latitude, and time.

  ## Analysing soil moisture data with the ncdf4 library

  ## Reading, Extracting and Transforming with the ncdf4 library
  library(ncdf4)

  bom <- nc_open("data/sm_pct.nc")

  ## Generate matrix
  longitude <- ncvar_get(bom, "longitude")
  latitude <- ncvar_get(bom, "latitude")
  month <- as.Date("1900-01-01") + ncvar_get(bom, "time")

  moisture <- ncvar_get(bom, "sm_pct")
  dimnames(moisture) <- list(longitude, latitude, month)

Visualising the data with ggplot

This first code snippet extracts and plots a matrix from the cube for 31 May 2025. This code creates a data frame with the grid for the required date. Although I use the Tidyverse, I still need reshape2 because the pivot_longer function does not like matrices.

  ## Visualising the data with ggplot
  library(ggplot2)
  library(dplyr)
  library(reshape2)
  library(ozmaps)


  plot_date <- "2025-05-31"

  ## Generate data frame
  soil_moisture <- moisture[, , which(month == plot_date)] %>%
    melt(varnames = c("longitude", "latitude")) %>%
    subset(!is.na(value))

  ## State boundaries
  sf_aus <- ozmap("states")

  ggplot(soil_moisture) + 
    geom_tile(aes(x = longitude, y = latitude, fill = value)) +
    scale_fill_gradient(low = "chocolate4", high = "dodgerblue", name = NULL) + 
    geom_sf(data = sf_aus, fill = NA, col = "white") + 
    labs(title = "Relative Total rootzone soil moisture (0-100 cm)",
         subtitle = format(as.Date(plot_date), "%d %B %Y"),
         caption = "Source: Australian Landscape Water Balance AWRA-L Model") + 
    theme_void()

Relative soil moisture in Australia, May 2025.

With the ggmap package we can add some mapping tiles. You will need a Google Maps API to enable this functionality.

  ## Plot on Google Maps
  library(ggmap)
  
  api <- readLines("case-studies/google-maps.api") # Text file with the API key
  register_google(key = api)

  ggmap(get_map("Australia", zoom = 4, color = "bw")) +
    geom_tile(data = soil_moisture, 
              aes(x = longitude, y = latitude, fill = value), alpha = 0.5) +
    scale_fill_gradient(low = "chocolate4", high = "dodgerblue", name = NULL) + 
    labs(title = "Relative Total rootzone soil moisture (0-100 cm)",
         subtitle = format(as.Date(plot_date), "%d %B %Y"),
         caption = "Source: Australian Landscape Water Balance AWRA-L Model") + 
    theme_void()

Relative soil moisture in Australia, May 2025.

Analysing the data

For my analysis, I am interested in the time series of moisture data for a specific point on the map. The previous code slices the data for all location at a specific month.

We can also pierce the data vertically for a particular coordinate to create a time series. The coordinate needs to be rounded to the nearest 0.05 decimal degrees to match the source data.

This time series was used to investigate the relationship between sewer main blockages and deep soil data, which can be a topic for a future post.

Soil moisture time series for Bendigo.

  ## Time series
  location <- round(geocode("Bendigo, Australia") / 0.05) * 0.05 

  soil_moisture_time <- tibble(month,
                               moisture = moisture[as.character(location$lon), 
                                                   as.character(location$lat), ]) %>% 
    filter(month >= "2000-01-01")

  ggplot(soil_moisture_time) + 
    geom_line(aes(x = month, y = moisture)) +
    labs(title = "Relative Total rootzone soil moisture (0-100 cm)",
         subtitle = paste(as.character(location), collapse = ", "),
         caption = "Source: Australian Landscape Water Balance AWRA-L Model",
         x = NULL, y = "Relative Soil Moisture") + 
    theme_minimal(base_size = 16)

Data Science for Water Utilities

Routledge Amazon

Pacific Island Hopping using R and the iGraph package

Peter Prevos — Thu, 31 Aug 2017 10:00:56 +0000

Last month I enjoyed a relaxing holiday in the tropical paradise of Vanuatu. One rainy day I contemplated how to go island hopping across the Pacific ocean visiting as many island nations as possible. The Pacific ocean is a massive body of water between, Asia and the Americas, which covers almost half the surface of the earth. The southern Pacific is strewn with island nations from Australia to Chile. In this post, I describe how to use R to plan your next Pacific island hopping journey.

The Pacific Ocean with UN Geoscheme regions (source: Oceania_ISO_3166-1.svg: User:Tintazulderivative work: Cruickshanks / CC BY-SA).

Geocoding the airports

My first step was to create a list of flight connections between each of the island nations in the Pacific ocean. I am not aware of a publicly available data set of international flights so unfortunately, I created the list manually by collecting route maps on Pinterest (if you do know of a fomal data set with international flights, then please leave a comment).

My manual research resulted in a list of international flights from or to island airports. This list might not be complete, but it is a start. My Pinterest board with Pacific island airline route maps was the information source for this list.

The first code section reads the list of airline routes and uses the ggmap package to extract their coordinates from Google maps. You will need a Google API to enable the geocoding function. The data frame with airport coordinates is saved for future reference to avoid repeatedly pinging Google for the same information.

  library(ggmap)
  library(ggplot2)
  library(ggrepel)
  library(geosphere)
  library(tidyverse)
  library(stringr)

  ## Google maps api
  api <- readLines("google.api") # Text file with the API key
  register_google(key = api)

  ## Read flight list and airport list
  flights <- read_csv("pacific/pacific-flights.csv")
  f <- "pacific/pacific-airports.csv"
  if (file.exists(f)) {
    airports <- read.csv(f)
  } else
    airports <- data.frame(airport = NA, lat = NA, lon = NA)

  ## Lookup coordinates for new airports
  all_airports <- unique(c(paste(flights$From, flights$From.Country, sep = ", "),
                           paste(flights$To, flights$To.Country, sep = ", ")))
  new_airports <- all_airports[!(all_airports %in% airports$airport)]

  while (length(new_airports) != 0) {
    coords <- geocode(new_airports)
    temp_airports <- data.frame(airport = new_airports, coords)
    airports <- rbind(airports, temp_airports) %>%
      filter(!is.na(lat), !is.na(lon))
    new_airports <- all_airports[!(all_airports %in% airports$airport)]
  }
  ## Remove countries and save
  airports$airport <- str_remove_all(airports$airport, ",.*")
  write_csv(airports, f)

  ## Add coordinates to flight list
  flights <- merge(flights, airports, by.x = "From", by.y = "airport")
  flights <- merge(flights, airports, by.x = "To", by.y = "airport")

Create the map

To create a map, I modified the code to create flight maps I published in an earlier article. This code had to be changed to centre the map on the Pacific.

Mapping the Pacific ocean is problematic because the -180 and +180 degree meridians meet around the dateline. Longitudes west of the antemeridian are positive, while longitudes east are negative. The world2 data set in the borders function of the ggplot2 package is centred on the Pacific ocean. To enable plotting on this map, all negative longitudes are made positive by adding 360 degrees to them and defining the antipode.

  ## Pacific centric
  flights$lon.x[flights$lon.x < 0] <- flights$lon.x[flights$lon.x < 0] + 360
  flights$lon.y[flights$lon.y < 0] <- flights$lon.y[flights$lon.y < 0] + 360
  airports$lon[airports$lon < 0] <- airports$lon[airports$lon < 0] + 360

  ## Plot flight routes
  worldmap <- borders("world2", fill = "grey", col = NA)
  ggplot() + worldmap +
    geom_point(data = airports, aes(x = lon, y = lat), col = "#970027") +
    geom_text_repel(data = airports, aes(x = lon, y = lat, label = airport), col = "black", size = 2,
                    segment.color = NA) +
    geom_curve(data = flights, aes(x = lon.x, y = lat.x, xend = lon.y, yend = lat.y, col = Airline), size = 1,
               curvature = .2) +
    xlim(90, 300) + ylim(-50, 50) +
    theme_void() +
    labs(title = "Pacific Island Hopping", subtitle = "lucidmanager.org")
  ggsave("pacific-flights.png", width = 6, height = 4)

Pacific island hopping - flight route map.

Pacific Island Hopping

This visualisation is aesthetic and full of context, but it is not the best visualisation to solve the travel problem. This map can also be expressed as a graph with nodes (airports) and edges (routes). Once the map is represented mathematically, we can generate travel routes and begin our Pacific Island hopping.

The igraph package converts the flight list to a graph that can be analysed and plotted. The shortest_path function can then be used to plan routes. If I would want to travel from Auckland to Saipan in the Northern Mariana Islands, I have to go through Port Vila, Honiara, Port Moresby, Chuuk, Guam and then to Saipan. I am pretty sure there are quicker ways to get there, but this would be an exciting journey through the Pacific.

  library(igraph)
  g <- graph_from_edgelist(as.matrix(flights[,1:2]), directed = FALSE)

  png("pacific-network.png", width = 600, height = 600)
  par(mar = rep(0, 4))
  plot(g, layout = layout.fruchterman.reingold,
       vertex.size = 0,
       vertex.label.cex = 1.2,
       edge.width = 2)
  dev.off()

  shortest_paths(g, "Auckland", "Saipan")

Pacific island hopping - network model.

This diagram looks a lot like a Polynesian stick chart. These are maps made from sticks bound together with strings. These maps allowed the per-colonial Polynesians to navigate between their islands. The maps are not mathematical projections of the earth, like our first map, but a logical model.

Stick charts indicate the local swell patterns that allowed the ancient mariners to find their way across the vast distances in the pacific. All we have to do is hop on a plane and we get to wherever we want withing a few hours.

Happy island hopping!

Project Euler 30: Digit Fifth Powers

Peter Prevos — Thu, 24 Aug 2017 00:00:00 +0000

Project Euler 30 is another number-crunching problem that deals with numbers to the power of five. Two other Euler problems dealt with raising numbers to a power. The previous problem looked at permutations of powers, and problem 16 asks for the sum of the digits of $2^{1000}$.

Project Euler 30 Definition

Surprisingly there are only three numbers that can be written as the sum of fourth powers of their digits:

$$1634 = 1^4 + 6^4 + 3^4 + 4^4$$

$$8208 = 8^4 + 2^4 + 0^4 + 8^4$$

$$9474 = 9^4 + 4^4 + 7^4 + 4^4$$

As $1 = 1^4$ is not a sum, it is not included.

The sum of these numbers is $1634 + 8208 + 9474 = 19316$. Find the sum of all the numbers that can be written as the sum of the fifth powers of their digits.

Proposed Solution

The problem asks for a brute-force solution, but we have a halting problem. How far do we need to go before we can be sure there are no sums of fifth power digits? Do we need to search until infinity, which means we can never solve the problem? We can limit the solution space because the solution will have lower and upper limits. An upper bound is a number with $log 9^5$ digits.

In base ten, the highest digit is 9 and $9^5=59049$, which has five digits. This implies that the highest possible number, which equals the sum of the fifth power of its digits is 5 times $9^5=295245$. The lowest possible number is 2 times $2^5 = 64$. All numbers with this property thus must range between these two values.

The code to solve this project first declares the exponent and determines the lower and upper boundaries of the solution space. Change the variable n to try other exponents. The loop cycles through the digits of each number and tests whether the sum of the fifth powers equals the number.

The digit sum of a number $x$ in base 10 is given by:

$$\sum_{n=0}^{\lfloor \log_{10} x\rfloor} \frac{1}{10^n}(x \bmod 10^{n + 1} - x \bmod 10^n)$$

  ## Project Euler 30: Digit Fifth Powers
  m <- 5
  highest <- round(log10(9^m)) * 9^m
  answer <- 0
  for (x in 2:highest) {
      n <- 0:log10(x)
      power.sum <- sum(((1/10^n) * (x %% 10^(n + 1) - x %% 10^n))^m)
      if (power.sum == x) {
          print(x)
          answer <- answer + x
      }
  }
  print(answer)

Digit sums

The mathematics in this problem is about the digit sum, which is the sum of all digits in the relevant number base.

Computer scientists use digit sums in cryptography and data validation. You can find the digital sum base-10 numbers in sequence A007953 of the Online Encyclopedia of Integer Sequences. Checksum algorithms use digit sums as a quality assurance method. For example, the final digit in the ten-digit International Standard Book Numbers is a check digit. You can compute the check digit by multiplying each digit by its position in the ISBN number (alternating by 3 and 1), and taking the sum of these products. The check digit is 0 when the modules 10 of this modulus is zero, otherwise it is 10 minus the remainder. In R code, determining the check digit for a 13-digit ISBN number works as follows:

  ## Check ISB number
  check_isbn13 <- function(isbn) {
      isbn <- gsub(" |-", "", isbn)
      check_sum <- 0
      for (i in 1:12) {
          m <- ifelse(i %% 2 == 0, 3, 1)
          check_sum <- check_sum + m * as.numeric(substr(isbn, i, i))
      }
      check_sum <- ifelse(check_sum %% 10 == 0, 0, 10 - check_sum %% 10)
      check_sum == substr(isbn, 13, 13)
  }
  check_isbn13("978-0-9875669-4-2")

Visualise Water Consumption with a Geographic Bubble Chart

Peter Prevos — Thu, 17 Aug 2017 00:00:00 +0000

A geographic bubble chart is a straightforward method to visualise quantitative information with a geospatial relationship. Last week, I was in Vietnam helping the Phú Thọ Water Supply Joint Stock Company with their data science. They asked me to create a map of a sample of their water consumption data. In this post, I share this little ditty to explain how to plot a bubble chart over a map using the ggmap package.

In this post, I share this little ditty to explain how to plot a bubble chart over a map using the ggmap package. Thanks to Ms Quy and Mr Tuyen of Phu Tho Water for their permission to use this data. Other posts on this blog detail how to analyse water consumption from digital metering data.

This map visualises water consumption in the targeted area of Việt Trì. The larger the bubble, the larger the consumption. It is no surprise that two commercial customers used the most water.

Viet Water Conference 2018.

Load and Explore the Data

The sample data contains a list of just over 100 readings from water meters in the city of Việt Trì, Vietnam, along with their geospatial locations. This data uses the World Geodetic System of 1984 (WGS84), compatible with Google Maps and similar systems.

  # Water consumption bubble chart

  water <- read.csv("data/viet-tri-consumption.csv")
  water$Consumption <- water$read_new - water$read_old

  ## Summarise the data
  summary(water$Consumption)

Consumption per connection ranges from 0 to 529 cubic metres, with an average of 23.45 cubic metres.

Visualise the data with a geographic bubble chart

With the ggmap extension of the ggplot package, we can visualise any spatial data set on a map. The only condition is that the spatial coordinates are in the WGS84 datum. The ggmap package adds a geographical layer to ggplot by providing a Google Maps or OpenStreetMap canvas. The first step is to download the map canvas. You need to know the centre coordinates and the zoom factor to do this. Determining the perfect zoon factor requires some trial and error. The ggmap package provides for various map types, which are described in detail in the documentation. You will need a Google Maps API to enable this functionality.

  ## Plot on Google Maps
  library(ggmap)

  api <- readLines("data/google-maps.api") # Text file with the API key
  register_google(key = api)

  centre <- c(mean(range(water$lon)), mean(range(water$lat)))
  viettri <- get_map(centre, zoom = 17, maptype = "hybrid")
  g <- ggmap(viettri)

The ggmap package follows the same conventions as ggplot. We first call the map layer and then add any required geom. The point geom creates a nice bubble chart when combined with the scale_size_area option. This option scales the points to a maximum size to ensure they are easily visible. The transparency (alpha) minimises the problem of overplotting. This last code snippet plots the map with water consumption.

Water consumption in Việt Trì.

  # Add data to ggmap  
  g + geom_point(data = water, aes(x = lon, y = lat, size = Consumption),
                 shape = 21, colour = "dodgerblue4", fill = "dodgerblue", 
                 alpha = .8) +
    scale_size_area(max_size = 30) + # Size of the biggest point
    ggtitle("Việt Trì sự tiêu thụ nước")

Data Science for Water Utilities

Routledge Amazon

Project Euler 29: Distinct Powers

Peter Prevos — Thu, 03 Aug 2017 00:00:00 +0000

Project Euler 29 is another permutation problem that is quite easy to solve using brute force. The now defunct MathBlog site by Kristian Edlund has a nice solution using only pen and paper.

Euler 29 Definition

Consider all integer combinations of: $a^b$ for $2 \leq a \leq 5$ and $b \leq b \leq 5$ .

$$2^2=4, \quad 2^3 = 8, \quad 2^4 = 16, \quad 2^5 = 32$$

$$3^2 = 9,\quad 3^3 = 27, \quad 3^4 = 81,\quad 3^5 = 243$$

$$4^2 = 16,\quad 4^3 = 64, \quad 4^4 = 256, \quad 4^5 = 1024$$

$$5^2 = 25,\quad 5^3 = 125, \quad 5^4 = 625,\quad 5^5 = 3125$$

If they are then placed in numerical order, with any repeats removed, we get the following sequence of 15 distinct terms:

$$4, \ 8, \ 9, \ 16, \ 25, \ 27, \ 32, \ 64, \ 81, \ 125, \ 243, \ 256,\ 625, \ 1024, \ 3125$$

How many distinct terms are in the sequence generated by $a^b$ for $2 \leq a \leq 100$ and $2 \leq b \leq 100$ ?

Proposed Solution

This code calculates all powers from $2^2$ to $2^{100}$ and determines the number of unique values. There is no need to use Multiple Precision Arithmetic because we are only interested in their uniqueness and not the precise value.

  ## Project Euler 29: Distinct Powers
  target <- 100
  terms <- vector()
  i <- 1
  for (a in 2:target) {
    for (b in 2:target) {
      terms[i] <- a^b
      i <- i + 1
    }
  }
  answer <- length(unique(terms))
  print(answer)

How Virtual Tags have transformed SCADA data analysis

Peter Prevos — Thu, 27 Jul 2017 00:00:00 +0000

Yesterday, I delivered the International Keynote at the Asset Data & Insights Conference in Auckland, New Zealand (the place where R was initially developed). My talk was about creating value from CADA data using a method I developed with my colleagues called "Virtual Tags." My address started with my views on data science strategy, which I also presented to the R User Group in Melbourne. In this post, I would like to explain what Virtual Tags are and how they can be used to improve the value of SCADA data.

SCADA Systems at Water Treatment Plants

Water treatment plants are mostly fully automated, using analysers and the SCADA system to communicate this data. For those of you not familiar with water treatment plants, the video below gives a cute summary of the process.

Water Treatment — SCADA Plant IQ

Water treatment plants rely on sensors to track a broad range of parameters. These instruments capture data around the clock, supporting real-time operational decisions. When everything runs smoothly, all values stay within a precise band. Typically, these values are archived by the SCADA system for a year, then purged to manage storage needs.

Turbidity measurements—essentially a gauge of water clarity—help assess filtration effectiveness at water treatment plants. Below, you’ll find a code snippet simulating five hours of turbidity readings from one such instrument. Most of the time, these water quality values follow a log-normal distribution and cluster tightly together, making them ideal for precise monitoring.

  # Virtual tag simulation

  # Generate turbidity measurements
  set.seed(1234)
  n <- 300
  wtp <- data.frame(timestamp = seq.POSIXt(ISOdate(1910, 1, 1), length.out = n, by = 60),
                    turbidity = rlnorm(n, log(.1), .01))

  library(ggplot2)

  p1 <- ggplot(wtp) +
    aes(x = timestamp, y = turbidity) +
    geom_line(colour = "grey") +
    ylim(0.09, 0.11) + 
    theme_bw(base_size = 10) + 
    labs(title = "Turbidity simulation", x = "Timestamp", y = "Turbidity")

  p1

Turbidity simulation.

SCADA Historian

The data generated by the SCADA system is used to make operational decisions. The data is created and structured to make decisions in the present, not to solve problems in the future. SCADA Historian systems archive this information for future analysis. Historian systems only store new values when the new reading is more or less than a certain percentage of the previous one. This method saves storage space without sacrificing much accuracy.

Imagine an instrument reads 0.20, and the limit is 5%: you’ll see new values recorded only when they fall below 0.19 or above 0.21. Each subsequent value gets stored only when it deviates by 5% from the last stored point. The following code snippet brings this process to life, simulating it using earlier turbidity readings. Notice: this Historian captures and stores only the data points shown in black.

  # Historise using dead banding
  threshold <- 0.03
  h <- 1 # First historised point

  wtp$historised <- FALSE
  wtp$historised[c(1, n)] <- TRUE # Testing for delta > threshold

  for (i in 2:nrow(wtp)) {
    delta <- wtp$turbidity[i] / wtp$turbidity[h] 
    if (delta > (1 + threshold) | delta < (1 - threshold)) {
      wtp$historised[i] <- TRUE
      h <- i
    }
  }

  historian <- subset(wtp, historised)

  p2 <- p1 + 
    geom_point(data = historian, 
               aes(x = timestamp, y = turbidity), 
               size = 3, alpha = .5, color = "blue") +
    labs(title = "Historised data")

  p2

Historised simulated turbidity data.

Virtual Tags

This standard method for generating and storing SCADA data works well for operational systems but not for post hoc analysis. The data in Historian are unequally spaced, making them harder to analyse. Using constant interpolation, the Virtual Tag approach expands these unequal time series into an equally spaced time series.

The vt() function undertakes the constant interpolation using the approx function. The function vt is applied to all the DateTime values using the historised data points. The red line shows how the value is constant until it jumps by more than 5%. This example demonstrates that we have a steady process with some minor spikes, which is the expected outcome of this simulation.

  # Virtual Tags extrapolation

  vt <- function(t) {
    approx(historian$timestamp, historian$turbidity, xout = t, method = "constant")
  }

  turbidity <- lapply(as.data.frame(wtp$timestamp), vt)

  wtp$virtual_tag <- turbidity[[1]]$y

  p3 <- p2 +
    geom_line(data = wtp,
              aes(x = timestamp, y = virtual_tag), colour = "red") +
    ggtitle("Virtual Tags")
  p3

Virtual tags for the simulated data.

The next step in Virtual Tags is to combine tags from different data points for richer insights. For instance, imagine you're curious about turbidity readings only when the filter is running. By combining this data with information such as the filter's valve status and flow, you can gain a clearer picture of system performance.

Although this approach may seem cumbersome at first glance, it makes analysing data from SCADA Historian much more straightforward. With Virtual Tags, you can easily access a catalogue of analytical processes that would otherwise be challenging to perform. This system also provides valuable context for your SCADA information by linking tags to one another and to the operational processes they reflect. Curious to learn more? Download the technical manual for an in-depth look at Virtual Tags and their implementation in SQL.

Interested in solving water challenges with R code? You might enjoy the book Data Science for Water Utilities, which goes deeper into practical techniques for the field.

Data Science for Water Utilities

Routledge Amazon

Project Euler 28: Number Spiral Diagonals

Peter Prevos — Thu, 20 Jul 2017 00:00:00 +0000

Project Euler 28 takes us to the world of the Ulam Spiral. This is a spiral that contains sequential positive integers in a square spiral, marking the prime numbers. Stanislaw Ulam discovered that a lot of primes are located along the diagonals. These diagonals can be described as polynomials. The Ulam Spiral is thus a method to generate quadratic primes (Euler Problem 27).

Ulam spiral (Wikimedia).

Project Euler 28 Definition

Starting with the number 1 and moving to the right in a clockwise direction, a 5 by 5 spiral is formed as follows:

21  22  23  24  25

20   7   8   9  10

19   6   1   2  11

18   5   4   3  12

17  16  15  14  13

It can be verified that the sum of the numbers on the diagonals is 101.

What is the sum of the numbers on the diagonals in a 1001 by 1001 spiral formed in the same way?

Proposed Solution

To solve this problem, we do not need to create a matrix. This code calculates the values of the corners of a matrix with size $n$. The lowest number in the matrix with size $n$ is $n(n-3)+4$. The numbers increase by $n-1$ . The code steps through all matrices from size 3 to 1001. The solution uses only the uneven sized matrices because these have a centre. The answer to the problem is the sum of all numbers.

  ## Project Euler 28: Number Spiral Diagonals
  size <- 1001 # Size of matrix
  answer <- 1 # Starting number

  ## Define corners of subsequent matrices
  for (n in seq(from = 3, to = size, by = 2)) {
      corners <- seq(from = n * (n - 3) + 3, by = n - 1, length.out = 4)
      answer <- answer + sum(corners)
  }
  print(answer)

Plotting the Ulam Spiral

We can go beyond Euler Problem 28 and play with the mathematics. This code snippet plots all the prime numbers in the Ulam Spiral. Watch the Numberphile video for an explanation of the patterns that appear along the diagonals.

The code creates a matrix of the required size and fills it with the Ulam Spiral. The system then identifies all primes using the is.prime() function from Euler Problem 7, as visualised on the top of this article.

Ulam Spiral prime numbers.

Prime Spirals - Numberphile.

  ## Prime Spirals
  source("problem007.R", verbose = FALSE)
  size <- 201 # Size of matrix
  ulam <- matrix(ncol = size, nrow = size)
  mid <- floor(size / 2 + 1)
  ulam[mid, mid] <- 1
  for (n in seq(from = 3, to = size, by = 2)) {
      numbers <- (n * (n - 4) + 5) : ((n + 2) * ((n + 2) - 4) + 4)
      d <- mid - floor(n / 2)
      l <- length(numbers)
      ulam[d, d:(d + n - 1)] <- numbers[(l - n + 1):l]
      ulam[d + n - 1, (d + n - 1):d] <- numbers[(n - 1):(n - 2 + n)]
      ulam[(d + 1):(d + n - 2), d] <- numbers[(l - n):(l - 2 * n + 3)]
      ulam[(d + 1):(d + n - 2), d + n - 1] <- numbers[1:(n - 2)]
  }
  ulam.primes <- apply(ulam, c(1, 2), is.prime)

  library(ggplot2)
  library(reshape2)
  ulam.primes <- melt(ulam.primes)

  ggplot(ulam.primes, aes(x = Var1, y = Var2, fill = value)) +
      geom_tile() +
      scale_fill_manual(values = c("white", "black")) +
      guides(fill = FALSE) +
      theme_void()
  ggsave("images/problem-028.png", widt = 6, height = 4)

Using the iGraph package to Analyse the Enron Corpus

Peter Prevos — Thu, 13 Jul 2017 10:00:01 +0000

The Enron scandal is one of the most famous corporate governance failures in the history of capitalism. The case itself is interesting for its own sake, but in this post, I am more interested in one of the data sets that the subsequent investigation has provided. The code below show s how to analyse the Enron corpus using the R language.

This blog post analyses an extensive collection of e-mails from former Enron employees. The Enron corpus is analysed using network analysis tools provided by the iGraph package. Network analysis is a versatile technique that can be used to add value to a lot of different data sets, including the complex corporate relationships of Donald Trump.

The Enron Scandal Explained in One Minute

The Enron Corpus

The Federal Energy Regulatory Commission used an extensive collection of emails from Enron employees as part of their inquiries. The Enron corpus is one of the only publicly available collections of emails available for research. This dataset also provides a fascinating playground for citizen data scientists.

The set has privacy issues as it contains messages from living people. When analysing this data set, we need to keep in mind that the majority of former Enron employees were innocent people who lost their jobs due to the greed of their overlords. The code in this post only analyses the e-mail headers, ignoring the content.

Laid-off Enron employees outside Enron headquarters as the company collapsed in 2001 (Source cnbc.com).

The Enron Corpus is an extensive database of half a million emails generated by more than 100 Enron employees. You can download the corpus from the Carnegie Mellon School of Computer Science. The first code snippet downloads the 7 May 2015 version of the dataset (about 423Mb, tarred and gzipped) and untars it to your working directory.

  if (!file.exists("enron_mail_20150507.tgz")) {
      download.file("https://www.cs.cmu.edu/~./enron/enron_mail_20150507.tar.gz",
                     destfile = "enron-mail-20150507.tgz")
  }

  if (file.exists("enron-mail-20150507.tgz")) {
      untar("enron-mail-20150507.tgz")
  }

Preparing the Enron email

The main folder is maildir, which holds all the personal accounts. Our first task is to create a list of available emails. This code results in 517,401 e-mail files with 44,859 emails in the inboxes of users. The E-mail corpus consists of nested folders per user with each e-mail as a text file.

  emails <- list.files("maildir/", full.names = TRUE, recursive = TRUE)
  emails <- emails[grep("/inbox", emails)]
  length(emails)

The bulk of the code creates a list of emails between Enron employees. The code performs a lot of string manipulations to extract the information from the text files. The content of the e-mails is ignored, the code only retrieves the sender and the receiver. The analysis is limited to e-mails between Enron employees. The result of this code is a data frame with the usernames of the sender and receiver for each email. The data frame contains 2779 emails that meet these criteria.

The code below looks extracts all internal emails in the Inox folders. It then retrieves a lis of usernames from the e-mails. The data is cleaned by removing users without sent e-mails, and emails which users sent to themselves.

  library(tibble)
  inboxes <- tibble(
      from = apply(as.data.frame(emails), 1, 
                 function(x){readLines(x, warn = FALSE)[3]}),
      to = emails,
      stringsAsFactors = FALSE
  )

  ## Keep only enron.com and strip all but username
  library(stringr) # String manipulation
  inboxes <- inboxes[grepl("@enron.com", inboxes$from),]
  inboxes$from <- str_remove_all(inboxes$from, "From: |@enron.com")
  inboxes$to <- str_remove_all(inboxes$to, "maildir//|/inbox/[0-9]*.")

  ## Create list of usernames
  users <- data.frame(user = paste0("maildir/", unique(inboxes$to)))

  ## Remove those without sent mails
  sent <- apply(users, 1, function(x) sum(grepl("sent", dir(x))))
  users <- subset(users, sent != 0)

  ## Replace username with e-mail name
  users$mailname <- NA
  i < -1
  for (i in 1:nrow(users)){
      sentmail <- dir(paste0(users$user[i], "/sent_items/"))
      name <- readLines(paste0(users$user[i], "/sent_items/", sentmail[1]), warn = F)[3]
      name <- str_remove_all(name, "From: |@enron.com")
      users$mailname[i] <- name
  }

  users$user <- str_remove(users$user, "maildir/")
  inboxes <- merge(inboxes, by.x = "to", users, by.y = "user")
  inboxes <- tibble(from = inboxes$from,
                    to = inboxes$mailname)

  ## Only e-mails between users inside the corpus
  inboxes <- inboxes[inboxes$from %in% inboxes$to,]

  ## Remove no.address
  inboxes <- subset(inboxes, from != "no.address" & to != "no.address")

  ## Remove emails to self
  inboxes <- subset(inboxes, inboxes$from != inboxes$to)
  inboxes

Analyse the Enron Corpus

The last code snippet defines a graph from the table of e-mails. Each employee is a node in the network, and each e-mail is an edge (line). The iGraph package is a powerful tool to analyse networks. The graph_from_edgelist function creates a network object that can be analysed using the iGraph package. The graph is directed because the information flows from the sender to the receiver.

In the next step, the Spingglass algorithm finds community structure within the data. A community is a group of nodes that are more connected with each other than with any other node.

The last step visualises the network. The diagram is drawn using the Fruchterman-Reingold algorithm, which places the most connected nodes at the centre of the picture. The background colours in the diagram indicate the eight communities.

The graph tells us a lot about the group of employees in the Enron corpus and how they relate to each other. Each of the communities represents a tightly connected group of employees that mainly e-mail each other. Any connections between communities are shown in red. When the vertex.label = NA line is removed, the usernames are displayed at each node.

We can see groups that never email each other, lonely hangers-on and tightly knit cliques within Enron. In the centre of the graph, we see a few individuals who are connectors between groups because they send a lot of emails to people outside their community. On the fringes of the graph are the hangers-on who only once or twice emailed somebody in the corpus but were still included in the investigation.

  library(igraph)
  g <- graph_from_edgelist(as.matrix(inboxes), directed = TRUE)
  coms <- spinglass.community(g)
  par(mar = c(0,0,2,0))
  plot(coms, g,
      vertex.label=NA,
      layout = layout.fruchterman.reingold,
      vertex.size = 3
  )

Enron email corpus network with communities.

This analysis provides only a quick glimpse into the knowledge contained in the Enron email corpus. An extensive range of tools is available to analyse such networks. An interesting exercise would be to overlap this network with the organisation chart to see the relationships between teams. Have fun playing with this fantastic, but somewhat sensitive, data set!

Project Euler 27: Quadratic Primes

Peter Prevos — Thu, 06 Jul 2017 00:00:00 +0000

Prime numbers are the engine of the Internet economy. There is no algorithm that can generate these numbers which makes them ideal for as encryption keys. This impossibility has not stopped mathematicians from trying to find formulas to generate prime numbers. Project Euler 27 deals with quadratic formulas that can be used to generate sets of prime numbers.

Project Euler 27 Definition

Euler discovered the remarkable quadratic formula: $n^2+n+41$

It turns out that the formula will produce 40 primes for the consecutive integer values $0 \leq n \leq 39$. However, when $n=40$, $40^2+40+41=40(40+1)+41$ is divisible by 41, and certainly when $n=41$, $41^2+41+41$ is clearly divisible by 41.

The incredible formula $n^2-79n+1601$ produces 80 primes for the consecutive values $0 \leq n \leq 79$. The product of the coefficients, -79 and 1601, is -126479.

Considering quadratics of the form:

$n^2 + an + b$, where $|a| < 1000$ and $|b| \leq 1000$.

$|n|$ is the absolute value of $n$, e.g. $|11|=11$ and $|-4|=4$.

Find the product of the coefficients, $a$ and $b$, for the quadratic expression that produces the maximum number of primes for consecutive values of $n$, starting with $n=0$.

Proposed Solution

The only way to solve this problem is through brute force and search for all possible combinations for $a$ and $b$. Without reducing the search space, that would mean testing four million options. We need to reduce the solution space to achieve a manageable running speed.

Because the outcome of the equation must be prime for $n = 0$, $b$ also has to be prime. We can use the prime sieve from Euler Problem 3 to use only prime numbers from 2 to 1000, which reduces it from 2000 to 168 permutations.

When we insert $n = 1$, it follows that $a$ has to be an even number. If $1+a+b$ has to be prime and $b$ is a prime number, then $a$ can only be an odd number. However, when $b = 2$, $a$, also has to be even.

We can use these deliberations to write some code to solve this problem. We cycle through $a$ and $b$ and increment a counter every time we find a prime number. The is.prime() function from Project Euler 7 helps to identify whether a number is prime.

  ## Project Euler 27: Quadratic Primes
  source("problem007.R", verbose = FALSE)

  seq.a <- seq(-999, 1001, 2) # a has to be odd
  seq.b <- (esieve(1000)) # b has to be prime
  max.n <- 0

  for (a in seq.a) {
    for (b in seq.b) {
      n <- 0 # Find sequence of primes for a and b
      while (is.prime(n^2 + a * n + b)) {
        n <- n + 1
      }
      ## Store maximum values
      if (n > max.n) {
        max.n <- n
        max.a <- a
        max.b <- b
      }
    }
  }

  answer <- max.a * max.b
  print(answer)
  
  ## Test the answer
  n <- 0:(max.count - 1)
  sapply(n^2 + max.a * n + max.b, is.prime)

Project Euler 26: Reciprocal Cycles

Peter Prevos — Thu, 29 Jun 2017 00:00:00 +0000

Project Euler 26 delves into a topic many kids in primary school dread: fractions. A few years ago, a fraction broke the internet. What happens when you divide 1 by 998001?

$$\frac{1}{998001} = 0.000001002003004005006007008009010011012013014015 \ldots$$

What is unique about this fraction is that it lists every three-decimal number except for 998. Look carefully at the sequence to see that is 000, 001, 0002, 003, 004, 005 and so on. After it has reached 999, the sequence continues from the start. This fraction thus has 2997 recurring decimals. James Grime from Numberphile explains this mathematical oddity with his usual enthusiasm.

998,001 and its Mysterious Recurring Decimals - Numberphile.

The decimal fraction of 1/998001 is a recurring decimal. These are decimal numbers with periodic digits (repeating its values at regular intervals).

Project Euler 33 is also about fractions and asks us to analyse recurring decimals (reciprocal cycles) and Ford Circles.

Project Euler 26 Definition

A unit fraction contains 1 in the numerator. The decimal representation of the unit fractions with denominators 2 to 10 are given:

1/2 = 0.5

1/3 = 0.(3)

1/4 = 0.25

1/5 = 0.2

1/6 = 0.1(6)

1/7 = 0.(142857)

1/8 = 0.125

1/9 = 0.(1)

1/10 = 0.1

Where 0.1(6) means 0.166666… and has a 1-digit recurring cycle. It can be seen that 1/7 has a 6-digit recurring cycle. Find the value of d < 1000 for which 1/d contains the longest recurring cycle in its decimal fraction part.

Proposed Solution

Sequence A051626 in the On-Line Encyclopaedia of Integer Sequences describes the length of the recurring numbers in 1/n. To solve Euler Problem 26, we need to generate the first 1000 numbers of this sequence and find out which number has the longest recurring cycle.

R can only display up to 22 decimals by using options(digits=22). The base R capability is unsuitable for solving this problem, so I wrote some code to perform long division the old-fashioned way.

The recur() function divides 1 by any arbitrary integer. The code continues until the decimal terminates, for example 1/4 = 0.25, or when a recurring pattern emerges, e.g. 1/7 = 0.(142857).

The function has two arguments: n is the input number. The output argument determines the outcome of the function: “=len=” for the length of the recurring decimals. Any other value shows the result of the calculation. The output of the function is a string. Using the European notation, the recurring part of the decimals is shown between brackets, e.g. 1/14 = 0.0(714285).

  ## Project Euler 26: Reciprocal Cycles
  recur <- function(x, output = "") {
      # Prepare variable
      if (x == 0) return(NaN)
      if (x == 1) return(0)
      x <- floor(abs(x))
      # Initiate vectors to store decimals and remainders
      dec <- vector()
      rem <- vector()
      # Initiate values
      i <- 1
      r <- 10
      rem <- r
      # Long division
      repeat {
          dec[i] <- floor(r / x)
          r <- 10 * (r %% x)
          # Test wether the number is terminating or repeating
          if (r == 0 | r %in% rem) break
          rem[i + 1] <- r
          i <- i + 1 
      }
      # Determine number of recurring digits
      rep <- ifelse(r != 0, length(rem) - which(r == rem) + 1, 0)
      # Output
      if (output == "len")
          return(rep)
      else {
          if (rep != 0) {
              if (rep == length(dec)) 
                  l <- "("
              else
                  l <- c(dec[1:(length(dec) - rep)], "(")
              dec <- c(l, dec[(length(dec) - rep + 1):length(dec)], ")")
          }
          return(paste0("0.", paste0(dec, collapse = "", sep = "")))
          }
  }

  A051626 <- sapply(1:1000, recur, "len")
  answer <- which.max(A051626)
  print(answer)

  recur(998001, "len")
  recur(998001)

Project Euler 25: Fibonacci number with 1000 digits

Peter Prevos — Thu, 22 Jun 2017 00:00:00 +0000

Project Euler 25 asks us back to the Fibonacci numbers. This famous number sequence describes many natural processes, such as the patterns in this beautiful sunflower. The seeds inside the sunflower grow in a spiral pattern, and the number of spirals tends to be a Fibonacci number. This pattern is the most efficient way to pack the centre with as many seeds as possible.

Famous mathematician and father of computer science Alan Turing was fascinated by this problem until his death. In 2016, the Royal Society published a journal article that builds on his work in the Open Science journal. They collected sunflowers from citizen scientists from around the world and confirmed that for the majority of sunflowers, the number of spirals is a Fibonacci number. They did, however, also discover other patterns. Euler Problem 25 is a bit less poetic as it only asks to generate a huge Fibonacci number.

Sunflowers and Fibonacci - Numberphile.

Project Euler 25 Definition

The Fibonacci sequence is defined by a recurrence relation:

$$F_n = F_{n-1} + F_{n-2} \quad | \quad F_1 = 1, F_2 = 1$$

The 12^th term, $F_{12}=144$, is the first term to contain three digits.

What is the index of the first term in the Fibonacci sequence to contain 1000 digits?

Large Integers in R

By default, numbers with more than 7 digits are shown in scientific notation in R, which reduces the accuracy of the calculation. You can change the precision of large integers with the options() function, but R struggles with integers with more than 22 digits. This example illustrates this issue:

factorial(24)

[1] 6.204484e+23

options(digits = 22)
factorial(24)

[1] 620448401733239409999872

Finding a number with 1000 digits is a problem in R with using a special function or package. This article explains how to solve this problem using the GMP package and in base R code.

Project Euler 25 Solution in R

We have already seen various ways to generate Fibonacci numbers in R using a recursive function, a while loop and the GMP library in Euler Problem 2.

Arbitrary-Precision Arithmetic

The first solution uses the GMP package to manage large integers. This package implements the GNU Multiple Precision Arithmetic Library for working with huge numbers. This package also contains a function to generate Fibonacci numbers. This solution cycles through the Fibonacci sequence until it finds a number with 1000 digits.

 ## Project Euler 25: Fibonacci number with 1000 digits
 ## GNU Multiple Precision Arithmetic Library
  library(gmp) 
  answer <- 1
  fib <- 1
  while (nchar(as.character(fib)) <= 1000) {
      fib <- fibnum(answer) # Determine next Fibonacci number
      answer <- answer + 1
  }
  print(answer)

R-base code

This is a very fast solution but my aim is to solve the first 100 Project Euler problems using only base-R code. The big.add() function I developed to solve Euler Problem 13 uses only basic R to add large numbers. This function multiplies large numbers in the same way you would do on a piece of paper. This code to solve Euler Problem 25 is much slower than the GMP library but it was fun to develop.

  ## Base-R Solution
  fib <- 1 # First Fibonaci number
  cur <- 1 # Current number in sequence
  pre <- 1 # Previous number in sequence
  answer <- 2
  while (nchar(fib) < 1000) {
      fib <- big.add(cur, pre) # Determine next Fibonacci number
      pre <- cur
      cur <- fib
      answer <- answer + 1
  }
  print(answer)

Project Euler 24: Lexicographic Permutations

Peter Prevos — Thu, 15 Jun 2017 00:00:00 +0000

Project Euler 24 asks to develop lexicographic permutations. These are ordered arrangements of objects in lexicographic order. Tushar Roy of Coding Made Simple has shared a great introduction on how to generate lexicographic permutations.

String Permutation Algorithm.

Project Euler 24 Definition

A permutation is an ordered arrangement of objects. For example, 3124 is one possible permutation of the digits 1, 2, 3 and 4. If all of the permutations are listed numerically or alphabetically, we call it lexicographic order. The lexicographic permutations of 0, 1 and 2 are:

$$012 \quad 021 \quad 102 \quad 120 \quad 201 \quad 210$$

What is the millionth lexicographic permutation of the digits 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9?

Proposed Solution

The digits 0 to 9 have $10! = 362880$ permutations (including combinations that start with 0). Most of these permutations are, however, not in lexicographic order. A brute-force way to solve the problem is to determine the next lexicographic permutation of a number string and repeat this one million times. This code takes the following steps:

Find largest index $i$ such that$a_{i-1} < a_i$.
If no such index exists, then this is already the last permutation.
Find largest index $j$ such that $j \geq i$ and $a_j > a_{i-1}$.
Swap $a_j$ and $a_{i-1}$.
Reverse the suffix starting at $a_i$.

  ## Project Euler 24: Lexicographic permutations
  nextPerm <- function(a) {
      # Find longest non-increasing suffix
      i <- length(a)
      while (i > 1 && a[i - 1] >= a[i])
          i <- i - 1
      # i is the head index of the suffix
      # Are we at the last permutation?
      if (i <= 1) return (NA)
      # a[i - 1] is the pivot
      # Frind rightmost element that exceeds the pivot
      j <- length(a)
      while (a[j] <= a[i - 1]) 
          j <- j - 1
      # Swap pivot with j
      temp <- a[i - 1]
      a[i - 1] <- a[j]
      a[j] <- temp
      # Reverse the suffix
      a[i:length(a)] <- rev(a[i:length(a)])
      return(a)
      }

  numbers <- 0:9
  for (i in 1:(1E6 - 1)) numbers <- nextPerm(numbers)
  answer <- paste(numbers, collapse = "")
  print(answer)

Combinatorics

A more efficient solution is to use combinatorics, thanks to the now defunct MathBlog. The last nine digits can be ordered in 9! = 362880 ways. So the first 9! permutations start with a 0. By extending this thought, it follows that the millionth permutation must start with a 2.

$$\lfloor (1000000 - 1) / 9 ! \rfloor = 2$$

From this rule, it follows that the 725761^st permutation is 2,013,456,789. We now need 274,239 more lexicographic permutations:

$$(1000000 - 1) - (2 \times 9!) = 274239$$

We can repeat this logic to find the next digit. The last 8 digits can be ordered in 40320 ways. The second digit is the 6th digit in the remaining numbers, which is 7 (2013456789).

$$\lfloor 274239 / 8 ! \rfloor = 6$$

$$274239 - (6 \times 7 !) = 32319$$

This process is repeated until all digits have been used.

  numbers <- 0:9
  n <- length(numbers)
  answer <- vector(length = 10)
  remain <- 1E6 - 1
  for (i in 1:n) {
      j <- floor(remain / factorial(n - i))
      answer[i] <- numbers[j + 1]
      remain <- remain %% factorial(n - i)
      numbers <- numbers[-(j + 1)]
  }
  answer <- paste(answer, collapse = "")
  print(answer)

Tic Tac Toe Simulation: The Intelligent Minimax Algorithm

Peter Prevos — Thu, 08 Jun 2017 00:00:00 +0000

One of my favourite movies is WarGames. One of the most memorable scenes is where the protagonist tries to teach the out-of-control AI that a nuclear war is unwinnable with a Tic Tac Toe simulation.

Tic Tac Toe might be a futile children's game but it can also teach us about artificial intelligence. Tic Tac Toe, or Noughts and Crosses, is a zero-sum game with perfect information. Both players have full information about each other. When nobody makes a mistake, the game always ends in a draw.

This article describes how to self-playing game of Tic Tac Toe, complete with a graphical interface, in the R language. This article is part of a series about computer games in the R Language.

The first two parts of the article describe how to create the user interface and add a human player. The second part delves into creating computer players. My ultimate aim is to recreate this iconic scene in the WarGames movie.

Tic Tac Toe With Joshua (1983).

The Game Board

This first code snippet draws the Tic Tac Toe simulation board. The variable xo holds the identity of the pieces and the vector board holds the current game. The game board is a vector of length nine consisting of either -1 (X), 0 (empty field) or 1 (O). The vector indices correspond with locations on the game board:

1 2 3
4 5 6
7 8 9

The second part of the code checks for three in a row and draws the corresponding line.

## Tic Tac Toe Simulation in the R language
## Peter Prevos
## lucidmanager.org

  draw.board <- function(game) {
      xo <- c("X", " ", "O") # Symbols
      par(mar = rep(1,4))
      plot.new()
      plot.window(xlim = c(0,30), ylim = c(0,30))
      abline(h = c(10, 20), col = "darkgrey", lwd = 4)
      abline(v = c(10, 20), col = "darkgrey", lwd = 4)
      text(rep(c(5, 15, 25), 3), c(rep(25, 3), rep(15,3), rep(5, 3)), 
           xo[game + 2], cex = 4)
      # Identify location of any three in a row
      square <- matrix(game, nrow = 3, byrow = TRUE)
      hor <- abs(rowSums(square))
      if (any(hor == 3)) 
          hor <- (4 - which(hor == 3)) * 10 - 5 
      else 
          hor <- 0
      ver <- abs(colSums(square))
      if (any(ver == 3)) 
          ver <- which(ver == 3) * 10 - 5 
      else
          ver <- 0
      diag1 <- sum(diag(square))
      diag2 <- sum(diag(t(apply(square, 2, rev)))) 
      # Draw winning lines 
      if (all(hor > 0))
          for (i in hor)
              lines(c(0, 30), rep(i, 2), lwd = 10, col="red")
      if (all(ver > 0))
          for (i in ver)
              lines(rep(i, 2), c(0, 30), lwd = 10, col="red")
      if (abs(diag1) == 3)
          lines(c(2, 28), c(28, 2), lwd = 10, col = "red")
      if (abs(diag2) == 3)
          lines(c(2, 28), c(2, 28), lwd = 10, col = "red")
  }

Human Player

This second code snippet lets a human player move by clicking anywhere on the graphic display using the locator function. The click location is converted to a number to denote the position on the board. The entered field is only accepted if it has not yet been used (the empty variable contains the available fields).

move.human <- function(game) {
    text(4, 0, "Click on screen to move", col = "grey", cex=.7)
    empty <- which(game == 0)
    move <- 0
    while (!move %in% empty) {
        coords <- locator(n = 1) # add lines
        coords$x <- floor(abs(coords$x) / 10) + 1
        coords$y <- floor(abs(coords$y) / 10) + 1
        move <- coords$x + 3 * (3 - coords$y)
    }
    return(move)
}

Computer Player

The computer uses a modified Minimax Algorithm to determine its next move. This article from the Never Stop Building blog explains this method in great detail.

If the computer moves first, the algorithm takes a while because of the large number of permutations. There are 255,168 possible legal games in Tic Tac Toe, 46,080 (18%) of which end in a draw.

The code for this part is contributed by a reader (see comments below). I originally made a mistake in the recurring function and Alberto was so kind to contribute a correct version.

The ganador (Spanish for winning) function assesses the board condition by assigning -10 or + 10 for a winning game and 0 for any other situation.

The minimax function returns a list with the move and its valuation through the ganador function. The function calls itself until it has filled the board and retains the best scoring move using the minimax method. Random variables are added to avoid the computer always playing the same move in the same situation.

## Minimax code by Alberto C.
ganador <- function(juego, player) {
    game <- matrix(juego, nrow = 3, byrow = T)
    hor <- rowSums(game)
    ver <- colSums(game)
    diag <- c(sum(diag(game)), sum(diag(apply(game, 1, rev))))
    if (-3 %in% c(hor, ver, diag))
        return(-10)
    if (3 %in% c(hor, ver, diag))
        return(10)
    else
        return(0)
}

minimax <- function(juego, player) {
    free <- which(juego == 0)
    if(length(free) == 1) {
        juego[free] <- player
        return(list(move = free, U = ganador(juego, player)))
    }
    poss.results <- rep(0, 9)
    for(i in free) {
        game <- juego
        game[i] <- player
        poss.results[i] <- ganador(game, player)
    }
    mm <- ifelse(player == -1, "which.min", "which.max")
    if(any(poss.results == (player * 10))) {
        move <- do.call(mm, list(poss.results))
        return(list(move = move, U = poss.results[move]))
    }
    for(i in free) {
        game <- juego
        game[i] <- player
        poss.results[i] <- minimax(game, -player)$U
    }
    random <- runif(9, 0, 0.1)
    poss.results[-free] <- 100 * -player
    poss.results <- poss.results + (player * random)
    move <- do.call(mm, list(poss.results))
    return(list(move = move, U = poss.results[move]))
}

Play Tic Tac Toe

The last bit of code defines a function to play the game. Each layer can be either a human or a computer. The function loops while there are still empty spots on the board and no winner is declared.

tic.tac.toe <- function(player1 = "human", player2 = "computer") {
    game <- rep(0, 9) # Empty board
    winner <- 0 # Define winner
    player <- 1 # First player
    players <- c(player1, player2)
    draw.board(game)
    while (0 %in% game & winner == 0) { # Keep playing until win or full board
        if (players[(player + 3) %% 3] == "human") # Human player
            move <- move.human(game)
        else { # Computer player
            move <- minimax(game, player)[[1]]
            }
        game[move] <- player # Change board
        draw.board(game)
        winner <- ganador(game, player)
        player <- -player # Change player
    }
    if (winner == 0)
        text(15, 15 , "DRAW", col = "red", cex = 10)
}

tic.tac.toe()

Wargames Tic Tac Toe Simulation

Now all elements are in place to recreate the simulation in the Wargames movie. The animation package helps to create an animated gif where the computer keeps playing itself. All games end in a draw.

Wargames Tic Tac Toe simulation.

Project Euler 23: Non-Abundant Sums

Peter Prevos — Thu, 01 Jun 2017 00:00:00 +0000

Project Euler 23 asks us to solve a problem with abundant or excessive numbers. These are numbers for which the sum of its proper divisors is higher than the number itself.

Twelve is an abundant number because the sum of its proper divisors (the aliquot sum) is larger than 12: (1 + 2 + 3 + 4 + 6 = 16). All highly composite numbers or anti-primes greater than six are abundant numbers. These numbers have so many divisors that they are the opposite of primes.

5040 and other Anti-Prime Numbers - Numberphile.

Project Euler 23 Definition

A perfect number is a number for which the sum of its proper divisors is exactly equal to the number. For example, the sum of the proper divisors of 28 would be 1 + 2 + 4 + 7 + 14 = 28, which means that 28 is a perfect number. A number n is called deficient if the sum of its proper divisors is less than n and it is called abundant if this sum exceeds n.

As 12 is the smallest abundant number, 1 + 2 + 3 + 4 + 6 = 16, the lowest number that can be written as the sum of two abundant numbers is 24. By mathematical analysis, it can be shown that all integers greater than 28123 can be written as the sum of two abundant numbers. However, this upper limit cannot be reduced any further by analysis, even though it is known that the highest number that cannot be expressed as the sum of two abundant numbers is less than this limit.

Find the sum of all the positive integers which cannot be written as the sum of two abundant numbers.

Proposed Solution

This solution recycles the divisors function that determines the proper divisors for a number, introduced for Euler Problem 21. The first code snippet creates the sequence of all abundant numbers up to 28123 (sequence A005101 in the OEIS). An abundant number is one where its aliquot sum (sum of proper divisors) is larger than itself.

  ## Project Euler 23: Non-Abundant Sums
  source("project-euler-021.R", verbose = FALSE)
  abundant <- function(x){
      abundant <- vector()
      a <- 1
      for (n in 1:x) {
          sigma <- sum(divisors(n))
          if (sigma > 2 * n) {
              abundant[a] <- n
              a <- a + 1
          }
      }
      return(abundant)
  }

The solution to this problem is provided in sequence A048242 of the Online Encyclopedia of Integer Sequences. This website states that the highest number in this sequence is 20161, not 28123, as stated in the problem definition.

The second section of code creates a list of all potential numbers that are not the sum of two abundant numbers. The next bit of code sieves any sum of two abundant numbers from the list. You find the answer by adding the remaining numbers in the sequence.

  abundant <- abundant(28123)
  A048242 <- 1:20161
  for (i in 1:length(abundant)) {
      for (j in i:length(abundant)) {
          if (abundant[i] + abundant[j] <= 20161) {
              A048242[abundant[i] + abundant[j]] <- NA
          }
      }
  }
  A048242 <- A048242[!is.na(A048242)]
  answer <- sum(A048242)
  print(answer)

Project Euler 22: Names Scores

Peter Prevos — Thu, 25 May 2017 00:00:00 +0000

Project Euler 22 is another trivial task that takes us to the realm of ASCII codes. ASCII is a method to convert symbols into numbers, initially invented for telegraphs.

Back in the 8-bit days, ASCII art was a method to create images without using lots of memory. Each image consists of a collection of text characters that give the illusion of an image.

        ,-""""""-.        
     /\j__/\  (  \`--.    
hjw  \`@_@'/  _)  >--.`.  
    _{.:Y:_}_{{_,'    ) ) 
   {_}`-^{_} ```     (_/ .

Art by Hayley Jane Wakenshaw.

Euler problem 22 is, unfortunately, a bit less poetic. This problem is similar to Euler problem 17 that asks to count the length of words expressing numbers.

Project Euler 22 Definition

Using names.txt, a 46K text file containing over five-thousand first names, begin by sorting it into alphabetical order. Then working out the alphabetic value for each name, multiply this value by its alphabetical position in the list to obtain a name score.

For example, when the list is sorted into alphabetical order, COLIN, which is worth 3 + 15 + 12 + 9 + 14 = 53, is the 938^th name in the list. So, COLIN would obtain a score of 938 × 53 = 49714.

What is the total of all the name scores in the file?

Proposed Solution

This code reads and cleans the file and sorts the names alphabetically. The charToRaw function determines the numerical value of each character in each name. This output of this function is the hex ASCII code for each character. The letter A is number 65, so we subtract 64 from each value to get the sum total.

  ## Project Euler 22: Names Scores
  names <- readLines("https://projecteuler.net/project/resources/p022_names.txt",
                     warn =FALSE)
  names <- unlist(strsplit(names, ","))
  names <- gsub("[[:punct:]]", "", names)
  names <- sort(names)

  # Total Name scores
  answer <- 0
  for (i in names) {
      value <- sum(sapply(unlist(strsplit(i, "")),
                   function(x) as.numeric(charToRaw(x)) - 64))
      value <- value * which(names==i)
      answer <- answer + value
  }
  print(answer)

Modern computers in a globalised world don't use ASCII because it cannot encode the tens of thousands of characters that are needed for the world's languages. The video below explains the Unicode system to encode the world's symbols.

Characters, Symbols and the Unicode Miracle - Computerphile.

Project Euler 21: Amicable Numbers

Peter Prevos — Thu, 18 May 2017 00:00:00 +0000

Project Euler 21 takes us to the realm of amicable numbers. Sequence A259180 in the OEIS lists all amicable, or friendly numbers. These are the most romantic numbers in all of maths. Amicable numbers

Recreational mathematicians love to find numbers with special properties. Other such special numbers are 'perfect numbers', which equal the sum of its proper divisors. Mathematicians have also defined sociable numbers and betrothed numbers, which are similar to amicable numbers. But perhaps these are for another Euler problem. Euler Problem 23 is about finding abundant numbers, which are numbers for which the sum of its proper divisors is higher than the number itself.

Amicable numbers were known to the Pythagoreans, who credited them with many mystical properties. Before we had access to computers, finding amicable numbers was a task that required a lot of patience. No algorithm can systematically generate all amicable numbers, and until 1946 only 390 pairs were known. Medieval Muslim mathematicians developed several formulas to create amicable numbers, but the only way to be complete is by using brute force.

Project Euler 21 Definition

Let $d(n)$ be defined as the sum of proper divisors of $n$ (numbers less than $n$ which divide evenly into $n$). If $d(a) = b$ and $d(b) = a$, where $a \neq b$, then $a$ and $b$ are an amicable pair and each of $a$ and $b$ are called amicable numbers.

For example, the proper divisors of 220 are 1, 2, 4, 5, 10, 11, 20, 22, 44, 55 and 110; therefore $d(220) = 284$. The proper divisors of 284 are 1, 2, 4, 71 and 142; so, $d(284) = 220$.

Evaluate the sum of all the amicable numbers under 10000.

220 and 284 (Amicable Numbers) - Numberphile.

Proposed Solution

The first part of the code provides for a function to list all proper divisors for a given integer x.

The loop determines the divisors for numbers 220 to 10,000, calculates their sum and then checks if these numbers are amicable.

  ## Project Euler 21: Amicable Numbers

  divisors <- function(x) {
    divisors <- vector()
    d <- 1
    for (i in 1:floor(sqrt(x))) {
      if (x %% i == 0) {
        divisors[d] <- i
        if (i != x/i) {
          d <- d + 1
          divisors[d] <- x / i
        }
        d <- d + 1
      }
    }
    return(divisors)
  }

  answer <- 0
  n <- 220
  while (n <= 10000) {
    div.sum <- sum(divisors(n)) - n
    if (n == sum(divisors(div.sum)) - div.sum & n != div.sum) {
      answer <- answer + n
      print(paste0("(", n, ", ", div.sum, ")"))
    } 
    n <- n + 1
    }
  print(answer)

Project Euler 20: Factorial Digit Sum

Peter Prevos — Thu, 11 May 2017 00:00:00 +0000

Project Euler 20 is the third problem that requires special consideration for working with huge integers. In this problem, we look at factorials. These numbers are useful in combinatorics if, for example, you like to know in how many ways you can arrange a deck of cards.

The problem with computing factorials is that they are mostly vast numbers, beyond the generic capabilities of computers to process. For example, a humble pack of playing cards can be shuffled in $52! = 80658175170943878571660636856403766975289505440883277824000000000000$ different configurations.

The largest number traditional handheld calculators can factorise is the mythical number 69.

69! - Numberphile.

Project Euler 20 Definition

$n!$ means $n \times (n − 1) \times \ldots \times 3 \times 2 \times 1$

For example, $10! = 10 \times 9 \times \ldots \times 2 \times 1 = 3628800$, and the sum of the digits in the number $10!$ is $3 + 6 + 2 + 8 + 8 + 0 + 0 = 27$.

Find the sum of the digits in the number 100!

Proposed Solution

The factorial of the number 100 contains 158 digits, which is a lot more digits than a 64-bit operating system can accurately produce. Using the standard function: factorial(100) = 9.332622e+157. Without using a specialised algorithm, we cannot determine the sum of all digits. We need to deploy arbitrary-precision arithmetic to solve this problem.

This problem can be solved using a specialised R package and using only base-R code. Many computer languages, including R, have special libraries to deal with such large numbers. The GMP package renders this problem almost trivial.

  ## Project Euler 20: Factorial Digit Sum
  library(gmp)
  digits <- factorialZ(100)
  digits <- as.character(digits)
  answer <- sum(as.numeric(unlist(strsplit(digits, ""))))
  print(answer)

Using base-R code

The problem becomes more interesting when only using basic R code. I developed the big.add function to solve Euler Problem 13 through the addition of very large integers. We can extend this function to also calculate factorials. A factorial can be replaced by a series of additions, for example:

$$3! = 1 \times 2 \times 3 = (((1+1) + (1+1)) + (1+1))$$

This can be mimicked in R using the Reduce function. This function reduces a vector to a single value by recursively calling a function. Reduce("+", rep(4, 5)) is the same as:

$$4 \times 5 = ((((4 + 4) + 4) + 4) + 4) = 20$$

Using a loop, we can use the Reduce function to calculate a factorial, using only additions:

## Base-R Code
fact <- 1
x <- 100
for (i in 2:x) {
    fact <- Reduce("+", rep(fact, i))
}
print(fact)

The big.factorial() function below implements this idea by combining the big.add() and Reduce() functions to calculate large integer factorials. The function returns a value of 1, for factorial of 0 or 1. This function does not calculate the Gamma-function for fractions. For all other values, it goes through a loop from 2 to the requested factorial. The temporary values are stored in the bf variable. The code loops through the factorials by using the result of the previous Reduce call into the current one.

  source("problem013.R", echo = FALSE)
  big.factorial <- function(x) {
      x <- floor(x)
      bf <- 1
      if (x > 1) {
          for (i in 2:x) {
              bf <- Reduce(big.add, rep(bf,i))
          }
      }
  return (bf)
  }

  digits <- big.factorial(100)
  answer <- sum(as.numeric(unlist(strsplit(as.character(digits), ""))))
  print(answer)

Project Euler 19: Counting Sundays

Peter Prevos — Thu, 04 May 2017 00:00:00 +0000

Project Euler 19 is so trivial it is almost not worth writing an article about. One exciting aspect of this problem is the naming of weekdays and deciding which day the week starts. This issue is more complicated than it sounds because data science is, in essence, not about data but about people.

Project Euler 19 Definition

You are given the following information, but you may prefer to do some research for yourself.

1 January 1900 was a Monday.
Thirty days has September, April, June and November.
All the rest have thirty-one,
Saving February alone, Which has twenty-eight, rain or shine. And on leap years, twenty-nine.
A leap year occurs on any year evenly divisible by 4, but not on a century unless it is divisible by 400.

How many Sundays fell on the first of the month during the twentieth century (1 January 1901 to 31 December 2000)?

Proposed Solution

The problem can be quickly solved with R base code and a tiny bit faster when using the lubridate package.

  ## Project Euler 19: Counting Sundays

  ## Base-R
  dates <- seq.Date(as.Date("1901-01-01"), as.Date("2000-12-31"), "days")
  days <- rep(1:7, length.out = length(dates))
  answer <- sum(days[substr(dates, 9, 10) == "01"] == 1)
  print(answer)

  ## Using Lubridate
  library(lubridate)
  dates <- seq.Date(as.Date("1901-01-01"), as.Date("2000-12-31"), "months")
  answer <- sum(wday(dates) == 1)
  print(answer)

To draw out this post a little bit further, I wrote some code to solve the problem without using any of the calendar functions in R.

  ## Calculate from scratch
  week.day <- 0
  answer <- 0
  for (y in 1901:2000) {
      for (m in 1:12) {
          max.day <- 31
          if (m %in% c(4, 6, 9, 11))
              max.day <- 30
          # Leap years
          if (m == 2) {
              if (y %% 4 == 0 & y %% 100 != 0 | y %% 400 == 0) 
                  max.day <- 29
              else 
                  max.day <- 28
          }
          for (d in 1:max.day) {
              week.day <- week.day + 1
              if (week.day == 8) week.day <- 1
              if (week.day == 1 & d == 1)
              answer <- answer + 1
          }
      }
  }
  print(answer)

Which day does the week start?

The only aspect remotely interesting about this problem is the conversion from weekdays to numbers. In R, Monday is considered day one, which makes sense in the Christian context of Western culture. Saturday and Sunday are the weekends, the two last days of the week so they are day 6 and 7. According to international standard ISO 8601, Monday is the first day of the week. Although this is the international standard, several countries, including the United States and Canada, consider Sunday to be the first day of the week.

The international standard is biased towards Christianity. The Christian or Western world marks Sunday as their day of rest and worship. Muslims refer to Friday as their day of rest and prayer. The Jewish calendar counts Saturday—the Sabbath—as the day of rest and worship. This idea is also shared by Seventh-Day Adventists.

This Euler problem shows that data science is not only about data: it is always how people interpret the world.

First day of the week around the world.

  ## First day of the week
  library(tidyverse)
  firstday <- read_csv("data/first_day.csv")

  map_data(map = "world") %>%
    full_join(firstday) %>%
    ggplot(aes(long, lat, group = group, fill = firstday)) +
    geom_polygon() +
    theme_void() +
    coord_fixed() +
    labs(title = "First day of the week")

  ggsave("images/problem-019.png", width = 6, height = 4)

Where do the Days of the Week Get Their Names?

R-Cade Games: Simulating the Legendary Pong Game in R

Peter Prevos — Thu, 20 Apr 2017 00:00:00 +0000

Pong is one of the earliest arcade games on the market, first released in 1972. From the day I first saw this miracle box, I wanted to know more about computers. There was something magical about being able to influence what happens on the television screen, even though the graphics left everything to t he imagintaion.

Original Atari PONG (1972) arcade machine gameplay video.

I learnt how to write code from the 1983 book Dr. C. Wacko's Miracle Guide to Designing and Programming your own Atari Computer Arcade Games. This book explains in a very clear and humorous way how to write computer games in Atari basic. I devoured this book and spent many hours developing silly games. This article is an ode to Dr Wacko, a computer geek's midlife-crisis and an attempt to replicate the software I developed thirty years ago.

Dr. C. Wacko's Miracle Guide to Designing and Programming your own Atari Computer Arcade Games

I showed in a other posts that R can be used for board games or to play a text adventure. The question is whether we create arcade games in R. My challenge is to recreate the look and feel of 1980s arcade games, or R-Cade games, using R? The code shown below simulates the legendary game of pong.

Playing Pong in R

The code is based on the Wacko's Boing Program in the above-mentioned book. The R code is fully commented and speaks for itself. Please note that the animation is very clunky when you run it in RStudio. Only the native R Terminal displays the animation correctly.

This version plays automatically with a randomiser routine to render the gameplay imperfect. The skill parameter indicates how good the computer player is between 0 and 1.

The beepr library by Rasmus Bååth provides some much-needed sound.

The main issue with this code is that I have not been able to ad a human Pong player. I am not aware of a way to use continuous input in the R language. Perhaps somebody can help me perfect this little ditty. I love to know how to read real-time USB input to control the game, so we get a step closer to the first R-Cade game.

R language Pong simulation.

  ## Simulating the Legendary Pong Game in R

  ## Sound library
  library(beepr) 

  ## Game parameters
  skill <- 0.87 # Skill (0-1)
  score <- 0
  high.score <- 0

  ## Define playing field
  x11()
  par(mar = rep(1,4), bg = "black")
  plot.new()
  plot.window(xlim = c(0,30), ylim = c(0,30))
  lines(c(1, 30, 30, 1), c(0, 0, 30, 30), type = "l", lwd = 5, col = "white")

  ## Playing field boundaries (depends on cex)
  xmin <- 0.5
  xmax <- 29.4
  ymin <- 0.5
  ymax <- 29.4

  ## initial position
  x <- sample(5:25, 1)
  y <- sample(5:25, 1)
  points(x, y, pch = 15, col = "white", cex = 2)

  ## Paddle control
  psize <- 4
  ypaddle <- y

  ## Set direction
  dx <- runif(1, .5, 1)
  dy <- runif(1, .5, 1)

  ## Game play
  while (x > xmin - 1) {
      sound <- 0 # Silence
      Sys.sleep(.05) # Pause screen
      points(x, y, pch = 15, col = "black", cex = 2) # Erase ball
      # Move ball
      x <- x + dx
      y <- y + dy
      # Collision detection
      if (x > xmax) {
          dx <- -dx * runif(1, .9, 1.1) # Bounce
          if (x > xmin) x <- xmax # Boundary
          sound <- 10 # Set sound
          }
      if (y < ymin | y > ymax) {
          if (y < ymin) y <- ymin
          if (y > ymax) y <- ymax
          dy <- -dy * runif(1, .9, 1.1)
          sound <- 10
      }
      # Caught by paddle?
      if (x < xmin & (y > ypaddle - (psize / 2)) & y < ypaddle + (psize / 2)) {
          if (x < xmin) x <- xmin
          dx <- -dx * runif(1, .9, 1.1)
          sound <- 2
          score <- score + 1
          }
      # Draw ball
      points(x, y, pch = 15, col = "white", cex = 2)
      if (sound !=0) beep(sound)
      # Move paddle
      if (runif(1, 0, 1) < skill) ypaddle <- ypaddle + dy # Imperfect follow
      # Draw paddle
      # Erase back line
      lines(c(0, 0), c(0, 30), type = "l", lwd = 8, col = "black")
      # Keep paddle in window
      if (ypaddle < (psize / 2)) ypaddle <- (psize / 2)
      if (ypaddle > 30 - (psize / 2)) ypaddle <- 30 - (psize / 2)
      # Draw paddle
      lines(c(0, 0), c(ypaddle - (psize / 2), ypaddle + (psize / 2)), type = "l", lwd = 8, col = "white")
  }
  beep(8)
  text(15,15, "GAME OVER", cex=5, col = "white")
  s <- ifelse(score == 1, "", "s")
  text(15,5, paste0(score, " Point", s), cex=3, col = "white")

Project Euler 18: Maximum Path Sum

Peter Prevos — Mon, 17 Apr 2017 00:00:00 +0000

Project Euler 18 and 67 are identical, other than that the data in the second version is more extensive than in the first one. In this post, I kill two Eulers with one code. These two problems are logically similar to Project Euler 15.

These two problems concern binary trees, which is a data structure where each node has two children. A practical example of a binary tree is a pedigree chart, where each person or animal has two parents, four grandparents and so on. In this Project Euler problem, we need to navigate through such a tree.

Project Euler 18 Definition

By starting at the top of the triangle below and moving to adjacent numbers on the row below, the maximum total from top to bottom is 23.

That is, 3 + 7 + 4 + 9 = 23.

Find the maximum total from top to bottom of the triangle below:

                            75
                          95  64
                        17  47  82
                      18  35  87  10
                    20  04  82  47  65
                  19  01  23  75  03  34
                88  02  77  73  07  63  67
              99  65  04  28  06  16  70  92
            41  41  26  56  83  40  80  70  33
          41  48  72  33  47  32  37  16  94  29
        53  71  44  65  25  43  91  52  97  51  14
      70  11  33  28  77  73  17  78  39  68  17  57
    91  71  52  38  17  14  91  43  58  50  2 7 2 9 48
  63  66  04  68  89  53  67  30  73  16  69  87  40  31
04  62  98  27  23  09  70  98  73  93  38  53  60  04  23

Solution

This problem seeks a maximum path sum in a binary tree. The brute force method, as indicated in the problem definition, is a very inefficient way to solve this problem.

A more efficient method is to define the maximum path layer by layer, starting at the bottom. The maximum sum of 2 + 8 or 2 + 5 is 10, the maximum sum of 4 + 5 or 4 + 9 is 13, and the last maximum sum is 15. These numbers are now placed in the next row. This process cycles until only one number is left. This algorithm solves the sample triangle into four steps:

Step 1:

7 4

2 4 6

8 5 9 3

Step 2:

7 4

10 13 15

Step 3:

20 19

Step 4:

In the code below, the data is a triangle matrix. The variables rij (row) and kol (column) drive the search for the maximum path. The triangle for Euler Problem 18 is manually created, and the triangle for Euler Problem 67 is read from the website.

  ## Project Euler 18: Maximum Path Sum
  path.sum <- function(triangle) {
      for (rij in nrow(triangle):2) {
          for (kol in 1:(ncol(triangle)-1)) {
              triangle[rij - 1,kol] <- max(triangle[rij,kol:(kol + 1)]) + triangle[rij - 1, kol]
          }
          triangle[rij,] <- NA
      }
  return(max(triangle, na.rm = TRUE))
  }

  triangle <- matrix(ncol = 15, nrow = 15)
  triangle[1,1] <- 75
  triangle[2,1:2] <- c(95, 64)
  triangle[3,1:3] <- c(17, 47, 82)
  triangle[4,1:4] <- c(18, 35, 87, 10)
  triangle[5,1:5] <- c(20, 04, 82, 47, 65)
  triangle[6,1:6] <- c(19, 01, 23, 75, 03, 34)
  triangle[7,1:7] <- c(88, 02, 77, 73, 07, 63, 67)
  triangle[8,1:8] <- c(99, 65, 04, 28, 06, 16, 70, 92)
  triangle[9,1:9] <- c(41, 41, 26, 56, 83, 40, 80, 70, 33)
  triangle[10,1:10] <- c(41, 48, 72, 33, 47, 32, 37, 16, 94, 29)
  triangle[11,1:11] <- c(53, 71, 44, 65, 25, 43, 91, 52, 97, 51, 14)
  triangle[12,1:12] <- c(70, 11, 33, 28, 77, 73, 17, 78, 39, 68, 17, 57)
  triangle[13,1:13] <- c(91, 71, 52, 38, 17, 14, 91, 43, 58, 50, 27, 29, 48)
  triangle[14,1:14] <- c(63, 66, 04, 68, 89, 53, 67, 30, 73, 16, 69, 87, 40, 31)
  triangle[15,1:15] <- c(04, 62, 98, 27, 23, 09, 70, 98, 73, 93, 38, 53, 60, 04, 23)
  answer <- path.sum(triangle)
  print(answer)

Project Euler 67

The solution for problem number 67 is exactly the same. The data is read directly from the Project Euler website.

  triangle.file <- read.delim("https://projecteuler.net/project/resources/p067_triangle.txt",
                              stringsAsFactors = FALSE, header = FALSE)
  triangle.67 <- matrix(nrow = 100, ncol = 100)
  for (i in 1:100) {
      triangle.67[i,1:i] <- as.numeric(unlist(strsplit(triangle.file[i,], " ")))
  }
  answer <- path.sum(triangle.67)
  print(answer)

This video visualises the quest for the maximum path as a brute-force solution, which takes eleven minutes of hypnotic animation.

Maximum path sum.

Project Euler 17: Number Letter Counts

Peter Prevos — Thu, 23 Mar 2017 00:00:00 +0000

Project Euler 17 asks to count the letters in numbers written as words. This is a skill we all learnt in primary school mainly useful when writing cheques—to those that still use them. The principle for this problem is similar to Project Euler 22, which analyses given names.

Each language has its own rules for writing numbers. My native language Dutch has very different logic to English. Both Dutch and English use compound words after the number twelve.

Linguists have theorised this is evidence that early Germanic numbers were duodecimal. The importance of a “dozen” as a counting word and the twelve hours in the clock support this idea. There is even a Dozenal Society that promotes the use of a number system based on 12.

The English language changes the rules when reaching the number 21. While we say eight-teen in English, we do no say “one-twenty”. Dutch stays consistent and the last number is always spoken first. For example, 37 in English is “thirty-seven”, while in Dutch it is written as “zevenendertig” (seven and thirty).

Project Euler 17 Definition

If the numbers 1 to 5 are written out in words: one, two, three, four, five, then there are 3 + 3 + 5 + 4 + 4 = 19 letters used in total. If all the numbers from 1 to 1000 (one thousand) inclusive were written out in words, how many letters would be used?

Do not count spaces or hyphens. For example, 342 (three hundred and forty-two) contains 23 letters, and 115 (one hundred and fifteen) contains 20 letters. The use of “and” when writing out numbers is in compliance with British usage.

Solution

The first piece of code provides a function that generates the words for numbers 1 to 999,999. This is more than the problem asks for, but it might be a useful function for another application. The last line concatenates all words and removes the spaces to count the number of letters.

  ## Project Euler 17: Number Letter Counts
  numword.en <- function(x) {
      if (x > 999999) return("Error: Oustide my vocabulary")
      # Vocabulary 
      single <- c("one", "two", "three", "four", "five", "six", "seven", "eight", "nine")
      teens <- c( "ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen")
      tens <- c("ten", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety")
      numword.10 <- function (y) {
          a <- y %% 100
          if (a != 0) {
              and <- ifelse(y > 100, "and", "")
              if (a < 20)
                  return (c(and, c(single, teens)[a]))
              else
                  return (c(and, tens[floor(a / 10)], single[a %% 10]))
          }
      }
      numword.100 <- function (y) {
          a <- (floor(y / 100) %% 100) %% 10
          if (a != 0)
              return (c(single[a], "hundred"))
      }
      numword.1000 <- function(y) {
          a <- (1000 * floor(y / 1000)) / 1000
          if (a != 0)
              return (c(numword.100(a), numword.10(a), "thousand"))
      }
      numword <- paste(c(numword.1000(x), numword.100(x), numword.10(x)), collapse=" ")
      return (trimws(numword))
  }

  answer <- nchar(gsub(" ", "", paste0(sapply(1:1000, numword.en), collapse="")))
  print(answer)

Writing Numbers in Dutch

I went beyond Euler Problem 17 by translating the code to spell numbers in Dutch. Interesting bit of trivia is that it takes 307 fewer characters to spell the numbers 1 to 1000 in Dutch than it does in English. It would be good if other people can submit functions for other languages in the comment section. Perhaps we can create an R package with a multi-lingual function for spelling numbers.

  ## Nederlandse nummers
  numword.nl <- function(x) {
      if (x > 999999) return("Error: Getal te hoog.")
      single <- c("een", "twee", "drie", "vier", "vijf", "zes", "zeven", "acht", "negen")
      teens <- c( "tien", "elf", "twaalf", "dertien", "veertien", "vijftien", "zestien", "zeventien", "achtien", "negentien")
      tens <- c("tien", "twintig", "dertig", "veertig", "vijftig", "zestig", "zeventig", "tachtig", "negengtig")
      numword.10 <- function(y) {
          a <- y %% 100
          if (a != 0) {
              if (a < 20)
                  return (c(single, teens)[a])
              else
                  return (c(single[a %% 10], "en", tens[floor(a / 10)]))
          }
      }
      numword.100 <- function(y) {
          a <- (floor(y / 100) %% 100) %% 10
          if (a == 1)
              return ("honderd")
          if (a > 1) 
              return (c(single[a], "honderd"))
      }
      numword.1000 <- function(y) {
          a <- (1000 * floor(y / 1000)) / 1000
          if (a == 1)
              return ("duizend ")
          if (a > 0)
              return (c(numword.100(a), numword.10(a), "duizend "))
      }
      numword<- paste(c(numword.1000(x), numword.100(x), numword.10(x)), collapse="")
      return (trimws(numword))
  }

  antwoord <- nchar(gsub(" ", "", paste0(sapply(1:1000, numword.nl), collapse="")))
  print(antwoord)

Project Euler 16: Power digit sum

Peter Prevos — Thu, 16 Mar 2017 00:00:00 +0000

Project Euler 16 is reminiscent of the fable of wheat and chess. Lahur Sessa invented the game of chess for King Iadava. The king was delighted with the game and asked Lahur to name his reward. Lahur asked the king to place one grain of rice on the first square of a chessboard, two on the next square, four on the third square and so on until the board is filled.

The king was happy with his humble request until his mathematicians worked out that it would take millions of tonnes of grain. Assuming there are 25 grains of wheat in a gramme, the last field will contain more than 461,168,602,000 tonnes of grain.

Project Euler 16 Definition

$2^{15} = 32768$ and the sum of its digits is $3 + 2 + 7 + 6 + 8 = 26$.

What is the sum of the digits of the number $2^{1000}$?

Possible Solutions

The most straightforward solution uses the GMP package for Multiple Precision Arithmetic to calculate big integers. The as.bigz() function results in a special class of arbitrarily large integer numbers. To add the digits, the results is converted to a string and split into numbers.

  ## Project Euler 16: Power digit sum
  library(gmp)
  digits <- as.bigz(2^1000) # Define number
  answer <- sum(as.numeric(unlist(strsplit(as.character(digits), ""))))
  print(answer)

We can also solve this problem in base-R with the bigg.add() used for Euler Problem 13. This function uses basic string operations to add to arbitrarily large numbers. Raising a number to the power of two can also be written as a series of additions:

$$2^4 = 2 \times 2 \times 2 \times 2 = ((2+2)+(2+2)) + ((2+2)+(2+2))$$

The solution to this problem is to add 2 + 2 then add the outcome of that equation to itself, and so on. Repeat this one thousand times to raise the number two to the power of one thousand.

  ## Base-R Method
  source("problem013.R", echo = FALSE)
  pow <- 2
  for (i in 2:1000)
      pow <- big.add(pow, pow)
  answer <- sum(as.numeric(unlist(strsplit(pow, ""))))
  print(answer)

Project Euler 15: Lattice Paths

Peter Prevos — Thu, 09 Mar 2017 00:00:00 +0000

Project Euler 15 analyses taxicab geometry. This system replaces the usual distance function with the sum of the absolute differences of their Cartesian coordinates. In other words, the distance a taxi would travel in a grid plan instead of the shortest distance between two points. In chess, the distance between squares on the chessboard for rooks is measured in taxicab distance.

The fifteenth Euler problem asks to determine the number of possible routes a taxi can take in a city. This problem is logically similar to Euler Problem 18 and 67.

Taxicab Geometry.

Project Euler 15 Definition

Starting in the top left corner of a 2 by 2 grid, and only being able to move to the right and down, there are exactly 6 routes to the bottom right corner.

Project Euler 15: Lattice Paths.

How many such routes are there through a 20×20 grid?

proposed Solutions

The matrix to solve the problem is one larger than the number of squares in the challenge. Along the edges of the matrix, only one pathway is possible: straight to the right or down. We can calculate the number of possible paths for the other squares by adding the number to the right and below the point under consideration. The code loops through the matrix backwards, adding the number of paths step by step.

$$p_{i,j}=p_{i,j-1}+p_{{i+1},j}$$

For the two by two lattice the solution space is:

6  3  1
3  2  1
1  1  .

The total number of pathways from the upper left corner to the lower right corner is thus 6. This logic can now be applied to a grid of any arbitrary size using the following code. The code defines the lattice and initiates the boundary conditions. The bottom row and the right column are filled with 1 as there is only one solution from these points. The code then calculates the pathways by working backwards through the matrix. The final solution is the number is the first cell.

  ## Project Euler 15: Lattice Paths
  # Define lattice
  nLattice <- 2
  lattice = matrix(ncol = nLattice + 1, nrow = nLattice + 1)

  # Boundary conditions
  lattice[nLattice + 1, -(nLattice + 1)] <- 1
  lattice[-(nLattice + 1), nLattice + 1] <- 1

  # Calculate Pathways
  for (i in nLattice:1) {
      for (j in nLattice:1) {
          lattice[i,j] <- lattice[i+1, j] + lattice[i, j+1]
      }
  }
  answer <- lattice[1,1]
  print(answer)

Create Air Travel Route Maps with ggplot2

Peter Prevos — Tue, 07 Mar 2017 07:24:24 +0000

I have been lucky to fly to a few countries around the world. Like any other bored traveller, I thumb through the airline magazines and look at the air travel route maps. These maps are beautifully stylised depictions of the world with gently curved lines between the destinations serviced by the airline. I always wanted such a map for my own travel adventures. This article explains how to create air travel route maps with ggplot2 in the style of the Emirates Airlines route map.

Create Air Travel Route Maps using ggplot2

The first step was to create a list of all the places I have flown between at least once. Paging through my travel photos and diaries, I managed to create a pretty complete list. The structure of this document is simply a list of all routes (From, To) and every flight only gets counted once. The next step finds the spatial coordinates for each airport by searching Google Maps using the geocode function from the ggmap package. You will need a Google API to enable the geocoding function.

In some instances, I had to add the country name to avoid confusion between places. To prevent errors from the Google maps API, I have added a while loop that runs until all destinations have been geocoded. We now we have a data frame of airports with their coordinates and can create air travel route maps. The data frames are merged so that we can create air travel route maps using the curve geom. The borders function of ggplot2 creates the map data. The ggrepel package helps to prevent overplotting of text. This code also removes any return flights and splits flights that crossed the date line.

My personal Air Travel Route Maps in ggplot: A Visual Travel Diary

In another article, I have used the same principle to create a route map of flights between islands in the Pacific Ocean using the schedules from several international airlines and to show demonstrate the nonsense of Flat Earth theories.

  ## Flightpath map
  ## https://lucidmanager.org/create-air-travel-route-map

  ## Init
  library(tidyverse)
  library(ggmap)
  library(ggrepel)
  
  api <- readLines("../google-maps.api") # Text file with the API key
  register_google(key = api)

  ## Read flight and airports lists 
  #flights <- read_csv("flights.csv")
  flights <- read_csv("flight-map/flights.csv")
  airports_file <- "flight-map/airports.csv"

  if (file.exists(airports_file)) {
    airports <- read_csv(airports_file)
    } else {
    airports <- tibble(airport = NA, lon = NA, lat= NA)
  }

  ## Lookup coordinates
  ## Some airports need country names to ensure Google finds the correct location
  ## The geocoding keeps looping till all coordinates have been found
  destinations <- unique(c(flights$From, flights$To))
  new_destinations <- destinations[!destinations %in% airports$airport]

  while (length(new_destinations) > 0) {
      new_airports <- geocode(new_destinations) %>%
        mutate(airport = new_destinations) %>%
        select(airport, lon, lat)
      airports <- rbind(airports, new_airports) %>%
        filter(!is.na(lon) | !is.na(lat))
      new_destinations <- destinations[!destinations %in% airports$airport]
  }

  write_csv(airports, "flight-map/airports.csv")

  ## Remove country names
  airports$airport <- as.character(airports$airport)
  comma <- regexpr(",", airports$airport)
  airports$airport[which(comma > 0)] <- substr(airports$airport[which(comma > 0)], 1, comma[comma > 0] - 1)

  ## Remove return flights
  d <- vector()
  for (i in 1:nrow(flights)) {
      d <- which(paste(flights$From, flights$To) %in%
                 paste(flights$To[i], flights$From[i]))
      flights$From[d] <- "R"
  }
  flights2 <- flights %>%
    filter(From != "R") %>%
    select(From, To)

  ## Add coordinates to flight list
  flights <- merge(flights, airports, by.x = "From", by.y = "airport")
  flights <- merge(flights, airports, by.x = "To", by.y = "airport")
  flights <- flights %>% 
    select(From, To, lon.x, lat.x, lon.y, lat.y) %>% 
    as_tibble()

  ## Split Circumnaviation Flights at -180/180 degrees
  circ <- which(abs(flights$lon.y - flights$lon.x) > 180)
  flights[circ,]
  flights$lon.y[circ] <- ifelse(flights$lon.y[circ] < 0, 180, -180)
  flights$lat.y[circ] <- rowSums(flights[circ, c("lat.x", "lat.y")]) / 2
  leg2 <- airports %>%
    filter(airport %in% flights$To[circ]) %>%
    mutate(From = rep("", length(circ))) %>%
    mutate(lon.x = -flights$lon.y[circ], lat.x = flights$lat.y[circ]) %>%
    select(From, To = airport, lon.x, lat.x, lon.y = lon, lat.y = lat)
  flights <- rbind(flights, leg2)

  ## Plot flight routes
  worldmap <- borders("world2", colour="#efede1", fill="#efede1") 
  ggplot() + worldmap + 
      geom_point(data = airports, aes(x = lon, y = lat), col = "#970027") + 
      geom_text_repel(data=airports, aes(x = lon, y = lat, label = airport), col = "black", size = 2, segment.color = NA) + 
      geom_curve(data = flights, aes(x = lon.x, y = lat.x, xend = lon.y, yend = lat.y), col = "#b29e7d", size = .4) + 
      theme_void()
  ggsave("flights_map.png", dpi = 300)

Project Euler 14: Longest Collatz sequence

Peter Prevos — Thu, 02 Mar 2017 00:00:00 +0000

Euler Problem 14 looks at the Collatz Conjecture. These playful sequences, named after German mathematician Lothar Collatz (1910–1990), cause mathematicians a lot of headaches. What is most interesting about this problem is how a simple set of rules can create intricate complexity.

Project Euler 14 Definition

The following iterative sequence is defined for the set of positive integers:

$f(n) = \begin{cases} \frac{n}{2} &\text{if } n \equiv 0 \pmod{2}\\[4px] 3n+1 & \text{if } n\equiv 1 \pmod{2} .\end{cases}$

Using the rule above, and starting with 13, we generate the following sequence:

$$13 \rightarrow 40 \rightarrow 20 \rightarrow 10 \rightarrow 5 \rightarrow 16 \rightarrow 8 \rightarrow 4 \rightarrow 2 \rightarrow 1 $$

It can be seen that this sequence (starting at 13 and finishing at 1) contains 10 terms. Although it has not been proved yet (Collatz Problem), it is thought that all starting numbers finish at 1.

Which starting number, under one million, produces the longest chain?

NOTE: Once the chain starts, the terms are allowed to go above one million.

Uncrackable? The Collatz Conjecture - Numberphile.

Proposed solutions

This problem is highly computationally intensive, and it highlights R's lack of speed. Generating one million Collatz sequences and finding the longest one requires a lot more than a minute of processing time allowed for in Project Euler.

Brute Force

The brute force solution tests every answer and takes a long time to run.

  ## Project Euler 14: Longest Collatz Sequence
  ## Brute Force
  collatz.chain <- function(n) {
      chain <- vector()
      i <- 1
      while (n != 1) {
          if (n%%2 == 0)
              n <- n / 2
          else
          n <- 3 * n + 1
          chain[i] <- n
          i <- i + 1
      }
      return(chain)
  }
  answer <- 0
  collatz.max <- 0
  for (n in 1:1E6) {
      collatz.length <- length(collatz.chain(n))
      if (collatz.length > collatz.max) {
          answer <- n
          collatz.max <- collatz.length
      }
  }
  print(answer)

Optimised solution

The optimised version of the code stores the length of all sequences in an array. When the code generates a sequence and lands on a number already analysed, then it adds that previous number to the current one and moves on to the next step. This approach requires more memory but saves a lot of computation time. A minor tweak to the code optimises the rule for uneven numbers. Tripling an uneven number and adding one will always result in an even number so we can skip one step. This solution is more than twice as fast as the first version.

  ## Optimised method
  collatz.length <- vector(length = 1E6)
  collatz.length[1] <- 0
  for (n in 2:1E6) {
      x <- n
      count <- 0 
      while (x != 1 & x >= n) {
          if (x %% 2 == 0) {
              x <- x / 2
              count <- count + 1
          }
          else {
              x <- (3 * x + 1) / 2
              count <- count + 2
          }
      }
      count <- count + collatz.length[x]
      collatz.length[n] <- count
  }
  answer <- which.max(collatz.length)
  print(answer)

Visualising Collatz Sequences

The Collatz sequence is an example of a simple mathematical rule that can create an unpredictable pattern. The number of steps required to reach one is listed in sequence A006577 of the Online Encyclopedia of Integer Sequences. The image below visualises the number of steps for the first 1000 positive numbers. The scatterplot shows some unusual patterns. Does this visualisation show that the Collatz Sequence does have a pattern after all?

Number of steps to terminate a Collatz sequence.

  ## Visualise
  library(ggplot2)
  collatz <- data.frame(n = 1:10000,
                        steps = collatz.length[1:10000])
  ggplot(collatz, aes(n, steps)) +
    geom_point(size = .5, col = "darkblue") +
    theme_minimal(base_size = 10) +
    labs(title = "Collatz Sequence",
         subtitle = "Number of steps to reach 1")
  ggsave("../../../../static/images/project-euler/problem-014a.png", width = 6, height = 4)

Collatz Chains

The Collatz sequences can also be visualised using networks. Each step between two numbers is an edge, and the numbers are the vertices. For example, the network for the Collatz sequence for number 10 is 5–16, 16–8, 8–4, 4–2, 2–1. When generating subsequent sequences, the network will start to overlap, and a tree of sequences appears. The tree below combines the Collatz sequences for the numbers 2 to 26. Number 27 has a very long sequence, making the tree much harder to read. The code to create this network uses the igraph package. The edge list is a data frame where each row indicates a connected set of numbers.

Collatz network $(n \leq 26)$.

## Collatz Chains
library(igraph)
edgelist <- data.frame(a = 2, b = 1)
for (n in 3:26) {
   chain <- as.character(c(n, collatz.chain(n)))
   chain <- data.frame(a = chain[-length(chain)], b = chain[-1])
   edgelist <- rbind(edgelist, chain)
}
g <- graph.edgelist(as.matrix(edgelist))
g <- simplify(g)

png("../../../../static/images/project-euler/problem-014b.png", 
    width = 1024, height = 768)
par(mar = rep(0,4))
V(g)$color <- degree(g, mode = "out") == 0
plot(g,
     layout = layout.auto,
     vertex.color = V(g)$color,
     vertex.frame.color = NA,
     frame.color = "white",
     vertex.size = 6,
     vertex.label.cex = 1.5,
     vertex.label.color = "black",
     edge.arrow.size = 0.2,
     edge.color = "grey"
     )
 dev.off()

Project Euler 13: Adding Very Large Integers

Peter Prevos — Thu, 23 Feb 2017 00:00:00 +0000

Project Euler 13 asks to add one hundred numbers with fifty digits. This seems like a simple problem where it not that most computers are not designed to deal with numbers with a lot of integers. For example: 2⁶⁴ = 18446744073709551616. When asking R to compute this value, we get 1.844674e+19, losing most of the digits and limiting the accuracy of the results. Computers solve this problem using Arbitrary-Precision Arithmetic. Many software libraries can process long integers without losing accuracy. Euler Problem 13 requires this type of approach.

Project Euler 13 Definition

Work out the first ten digits of the sum of the following one-hundred 50-digit numbers.

Solution

The easy way to solve this problem is to use the GMP package for working with huge integers. This package uses a special number types such as Big Rational and Big Integer. The number of digits in these number types is only limited by the size of your computer's memory.

  ## Project Euler 13: Adding Very Large Numbers
  ## GMP Library
  library(gmp)
  numbers <- readLines("data/p013_numbers.txt")

  gmp_method <- function() {
      digits <- sum(as.bigz(numbers))
      answer <- substr(as.character(digits),1,10)
      return(answer)
  }
  gmp_method()

Using Base-R

To find the solution to this problem using only base R, I wrote a function to add numbers using strings instead of integers. The function adds leading zeros to the smallest number to make them both the same length. The function then proceeds to add numbers in the same way we were taught in primary school.

  ## Base-R Solution
  big.add <- function(a, b) {
    ## Add leading zeros to smallest number
    if (nchar(a)<nchar(b)) 
      a <- paste0(paste(rep(0, nchar(b) - nchar(a)), collapse = ""), a)
    if (nchar(a)>nchar(b)) 
      b <- paste0(paste(rep(0, nchar(a) - nchar(b)), collapse = ""), b)
    solution <- vector()
    remainder <- 0
    for (i in nchar(b):1) {
      p <- as.numeric(substr(a, i, i))
      q <- as.numeric(substr(b, i, i))
      r <- p + q + remainder
      if (r >= 10 & i != 1) {
        solution <- c(solution, r %% 10)
        remainder <- (r - (r %% 10)) / 10
      } else {
        solution <- c(solution, r)
        remainder <- 0
      }
    }
    return(paste(rev(solution), collapse = ""))
  }

With this function, the problem is easy to solve. The second part of the code runs this function over the one hundred numbers provided on the Euler Problem page and calculates the answer.

    base_method <- function() {
        answer <- 0
        for (i in numbers)
            answer <- big.add(answer, i)
        answer <- substr(answer, 1, 10)
        return(answer)
    }

    base_method()

This solution for managing very large numbers is reused in Project Euler 25 which asks for a Fibonacci number with 1000 decimals.

Project Euler 12: Highly Divisible Triangular Number

Peter Prevos — Thu, 16 Feb 2017 00:00:00 +0000

Project Euler 12 takes us to the realm of triangular numbers and proper divisors. The image below shows a hands-on method to visualise the number of divisors of ten. Cuisenaire rods are learning aids to explore mathematics. Each of the coloured rods has a different integer length, which allows students to explore basic arithmetic. In this article, I discuss a method to generate triangular numbers using the R language.

Cuisenaire-rods (Source: Wikimedia user Celcom / CC BY).

Project Euler 12 Definition

The sequence of triangle numbers is generated by adding the natural numbers. So the 7th triangle number would be $1 + 2 + 3 + 4 + 5 + 6 + 7 = 28$. The first ten terms would be: $1, 3, 6, 10, 15, 21, 28, 36, 45, 55, \ldots$

Let us list the factors of the first seven triangle numbers:

 1: 1
 3: 1, 3
 6: 1, 2, 3,  6
10: 1, 2, 5, 10
15: 1, 3, 5, 15
21: 1, 3, 7, 21
28: 1, 2, 4,  7, 14, 28

We can see that 28 is the first triangle number to have over five divisors.

What is the value of the first triangle number to have over five hundred divisors?

Solution

In the video below, Vishal Kataria explains a simple method to determine the number of divisors using prime factorization. The prime factorization of $n$ is given by:

$$n = p^{\alpha_1}_1 \times p^{\alpha_2}_2 \times \ldots \times p^{\alpha_k}_k$$

Where $p$ is a prime factor, and $\alpha$ is the number of times the factor is repeated.

The number of proper divisors is for $n$ is:

$$d = (\alpha_1 + 1) (\alpha_2 + 1) \times \ldots \times (\alpha_k + 1)$$

The code reuses the prime factorisation function developed for Euler Problem 3. This function results in a vector of all prime factors, e.g. the prime factors of 28 are 2, 2 and 7.

The code to solve this problem determines the values for alpha using the run-length rle() function. This function counts the number of times each element in a sequence is repeated. The outcome of this function is a vector of the values and the number of times each is repeated. The prime factors of 28 are 2, 2 and 7, and their run lengths are 2 and 1. The number of divisors can now be determined.

$$ 28 = 2^2 \times 7^1$$

$$ d = (2+1)(1+1) = 6$$

The six divisors of 28 are 1, 2, 4, 7, 14 and 28.

The code to solve Project Euler 12 is shown below. The loop continues until it finds a triangular number with 500 divisors. The first two lines increment the index and create the next triangular number. The third line in the loop determines the number of times each factor is repeated (the run lengths). The last line calculates the number of divisors using the above formula.

  ## Project Euler 12: Highly Divisible Triangular Number
  i <- 0
  divisors <- 0
  while (divisors < 500) {
      i <- i + 1
      triangle <- (i * (i + 1)) / 2
      pf <- prime.factors(triangle)
      alpha <- rle(pf)
      divisors <- prod(alpha$lengths + 1)
  }
  answer <- triangle
  print(answer)

Vishal Kataria. Number Theory - Number of Divisors.

Percentile Calculations in Water Quality Regulations

Peter Prevos — Fri, 10 Feb 2017 00:00:00 +0000

Percentile calculations can be more tricky than at first meets the eye. A percentile indicates the value below which a percentage of observations fall. Some percentiles have special names, such as the quartile or the decile, both of which are quantiles. This deceivingly simple definition hides the various ways to determine this number. Unfortunately, there is no standard definition for percentiles, so which method do you use?

The quantile function in R generates sample percentiles corresponding to the given probabilities. By default, the quantile function provides the quartiles and the minimum and maximum values. The code snippet below creates semi-random data, plots the histogram and visualises the third quartile.

The code for this page is available on GitHub in case-studies/percentiles.R.

  ## Percentile calculations

  ## Example
  set.seed(1969)
  test.data <- rnorm(n = 10000, mean = 100, sd = 15)

  library(ggplot2)

  ggplot(as.data.frame(test.data), aes(test.data)) +
    geom_histogram(binwidth = 1, aes(y = ..density..), fill = "dodgerblue") +
    geom_line(stat = "function", fun = dnorm, args = list(mean = 100, sd = 15), colour = "red", size = 1) +
    geom_area(stat = "function", fun = dnorm, args = list(mean = 100, sd = 15),
              colour = "red", fill = "red", alpha = 0.5, xlim = quantile(test.data, c(0.5, 0.75))) +
    theme_bw(base_size = 8)

Percentile Calculations example.

The quantile function with default settings and the 95^th percentile give the following results:

  quantile(test.data)

       0%       25%       50%       75%      100% 
 39.91964  89.68041 100.16437 110.01910 153.50195

  quantile(test.data, probs = 0.95)

     95% 
124.7775

Methods of percentile calculation

The quantile function in R provides nine different ways to calculate percentiles. Each of these options uses another method to interpolate between observed values. I will not discuss the mathematical nuances between these methods. Hyndman and Fan (1996) provides a detailed discussion of these methods.

The differences between the nine available methods only matter in highly skewed distributions, such as water quality data. All methods provide the same outcome for the normal distribution simulated above, as illustrated by the following code.

  sapply(1:9, function(m) quantile(test.data, 0.95, type = m))

     95%      95%      95%      95%      95%      95%      95% 
124.7775 124.7775 124.7775 124.7775 124.7775 124.7775 124.7775 
     95%      95% 
124.7775 124.7775

Percentile calculations in water quality

The Australian Drinking Water Quality Guidelines (November 2016) specify that: "based on aesthetic considerations, the turbidity should not exceed 5 NTU at the consumer's tap". The Victorian Safe Drinking Water Regulations(2015) relax this requirement and require that:

The 95^th percentile of results for samples in any 12 months must be less than or equal to 5.0 NTU.

The Victorian water quality regulator also specifies that the percentile should be calculated with the Weibull Method. This requirement raises two questions: What is the Weibull method? How do you implement this requirement in R?

The term Weibull Method is confusing as this is not a name generally used by statisticians. In Hyndman & Fan (1996), this method has the less poetic name $\hat{Q}_8(p)$. The Victorian water quality regulator references McBride (2005), who calls it the Weibull method. Waloddi Weibull, a Swedish engineer famous for his distribution, was one of the first to describe this method.

This theoretical interlude aside, how can we practically apply this to water quality data?

If you are interested in how the Weibull method works, the weibull.quantile() function below calculates a quantile p for a vector x using this method. This function gives the same result as quantile(x, p, type = 6).

  ## Weibull method
  weibull.quantile <- function(x, p) {
    # Order Samples from large to small
    x <- x[order(x, decreasing = FALSE)]
    # Determine the ranking of percentile according to Weibull (1939)
    r <- p * (length(x) + 1)
    # Linear interpolation
    rfrac <- (r - floor(r))
    return((1 - rfrac) * x[floor(r)] + rfrac * x[floor(r) + 1])
  }
  weibull.quantile(test.data, 0.95)

  quantile(test.data, 0.95, type = 6)

124.777545386916

Turbidity Data Example

Turbidity data is not normally distributed as it is always greater than zero. In this example, we use simulated data from a fictional water system called Gormsey. This data is also used in book Data Science for Water Utilities.

  ## Visualise Data
  library(tidyverse)

  gormsey <- read_csv("~/Documents/projects/r4h2o/data/water_quality.csv")
  turbidity <- filter(gormsey, Measure == "Turbidity" &
                               Suburb %in% c("Blancathey", "Tarnstead"))

  ggplot(turbidity, aes(Result)) +
    geom_density(fill = "dodgerblue", aes(y = ..density..)) +
    scale_x_continuous(trans = "log10") + 
    facet_wrap(~Suburb) +
    theme_bw() +
    labs(title = "Turbidity Distribution",
         x= "log Results")

Turbidity at customer tap in two towns in the fictional country of Gormsey.

When we calculate the percentiles for all nine methods available in the base-R function, we see that the Weibull method generally provides the most conservative result for heavily skewed data.

  # Calculate all percentile methods
  tapply(turbidity$Result, turbidity$Suburb,
         function(x) sapply(1:9, function(m) quantile(x, 0.95, type = m)))

$Blancathey
   95%    95%    95%    95%    95%    95%    95%    95%    95% 
6.0200 6.0200 6.0200 5.8820 6.2720 6.6500 5.9165 6.3980 6.3665 

$Tarnstead
     95%      95%      95%      95%      95%      95%      95% 
4.480000 4.480000 4.480000 4.432500 4.607500 4.837000 4.442000 
     95%      95% 
4.684000 4.664875

Chapter 4 of my book Data Science for Water Utilities discusses calculating descriptive statistics with R

Data Science for Water Utilities

Routledge Amazon

Project Euler 11: Largest product in a grid

Peter Prevos — Thu, 09 Feb 2017 00:00:00 +0000

Project Euler 11 is a relatively straightforward application of vector arithmetics in R. Just like problem 8, it asks for the largest product.

Project Euler 11 Definition

In the 20×20 grid below, four numbers along a diagonal line have been marked in red. The product of these numbers is 26 × 63 × 78 × 14 = 1788696.

What is the greatest product of four adjacent numbers in the same direction (up, down, left, right, or diagonally) in the 20×20 grid?

Project Euler 11.

Solution

The solution applies straightforward vector arithmetic to multiply the verticals, horizontals and diagonals. The product of all verticals is an array of the product of rows 1 to 4, rows 2 to 5 and so on. The code uses a similar logic for the horizontals and the diagonals.

  ## Project Euler 11: Largest product in a grid

  square <- as.matrix(read.delim("data/p011_matrix.txt", header = FALSE, sep = " "))

  prod.vert <- square[1:17, ] * square[2:18, ] * square[3:19, ] * square[4:20, ]
  prod.hori <- square[ ,1:17] * square[ ,2:18] * square[ ,3:19] * square[ ,4:20]
  prod.dia1 <- square[1:17, 1:17] * square[2:18, 2:18] * square[3:19, 3:19] * square[4:20, 4:20]
  prod.dia2 <- square[4:20, 1:17] * square[3:19, 2:18] * square[2:18, 3:19] * square[1:17, 4:20]

  answer <- max(prod.vert, prod.hori, prod.dia1, prod.dia2)
  print(answer)

Project Euler 10: Summation of Primes

Peter Prevos — Thu, 02 Feb 2017 00:00:00 +0000

Project Euler 10 asks for the summation of primes. Computationally this is a simple problem because we can re-use the prime sieve developed for Project Euler 3. Mathematicians consider primes the basic building blocks of number theory. No matter how hard we look, however, they do not seem to obey any logical sequence. There is thus no algorithm to create prime numbers, all we can do is mine them from the infinite field of integers.

Project Euler 10 Definition

The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17. Find the sum of all the primes below two million.

Solution

The sieve of Eratosthenes function from Project Euler 3 generates the prime numbers between two and two million, which can then easily be summed.

  ## Project Euler 10: Summation of Primes
  source("problem003.R", echo = FALSE)

  primes <- esieve(2e6)
  answer <- sum(primes)
  print(answer)

Goldbach's Conjecture

The summing of primes reveals an interesting problem in mathematics. Goldbach's conjecture is one of the oldest and best-known unsolved problems in number theory. In 1742 Goldbach wrote a letter to Euler in which he states that:

Every even integer greater than 2 can be expressed as the sum of two primes.

A Goldbach partition is the expression of a given even number as a sum of two primes, for example:

$$100 = 3 + 97 = 11 + 89 = 17 + 83 = 29 + 71 = 41 + 59 = 47 + 53$$

All even numbers greater than 12 can be written as the sum of two primes in more than two ways. Sequence A045917 in the Online Encyclopedia of Integer Sequences lists the number of ways in which $2n$ can be written as the sum of two primes ($n\geq1$). This conjecture is as yet unproven and one of the most persistent problems in mathematics.

Goldbach Conjecture - Numberphile.

We can write some code to verify that every integer greater than two and less than $n$ can be expressed as the sum of two primes. This code recreates sequence A045917 and visualises the first 1000 results. There does not seem to be a pattern in the number of ways to sum two primes. However, When visualising the sequence with a scatter plot, horizontal bands of points appear.

  ## Goldbach's Conjecture
  goldbach <- function(n) {
      if (n%%2 != 0) return("This only works for even numbers")
      if (n == 2) return(0)
      primes <- rev(esieve(n)) # All primes below n
      k <- length(primes)
      g <- 0
      for (i in 1:k) {
        for (j in i:k) {
          if((primes[i] + primes[j]) == n) {
            # print(paste(n, "=", primes[i], "+", primes[j]))
            g <- g + 1
          }
        }
      }
      return(g)
    }

    library(ggplot2)
    n <- seq(from = 2, by = 2, length.out = 1000)
    A002375 <- data.frame(n,
                          g = sapply(n, goldbach))
    ggplot(A002375, aes(n, g)) +
      geom_point(col = "red", size = .5) +
      theme_minimal(base_size = 10) +
      labs(title = "Goldbach's Expressions",
           subtitle = "Number of ways to write an even number as the sum of two primes",
           x = "Even number", y = "Number of prime sums")
    ggsave("images/problem-010.png", width = 6, height = 4)

Number of ways to write an even number $n$ as the sum of two primes $(2 \leq n \leq 1000)$, (sequence A002375 in the OEIS).

Spike Detection in Water Treatment Data in the R Language

Peter Prevos — Mon, 30 Jan 2017 00:00:00 +0000

Spike detection in time series is essential to water treatment plant control. Spikes in SCADA data are events in the data stream of water treatment plants or similar installations. These spikes can indicate problems with the process and could result in an increased risk to public health. The WSAA Health Based Targets Manual specifies a series of decision rules to assess the performance of filtration processes. For example, this rule evaluates the performance of conventional filtration:

Individual filter turbidity ≤ 0.2 NTU for 95% of the month and not > 0.5 NTU for ≥ 15 consecutive minutes.

Turbidity is a measure for the cloudiness of fluid because of large numbers of individual particles otherwise invisible to the naked eye. Turbidity is an essential parameter in water treatment because a high level of cloudiness strongly correlates with the presence of microbes. This article uses the R language to show how to implement this specific decision rule.

Simulation

I first create a simulated SCADA feed for turbidity to create a minimum working example. The turbidity data frame contains 24 hours of data. The seq.POSIXt() function creates 24 hours of timestamps at a one-minute spacing. In addition, the rnorm() function creates 1440 turbidity readings with an average of 0.1 NTU and a standard deviation of 0.01 NTU. The image below visualises the simulated data. The next step is to assess this data following the decision rule.

  # Simulate turbidity data
  
  set.seed(1234)
  turbidity <- data.frame(DateTime = seq.POSIXt(as.POSIXct("2017-01-01 00:00:00"),
                                                by = "min", length.out = 24 * 60),
                          Turbidity = rnorm(n = 24*60, mean = 0.1, sd = 0.01)
                          )

The second section simulates five spikes in the data. The first line picks a random start time for the spike. The second line in the for-loop picks a duration between 10 and 30 minutes. In addition, the third line simulates the value of the spike. The mean value of the spike is determined by the rbinom() function to create either a low or a high spike. The remainder of the spike simulation inserts the new data into the turbidity data frame.

  ## Spike detection

  ## Simulate data
  for (i in 1:5) {
    time <- sample(turbidity$DateTime, 1)
    duration <- sample(10:30, 1)
    value <- rnorm(1, 0.5 * rbinom(1, 1, 0.5) + 0.3, 0.05)
    start <- which(turbidity$DateTime == time)
    turbidity$Turbidity[start:(start+duration - 1)] <- rnorm(duration, value, value/10)
  }

  library(ggplot2)
  ggplot(turbidity, aes(x = DateTime, y = Turbidity)) + 
    geom_line(size = 0.2) +
    geom_hline(yintercept = 0.5, col = "red") + 
    ylim(0,max(turbidity$Turbidity)) +
    theme_bw(base_size = 10) + 
    ggtitle("Simulated SCADA data")
  ggsave("spikes.png", width = 4, height = 3)

The image below visualises the simulated data using the mighty ggplot2 package. Only four spikes are visible because two of them overlap. The next step is to assess this data following the decision rule.

Simulated turbidity spikes.

SCADA Spike Detection

The following code searches for all spikes over 0.50 NTU using the runlength function. This function transforms a vector into a vector of values and lengths. For example, the run length of the vector c(1, 1, 2, 2, 2, 3, 3, 3, 3, 5, 5, 6) is:

lengths: int [1:5] 2 3 4 2 1
values : num [1:5] 1 2 3 5 6

The value 1 has a length of 1. The value 2 has a length of 3, and so on. The spike detection code creates the run length for turbidity levels greater than 0.5, which results in a boolean vector. The cumsum function calculates the starting point of each spike which allows us to calculate their duration.

The code results in a data frame with all spikes higher than 0.50 NTU and longer than 15 minutes. The spike at 11:29 was higher than 0.50 NTU and lasted 24 minutes. The other three spikes are either lower than 0.50 NTU. The first high spike lasted less than 15 minutes.

  # Spike Detection Function
  
  spike.detect <- function(DateTime, Value, Height, Duration) {
    runlength <- rle(Value > Height)
    spikes <- data.frame(Spike = runlength$values,
                         times = cumsum(runlength$lengths))
    spikes$Times <- DateTime[spikes$times]
    spikes$Event <- c(0,spikes$Times[-1] - spikes$Times[-nrow(spikes)])
    spikes <- subset(spikes, Spike == TRUE & Event > Duration)
    return(spikes)
  }
  spike.detect(turbidity$DateTime, turbidity$Turbidity, 0.5, 15)

This approach was used for prototyping a software package to assess water treatment plant data following the Health-Based Targets Manual. The finished product has been written in SQL and is available under an Open Source sharing license.

The book Data Science for Water Utilities discusses anomaly detection in more detail.

Data Science for Water Utilities

Routledge Amazon

Project Euler 9: Special Pythagorean Triplet

Peter Prevos — Thu, 26 Jan 2017 00:00:00 +0000

Project Euler 9 is about the ancient problem or Pythagorean triples. My background is in carpentry and the essential Pythagorean triple 3:4:5 is a standard tool to check whether an angle is straight. When building a structure, simply mark 3 and 4 units of length on the two sides and check whether the diagonal is precisely five units to ensure your angles are straight. There are, however, an infinite number of triples that form right angles.

Project Euler 9 Definition

A Pythagorean triplet is a set of three natural numbers, $a < b < c$, for which, $a^2 + b^2 = c^2$.

For example, $3^2 + 4^2 = 9 + 16 = 25 = 5^2$.

There exists exactly one Pythagorean triplet for which $a + b + c = 1000$.

Find the product $abc$.

Brute Force Solutions

This solution uses brute force and checks all combinations of $a$, $b$ and $c$. To limit the solution space, I used the fact that $a < b < c$, which implies that $a < s/3$, and $a < b < s/2$, where s is the sum of the three sides.

  ## Project Euler 9: Special Pythagorean Triplet
  a <- 0
  b <- 0
  c <- 0
  s <- 1000
  found <- FALSE
  for (a in 1:floor((s / 3))) {
      for (b in (a + 1):floor(s / 2)) {
          c <- s - a - b
          if (a^2 + b^2 == c^2) {
              found <- TRUE
              break
          }
      }
      if (found) break
  }
  answer <- a * b * c
  print(answer)

Alan Jordan provided a faster brute force method. Since $min(a)=1$ and $max(b)=499$, we can walk $a$ and $b$ towards each other. That way there’s a max of 498 possibilities instead of 292*497 in the first solution.

  # Improved Brute Force
  a <- 1
  b <- 499
  repeat{
      c <- sqrt(a^2 + b^2) 
      if (a + b + c > 1000) {b <- b - 1}
      else if (a + b + c < 1000) {a <- a + 1}
      else if (a + b + c == 1000) {break}
  }
  answer <- a * b * c
  print(answer)

Euclid’s Method

Euclid’s formula generates Pythagorean triples for an arbitrary pair of integers m and n with $m > n > 0$. The formula states that the integers $a = m^2 - n^2, b = 2mn, c = m^2 = n^2$ form a Pythagorean triple.

  ## Euclid's Method
  abc_sum <- 1000
  x <- abc_sum / 2
  min <- floor(sqrt(x / 2))
  max <- ceiling(sqrt(x))
  m <- min:max
  m <- m[x %% m == 0]
  n <- ((x / m) - m)
  a <- 2 * m * n
  b <- m^2 - n^2
  c <- m^2 + n^2
  answer <- a * b * c
  print(answer)

Scatter plot of the legs (a, b) of the first Pythagorean triples with a and b less than 6000. Negative values are included to illustrate the parabolic patterns. By Dearjean13 CC BY-SA 4.0.

Trumpworld Analysis: Ownership Relations in his Business Network

Peter Prevos — Mon, 23 Jan 2017 00:00:00 +0000

You do not need a machine learning algorithm to predict that the presidency of Donald Trump will be controversial. Judging by his pointed language in the campaign, we can rest assured that Trump will break many conventions of world politics. One of the most discussed aspects of his reign is the massive potential for conflicts of interest. Trump's complex business empire is entangled with national and international politics. Buzzfeed has mapped many of the relationships between businesses and people in what they have dubbed Trumpworld.

They provided the data to enable citizens data science into the wheeling and dealings of Donald J. Trump. The raw data set consists of three subsets of connections between:

Organisations
People
People and organisations

This article presents a quick analysis of the data using the R language for statistical computing and the iGraph package for analysing networks.

Trumpworld Analysis

This article analyses the connections between organisations using the mighty igraph package.The package implements social network analysis for the R language which can be used for many purposes, such as analysing email traffic or flight paths.

The code snippet below reads the data and converts it to a graph that can be examined using social network analysis techniques. I have saved the raw data available on Google Sheets file as CSV files. This data is filtered to contain only ownership relationships and the names of the organisations.

  ## Analysing Trumpworld
  library(tidyverse)
  library(igraph)
  trumpworld.org <- read_csv("trumpworld/TrumpWorld Data org-org.csv") %>%
      filter(Connection == "Ownership") %>%
      select(OrgA = "Organization A", OrgB = "Organization B")

  org.ownership <- trumpworld.org %>%
      as.matrix %>%
      graph.edgelist()

  par(mar=rep(0,4))
  plot(org.ownership,
  layout = layout.fruchterman.reingold,
      vertex.label = NA,
      vertex.size = 2,
      edge.arrow.size = .1
  )

Trumpworld Analysis: Ownership Relations in his Business Network

Network Analysis

This network contains 309 ownership relationships between 322 distinct organisations. When we plot the data, we see that most connections are dyadic connections between two firms. The plot is organised with the Fruchterman-Reingold algorithm to improve its clarity. We can also see a large cluster in the centre. The names have been removed for clarity. Our excursion into Trumpland continues by zooming in on the subnetwork. The second code section excises this connected subnetwork so we can analyse it in more detail.

Digging Deeper

The node with the highest degree identifies the business with the most holdings. This analysis shows that DJT Holdings LLC owns 33 other organisations. These organisations hold other organisations. We can now use the cluster function to investigate this subnetwork using the decompose function, which splits the network into its unconnected subnetworks. The code then selects and visualises the largest of the subnetworks.

This Trumpworld analysis shows that the ownership network is a star network. DJT Holdings LLC centrally controls all organisations. Perhaps this graph visualises the management style of the soon to be president Trump. Trump centrally controls his empire, which is typical of a family business. Does this chart visualise Trump's leadership style? Is the star network an expression of his lack of trust and thus desire to oversee everything directly?

  which.max(degree(org.ownership))

  org.ownership.d <- decompose(org.ownership)

  largest <- which.max(sapply(org.ownership.d, diameter))

  plot(org.ownership.d[[largest]],
       layout=layout.fruchterman.reingold,
       vertex.label.cex = .5,
       vertex.size = 5,
      edge.arrow.size = .1
  )

Mapping antipodes using the ggmap package

Peter Prevos — Fri, 20 Jan 2017 00:00:00 +0000

When I was much younger, growing-up in the Netherlands I was fascinated by the conundrum of what happens when you drill a hole straight through the centre of the earth. I always believed that I would turn up in Australia. Well, I did end up in Australia, but not by drilling a hole. But is this really the case? This article is about mapping antipodes using the ggmap package of the R language.

The antipodes of any place on earth is the place that is diametrically opposite to it. A pair of antipodes are connected by a straight line running through the centre of the Earth. These points are as far away from each other as is possible on this planet. Two people are antipodes when they live on opposite sides of the globe.

The term antipodes was first coined by the ancient Greeks. Antipodes are people whose feet (πούς/pouch in Ancient Greek) are directly opposite each other. This concept was a direct consequence the Hellenistic understanding of geography, but the existence of such lands or peoples was often debated or outright denied.

How can we use coding in R to solve this conundrum?

Mapping antipodes with ggmap

We can roughly recreate the antipodean map on Wikipedia with the globe package. This package, written by Adrian Baddeley, plots 2D and 3D views of the earth. The package contains a data file with major coastlines that can be used to create a flipped map of the world. The ggplot2 package uses the globe to visualise maps.

The package contains a data file with major coastlines that can be used to create a flipped map of the world. To turn a spatial location into its antipode you subtract 180 degrees from the longitude and reverse the sign of the latitude, shown below.

Antipodean globe.

  library(ggplot2)
  
  ## Antipodean globe
  world <- map_data("world")
  anti_world <- world %>%
      mutate(long = long - 180,
             lat = - lat)           
  ggplot() +
      geom_polygon(data = world,  aes(long, lat, group = group), fill = "grey") +
      geom_polygon(data = anti_world, aes(long, lat, group = group),
                   fill = "blue", alpha = 0.2) + 
      coord_map("ortho", orientation = c(0, 100, 00)) +
      theme_void()

We can also use the ggmap package to visualise antipodes. This package, developed by David Kahle antipodean R-guru Hadley Wickham, has a neat geocoding function to obtain a spatial location. You will need a Google API to enable the geocoding function.

The antipode function takes the description of a location and a zoom level to plot a dot on the antipode location. The gridExtra package is used to create a faceted map, which is not otherwise possible in ggmap.

  library(gridExtra)
  library(ggmap)

  api <- readLines("google.api") # Text file with the API key
  register_google(key = api)
  
  ## Antipode function
  antipode <- function(location, zm = 6) {
      # Map location
      lonlat <- geocode(location)
      loc1 <- get_map(lonlat, zoom = zm)
      map1 <- ggmap(loc1) +
          geom_point(data = lonlat, aes(lon, lat, col = "red", size = 10)) + 
          theme(legend.position = "none")
      # Define antipode
      lonlat$lon <- lonlat$lon-180
      if (lonlat$lon < -180) 
          lonlat$lon <- 360 + lonlat$lon
      lonlat$lat <- -lonlat$lat
      loc2 <- get_map(lonlat, zoom = zm)
      map2 <- ggmap(loc2) +
          geom_point(data = lonlat, aes(lon, lat, col = "red", size = 10)) + 
          theme(legend.position = "none")
      grid.arrange(map1, map2, nrow = 1)
  }

  antipode("Rector Nelissenstraat 47 Hoensbroek", 4)

This code solves the problem I was thinking about as a child. Running the code shows that the antipodes location of the home I grew up in is not in Australia, but quite a way south of New Australia. Although I did end-up living in Australia withouth having to dig a hole in the earth.

Antipode map of my birthplace, Hoensbroek, the Netherlands.

Project Euler 8: Largest product in a series

Peter Prevos — Thu, 19 Jan 2017 00:00:00 +0000

Project Euler 8 combines mathematics with text analysis. Just like problem 11, it asks for the largest product in a grid.

Project Euler 8 Definition

The four adjacent digits in the 1,000-digit number below with the greatest product are 9 × 9 × 8 × 9 = 5832.

73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450

Find the thirteen adjacent digits in the 1,000-digit number that have the greatest product. What is the value of this product?

Euler Problem 8 Proposed Solution

The first step is to define the digits as a character string with the 1000 digits. The answer is found by cycling through all 13-character n-grams to find the highest product using the prod function. The next step strips the 13 numbers into separate digits and multiplies them with each other with the prod() function.

  ## Project Euler 8: Largest product in a series
  digits <- "7316717653133062491922511967442657474235534919493496983520312774506326239578318016984801869478851843858615607891129494954595017379583319528532088055111254069874715852386305071569329096329522744304355766896648950445244523161731856403098711121722383113622298934233803081353362766142828064444866452387493035890729629049156044077239071381051585930796086670172427121883998797908792274921901699720888093776657273330010533678812202354218097512545405947522435258490771167055601360483958644670632441572215539753697817977846174064955149290862569321978468622482839722413756570560574902614079729686524145351004748216637048440319989000889524345065854122758866688116427171479924442928230863465674813919123162824586178664583591245665294765456828489128831426076900422421902267105562632111110937054421750694165896040807198403850962455444362981230987879927244284909188845801561660979191338754992005240636899125607176060588611646710940507754100225698315520005593572972571636269561882670428252483600823257530420752963450"
  ngram <- 13 # Define length
  answer <- 0
  # Clycle through digits
      for (i in 1:(nchar(digits) - ngram + 1)) {
      adjecent <- substr(digits, i, i + ngram - 1) # 13 consecutive digits
      mult <- prod(as.numeric(unlist(strsplit(adjecent, "")))) # Define product
      if (mult > answer) # Largest? 
          answer <- mult
  }
  print(answer)

Project Euler 7: 10,001st Prime Number

Peter Prevos — Thu, 12 Jan 2017 00:00:00 +0000

Project Euler 7 delves into the wonderful world of prime numbers. These numbers are interesting because they don't follow a predictable pattern. There is no algorithm to calculate primes, which is what makes them valuable in cryptography.

As the numbers get larger, the gaps between consecutive primes also increase. There are, however, some interesting patterns. There seem to be infinitely many twin primes, which are prime numbers that with a difference of two, such as (3, 5), (5, 7) and (11, 13). Another pattern in prime numbers are sexy primes, which differ by six: e.g. (5,11), (7, 13) and (11, 17). They are called sexy not because these primes are particularly attractive but because of the Latin word for six.

Project Euler 7 Definition

By listing the first six prime numbers: 2, 3, 5, 7, 11, and 13, we can see that the 6th prime is 13. What is the 1,0001st prime number?

Solution

Finding the n^th prime number can only be solved using brute force because primes do not follow a predictable pattern. We don't have to check whether a number is divisible by any number between 1 and itself. We only have to check whether a number is divisible by all primes between 2 and its square root. The is.prime() function determines whether a number is a prime number by checking that it is not divisible by any prime number up to the square root of the number. The sieve of Eratosthenes used in Euler Problem 3 generates the prime numbers. The next step cycles through all odd numbers and increments a counter when we find a prime, stopping at 10001.

  ## Project Euler 7: 10,0001st Prime Number
  source("problem003.R", echo = FALSE)
  is.prime <- function(n) {
      if (n <= 1) return(FALSE)
      if (n == 2) return(TRUE)
      primes <- esieve(ceiling(sqrt(n)))
      return(prod(n %% primes != 0) == 1)
  }

  answer <- 1
  n <- 1 # Start counter
  while (n < 10001) { # Find 10001 prime numbers
      answer <- answer + 2 # Next number
      if(is.prime(answer)) { 
          n <- n + 1 # Increment counter
      }
  }
  print(answer)

Sexy Primes

The largest gap between two primes for the first 10,001 is 72. Sexy primes with a gap of 6 are the most common, as shown in the figure below.

Frequency or prime gaps for the first 10,001 prime numbers.

  ## Sexy Primes
  library(ggplot2)
  library(dplyr)

  primes <- esieve(answer)
  p <- length(primes)
  gaps <- tibble(Gap = primes[2:p] - primes[1:(p - 1)],
                     Sexy = Gap %% 6 == 0) %>%
      count(Gap, Sexy)

  ggplot(gaps, aes(factor(Gap), n, fill = Sexy)) +
      geom_col() +
      scale_fill_manual(values = c( "#7391AB", "#A62102"), name = "Sexy Prime Gap") +
      theme_minimal(base_size = 10) + 
      labs(title = "Frequency of prime gaps for the first 10,000 primes",
          x = "Prime Gap",
          y = "Frequency")
          guide_legend(title = "Sexy Primes")

  ggsave("images/problem-007.png", width = 6, height = 4)

Sexy Primes - Numberphile.

Project Euler 6: Sum Square Difference

Peter Prevos — Thu, 05 Jan 2017 00:00:00 +0000

Project problem 6 is another trivial one that only requires a basic understanding of coding in R, but it does teach us something about computer science.

Project Euler 6 Definition

The sum of the squares of the first ten natural numbers is:

$$1^2 + 2^2 + \ldots + 10^2 = 385$$

The square of the sum of the first ten natural numbers is:

$$(1 + 2 + \ldots + 10)^2 = 552 = 3025$$

The difference between the sum of the squares of the first ten natural numbers and the square of the sum is $3025 - 385 = 2640$. Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.

Solution

This is a straightforward problem for vector processing capabilities in R.

## Project Euler 6: Sum square difference
answer <- sum(1:100)^2 - sum((1:100)^2)
print(answer)

This problem can also be solved arithmetically. When Carl Friedrich Gauss (1777–1855) when he was a child his teacher challenged his students to add all numbers from 1 to 100. All of his friends struggled to add all the 100 numbers one by one, but Carl completed the task in a few seconds. Gauss found that the sum of natural numbers 1 to $n$ is:

$$\sum_{i=1}^{n} i = \frac{n(n + 1)}{2}$$

The code shown above does what Carl’s classmates would have done, using brute force to count the total. Gauss’ solution only required a straightforward step. Computer scientists use Big O notation to describe the steps it takes to complete an algorithm. In Big-O Notation, the first solution is $O(n)$ which means that the number of steps in the algorithm grows in direct proportion to the data set. In this case, we only look at 100 numbers. But solving this problem for many billions of numbers would pose some practical issues. The brute-force solution is notated as $O(n)$ as it takes as many steps as there are elements. Gauss’ solution is $O(1)$, which means that the solution only requires one step, regardless of the size of the data set.

The sum of the squares of the first $n$ natural numbers is:

$$\sum_{i=1}^{n} i^2 = \frac{n(n+1)(2n+1)}{6}$$

With these two formulas, a fast solution without having to use loops can be written.

n <- 100
answer <- ((n * (n + 1)) / 2)^2 - (n * (n + 1) * (2 * n + 1)) / 6
print(answer)

Project Euler 5: Smallest Multiple

Peter Prevos — Thu, 29 Dec 2016 00:00:00 +0000

Project Euler 5 relates to the divisibility of numbers and is an almost trivial problem that is included here only for completeness.

Project Euler 5

2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder. What is the smallest positive number that is evenly divisible by all of the numbers from 1 to 20?

Solution

The solution will also be divisible by the numbers 1 to 10, so we can start at 2520 and increment by 2520. The loop checks whether the number is divisible by the numbers 1 to 20.

  ## Project Euler 5: Smallest Multiple

  answer <- 2520
  while (sum(answer %% (1:20)) != 0) {
      answer <- answer + 2520 # Increase by smallest number divisible by 1:10
  }
  print(answer)

David Radcliffe provided a more generalised way to derive the answer. To solve this problem, we need the Least Common Denominator, which is the smallest number that two or more numbers will divide into evenly. To find the LCD, we first need to know the Greatest Common Divisor $gcd(x, y)$. The Lowest Common Denominator is the product two numbers, divided by the $gcd$.

The first function calculates the Greatest Common Divisor of tho numbers, $gcd(x, y)$, using the Euclidean Algorithm in a recursive function. The second function derives the $lcm$.

The Reduce() function takes a function, and a list of data items and recursively applies the function to the elements of the list. Reduce performs the function in the first parameter over the first two elements, then over that result and the third element, and so on. A nice way to see what Reduce() does is to use accumulate = TRUE. This option returns its state after processing the elements of the list.

This solution thus finds the Lowest Common Denominator for the numbers 1 to 20.

  ## Generalised method 

  gcd = function (x, y) ifelse(x == 0, y, gcd(y %% x, x))
  lcm = function (x, y) x * y / gcd(x, y)
  Reduce(lcm, 1:20)
  Reduce(lcm, 1:20, accumulate = TRUE)

Project Euler 4: Largest Palindromic Product

Peter Prevos — Thu, 22 Dec 2016 00:00:00 +0000

Project Euler 4 takes us to the palindromic numbers, a number that stays the same when reading it backwards. Palindromic numbers serve no practical purpose other than recreational mathematics.

Project Euler 4 Definition

A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 × 99. Find the largest palindrome made from the product of two 3-digit numbers.

Solution

The formal definition for Palindromic numbers in base-$b$, with $k$ digits $a_i$ is:

$$\sum_{i=0}^ka_ib^i = \sum_{i=k}^0a_ib^i$$

The solution code uses a helper-function to reverse numbers. The reverse() function takes the modulus of 10 to extract the last digit until all digits have been processed. The solution code searches for palindromic numbers, starting at the highest product of three-digit numbers.

  ## Project Euler 4: Largest Palindromic Product

  ## Reverse a number
  reverse <- function(n) {
      reverse <- 0
      while (n > 0) {
          reverse <- 10 * reverse + (n %% 10)
          n <- floor(n / 10)
      }
      return(reverse)
  }

  ## Euler problem 4
  for (i in 999:900) {
      for (j in 990:900) {
          p <- i * j
          if (p == reverse(p)) 
              break
      }
      if (p == reverse(p)) {
          break
      }
  }
  answer <- i * j
  print(answer)

Lychrel Numbers

The Lychrell process can convert almost all non-palindromic numbers to a palindromic number. The process reverses the non-palindromic number and adds the result to the original number. If the result is not a palindromic number, the process repeats until we reach a palindrome. For example, 59 becomes a palindrome after 3 iterations: 59 + 95 = 154, 154 + 451 = 605, 605 + 506 = 1111.

Any number that will never become a palindrome using the Lychrel process is a so-called Lychrel number. Most numbers will convert to a palindrome after a few iterations. However, the number 196 apparently never becomes a palindrome, even after hundreds of thousands of repetitions. There is no proof that Lychrel numbers actually exist. Any code to undertake the Lychrel process could run forever, which is the infamous halting problem in computer science. Sequence A023108 of the Online Encyclopedia of Integers lists numbers that apparently never iterate to become a palindrome.

The lychrel() function tests a number for Lychrel property by following the palindrome process. The output of this function is the resulting palindrome. As these numbers can be huge, we maximise the number of available digits. Unfortunately, there is no way of telling if it will ever stop for certain numbers. We know that the first 195 will terminate, but the number 196

  ## Lychrel Numbers
  lychrel <- function(l, verbose = FALSE) {
    while(l != reverse(l)) {
      l <- l + reverse(l)
      print(l)
    }
  }

  lychrel <- function(l) {
    while(l != reverse(l))
      l <- l + reverse(l)
    return(l)
  }

  ## First 195 non-Lychrel numbers
  options(digits = 22)
  sapply(10:195, lychrel)

  ## This function call apparently never halts - run at your own risk!
  lychrel(196)

What's special about 196? - Numberphile.

Project Euler 3: Largest Prime Factor

Peter Prevos — Fri, 09 Dec 2016 00:00:00 +0000

Project Euler 3 is about prime factors, which are some of the most important numbers of the digital economy. RSA encryption is based on the fact that determining the prime factors of huge numbers takes a very long time. So much time in fact that the cost of breaking the encryption outweigh the benefits of obtaining the secret.

Prime numbers are the basic building blocks of the natural numbers as every number, except the primes themselves. Every natural number can be written as the product of a series of primes. The RSA encryption system uses large primes to keep electronic messages away from prying eyes, as explained in the Numberphile video below.

Prime numbers are a favourite topic in Project Euler. Problem 7 asks for the first ten thousand primes, and Problem 10 looks at the sum of all the primes below two million.

Project Euler 3 Definition

The prime factors of 13195 are 5, 7, 13 and 29. What is the largest prime factor of the number 600851475143?

Generating Prime Numbers

This solution relies on two functions that can be used for multiples problems. The Sieve of Eratosthenes generates prime numbers from 2 to n. The code is commented to explain the sieve. The image shows how numbers from 1 to 100 are sieved to find the primes.

Sieve of Eratosthenes (Source: SKopp at German Wikipedia / CC BY-SA).

This problem uses two functions: prime.factors() and esieve(). The prime.factors() function generates the list of unique prime divisors and then produces the factors. The factors are identified by dividing the number by the candidate prime factors until the result is 1.

Base R Solution

  ## Project Euler 3: Largest Prime Factor

  ## Base R solution
  esieve <- function(n) {
      if (n == 1) return(NULL)
      if (n == 2) return(n)
      ## Create a list of consecutive integers {2,3,... n}.
      l <- 2:n
      ## Start counter
      i <- 1
      ## Select p as the first prime number in the list, p = 2.
      p <- 2
      while (p^2 <= n) {
          ## Remove all multiples of p from the l.
          l <- l[l == p | l %% p!= 0]
          ## set p equal to the next integer in l which has not been removed.
          i <- i + 1 ## Repeat steps 3 and 4 until p^2 > n,
          ## all the remaining numbers in the list are primes
          p <- l[i]
      }
      return(l)
  }

  prime.factors <- function (n) {
      ## Define list of factors
      factors <- c()
      ## Define primes to be tested
      primes <- esieve(floor(sqrt(n)))
      ## Idenitfy prime divisors
      d <- which(n %% primes == 0) 
      ## No prime divisors
      if (length(d) == 0) 
          return(n)
      ## Test candidate primes
      for (q in primes[d]) {
          ## Generate list of factors
          while (n %% q == 0) {
              factors <- c(factors, q)
              n <- n/q } } 
              if (n > 1) 
                  factors <- c(factors, n)
      return(factors)
  }
  
  max(prime.factors(600851475143))

Number package

The solution can also be found by using the primeFactors() function in the numbers package. This package provides a range of functions related to prime numbers and is much faster than the basic R code.

  ## Numbers package
  
  library(numbers)
  max(primeFactors(600851475143))

Project Euler 2: Sum of Even Fibonacci Numbers

Peter Prevos — Thu, 08 Dec 2016 00:00:00 +0000

Project Euler 2 looks at Fibonacci numbers. This number sequence seems to describe our sense of natural beauty and aesthetics. The spiral staircase uses Fibonacci numbers as part of its geometry. Euler Problem 2 is a bit less poetic as it only asks to generate and sum even numbers. Euler Problem 25 also deals with Fibonacci numbers and asks to find the first such number with 1000 digits.

Project Euler 2 Definition

Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:

$1, 2, 3, 5, 8, 13, 21, 34, 55, 89, \ldots$

By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.

Euler Problem 2 Solution in R

The code generates Fibonacci numbers until it reaches the value of four million. The code then sums the even numbers in the sequence.

The intuitive way to calculate Fibonacci numbers is to use a recursive function. This function calls itself until it reaches the first two numbers.

  ## Project Euler 2: Sum of Even Fibonacci Numbers

  ## Recursive function

  fib <- function(n) {
      if (n <= 2) return(1)
      return(fib(n - 1) + fib(n - 2))
  }
  
  fib(15)

This function calculates the $n^{th}$ Fibonacci number by calling itself with a reduced index until it reaches 1. Recursive functions are, however, very memory hungry and not very fast. A much faster method is to create a vector of Fibonacci numbers and add the even ones. The code below generates Fibonacci numbers until it reaches the value of four million. The code then sums the even numbers in the sequence.

  ## Sequential function

  fib <- c(1, 2)

  while (max(fib) < 4E+06) {
      len <- length(fib)
      fib <- c(fib, fib[len - 1] + fib[len])
  }
  
  answer <- sum(fib[fib %% 2 == 0])
  print(answer)

Using the GMP Package

A series of R packages exist to generate Fibonacci numbers. The GMP package for Multiple Precision Arithmetic provides the fibnum() function to calculate the n^th Fibonacci number. This package is also able to work with huge numbers. Using this package is much faster than the base R code used above. Interestingly, the GMP library uses tables for Fibonacci numbers up to twelve quintillion (10¹⁸).

  # Using the GMP Package

  library(gmp)
  i <- 1

  answer <- 0
  fib <- fibnum(i)

  while (fib <= 4E6) {
    if (fib %% 2 == 0) {
      answer <- answer + fib
    }
    i <- i + 1
    fib <- fibnum(i)
  }

  print(answer)

Analytic Solution

An analytical solution prevents using recursion and a brute force method to find the answer. Binet's formula for the n^th Fibonacci number lets us determine a Fibonacci number without recursion:

$$F_n=\frac{\varphi^n-\frac{1}{(-\varphi)^n}}{\sqrt{5}}$$

We know that $\varphi$ is the Golden Ratio, so we can simplify this formula:

$$F_n\sqrt{5}=\frac{(1+\sqrt{5})^n}{2^n}$$

This formula results in a simple function to generate a sequence of Fibonacci numbers:

  ## Analytical solution

  binet <- function(n) {
    round((((1 + sqrt(5))^n) / 2^n) / sqrt(5))
  }

  binet(1:10)

The inverse of the Binet formula gives us:

$n=\left[ \log_\phi \sqrt{5}(F_n-\frac{1}{2}) \right]$ holds for all $n\ge 3$

This formula allows us to work out which Fibonacci number is less than or equal to 4 million:

  phi <- (1 + sqrt(5)) / 2

  n_4e6 <- floor(log(sqrt(5) * (4E6 - 0.5), base = phi))

Using this formula, we find that the first 33 Fibonacci numbers are equal or less than four million. Next step is to sum all the even numbers.

According to the Online Encyclopedia of Integer Sequences (lemma A014445), every third Fibonacci number is even. We can now use Binet's formula to sum every third Fibonacci number.

  sum(binet(seq(3, n_4e6, 3)))

Project Euler 1: Multiples of 3 and 5

Peter Prevos — Thu, 01 Dec 2016 00:00:00 +0000

This first Euler problem provides a gentle introduction to solving mathematical problems with code. This problem involves finding the sum of two related arithmetic sequences.

The natural numbers below 10 that are multiples of 3 or 5 results are 3, 5, 6 and 9. The sum of these numbers is $3+5+6+9=23$. Your assignment is to find the sum of all the multiples of 3 or 5 below 1000.

This problem is similar to one posed to a mathematics child prodigy that lived around the same time as Euler. When Carl Friedrich Gauss (1777–1855) was ten years old, his teacher gave his pupils a maths problem to ponder on. The teacher was perhaps seeking some quiet time for himself by giving the kids a hard problem. He asked them to sum all the numbers from 1 to 100, which surely would give him enough time to enjoy a cuppa.

Young Carl Gauss stood up after just a minute and confidently proudly raised his slate with 5050 written in large chalk letters to the teacher. All the other kids were still diligently adding numbers. How was Carl able to finish his task so quickly? The details and full truth of this story are lost in history. Perhaps Carl realised the solution while writing the series forwards and backwards:

$$1 + 2 + 3 + \ldots + 98 + 99 + 100$$

$$100 + 99 + 98 + \ldots + 3 + 2 + 1$$

Little Carl calculated the sum of some of the pairs and noticed a regularity:

$$1 + 100 = 2 + 99 = 3 + 98 = \ldots 100 + 1 = 99 + 2 = 98 + 3 = 101$$

Each of the pairs adds up to 101 and there are 50 pairs, therefore the sum of all numbers is half the sum of all pairs. Gauss was not the first to discover this method. Some thirteen hundred years earlier, Indian mathematician and astronomer Aryabhata had already discovered this method to sum a arithmetical sequence.

Proposed Solutions

This anecdote shows that there are two ways to solve this first problem. The smart method that Gauss used or the slow brute force method that his fellow pupils used. The brute force solution requires many steps to find the answer. The smart solution uses a one-step formula, such as the young Gauss and Aryabhata had figured out.

The most inefficient solution is to test whether all numbers between 1 and 999 are divisible by 3 or 5 and then add all these number. The R language provides several ways to step through a sequence of numbers.

The code below loops trough the numbers between 1 and 999.

  # Project Euler 1: Multiples of 3 and 5
  
  # Loop

  answer <- 0
  for (i in 1:999) {
    if (i %% 3 == 0 | i %% 5 == 0) {
      answer <- answer + i
      print(i)
    }
  }
  print(answer)

This first solution is elaborate. Using for-loops in R should always be a last resort in R coding because they are the slowest way to compute over a series of steps. The next two solutions solve this problem by using vector arithmetic. These methods are much faster because they run the loop directly in machine code instead of having to compile each step.

The solution for the first problem only requires one line of code when using vector arithmetic.

  # Vector aritmetic
  
  sum((1:999)[((1:999) %% 3 == 0) | ((1:999) %% 5 == 0)])

Analytical solution

Although the vector arithmetic is much faster than the for-loop, it is still not as fast as using the methods discovered by Aryabhata and Gauss. The two brute-force solutions are very fast because of the low number of steps to find a solution. An analytical solution reduces the processing time significantly, as it only requires one step.

The problem involves arithmetic progressions, which are sequences of numbers with a constant difference. The sequence 1, 4, 7, 10, 13, … is an arithmetic progression with a common difference of three.

We can generalise the method discovered by Gauss. The sum of an arithmetic progression ($S_n$), where $n$ is the number of elements and $a_1$ and $a_n$ are the lowest and highest value, is:

$$S_n= \frac{n(a_{1} + a_n)}{2}$$

The numbers divisible by $n=3$ can be expressed as:

$$\mathrm{sum}_3(999)=3+6+9+12+ \ldots + 999 = 3(1+2+3+4+ \ldots + 333)$$

We can now calculate the sum of all divisors by combining the above progression with the formula for arithmetic progressions as expressed in the above code, where $m$ is the divisor and $n$ the extent of the sequence. $p$ is the highest number less than $n$ divisible by $m$. In the case of 5, this number is 995.

$$p = n \lfloor (m/n) \rfloor$$

Substitution gives:

$$\mathrm{sum}_m(n) = p \frac{1+(p/m)}{2}$$

  # Analytical solution
  
  SumDivBy <- function(m, n) {
      p <- floor(n / m) * m # Round to multiple of n
      return (p * (1 + (p / m)) / 2)
  }
  SumDivBy(3, 999) + SumDivBy(5, 999) - SumDivBy(15, 999)