2018-02-03
Introduction
In this tutorial, we are going to use the leaflet package to map the most vigorous metal scenes in the world. To do so, we will use the excellent rvest library to scrape band data from the most comprehensive online data source for all things metal: The Metal Archives. We will then clean the text data, geocode the band cities of origin, and create an interactive HTML widget to map “metal activity” across the globe.
Part I: Scrape Metal Archives Band Pages
Note: Always make sure you have permission to scrape a website by checking the robots.txt or contacting the webmaster.
The Metal Archives website has data on over 110,000 metal bands from all over the world. Individual band pages include information on band name, country of origin, location, year of formation, current label and much more. We will begin by writing a simple scraper that targets the HTML tag that has the relevant data.
library(data.table)
library(rvest)
library(pbapply)
library(splitstackshape)
library(tidyverse)
library(ggmap)
library(leaflet)# Use %s placeholder to loop through page index
url <- "https://www.metal-archives.com/bands/Slayer/%s"
# Function tells the scraper to skip empty pages (404 error)
# Otherwise target the specified tags to extract album information
location_scraper <- function(id) {
page <- try(html(sprintf(url, id)), silent = TRUE)
if (inherits(page, "try-error")) {
data.frame(info = character(0))
} else {
data.frame(info = page %>% html_nodes(xpath = '//*[@id="band_info"]') %>% html_text())
}
}
# Set sleep interval greater than 5 to prevent fast scrape and getting booted
Sys.sleep(7)
# Specify number of pages to scrape, only doing 10 for demo purposes
# Depending on the number of broken links, this could take a few hours
location_data <- rbindlist(pblapply(1:10, location_scraper))
head(location_data)## info
## 1: \n\t\t\n\t\t\tAmorphis\n\n\t\t\n \t\t \n \n\t\t Country of origin:\n\t\t Finland\n\t\t Location:\n\t\t Helsinki\n\t\t Status:\n\t\t Active\n\t\t Formed in:\n\t\t 1990 \n\t\t Genre:\n\t\t Progressive/Death/Doom Metal (early), Melodic Heavy Metal/Rock (later)\n\t\t Lyrical themes:\n\t\t War (early), Death, Finnish legends, Kalevala\n\t\t Current label:\n\t\t Nuclear Blast\n\t\t Years active:\n\t\t \n\t\t \t\t\t \t\t\t\t \t\t\n\t\t \t\t\t1990-present\t\t \t\t\t \n\t\t \n\t \n\t \n\t \t\t \tFirst band added to the Metal Archives!\rAmorphis is derived from the Greek 'amorphous' which means without determinate form, shapeless.\rWith Violent Solution's break-up, Jan Rechberger and Esa Holopainen formed Amorphis in 1990 with Jan on drums, Esa on lead guitars, Tomi Koivusaari as vocalist, and Olli-Pekka Laine as bassist. Soon after signing with Relapse Records (which would remain their ... \n\t \t\t \t\tRead more\n\t \t\t\t\n\t
## 2: \n\t\t\n\t\t\tBlind Guardian\n\n\t\t\n \t\t \n \n\t\t Country of origin:\n\t\t Germany\n\t\t Location:\n\t\t Krefeld, North Rhine-Westphalia\n\t\t Status:\n\t\t Active\n\t\t Formed in:\n\t\t 1986 \n\t\t Genre:\n\t\t Speed Metal (early), Power Metal (later)\n\t\t Lyrical themes:\n\t\t Epic tales, Legends/Myths, Fantasy, Tolkien, Literature\n\t\t Current label:\n\t\t Nuclear Blast\n\t\t Years active:\n\t\t \n\t\t \t\t\t \t\t\t\t \t\t\n\t\t \t\t\t1984-1986\t\t \t\t\t\t(as Lucifer's Heritage),\t\t \t\t\n\t\t \t\t\t1986-present\t\t \t\t\t \n\t\t \n\t \n\t \n\t \t\t \tMarcus Siepen had a more active role in Blind Guardian's early days, co-writing a good part of the material and contributing as a lead guitarist. He progressively relinquished these roles to Hansi Kürsch and André Olbrich, however he still shares lead guitar parts live with the latter.Thomen Stauch left in 2005, after 20 years with the band, claiming dissatisfaction with the direction the ... \n\t \t\t \t\tRead more\n\t \t\t\t\n\t
## 3: \n\t\t\n\t\t\tIced Earth\n\n\t\t\n \t\t \n \n\t\t Country of origin:\n\t\t United States\n\t\t Location:\n\t\t Tampa, Florida\n\t\t Status:\n\t\t Active\n\t\t Formed in:\n\t\t 1988 \n\t\t Genre:\n\t\t Power/Thrash Metal\n\t\t Lyrical themes:\n\t\t Horror, Fantasy, Literature, Mythology, History, War\n\t\t Current label:\n\t\t Unsigned/independent\n\t\t Years active:\n\t\t \n\t\t \t\t\t \t\t\t\t \t\t\n\t\t \t\t\t?\t\t \t\t\t\t(as The Rose),\t\t \t\t\n\t\t \t\t\t1985-1988\t\t \t\t\t\t(as Purgatory),\t\t \t\t\n\t\t \t\t\t1988-present\t\t \t\t\t \n\t\t \n\t \n\t \n\t \t\t \tThe original name of the band formed in 1985 was Purgatory but was later changed to Iced Earth before they released their last demo.Gene Adam was originally going to perform on "Night of the Stormrider". Jon Schaffer was concerned that Adam wouldn't be able to handle the vocal melodies for the new material and asked him to take singing lessons. Adam refused and as a result he was replaced by\n ... \n\t \t\t \t\tRead more\n\t \t\t\t\n\t
## 4: \n\t\t\n\t\t\tKatatonia\n\n\t\t\n \t\t \n \n\t\t Country of origin:\n\t\t Sweden\n\t\t Location:\n\t\t Stockholm\n\t\t Status:\n\t\t On hold\n\t\t Formed in:\n\t\t 1991 \n\t\t Genre:\n\t\t Doom/Death Metal (early), Depressive Rock/Metal (later)\n\t\t Lyrical themes:\n\t\t Depression, Bereavement, Death, Suicide\n\t\t Last label:\n\t\t Peaceville Records\n\t\t Years active:\n\t\t \n\t\t \t\t\t \t\t\t\t \t\t\n\t\t \t\t\t1987-1991\t\t \t\t\t\t(as Melancholium),\t\t \t\t\n\t\t \t\t\t1991-1994,\t\t \t\t\n\t\t \t\t\t1996-2017\t\t \t\t\t \n\t\t \n\t \n\t \n\t \t\t \tKatatonia (Catatonia) is a psychiatric condition often associated with schizophrenia. It is characterized by a general absence of motor activity or violent, hyperactive behavior. The word "catatonia" (κατατονία) is also a Greek word for low tone, under tone or commonly down-tempo.\r\nThe band's official autobiography, from the very beginning up to "Night Is the New Day", was available as the tour ... \n\t \t\t \t\tRead more\n\t \t\t\t\n\t
## 5: \n\t\t\n\t\t\tEntombed\n\n\t\t\n \t\t \n \n\t\t Country of origin:\n\t\t Sweden\n\t\t Location:\n\t\t Stockholm\n\t\t Status:\n\t\t Active\n\t\t Formed in:\n\t\t 1989 \n\t\t Genre:\n\t\t Death Metal/Death 'n' Roll\n\t\t Lyrical themes:\n\t\t Horror, Death, Violence, Religion\n\t\t Current label:\n\t\t Threeman Recordings\n\t\t Years active:\n\t\t \n\t\t \t\t\t \t\t\t\t \t\t\n\t\t \t\t\t1987-1989\t\t \t\t\t\t(as Nihilist),\t\t \t\t\n\t\t \t\t\t1989-2014,\t\t \t\t\n\t\t \t\t\t2016-present\t\t \t\t\t \n\t\t \n\t \n\t \n\t \t\t \tDespite the fact that he appears in the album credits, Johnny Dordevic didn't sing on Clandestine or Stranger Aeons. The vocals were actually recorded by Nicke Andersson and Ulf "Uffe" Cederlund), just as he didn't record bass on Carnage's Dark Recollections. He only performed with them live.\rAfter Alex Hellid didn't participate on an Entombed tour in 2013, it was announced that he'd be ... \n\t \t\t \t\tRead more\n\t \t\t\t\n\t
## 6: \n\t\t\n\t\t\tDark Tranquillity\n\n\t\t\n \t\t \n \n\t\t Country of origin:\n\t\t Sweden\n\t\t Location:\n\t\t Gothenburg\n\t\t Status:\n\t\t Active\n\t\t Formed in:\n\t\t 1991 \n\t\t Genre:\n\t\t Melodic Death Metal\n\t\t Lyrical themes:\n\t\t Death, Memories, Despair, Chaos theory\n\t\t Current label:\n\t\t Century Media Records\n\t\t Years active:\n\t\t \n\t\t \t\t\t \t\t\t\t \t\t\n\t\t \t\t\t1989-1991\t\t \t\t\t\t(as Septic Broiler),\t\t \t\t\n\t\t \t\t\t1991-present\t\t \t\t\t \n\t\t \n\t \n\t \n\t \t\t \tRegarded as one of the pioneers of the Gothenburg melodic death metal style.\rThey have kept a mostly consistent line-up. Rhythm guitarist Mikael Stanne switched to vocals after Anders Fridén's departure. When his replacement Fredrik Johansson left in 1998, bassist Martin Henriksson took up the guitar duties and Michael Nicklasson came in on bass. They also added an "electronics" man Martin\n ... \n\t \t\t \t\tRead more\n\t \t\t\t\n\tPart II: Clean The Data
This is good but very messy! We need to get rid of all the whitespace and labels such as “Country of origin”.
# Delete excess whitespace
location_data <- lapply(location_data, function (x) gsub("\\s+"," ", x))
# Delete recurring variable names from columns
delete_fun <- function(x) {
gsub(" Country of origin| Location| Status| Formed in| Genre|
| Lyrical themes| Current label| Years active", "", x)
}
location_data <- sapply(location_data, delete_fun)
head(location_data)## info
## [1,] " Amorphis: Finland: Helsinki: Active: 1990: Progressive/Death/Doom Metal (early), Melodic Heavy Metal/Rock (later): War (early), Death, Finnish legends, Kalevala: Nuclear Blast: 1990-present First band added to the Metal Archives! Amorphis is derived from the Greek 'amorphous' which means without determinate form, shapeless. With Violent Solution's break-up, Jan Rechberger and Esa Holopainen formed Amorphis in 1990 with Jan on drums, Esa on lead guitars, Tomi Koivusaari as vocalist, and Olli-Pekka Laine as bassist. Soon after signing with Relapse Records (which would remain their ... Read more "
## [2,] " Blind Guardian: Germany: Krefeld, North Rhine-Westphalia: Active: 1986: Speed Metal (early), Power Metal (later): Epic tales, Legends/Myths, Fantasy, Tolkien, Literature: Nuclear Blast: 1984-1986 (as Lucifer's Heritage), 1986-present Marcus Siepen had a more active role in Blind Guardian's early days, co-writing a good part of the material and contributing as a lead guitarist. He progressively relinquished these roles to Hansi Kürsch and André Olbrich, however he still shares lead guitar parts live with the latter.Thomen Stauch left in 2005, after 20 years with the band, claiming dissatisfaction with the direction the ... Read more "
## [3,] " Iced Earth: United States: Tampa, Florida: Active: 1988: Power/Thrash Metal: Horror, Fantasy, Literature, Mythology, History, War: Unsigned/independent: ? (as The Rose), 1985-1988 (as Purgatory), 1988-present The original name of the band formed in 1985 was Purgatory but was later changed to Iced Earth before they released their last demo.Gene Adam was originally going to perform on \"Night of the Stormrider\". Jon Schaffer was concerned that Adam wouldn't be able to handle the vocal melodies for the new material and asked him to take singing lessons. Adam refused and as a result he was replaced by ... Read more "
## [4,] " Katatonia: Sweden: Stockholm: On hold: 1991: Doom/Death Metal (early), Depressive Rock/Metal (later): Depression, Bereavement, Death, Suicide Last label: Peaceville Records: 1987-1991 (as Melancholium), 1991-1994, 1996-2017 Katatonia (Catatonia) is a psychiatric condition often associated with schizophrenia. It is characterized by a general absence of motor activity or violent, hyperactive behavior. The word \"catatonia\" (κατατονία) is also a Greek word for low tone, under tone or commonly down-tempo. The band's official autobiography, from the very beginning up to \"Night Is the New Day\", was available as the tour ... Read more "
## [5,] " Entombed: Sweden: Stockholm: Active: 1989: Death Metal/Death 'n' Roll: Horror, Death, Violence, Religion: Threeman Recordings: 1987-1989 (as Nihilist), 1989-2014, 2016-present Despite the fact that he appears in the album credits, Johnny Dordevic didn't sing on Clandestine or Stranger Aeons. The vocals were actually recorded by Nicke Andersson and Ulf \"Uffe\" Cederlund), just as he didn't record bass on Carnage's Dark Recollections. He only performed with them live. After Alex Hellid didn't participate on an Entombed tour in 2013, it was announced that he'd be ... Read more "
## [6,] " Dark Tranquillity: Sweden: Gothenburg: Active: 1991: Melodic Death Metal: Death, Memories, Despair, Chaos theory: Century Media Records: 1989-1991 (as Septic Broiler), 1991-present Regarded as one of the pioneers of the Gothenburg melodic death metal style. They have kept a mostly consistent line-up. Rhythm guitarist Mikael Stanne switched to vocals after Anders Fridén's departure. When his replacement Fredrik Johansson left in 1998, bassist Martin Henriksson took up the guitar duties and Michael Nicklasson came in on bass. They also added an \"electronics\" man Martin ... Read more "Ah, much better but there is a little more cleaning left to do. Let’s split the text into multiple columns (i.e. create variables) using the : delimiter. We will also drop the unwanted variables.
# Split into columns and clean
location_data <- cSplit(location_data, "info", sep = ":", type.convert = FALSE)
# Change column names and turn into data frame
colnames(location_data) <- c("band", "country", "city", "status", "formed", "genre",
"themes", "label", "history", "misc")
# Drop unwanted variables
location_data <- select(location_data, -c(themes, label, history, misc))
head(location_data)## band country city status
## 1: Amorphis Finland Helsinki Active
## 2: Blind Guardian Germany Krefeld, North Rhine-Westphalia Active
## 3: Iced Earth United States Tampa, Florida Active
## 4: Katatonia Sweden Stockholm On hold
## 5: Entombed Sweden Stockholm Active
## 6: Dark Tranquillity Sweden Gothenburg Active
## formed
## 1: 1990
## 2: 1986
## 3: 1988
## 4: 1991
## 5: 1989
## 6: 1991
## genre
## 1: Progressive/Death/Doom Metal (early), Melodic Heavy Metal/Rock (later)
## 2: Speed Metal (early), Power Metal (later)
## 3: Power/Thrash Metal
## 4: Doom/Death Metal (early), Depressive Rock/Metal (later)
## 5: Death Metal/Death 'n' Roll
## 6: Melodic Death MetalPart III: Geocoding
And that does it! We will now move onto the next stage, which includes geocoding the city names. First we will need to get city frequencies so we can create density bubbles in our leaflet widget/map.
# Get city frequencies
freqs <- count(location_data, city)This is where it gets tricky! Google only allows you to geocode 2,500 rows at a time so you can either (1) run this code multiple times using a loop/function or purchase the basic plan that allows up to 100,000 rows per day.
# Geocode band locations
# Can only do up to 2500 rows at a time with free account
# We have over 19,000 unique cities in the frequency dataset
# Either geocode 2500 at a time or up to 100,000 at a time with basic plan
# Leave client and signature values as is
for (i in 1:nrow(location_data)) {
latlon <- geocode(location_data$city[1:19242], client = "123", signature = "123", output = "latlon", source = "dsk")
freqs$lat <- as.numeric(unlist(latlon[1]))
freqs$lon <- as.numeric(unlist(latlon[2]))
}Part IV: Creating Interactive Leaflet Widget
This is the fun part! We will use the leaflet package to create an awesome interactive map of metal activity and identify the most “happening” metal scenes across the globe. The leaflet package has a number of beautiful styles (i.e. see addProviderTiles argument). We will use the black and white theme for this map because metal is dark and awesome!
# Use frequencies data
map <- leaflet(data = freqs) %>% addTiles() %>% addProviderTiles("CartoDB.Positron") %>%
addCircleMarkers(~lon, ~lat,
# can play with the radius value for resizing
radius = ~n/100,
# specifying colors based on city frequency
# don't show bubbles for cities with fewer than 10 bands
color = ~ifelse(freqs$n >= 500 , "red",
ifelse(freqs$n >= 50 & freqs$n < 500 , "green",
ifelse(freqs$n >= 10 & freqs$n < 50, "black",
"none"))),
stroke = TRUE,
fillOpacity = 0.3,
popup = ~as.character(city)
)
map
Conclusion
Interesting! Athens, Moscow, Stockholm, Paris, Los Angeles, and Mexico City all have pretty vibrant metal scenes indicated by the number of bands that were founded. In the US, LA, Chicago, and of course, Tampa (home of “death metal”) are the leading cities. Go ahead and zoom in and out to explore all the scenes on the map. Let me know what you think about the this tutorial, R, or metal music in general in the comments section below!