That means, to be considered in our tf-idf computation, a term had to be used at least once by 10% of the artists in our dataset. 2) We also set a “cut-off” for document frequency of 0.1. You can read more about why you might want to do sublinear scaling here. 1) We used sublinear scaling on the term frequencies, giving us a little more variation across our lists. We made two slight modifications to the traditional formula. The words with the ten highest tf-idf scores for each artist were deemed the words “most unique” to him or her. For a given word, we count the number of times it occurs in one rapper’s catalogue (its term frequency) and divide by the number of artists that use it across the hip-hop corpus (its document frequency). Each rapper gets assigned a tf-idf score for every word in the hip-hop corpus. TF-IDF: to determine the words that characterize each hip-hop artist, we used a technique called term frequency-inverse document frequency (tf-idf). other genres, but was only used 116 times in 26 million words. For example “lowrider” had a 255:1 ratio in hip hop vs. These all had fewer than 1,000 occurances in the hip hop corpus.
the general corpus, were still rather rare words. Some words were filtered from this list that, while indexing high in hip hop vs. We then compare that to the same math for the general corpus. For example, this is # of appearences in hip hop corpus divided by total words in hip hop corpus. Most Hip Hop: To find the words most “characteristic” of hip-hop, we computed the odds that a word appeared in the hip hop corpus vs. This included efforts to standardize spelling, remove capitalization, and apply light lemmatization. We filtered hip-hop artists by cross-referencing their primary genre on MusixMatch.įor consistency, The hip hop data was cleaned using the same script as the LyricFind corpus. Hakeem Olajuwon, with the easy to rhyme Hakeem the Dream moniker, has been appearing in verses for more than a quarter of a century, way back to Chubb Rock's 1990 mention up to the contemporary mentions from Jay Z and Nicki Minaj.īrowse through the photos above for all the Houston sports references in rap lyrics.The general music corpus was formed using data from LyricFind. Unfortunately – or probably, fortunately – Houston newspaper reporters don't get nearly as much love in hip hop as Houston athletes. Exxon's ban on LGBTQ Pride, Black Lives Matter flags at Houston Headquarters sparks employee backlashĪnd, salute to Lil Keke for name-checking his favorite newspaper in Paul Wall's Break Em Off: "Chasin' paper in the morning, call me Houston Chronicle.".Mysterious Yankees Letter from Major League Baseball to be released soon.Angels broadcasters get duped by fake Gerrit Cole story during Astros game.
Train through Houston makes the list for most scenic Amtrak route.