--- title: "scholar: Analyse Citation Data from Google Scholar" author: "Guangchuang Yu, James Keirstead and Gregory Jefferis" date: "`r Sys.Date()`" format: html: toc: true theme: cosmo code-fold: false execute: message: false warning: false freeze: auto vignette: > %\VignetteIndexEntry{scholar introduction} %\VignetteEngine{quarto::html} %\usepackage[utf8]{inputenc} --- ```{r} #| label: setup #| echo: false #| results: hide knitr::opts_chunk$set( tidy = FALSE, message = FALSE, warning = FALSE, fig.width = 7, fig.height = 4 ) has_scholar <- yulab.utils::has_internet("https://scholar.google.com") run_examples <- has_scholar && identical(tolower(Sys.getenv("SCHOLAR_RUN_VIGNETTE_EXAMPLES")), "true") ``` ```{r} #| label: libraries #| echo: false #| results: hide library("scholar") library("ggplot2") theme_set(theme_minimal()) ``` # Overview The **scholar** package extracts citation and publication data from Google Scholar profiles. It can be used to: * find Google Scholar profile IDs; * retrieve profile metadata, publication records, and citation histories; * calculate citation metrics such as h-index, g-index, i10-index, i50-index, and i100-index; * inspect article-level metadata and article citation histories; * complete truncated author lists; * compare multiple scholars; * explore coauthor networks; * format publication lists for CVs; * match publication venues against SCImago journal rankings; and * predict future h-index values. Google Scholar does not provide an official public API. Results can change when Google changes page markup, and repeated requests may be rate limited. The examples below use small requests and are skipped automatically when Google Scholar is not reachable. # Scholar IDs Most functions need a Google Scholar profile ID. In a profile URL such as `https://scholar.google.com/citations?user=B7vSqZsAAAAJ`, the ID is `B7vSqZsAAAAJ`. ```{r ids-basic} id <- "B7vSqZsAAAAJ" ``` If you have a URL or an ID with trailing query parameters, `tidy_id()` keeps only the Scholar ID. ```{r tidy-id} tidy_id("https://scholar.google.com/citations?user=B7vSqZsAAAAJ&hl=en") tidy_id("B7vSqZsAAAAJ&hl=en") ``` Author search is the endpoint Google has put behind a login, so `get_scholar_id()` and `search_scholar_ids()` can come back empty on a machine where a browser shows the profile plainly; the package warns when the result is a refusal rather than an absence of matches. If you have the profile URL, take the ID from it. If you are matching authors programmatically, OpenAlex is the sanctioned source. ```{r get-scholar-id, eval=run_examples} get_scholar_id(first_name = "Richard", last_name = "Feynman") ``` For ambiguous names, `search_scholar_ids()` returns all author-search matches it can find, including IDs, affiliations, verified email text, research interests, and profile URLs. `max_pages` limits pagination so that examples and scripts do not make excessive requests. ```{r search-scholar-ids, eval=run_examples} search_scholar_ids("hao xu", max_pages = 1) ``` # Profile Data `get_profile()` returns a list with name, affiliation, citation metrics, fields, homepage, coauthors, and availability counts. ```{r profile, eval=run_examples} profile <- get_profile(id) profile$name profile[c("total_cites", "h_index", "i10_index")] profile$fields ``` # Publications `get_publications()` returns a data frame of publications from a Scholar profile. It includes the publication title, displayed author list, venue text, volume/page details, citation count, year, the Google Scholar citation ID (`cid`), and the publication ID (`pubid`). ```{r publications, eval=run_examples} pubs <- get_publications(id, pagesize = 20) head(pubs, 3) ``` You can sort the profile page by citation count or by publication year. ```{r publications-sort, eval=run_examples} head(get_publications(id, pagesize = 10, sortby = "year"), 3) ``` Google Scholar sometimes truncates long author lists with `...`. Use `get_publications_all_authors()` when you need the complete author list for those rows. It only calls `get_complete_authors()` for rows that appear to be truncated. ```{r all-authors, eval=FALSE} pubs_full <- get_publications_all_authors(id, pagesize = 20, delay = 1) head(pubs_full, 3) ``` For a single publication ID, call `get_complete_authors()` directly. ```{r complete-authors, eval=run_examples} get_complete_authors(id, pubs$pubid[1], delay = 0) ``` `author_position()` helps analyse where a named author appears in publication author lists. `Position_Normalized` is 0 for first author, 1 for last author, and values between 0 and 1 for middle positions. ```{r author-position, eval=run_examples} author_position(pubs$author[1:5], profile$name) ``` # Citation Histories `get_citation_history()` retrieves the yearly citation bar chart for a scholar. ```{r citation-history, eval=run_examples} ct <- get_citation_history(id) ct ``` ```{r citation-history-plot, eval=run_examples} ggplot(ct, aes(year, cites)) + geom_line() + geom_point() + labs(x = "Year", y = "Citations") ``` `get_article_cite_history()` retrieves the yearly citation history for one publication. Missing years are filled with zero citations. ```{r article-citation-history, eval=run_examples} article_history <- get_article_cite_history(id, pubs$pubid[1]) article_history ``` ```{r article-citation-history-plot, eval=run_examples} ggplot(article_history, aes(year, cites)) + geom_segment(aes(xend = year, yend = 0), linewidth = 1, colour = "grey70") + geom_point(size = 3, colour = "firebrick") + labs(x = "Year", y = "Citations") ``` `get_citing_articles()` goes one step further than a citation count and lists the works citing a publication, with their venue and year, which is what you need to split a citation curve by community or to measure an autocitation rate. Google caps the list the same way the "cited by" link does, and guards it more closely than a profile page, so expect it to be refused from a server. ```{r citing-articles, eval=FALSE} pubs <- get_publications(id) top <- pubs[which.max(pubs$cites), ] get_citing_articles(id, top$pubid, publications = pubs, max_pages = 2) ``` # How the h-index Grew `get_profile()` reports one h-index at one moment. `get_h_index_history()` rebuilds it year by year from the citation history of each publication, which is the difference between knowing someone is at h-index 39 and knowing whether they got there in five years or thirty. ```{r h-index-history, eval=FALSE} history <- get_h_index_history(id, delay = 1) head(history) ``` ```{r h-index-history-plot, eval=FALSE} ggplot(history, aes(year, h_index)) + geom_step(direction = "hv") + labs(x = NULL, y = "h-index", title = "How the h-index accumulated", subtitle = "each step is a year in which another paper reached the threshold") ``` The cost is one request per cited publication, so the histories are cached with `R.cache` and a second call for the same profile makes no requests. Pass `flush = TRUE` to refresh. Publications with no citations are skipped, since they cannot enter an h-index in any year, and the curve is only as long as the years Google reports per publication. # Citation Metrics `get_publication_metrics()` calculates metrics from a publication data frame. `get_scholar_metrics()` fetches publications and calculates the same summary in one call. ```{r publication-metrics, eval=run_examples} get_publication_metrics(pubs) get_scholar_metrics(id, pagesize = 20) ``` The package also includes focused helpers used by the h-index prediction model. ```{r publication-helpers, eval=run_examples} get_num_articles(id) get_oldest_article(id) get_num_distinct_journals(id) get_num_top_journals(id) ``` # Publication Details Google Scholar publication detail pages can expose additional fields. These helpers accept a Scholar ID and a publication ID. ```{r publication-detail, eval=run_examples} pubid <- pubs$pubid[1] get_article_scholar_url(id, pubid) get_publication_url(id, pubid) get_publication_date(id, pubid) get_publication_abstract(id, pubid) get_publication_data_extended(id, pubid) ``` Some publications have no linked PDF, abstract, or publication date on Google Scholar. In those cases the corresponding helper may return an empty value or `NA`. # Comparing Scholars `compare_scholars()` compares citation totals from publication records by publication year. ```{r compare-scholars, eval=run_examples} ids <- c("B7vSqZsAAAAJ", "DO5oG40AAAAJ") cs <- compare_scholars(ids) head(cs, 3) ``` ```{r compare-scholars-plot, eval=run_examples} cs <- dplyr::filter(cs, !is.na(year) & year > 1900) ggplot(cs, aes(year, cites, colour = name, group = name)) + geom_line() + labs(x = "Publication year", y = "Citations", colour = NULL) + theme(legend.position = "bottom") ``` `compare_scholar_careers()` compares citation histories after aligning each scholar by career year. ```{r compare-careers, eval=run_examples} csc <- compare_scholar_careers(ids) head(csc, 3) ``` ```{r compare-careers-plot, eval=run_examples} ggplot(csc, aes(career_year, cites, colour = name, group = name)) + geom_line() + geom_point() + labs(x = "Career year", y = "Citations", colour = NULL) + theme(legend.position = "bottom") ``` # Coauthor Networks `get_coauthors()` retrieves the coauthors listed by Google Scholar for a profile and can optionally expand the network by following coauthors' profiles. Keep `n_coauthors` and `n_deep` small for exploratory work; network plots get messy quickly and repeated requests can be rate limited. ```{r coauthors, eval=run_examples} coauthor_network <- get_coauthors("DO5oG40AAAAJ", n_coauthors = 4, n_deep = 0) coauthor_network ``` ```{r coauthors-plot, eval=run_examples} if (nrow(coauthor_network) > 1) { plot_coauthors(coauthor_network) } ``` # Journal Rankings `get_journalrank()` is deprecated and fetches nothing. SCImago has served its ranking page behind a JavaScript challenge since late 2025, so a script is answered with HTTP 403 and cannot read the table, and the table carries no published licence. The function remains so that old code fails with an explanation rather than a stack trace: ```{r journalrank-deprecated, error=TRUE} get_journalrank(c("Nature", "Science")) ``` For journal metrics, use OpenAlex, which is openly licensed, built for programmatic use, and needs no session or scraping. It publishes the h-index, i10-index, work and citation counts, and the ISSNs of a journal: ```{r openalex-sources, eval=FALSE} openalex <- jsonlite::fromJSON( "https://api.openalex.org/sources?search=Nature&per-page=1") openalex$results[, c("display_name", "works_count", "cited_by_count")] openalex$results$summary_stats ``` If you need SCImago's SJR quartiles specifically, save the table by hand in a browser from `https://www.scimagojr.com/journalrank.php` and read it with `utils::read.csv2()` — the separators are semicolons. # Predicting Future h-index `predict_h_index()` implements the model from Acuna et al. for predicting future h-index values. The original model was calibrated on neuroscientists, so results for other fields should be treated as illustrative rather than authoritative. ```{r predict-h-index, eval=run_examples} predict_h_index("GAi23ssAAAAJ") ``` You can provide a custom list of top journals for the model's top-journal predictor. ```{r predict-h-index-custom, eval=FALSE} predict_h_index( id, journals = c("Nature", "Science", "Proceedings of the National Academy of Sciences") ) ``` # Formatting Publications for CVs `format_publications()` returns publication strings that can be printed in R Markdown, Quarto, or a CV workflow. The `author.name` argument highlights a chosen author in bold after converting the supplied name to the abbreviated format used by the output. ```{r format-publications-asis, results="asis", eval=run_examples} format_publications("DO5oG40AAAAJ", "Guangchuang Yu") |> head() |> cat(sep = "\n\n") ``` For numbered output, print the returned vector directly. ```{r format-publications-print, eval=run_examples} format_publications("DO5oG40AAAAJ", "Guangchuang Yu") |> head() |> print(quote = FALSE) ``` # Mirrors, Retries, and Caching Use `set_scholar_mirror()` if you need to point the package at a compatible Scholar mirror. ```{r mirror, eval=FALSE} set_scholar_mirror("https://scholar.google.com") ``` All network requests go through `get_scholar_resp()`, which stores Google Scholar session cookies and retries transient failures. `get_publications()` also caches profile publication data with `R.cache` in a temporary cache directory. Use `flush = TRUE` in `get_publications()` or functions that forward to it when you need fresh data. ```{r flush-cache, eval=FALSE} get_publications(id, flush = TRUE) get_scholar_metrics(id, flush = TRUE) ``` ## When Scholar refuses: servers, CI, and hosted apps Google guards Scholar against automated clients, and how it responds depends on where the request comes from. On your machine a profile page opens; on shinyapps.io, a GitLab runner or a GitHub Actions job the same request can come back as a sign-in redirect, a captcha, or HTTP 403. The package no longer describes that as a bad ID: `get_scholar_resp()` inspects the address the request finally landed on and the body it carries, then warns with the reason. ```{r refused, eval=FALSE} get_profile("B7vSqZsAAAAJ") #> Warning: Google Scholar returned a captcha page that needs JavaScript, so the #> response contains no data. ... Carry the session of a browser that opens the #> page with set_scholar_cookie(), ... or run it from a workstation. ``` Four ways out, in the order they are worth trying: 1. **Fetch outside the app.** Pull the data where a person browses, save it, and have the deployed app read the saved table. Scholar data changes slowly and an app rarely needs a live scrape per visit. 2. **Pace the requests.** `options(scholar_request_delay = 1)` (or `SCHOLAR_REQUEST_DELAY=1` in CI) puts a pause in front of every request, and rate limits already back off geometrically within `scholar_retry_max_delay`. Loops over many publications are what get an IP flagged. 3. **Present a session.** `set_scholar_cookie(file = "cookies.txt")`, or `SCHOLAR_COOKIE_FILE` on a server, reuses the session of a browser that opens the page. This is what author search now requires. A session is a credential: keep it out of the command history and out of version control, and note that the package sends it only to Google hosts, never to a mirror. 4. **Use another source.** For anything at scale, OpenAlex (`https://api.openalex.org`) is sanctioned, needs no session, and returns h-index and citation counts alongside the works. The settings the package reads, each with an environment variable of the same name in upper case: `scholar_site`, `scholar_user_agent`, `scholar_accept_language`, `scholar_request_delay`, `scholar_retry_delay`, `scholar_retry_max_delay`, `scholar_timeout`, `scholar_cookie`, and `scholar_cookie_file`. For large workflows, prefer fetching each scholar's publications once and then reusing the returned data frame for multiple summaries. ```{r reusable-publications, eval=FALSE} pubs <- get_publications(id) recent <- subset(pubs, !is.na(year) & year >= 2020) nrow(recent) sum(recent$cites, na.rm = TRUE) get_publication_metrics(recent) ```