Meaningful Measures? Bibliometrics and Assessing Latin American Publications (2014)
Tuesday, May 21, 2013, 8:30am-10:00am
Moderator: Melissa Gasparotto, Rutgers University
Rapporteur: Emma Marschall, Tulane University
- Not Ready for Prime Time: Measuring Publications/Citation Impact for Latin American Titles — Amelia Craig, United Nations Economic Commission for Latin America and the Caribbean, Subregional Headquarters, Mexico and Mirian Ramírez, United Nations Economic Commission for Latin America and the Caribbean, Biblioteca Hernán Santa Cruz
- Seeking Stability Online: Analyzing the Online Availability of a Latin American Serials Collection — Lisa Gardinier, University of Iowa
- Evaluating the Content of the Hispanic American Periodicals Index (HAPI): A Bibliometric Analysis of Latin American Serials — Bruce Bachand, University of Kentucky and Orchid Mazurkiewicz, University of California, Los Angeles
- Documenting Pan-American Scholarly Communications: A Citation Study of Less Commonly Taught and Indigenous Languages — Marina Todeschini Crumbacher, University of New Mexico and Suzanne Schadl, University of New Mexico
Melissa Gasparotto introduces the first presenters and thanks them for their willingness to be the trial for virtual presentations at SALALM. Amelia Craig (United Nations Economic Commission for Latin America and the Caribbean, Subregional Headquarters, Mexico) and Mirian Ramírez (United Nations Economic Commission for Latin America and the Caribbean, Biblioteca Hernán Santa Cruz) -- Not Ready for Prime Time: Measuring Publications/Citation Impact for Latin American Titles
Mirian introduces herself and Amelia over the Skype chat and begins the presentation using Skype chat and Prezi PowerPoint. They will be discussing the findings of their study titled “Study about the Impact of ECLAC Publications in Academics”. The purpose of their study was to gather information about coverage and impact of ECLAC publications. The study was conducted between 2003-2012 by searching 5 selected databases – Scopus, Scimago, Google Scholar, SciELO, and Publish or Perish. Findings show low coverage; almost none in the open access platform, SciELO. The study also finds that while there was poor ranking in the world, there was good ranking in Latin America. One of the limitations of this study was that it only measured the impact on academic, not on public policy or the political world, where CEPAL has an arguably greater audience. These findings lead to the following suggestions: to start a new internal discussion; come up with strategies to increase visibility; develop a methodology for continued impact assessment. A review team was formed to work on these issues. Mirian’s chat function is dropped due to technical difficulties. Amelia continues for her. CEPAL is trying to increase visibility in selected databases including: Econolit, ISIThomson, HAPI, Scopus, Dialnet, DOAJ, IBSS, Pais International, EBSCO Open Access Journals, CLASE, and Latindex. Mirian comes back online and begins again, stating that CEPAL has a long tradition of freely available information, but they are working to make this information more accessible and stable, by: reviewing international standards, including international and regional indexes, redesigning the journal website. They will also be taking steps to: evaluate alternative indicators for impact factors, implement a new institutional repository, working on new marketing strategies that utilize social media and apps, and the library is planning new services and user training on the impact of researchers’ publication.
Amelia takes over, saying that Mirian has presented a review of the pilot program, and she will continue by talking about alternative ways of thinking about impact factors beyond the traditional citation method. She reads a quote by Jason Priem about problems with traditional impact factors. The academic world has different ways of sharing data and sharing research, professor to professor, through gray literature that the sharing of information doesn’t wait for publication; it happens through networks of people; so it’s worth considering the impact of research before it hits the traditional publication model. Problems with traditional research include: it is slow and conventional, it is retrospective, the quantity of citations is valued over quality, there is no way to distinguish between positive and negative citations, the lack of context from on discipline to another, and there and language and regional biases and does not include publications in all regions. Amelia talks about tools that use traditional citation metrics: Google Scholar Citations, SCIMago, Publish or Perish, SciELO. She reiterates that there are scholarly citations in formats outside the traditional measures used by these tools: through blogs, shared software, repurposed data, public peer-review, pre-prints, record management tools, and twitter. Therefore, it is worth looking at alternative metrics, promoted by people like Jason Priem, readership and diffusion and reuse through the web; Impact Factor is an example of a tool created through a Sloane foundation grant, that does this; Altmetric looks at social media; Mendeley looks at tags, etc.; Plum Analytics looks at likes, tweets, etc. Amelia offers up there contact information and invites further discussion.
Melissa Gasparotto introduces the presenter. Lisa Gardinier (University of Iowa) -- Seeking Stability Online: Analyzing the Online Availability of a Latin American Serials Collection
Lisa thanks Mirian and Amelia for doing this pilot program with the Skype presentation and for her opportunity to work with the Biblioteca Hernán Santa Cruz, where she worked on the project that she is presenting on. The Biblioteca Hernán Santa Cruz received 316 print-only periodical subscriptions. The study was to find out how many of these titles were available online, either in proprietary databases or freely accessible. The question was, if these resources were freely available online were they stable enough to stop receiving print copies of these titles? The CEPAL collection is mainly focused on the social sciences, economics, business, government documents, and includes formats such as academic journals, trade magazines, and government documents, received by subscription, exchange, and donation. The procedure was to check each title in the catalog, using Ulrich’s, although there were problems with title search, then searched by publisher. In the initial 2011 results, of the titles, 182 were from Latin America, 96 with some online availability, and 46 evaluated to be reasonably stable online. At the time the study these titles where not broken down by region, but this was done for this presentation. Recommendations from this study include: need to update the catalog records for titles that were not being received; while 172 were available online, of those only 96 could be considered stable. This left 220 titles that could only be received in print. These titles were ranked, with the limitation of Lisa not being a digital preservation expert, so her professional judgment as a librarian was the key factor in ranking the stability of the online availability in the following categories: Yes/Probably/Maybe/No/Not satisfactory/By subscription. Latin American titles represent 46 of the 96 stable titles.
Currently in 2013, Lisa decides to revisit the findings that are relevant to Latin America for this presentation at SALALM. She reviewed the academic journals and government documents from Latin America. In 2011, she had seen 40 academic journal titles and judged 26 of them to be stable; in 2013, 20 were still stable; 7 of these put their publications on Open Journal Source. She also checked holdings on Redalyc and SciELO. In terms of government publications, in 2011 Lisa was struck by the apparent transparency and availability of the Chilean government publications, citing an example of statistical information published by the Ministerio de Educación from 1986- in many formats; it was stable, had experienced titles changes and had two urls. 2011 coincided with the rise of the student movement. In 2013, there have been 4 education ministers since 2011. The urls for the Ministerio de Educación are gone; the information only goes back to 2001; not captured by Internet Archive. Of the original 2011 study, 42 of the 316 titles were government documents; 26 of those were available on online; 12 were stable at the time. In 2013, only two have declined; of the instable government documents publications sites from 2011, 8 had improved. In conclusion Lisa there is stability and some stagnation, Latin American journals have support through open access, but the governments may not be ready to rely on online publication.
Melissa Gasparotto introduces the presenters. Bruce Bachand (University of Kentucky) and Orchid Mazurkiewicz (University of California, Los Angeles) -- Evaluating the Content of the Hispanic American Periodicals Index (HAPI): A Bibliometric Analysis of Latin American Serials
Orchid notes this study is the result of an internship program with HAPI that Bruce did. This internship was set up to analyze the steady stream of new titles that are up for inclusion in HAPI, to find with titles represent the best candidates for inclusion. Most years, there are a number of titles that are indexed by HAPI that are ceased, so there is an opportunity to add new titles. Every year there are few days that HAPI staff considers the new titles and deliberates which to add based on fit with the existing titles, and strengths and weaknesses of coverage of subjects. The big problem is, for example, that if HAPI indexes 10 Brazilian economics journals, and in the new titles there are 2 new Brazilian economics journals, the decision may be made to exclude both, to favor a subject area that is not as well covered as Brazilian economics, even though they are both excellent. Currently, the process does not include a review of all 12 titles, the 10 already in HAPI, and the 2 new titles, as a whole, to evaluate the quality and choose the 10 best. In the long term, the potential consequence is that the subject coverage may not represent the best scholarship in that area. The internship was created to evaluate the titles indexed by HAPI for quality, excellence, and value. Orchid turns the presentation over to Bruce.
Bruce cites Jean-Claude Guédon’s definition of quality and excellence: Quality--Peer review, editorial board, style guide; Excellence--Impact factor, use. Bruce recognizes problems of cultural bias and industry practices that may affect these measures; many Lain American journals of high quality have low impact factors. Because of this, a third factor, that of value, defined as Journal’s influence relative to other titles in its field, was incorporated into the evaluation. The methodology for evaluation for HAPI’s 367 titles involved the use of the Latindex score (for quality) Impact Factor, Redalyc downloads, Global Use Measure, SciELO Visits (for excellence) and a survey of SALALM members (for value), although there were too few responses to incorporate this into study. The Latindex score found for 198 of the HAPI titles; Chile has the highest score, Mexico has the most titles rated, most in the near perfect, while at the same time having many in the lower ranks, more than most countries. This shows that Latindex is providing straightforward evaluation of quality of its own country’s production. Impact factor was available for 93 titles from the HAPI index; many came from SciELO and SCIMago. In these, the US publications had the highest impact factor (21 titles), followed by Brazil and Mexico. Providing the median, the impact factor for Latin American journals tended to be low. Also, the fact that only 25% of the titles in HAPI had information about the impact factor indicates that much citation work remains for Latin American bibliometric compilers. Readings are so few that we hesitate to draw conclusions. For one, it is quite likely that IFs range differently in each subject area, recalling Garfield’s statement that "the size of the scientific community that a journal serves significantly affects impact factor." To test this, we compared the IFs for anthropology, economics, history, and the social sciences and humanities. When the IFs were averaged by subject area we discovered that economics journals have IFs that are, on average, twice as high as IFs for journals in anthropology, history, or the social sciences and humanities. Again, it is preferable to have more IF data before making such inferences, but preliminary findings seem to confirm Garfield's observation. Despite the small sample size, we can be fairly confident that vast differences in excellence exist between titles ranked in the top 20 and those ranked in the bottom 20. It logically follows that those in the bottom 20 could become candidates for de-selection, or at least placed on a "need further investigation" list.
Bruce notes that they are running short on time and says he is going to skip through a few of the following slides. There are findings on excellence (based on use), based on Redalyc and SciELO; findings suggest that users of Redalyc are primarily Mexican and users of SCIelo are primarily Brazilians. The top ten titles by use in the respective databases reflect the regional bias; 8 of the top 10 are from that country. There is also information on the Global Use Metric, created for the purposes of this study by Bruce and Orchid, trying to get a handle on what regions the Redalyc downloads are coming from. The next slide discusses value. Bruce states that due to his background in anthropology, he has familiarity, through publication or serving on the board, with 4 titles in HAPI. What are the differences among these four journal titles? The crucial distinctions are: (1) feeder vs. synthesizer journals, and (2) invited vs. non-invited submissions (formally only), and (3) different degrees/levels/manners of cronyism.
We know that HAPI contains a large number of US titles, many with high IFs, and some of HAPI’s Latin American titles receive high marks in both quality and excellence. But a huge gap in our knowledge remains because no data or partial data are available for many titles. Even though HAPI comprises a miniscule slice of the Latin American serials universe, complete bibliometric data are available for only 10% of its titles. This makes it extremely difficult to develop an objective strategy for selecting and deselecting titles. Our overall impression is that HAPI's content is very good, but there is ample room for improvement. HAPI most likely indexes a small number of titles that should be replaced with titles of higher quality/excellence relating to the same fields. The data we’ve collected helps us identify HAPI titles that should be safe from de-selection, but the absence of comprehensive data makes it difficult to confidently identify titles for de-selection. Going forward, we hope to combine the limited data we have with a strategy to assess the value. Our brief foray into the labyrinth of "value” suggests a new path for acquiring relevant qualitative information that could help HAPI make informed decisions about journal worth. This strategy would seek expert scholarly opinion on the nature of a specified group of journals within a narrowly defined field. With such information, the final piece of the evaluation puzzle would be in place. Because HAPI covers such a small slice of the Latin American literature, it is well-positioned to develop itself as a highly refined, authoritative information resource like none other for Latin America.
Melissa Gasparotto introduces the presenters. Marina Todeschini Crumbacher (University of New Mexico) and Suzanne Schadl (University of New Mexico) -- Documenting Pan-American Scholarly Communications: A Citation Study of Less Commonly Taught and Indigenous Languages
Suzanne Schadl introduces Marina, a student in Latin American Studies, who worked on this project to evaluate how language materials are being used in dissertation research at UNM. This presentation represents a part of a larger research project. This study, as the others, found problems with the metrics; the study began by looking at Spanish and Portuguese materials to determine which departments where using these materials, and they found the problem that, being a part of Spanish America, there were students using Spanish and Portuguese materials from the perspective of the New Mexico locality; there is a large indigenous population and there is a large group of Native American studies. This information is being separated for the larger study, but this presentation focuses on Navajo, Mayan, and Portuguese.
Marina continues to explain the methodology: checking citations in UNM dissertations in Google Scholar Index for these languages, as well as circulation statistics for these language materials at UNM. The goal was to understand usage and embeddedness of language materials in order to propose outreach and collection development practices. She outlines the community profile at UNM: Population Spanish Presence (35% @ UNM) and Native Presence (6% @ UNM); Student Success Services; Academic Emphasis on Latin American & Native American studies and Area and including Programmatic Support of the Latin American and Iberian Institute, the Institute of American Indian Research, the Indigenous Nations Library Program, and the Inter-American studies Library Program. Collection coverage of language includes: Portuguese 61,300 Volumes (only Portuguese); Navajo 924 Volumes (mixed Navajo and Bilingual); Mayan 478 Volumes (mixed Mayan language groups and bilingual). Methodology: Small Sample, limited to one year (2010); searched for words in the title or abstract Dissertations with Indigenous or Brazilian content. They found 15 total = 2 Brazilian and 13 Indigenous (mixed New Mexico Pueblo, Navajo, Apache, Latin American peoples) In these dissertations, they then conducted a citation analysis for foreign language usage and compared the citations against the collection in order to analyze the availability of the works cited. Circulation analysis for usage in general and citation index analysis on circulating titles in Google Scholar. The findings suggest that limited accessibility, defined as limitations in publications or in collections, lead to limited impact, defined as the repeatability of references in scholarly communications, but not limited usage.
Marina proceeds to break down the findings by language. Findings for Portuguese, Navajo, and Mayan, referring to the measurements for Citations in dissertations, Circulation of materials from the UNM collection, and Citation Indexing of the cited titles. What can librarians do?
Suzanne reviews the implications for alternatives to collection development. Work actively with local scholars, and organizations to get foreign language materials into their hands. Identify opportunities for building collections. Library hosted institutional repositories allow for individuals and organizations to determine what resources illustrate their organizational goals and priorities. There is precedent for this practice in special collections (oral histories and interviews), with the example of the France V. Scholes Collection; the American Indian Oral history Collection and with projects that are ongoing, such as the Latin American and Iberian Institute’s Lobo Vault and the K’iche’ Maya Oral History Project.
Peter Johnson (Princeton University) states that Bruce is one of the SALALM scholarship awardees. Peter cites the case of Chile, a country that prides itself on access to education and information about Argentina. The fact that a decade of information can disappear, especially if paper publications are not available, this is a concern in this area of information and has implications for other Latin American countries.
Lisa Gardinier (University of Iowa) has heard that other colleagues are ceasing to collect print government collections. She was shocked to find that this information had disappeared in Chile. With academic journals, she feels that these are stable enough, especially with platforms like OJS.
Emma Marschall (Tulane) states that she doesn’t have a background in Government Documents and is not knowledgeable of how the governments of different Latin American countries make information available, in what formats and how it is distributed, and that this presentation opens the way for a discussion of that topic. She asks how we can become more informed about government publications as librarians.
Lisa Gardinier (University of Iowa) that it does seem that government publications were often irregular, this is continuing in online formats, and they are ceasing print publication and distribution, and online is unreliable.
Paul Losch (University of Florida) trying to bring together some of the points discussed in this panel and another panel on e-books, he posits that if freely available electronic information is considered to have increased value, through adding metadata, accessibility, maybe vendors can be encouraged to deal with these kinds of materials.
Allison Hicks (University of Colorado) has a question for Amelia. Have they encountered regional bias for Impact Story and have had any success introduction impact factor metrics to researchers? Amelia (very hard to hear) can’t say for sure, it would need more study, but believes that while there may be some bias towards the U.S. right now, there is potential for similar application in Latin America. She asks Allison to repeat the second part of the question.
Allison asks Amelia to write that study.
Amelia talks about potential to study the government documents, too and her experience when she was previously in Chile, where the question of whether or not to archive online government publications came up.
Allison repeats her question: have you had any success introduction impact factor metrics to researchers?
Amelia has not had much experience, but has some researchers who might be interested. She cites a study by Jason Priem that dispels the idea that there is an age-gap for researchers using Twitter as an academic tool, and believes that this supports the idea that many researchers would be interested in these newer tools and invites further conversation.
Melissa says that there is time for one more question, but there are no more questions. She asks that the presenters put their Power Points in the SALALM repository because they offer helpful models. Melissa thanks the presenters.