Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


GS Seminar 2010

Using Google Scholar to Estimate the Impact of Journal Articles in Education

Jan van Aalst

on 3 June 2011

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of GS Seminar 2010

Using Google Scholar to Estimate the Impact of Journal Articles in Education Dr. J. van Aalst, vanaalst@hku.hk
Associate Professor
Faculty of Education
The University of Hong Kong Acknowledgments
Supported by Education Faculty Research Fund
Technical assistance: Li Dongyu
Full details: Educational Researcher, 39, 387-400 (2010) Web of Science ® provides researchers, administrators, faculty, and students with quick, powerful access to the world's leading citation databases. Authoritative, multidisciplinary content covers over 10,000 of the highest impact journals worldwide, including Open Access journals and over 110,000 conference proceedings. You'll find current and retrospective coverage in the sciences, social sciences, arts, and humanities, with coverage available to 1900. ISI impact factor Institute for Scientific Information / WEB OF SCIENCE (SSCI, JCI)
Maintains small database of journals
105 for education, 2007
139 in 2009
Annually reports many metrics based on these collections
Impact factor
Hirsch index (for authors)
Etc. Definition - ISI IMPACT FACTOR # citations in the database in current year, divided by the # articles published in the target journal in previous 2 years Eg., IF of Journal of the Learning Sciences, 2009 Journal Impact factor = 53/30 = 1.767 How are impact factors used? In individual performance reviews
In system-wide research assessment exercises EDUCATION AND EDUCATIONAL RESEARCH Top impact factor 3.00
20 journals with impact factors ≥ 1.0
scientific studies
intended for general audiences
Lack of high-impact factor journals for
teacher education
curriculum studies
policy analysis
higher education, etc. EXPERIMENTAL PSYCHOLOGY Top impact factor 9.37
56 journals with ISI impact factor ≥ 1.0
Many areas of psychology included Problems with assessment based on ISI impact factors Gate-keeping: journals selected for quality
Practicality: not feasible to conduct analyses for large numbers of journals Number of journals small ! Exclusion of other publication formats Dissertations Books/book chapters Conference papers Conservative estimates of impact Impact factors low for many fields >> Google Scholar citations Since 2004 http://scholar.google.com

“Organize the world’s information and make it universally accessible and useful.”

http://www.google.com/corporate/facts.html Fits with 21 st century skills! How Google Search/Scholar work? Brin & Page (1998); Maslov & Redner (2008) PAGERANK Current PR of Web page = sum of its shares of the PRs of all the pages that link to it
Current values of all the PRs are calculated from the most recent values
Process is repeated until all the PRs settle on steady state values
(All values reset each step)
Appr. Markov process Combine with query relevance How publications come up high? Be cited by influential papers
Don’t cite too many obscure papers
First results from Google give works that have most influenced the field – the papers that the best scholars find important to cite NOTE: ISI uses only citation counts to a paper, GS all papers and their citations Strengths and weaknesses of GS STRENGTHS Freely available on Web
Easy to use
- Widgets on publisher Web sites
- Perish or Publish software (Harzing)
May compensate for problems with ISI WEAKNESSES As a database of scholarship
- Incomplete
- Lack of knowledge of what’s included
- Failure to utilize extensive indexing
For citations
High correlations with ISI impact (redundancy)
Citations from ghost papers Derek Hodson
Science education Paul Black
Assessment for learning Can GS provide
useful information about impact? To what extent does GS citations render a valid picture of the influence of specific journals?
What is the nature of the citing documents?
How fast do GS citations grow? Study: Data collection Selected lead authors from chapters of one international handbook on research in
Science education
English language education
Educational technology

Did a Google Scholar search for each author
Basic data set:
Up to 1st five unique results for an author that were journal articles published between 2003 and 2007
1st 100 unique citing documents for each article The sample
Lead authors of handbook chapters Science Education English Language Education Educational Technology Interpretation Similarities between areas 1.Clear advantage to publishing in ISI journals in all 3 areas—if ISI journals were available
2.3 x as many GS citations if article published in ISI for science education and educational technology
1.7 x as many for English language education

2.Publishing in ISI journal associated with more GS citations across most document types Differences between areas 1. In English language education more GS citations from books and book chapters

2. In educational technology this was somewhat true of conference papers GS, ISI, and Scopus citation counts Mean (over area) ± standard error Growth of citations over time
Similar in 3 areas; data combined 7.3 after 3 years
18.9 after 6 years

✔-> 19 after 1 year high

LINEAR (no compound interest on citations) Characterizing a specific area
Knowledge building Led by Bereiter & Scardamalia
Transfer of control over learning to students
Advancement of community knowledge
Online discourse
Top article: 1,678 GS citations (1994)
167 publications since 1996 Most cited journal articles Knowledge building, 1996-2010 Knowledge building, last 5 years Conclusions 1. GS promising tool for characterizing scholarship in education
Free and easy to use
Acceptable “big picture” of each area
Characterizes differences in publication patterns between specializations
More equitable measure of citations across areas
Relatively few non-academic web pages (<13%) 2. Some important cautions
Some invalid results (web page elements)
Some “papers” may no longer exist
Some results not academic The future of impact Capture citations from non-traditional sources
Course outlines
Evaluate impact of additional outputs
Curricular materials COMP & EDUC missing (Elsevier not incl. before 2007)
JCAL missing (new editor in 2005) (A) Science Education (B) English Language Education (C) Educational Technology
Full transcript