Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Fall 12: INF1240 Peer Review Workshop

Lecture Slides: Week 8, Oct. 29, 2012
by

Sara Grimes

on 28 October 2012

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Fall 12: INF1240 Peer Review Workshop

INF1240H
Research Methods Assignment #3: Peer Review




Using course readings and any relevant published (peer-reviewed) academic literature that you find, you will answer the following questions:

What is the article about? What is its thesis statement, area of inquiry, research questions and objectives?
Does the author(s) draw upon previous research in establishing their methodological approach or to justify the use (or non-use) of a particular method?
What research methods are used? To collect the data? To analyze the data?
Is the research design appropriate to the thesis statement and/or research question(s)? Why or why not?
Do you think the research design has “reliability”? Do you think the data has “internal and external validity”?
In terms of the methods and research design, how could this study or project be improved upon?
What does this article teach you about research methods (either generally, or in relation to the specific methods applied)? Due Nov.9 noon: Value 25%
1,500 – 2,000 words Group Brainstorming Session In groups of 5-6, discuss preliminary ideas for how to go about conducting a "peer review".

Come up with 2 to 3 additional strategies that you think will be helpful.

Present them to the rest of the class. http://www.michaeltyworth.com/?p=11 "Ramblings from the Ivory Tower" Peer Review Workshop Oct. 29, 2012 You have 5 different articles to choose from (see Blackboard). They span a variety of disciplinary and methodological approaches.

You will select ONE PAPER ONLY for your review.
The peer review process is a crucial component of academic research. Ideally, it supplies a mechanism for independent, anonymous evaluation and assessment of new contributions to a particular field by a number of its established experts (usually between 3 and 5). For this assignment, you will engage in a peer review exercise, by carefully reading and assessing the research design and methodology of an as-yet-"unpublished" social science research article.

Drawn from SSRN - working papers series.
“For better or worse,” as Luker (2008) describes, peer-reviewed journals “are the gold standard for much of the academic system” (p.69). When to Publish?
Start as early as possible.
Students are admitted into graduate studies for their potential, but ongoing progress is based on accomplishment (key when applying for PhDs, academic jobs). Getting involved in the peer review process is a great way to dispel some of your more primal fears about the whole thing. Publishing Tips for Grad Students It can also help you develop more sympathy and patience for the length of time it can take for the reviewers and editors to get back to you. Seeing how it all works from the perspective of a peer reviewer can really help you to identify areas in your own work that might need tweaking (or further development and explanation), AND to understand the context from which comments and revisions are drawn. "An overall lack of effort to really read the manuscript and understand what it is the author is talking about. Too often reviewers tend to try and crowbar the subject of a paper into their own paradigm instead of evaluating the paper on its own merit." "Making a strong statement of fact without supporting evidence; or essentially offering the criticism ""You're wrong, I'm right."" Making contradictory criticisms; for example: "This topic requires a broader and more specific discussion." Personal nastiness. Common characteristics of poor quality reviews: Dr. Michael Tyworth Go into the review with a positive perspective - "you want the paper to get published; that the review process is, to a degree, a collaboration between you (the reviewer) and the author to work to make the paper better." Tyworth's "How to Conduct a Peer Review" State outright if/when you lack expertise on a particular area or method. Limit your comments to things you do know about, feel you have expertise in (construction of argument, description of research design, etc.) If you're "only tangentially aware" of the paper topic/subject matter, or not up to date on the current literature/trends, take the time to familiarize yourself with the area, foundational and current literature, so that you can make informed assessments AND reviewer suggestions. Most importantly: "write the kind of review you would like to receive yourself" in terms of the style and tone, the depth of insight and content that YOU would find useful. Don't try to turn the paper into your own - i.e. don't "offer revision suggestions that are, in essence, ways to re-frame the paper to one that YOU would write on the topic." Look for ways to offer suggestions that strengthen the paper.

e.g. know a great article on the topic that the author overlooked? recommend it with specific citation info so they can find it, and explain why you think it would be useful. vs. starting out negative:
e.g. most papers are terrible, flawed, failures and this one has to prove it's not. vs. vague or sweeping criticisms vs. making criticisms or offering advice you're not sure about (e.g. suggesting articles to add based on a cursory database search and not actual knowledge of topic or contents of the lit). Bridging and Bonding in Emergency Management Networks (interviews)
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2066935

An Insight into the Networking Approaches of Women Entrepreneurs in Mauritius (focus group)
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2130451

A Pocket Full of Learning: Podcasting in American Colleges and Universities (survey)
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2014887

President or Dictator? A Comparison of Cuban-American Media Coverage of Cuban News (content analysis)
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2107964

Privacy and Modern Advertising: Most US Internet Users Want 'Do Not Track' to Stop Collection of Data about their Online Activities (survey)
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2152135 Assignment Guidelines Strategies Guidelines Applications The Peer Review's (2007) "The Ultimate Guide to Scholarly Publishing" (key points for grad students) Where to Publish?
Spend time in choosing where to publish. You need to match the publication with quality of your work. Top tier journals are great, but other venues are worthwhile too - work your way up, engage with different audiences, etc.
Do Your Homework:
Key to picking a journal and prepping to send something to a journal is....READING that journal. This is the only way to get a real sense of the fit, as well as the expectations, tone, citation style and quality of the journal itself. Pay attention to: who is publishing in the journal (names you know? disciplines? geographical region?) Who is on the editorial board (should know some of them, if not...perhaps not the best fit)? Quality: "It’s amazing the number of articles that get tossed out at the beginning because the quality is just so poor,"" [...] "The less editors need to work on your piece, the more likely it is to get selected for publication." Clarity: ""What did you do? Why did you do it? What does your study or idea contribute to what we already know about the subject? State this clearly in the abstract, introduction and cover letter...""
Context: ""If there is one mistake authors make, it is not to describe the big picture," says Dhand (editor at Nature). "What is known, what this piece of work is adding to the field and how this can be applied or take the field further."" Making Sense of....Peer Review Comments:
"[F]eedback is sometimes totally tactless because reviewers are in a hurry. And note that ‘revise and resubmit’ does not mean ‘reject.’ It’s very common for a paper to be accepted contingent on minor changes."

Often, you will need to make the changes before you find out if it's accepted (formally). Other times, the paper needs to go through the review process again after the changes are made. Making Sense of...Publishing as a Process:
"There is no great mystery to the process, it just seems mysterious because it is drawn out, and can be derailed by any number of very tiny mistakes, errors that are not just technical, like poor editing, but also mental mistakes..."

e.g. assuming harsh comments mean the revisions are impossible or that your work is hopeless Making Sense of...the Process of Publishing:
"...journals are picky, reviewers are overworked, publication schedules are slow, and competition is plentiful. If your work is of relevance and importance, you can be assured that there is a journal that will publish it." Making Sense of the Process The Three Keys to Editing your Submission
(from "The Peer Review," 2007) Tips on Publishing Your Work "[S]tay productive, be out there, be putting your work out in places that people can see it and often that means in a variety of places."

Consider industry publications (magazines), online publications, policy consultations, etc. And throughout the whole thing, keep reminding yourself that the revisions and criticisms you've received are (deep down ;) meant to help you make your paper even better, and that the reviewers wouldn't have taken the time to craft detailed feedback if they didn't think your contribution was worthwhile. On the Receiving End Use your lists to write up your report of changes made (& any not made). Tackle/check things off the (first) list one at a time. Read through the comments a second time (2 days later) w/ a clear head & fury quelled. Promptly close the reviewer document (or email) and set it aside for at least 2 days. Read through them ONCE - thoroughly but very quickly. What to do when you get your reviewer comments back: Create 3 lists:
1. a list of the issues you can actually attend to (in your own, nice, helpful, non-snarky words),
2. a list of things that are beyond your capabilities (e.g. "instead of surveys, you should do interviews"),
3. and a list places/issues where the reviewers contradict each other. When in crippling doubt, don't be afraid to ask the editor for clarification (e.g. what to do about contradictory advice, etc.) Group Exercise 2:

In groups of 5-6 discuss the following:

How does knowing about the publishing process help you to develop strategies for conducting a peer review?

How does a better understanding of peer review help us to better understand & assess academic research? iSchool, UofT
Full transcript