Kappa calculation at level 1 screening level

Exclusion reports, activity reporting and datarama
Posts: 2
Joined: Thu Nov 08, 2018 5:26 pm

Kappa calculation at level 1 screening level

Postby cmoran12 » Fri Nov 09, 2018 7:46 pm

When looking at the Kappa report, it seems that the kappa is calculated at the question-level of a form. Is there a way to calculate kappa at the form-level, meaning that the kappa calculation would ignore agreement of each question and rather calculate concordance between the overall decision to include or exclude the reference?

I see that there is also an "overall kappa" score - does this take into account once away question-level agreement? Could I have some clarity about how the overall kappa is calculated? Thank you in advance!

Site Admin
Posts: 25
Joined: Wed Nov 09, 2016 12:04 pm

Re: Kappa calculation at level 1 screening level

Postby evidencepartners » Sun Dec 02, 2018 8:40 pm

DistillerSR calculates Kappa at the question level, as you point out.

What you could do is add a question that says Article Included, Yes/No to your screening form and just look at the Kappa for that question.
You could have your screeners answer that question or you could use a formula to automatically tick the correct button depending on what answers the user clicks.

Obviously, option 1 is a easier to implement but is a bit more work for your screeners.

Hope this helps.

Return to “Reporting”

Who is online

Users browsing this forum: No registered users and 1 guest