Look inside the methodology.
Transparent questions. Traceable passages. Explicit uncertainty.
The score below uses a fictional teaching example, not real research. Its calculation is the same function used to validate reports.
Appraisal confidence
A score you can look inside.
9
/ 10
Source coverage, matched quotations, and assessable questions. Not a measure of a paper’s truth, accuracy, or clinical usefulness.
Source coverage
4.0 / 4
full text
Quotation matching
2.5 / 3
5 of 6 passages matched
Assessable questions
1.6 / 2
4 of 5 questions
Formula: round(1 + 4 × coverage + 3 × match ratio + 2 × assessable ratio), bounded to 1–10. Without matched evidence or assessable questions, the score is capped at 3.
Full text gives 1.0 coverage; an abstract gives 0.45. Abstract-only reports retain a “lower” confidence label even if the numeric heuristic is higher. A matched quotation does not prove the interpretation is correct.
The complete appraisal process
1. Retrieval and source coverage
MedLit accepts a PubMed, PubMed Central, DOI, or supported preprint identifier. It attempts authoritative metadata and lawful open-access sources, including bounded server-side extraction of machine-readable PDF page text, HTML tables, and a small number of allowlisted supplements. Every report distinguishes full text, abstract only, and metadata only. Metadata-only records are not analyzed.
•
PDF reading order and table-row candidates can be wrong; scans/OCR, figures, equations, and complex table meaning still require source review.
•
Abstract-only reports deliberately lower confidence and leave unsupported domains unable to assess.
•
Preprints are labeled; publication status is not treated as evidence of validity.
2. Untrusted source boundary
Article text is data, not an instruction. The analysis prompt tells the provider to ignore instructions embedded in an article, avoid outside facts, and use exact passages from the supplied source.
•
Generated quotations are normalized and matched against the retrieved title, abstract, or full text.
•
A domain cannot receive a determinate judgment when none of its linked passages can be matched.
•
Unmatched quotations are counted and disclosed in the report.
3. Design-specific question set
The detected design selects a concise, published-framework-aligned question set. These are transparent MedLit questions informed by RoB 2, ROBIS, ROBINS-I, QUADAS-2, JBI, or CASP concepts; MedLit does not claim to reproduce or validate the official instruments.
•
Each report shows the exact rubric ID, version, domain, question, judgment, rationale, and supporting passage.
•
Possible judgments are low, some concerns, high, unclear, or not applicable.
•
Design classification includes a rationale and can be wrong; readers should verify it.
4. Deterministic safeguards
The language model proposes structured evidence, summaries, and domain rationales. Application code verifies quotations, fills missing rubric questions, derives source coverage, and computes the displayed confidence and concern summaries.
•
Appraisal confidence measures source completeness, evidence traceability, and domain assessability—not whether a paper is true. No matched evidence or no assessable domains means insufficient confidence.
•
Risk-of-bias concern and the overview are derived from visible domain judgments—not a hidden model verdict. No assessable domains is unclear, never low risk.
•
Narratives, methods, registration, and disclosure descriptions are model-generated and must be checked against the source. Clinical applicability is not assessed.
5. Reference and disclosure scope
When structured references contain DOI identifiers, MedLit attempts a bounded Crossref resolution and checks indexed update relationships for retraction or withdrawal notices. This is not a complete citation audit or a guarantee that an article is unretracted.
•
The number of references found, identifiers present, identifiers checked, and notices found are reported.
•
Conflict-of-interest and funding fields only summarize disclosures visible in the supplied text.
•
Missing disclosure text is reported as missing or unclear, never assumed clear.
6. Storage, revisions, and export
Saved reports preserve the full structured analysis, analysis contract version, timestamp, and revision. Users can reopen reports without spending provider tokens, compare papers, review individual judgments, organize collections, run local evidence alerts, create bounded read-only snapshots, and export or back up the library.
•
Guest libraries and saved searches stay on the device.
•
Signed-in libraries can sync through Firebase when configured.
•
AI-provider keys never sync with the library, and portable links omit private notes and feedback.
Known limitations
MedLit is an AI-assisted reading aid. It cannot reanalyze raw data, reliably interpret every table or figure, detect all statistical errors, establish causal truth, perform a complete retraction search, or replace domain expertise and peer review.
•
English scientific prose is the primary supported language.
•
Paywalled full text is not bypassed.
•
Expert validation is an external release-evidence step; no expert agreement result is claimed until real blinded reviews are collected.
Analysis contract v4.1.0 · Last reviewed September 10, 2026