Home Bibliometrics and research assessment

Bibliometrics and research assessment

Euan
By Euan
• 6 articles

Benchmarking %age cited in policy

How Overton benchmarks the percentage of documents in a set that are cited in policy Given a large set of articles it can be helpful to know if the %age of articles cited in policy matches what you might expect – what’s “normal”. But determining expectations is difficult as it depends on the age, subject area and type of the articles in question. As a very general rule we sometimes say that you should expect around 5% of scholarly articles to be cited in policy: in reality for documents in public health, economics or international development, for example, it could be as high as 60%+, and for quantum physics or drama as low as 0.1%. When we’re helping users understand their data we’ll sometimes run a simple benchmarking experiment that takes some of this into account by compensating for publication date and venue (as a stand in for subject area) biases – we compare the set of articles to another, randomly selected set of articles from the same journals and published in the same kind of timeframe. More specifically the steps we take are: - For each article in the input list - We use OpenAlex to find all articles published in the same journal in the same short time window – a month before publication or a month after publication. This is the possible comparands set for this article - If the possible comparands set contains fewer than 80 articles – perhaps the journal only publishes a few articles a month – the time window is expanded to between two months before publication and two months after publication - If the possible comparands set still contains fewer than 80 articles the time window is expanded to three months before and after the article publication date - If there are no articles to compare to in this larger time window then we skip the article and exclude it from analysis - We treat everything with a DOI as an article, and don’t include items that OpenAlex considers paratext (e.g. tables of contents, front covers, mastheads, editorial board listsings) - At the end of the process for an input list containing n articles we have n corresponding possible comparand sets - We then run the following experiment 1,000 times: - Create a new empty list - Pick a single entry at random from each of the possible comparand sets in turn and add it to the list we created above - The list now contains n items. This is the matched comparands set (each article in the matched comparand set has the same journal and approximately the same publication date as one of the articles in the original input list) - Calculate the %age of articles on the matched comparands set that are cited in policy - We then plot the output (the %age of comparand articles cited) of each run and from that we can derive the median outcome which we can use to compare to the %age cited figure for the input set. Example output of a benchmarking run. The red line shows the %age of documents cited in policy of the input set – in this case that’s better than you might reasonably expect Caveats and limitations This is still a very rough, indicative approach. In particular some shortcomings to be aware of are: - Multidisciplinary journals like Nature & PLoS One, for example, may publish articles from a mix of subject areas, so the articles of a similar age picked randomly from these venues are less likely to be on comparable topics. - Article types – we look for articles published in the same journal around the same time and ignore paratext (see above), but there is no easy way to tell if a comparand article we’ve selected is, for example, a book review or an editorial, or a systematic review or a short letter. Ideally a proper comparand would be of the same type as its input article. - Incorrect publication dates – getting accurate publication dates is a long standing bugbear for anybody working with scholarly data. We rely on the dates in OpenAlex (which in turns collects data from publishers and Crossref) being correct.

Last updated on Jun 25, 2026

NISO Altmetrics Code of Conduct self-reporting table for Overton

NISO Altmetrics Working Group C “Data Quality” Code of Conduct Self-Reporting Table | **Description ** | | **Supports CoC Recommendation ** | | **Aggregator / Provider Submission ** | | | List all available data and metrics (providers and aggregators) and altmetric data providers from which data are collected (aggregators). | | T1 | | Overton collects and processes documents from policy sources around the world such as governments, think tanks and IGOs. We link various entities to the policy documents we track using third party data sources (Crossref, OpenAlex, Wikipedia) | | | Provide a clear definition of each metric. | | A1 | | Overton currently provides one quantitative indicator ( the number of policy documents citing either research or policy outputs). | | | Describe the method(s) by which data are generated or collected and how data are maintained over time. | | T1, T2, R1 | | Data are collected in various ways including web scraping and third party APIs. New data points are created using our own AI/machine learning systems. We have checks in place for broken sources that are investigated by the team as soon as possible. More information is available on our support pages. | | | Describe all known limitations of the data. | | A3 | | We do not claim to track every policy source in the world and we add more sources all the time, which may lead to slightly different results depending on when a search of the database has been run.We believe our coverage of the sources we do track is broadly complete back to 2015, before which the coverage may become more patchy. This is due to source links changing over time and the fact that many organisations started publishing their documents online relatively recently.There may be errors in the affiliation and funding data which exist on the third party data sources we use, over which we have limited control. Funding data in particular is geographically patchy because of reporting different norms between countries, with more reliable information in the UK and US. Although we do have global coverage, there is a known bias in global policy document availability towards knowledge economies, which tend to have a wider range of organisations engaged in the policy process and which have norms of publishing policy documents publicly and in a digital format. | | | Provide a documented audit trail of how and when data generation and collection methods change over time and list all known effects of these changes. Documentation should note whether changes were applied historically or only from change date forward. | | R1, R2, R3 | | Opted out due to complexity but feel free to get in touch | | | Describe how data are aggregated. | | T2 | | We enable searches by DOI, ORCID, PMID, PMCID, ISBN, researcher name, policy organisation/source, topic, SDG, COFOG, funder, publisher and journal. | | | Detail how often data are updated. | | T3 | | Depending on the data point, we refresh data daily, weekly or monthly. More information is available on our support pages. | | | Describe how data can be accessed. | | T4 | | Overton provides access via the web app, API and data snapshots. | | | Confirm that data provided to different data aggregators and users at the same time are identical and, if not, how and why they differ. | | R4 | | Yes the data provided is identical. | | | Confirm that all retrieval methods lead to the same data and, if not, how and why they differ. | | R4 | | Yes all methods lead to the same data. | | | Describe the data-quality monitoring process. | | T5, A2 | | We monitor the database for sources that seem inactive and check them for broken links. For institutions that are duplicated, we have a method for merging records when these are discovered. | | | Provide a process by which data can be independently verified. | | R5 | | Our data is accessible via the web app, API and data snapshots, so users are able to see the underlying data. | | | Provide a process for reporting and correcting data or metrics that are suspected to be inaccurate. | | A2 | | Any suspected issues or inaccuracies within the data can be reported via [[email protected]] Clients have the opportunity to request that we add additional policy sources | | For more information about the NISO Altmetrics standards, visit niso.org/publications/rp-25-2016-altmetrics

Last updated on Jun 25, 2026

Our commitment to DORA: what this means for Overton

Overton is a proud signatory of the San Francisco Declaration on Research Assessment (DORA) and we are committed to ensuring we operate within the framework it outlines. We want to be transparent about what that means in practice, so we have described below how each relevant section applies in Overton’s context and will keep this updated as the platform evolves. 11. Be open and transparent by providing data and methods used to calculate all metrics. We are and will continue to be open about how we gather our data (see our commitments and our explanatory articles on help.overton.io). We do not currently provide any metrics, but if and when we do, where possible we will adhere to this principle. ***12. Provide the data under a licence that allows unrestricted reuse, and provide computational access to data, where possible. *** As a small company whose business model relies on proprietary data, we are not able to allow unrestricted reuse under the terms of our licence. However, we do provide computational access via API. ***13. Be clear that inappropriate manipulation of metrics will not be tolerated; be explicit about what constitutes inappropriate manipulation and what measures will be taken to combat this. *** We do not currently provide indicators or metrics within our platform but we know that some users want to use the data to create their own so we want to work closely with our clients and the wider sector to understand how this new type of data can be used responsibly. We’re at the start of our journey, and will be actively working to pioneer approaches and to create and share best practice around this. ***14. Account for the variation in article types (e.g., reviews versus research articles), and in different subject areas when metrics are used, aggregated, or compared. *** As our primary focus is on policy documents this criteria applies slightly differently to us. We do differentiate policy document types and any future indicators or metrics we produce are likely to be on this side rather than for journal articles. This is something we’ll be looking at in future. Overton is committed to being a responsible data custodian – read about what this means here.

Last updated on Jun 25, 2026

Our commitment to the Leiden Manifesto principles: what this means for Overton

Overton is proud to support the principles in the Leiden Manifesto for Research Metrics and we are committed to ensuring we operate within the framework it outlines. We want to be transparent about what that means in practice, so we have described below how each relevant section applies in Overton’s context and will keep this updated as the platform evolves. ***1. Quantitative evaluation should support qualitative, expert assessment *** Yes, we agree with this and have taken steps to support it (see our [responsible metrics commitments](https://www.overton.io/overton’s-commitment-to-being-a-responsible-data-custodian/(opens in a new tab))). 4. Keep data collection and analytical processes open, transparent and simple Yes, this is something we have done and will continue to do (see our [responsible metrics commitments](https://www.overton.io/overton’s-commitment-to-being-a-responsible-data-custodian/(opens in a new tab))). 6. Account by variation by field in publication and citation practices Although we don’t have indicators or metrics at the moment that would be affected by these variations, this is something we are aware of and will take into consideration in any future developments. 8. Avoid misplaced concreteness and false precision At the moment, we do not have any indicators or metrics within the platform but if any users are planning to use the data in this way, we provide additional information via data cards within the platform to let them know its limitations. For consultancy projects, we are careful to ensure our analysis avoids these pitfalls and we communicate the limitations of the analysis clearly to clients. 10. Scrutinise indicators regularly and update them We do not have any indicators at the moment, but we do keep a close eye on our coverage and let users know about how they should use the data in their work. Overton is committed to being a responsible data custodian – [read about what this means here](https://www.overton.io/overton’s-commitment-to-being-a-responsible-data-custodian/(opens in a new tab)).

Last updated on Jun 25, 2026

Using policy related metrics responsibly in research assessment

Important things to know when planning to use metrics or indicators based on Overton’s data Overton’s search and filtering capability makes it quick and easy to find policy documents mentioning or citing outputs from research organisations. It is important to focus on the qualitative aspects of the data – to find potential case studies, alert researchers to interesting uses of their work or to highlight successes in particular collaborations or topics, for example. But sometimes this needs to be combined with quantitative data: to measure change, see trends over time, compare yourself to other institutions or for a policy related KPI. It is also possible to use Overton’s data to assess a grant, a group or an academic. In these cases it’s important to understand the caveats and limitations of the data. We’ve collected some useful notes below, but would also encourage Overton users to reach out with any questions, big or small – we’re very happy to advise and answer data questions, or explain how to pull out results. We can be contacted directly through the app or at [email protected]. What does the database represent? - Remember that when looking at Overton data you’re looking at policy documents that have been indexed by Overton, not all policy documents ever written. We may be missing older documents, for example, or the document might never have been made publicly available online. - The definition of a policy document is subjective. The one we use is quite broad: public documents written primarily by or for policymakers (more here). - There are always new policy sources to add and we’re still working through the list. There are some kinds of policy source that we wouldn’t automatically add: you can see our minimum criteria for inclusion here. - In general the data for better developed countries is more complete. Some countries are better represented than others. Reference extraction - Overton currently matches references to other policy documents, scholarly works with a DOI and links to mainstream news sources. - Some outputs fall into both categories: for example the OECD may publish work in a journal (so it’ll appear in the scholarly works set) and also make it publicly available on their website (so it’ll appear in the policy document set). - Note that many books – especially older books and monographs – have never been assigned DOIs and so references to them will not be picked up by Overton. This is something we’re actively working on. - Generally speaking Overton is language agnostic but we may miss references from documents in some languages depending on the style of citations they are using. - Reference matching is one of our core competencies and Overton’s reference matching compares very favourably to other systems, but we can’t always resolve references to a DOI and can sometimes match to an incorrect DOI, especially if there are multiple articles with very similar titles and publication dates in the same journal. Author and institution disambiguation - By default we use institutional affiliation data from OpenAlex but can support other systems too with some custom work – please get in touch if this is something you’d like to explore. - The institution list isn’t a hierarchy: if a medical school appears as a separate institution in OpenAlex then it won’t appear in searches for the parent institution and vice versa. - We don’t yet disambiguate authors e..g if an author has a middle initial in the author list of one paper and not in another, they’ll appear as two different people in Overton.

Last updated on Jun 25, 2026