Introduction
The so-called ‘serials crisis’ that was deeply experienced in Europe and the USA at the beginning of the last decade did not only have impacts on its own institutions, on publisher business models and on the redesign of the instruments to ensure access to scientific information resources. In Latin America, following the pioneering experience of Brazil's journals portal, CAPES, in 2000, countries such as Argentina, Uruguay and Colombia also began to develop interventions (arising from public agency initiatives) to ensure access to scientific information for their research communities.
The Chilean case is part of this trend, but with a few differences. Firstly, the public agency in charge, the National Commission of Scientific and Technological Research (CONICYT), was in favour of creating a separate institution – a non-profit corporation – to contract subscriptions without the constraints of public purchasing.
A second aspect in Chile was that CONICYT determined that the senior personnel of the corporation‘s founding organizations should participate in the strategic decision-making of this new body. This was to ensure that decisions were taken at the highest level.
A third aspect of the experience – for better or for worse – was the decision to avoid, as far as possible, the conflicts of interest among very heterogeneous institutions, making content selection procedures based on objective studies of the citation patterns of the Chilean scientific community.
Finally, a co-payment model was developed in an attempt to minimize cross-subsidies and reflect the potential use of each institution. This was to ensure the governance of the primary agreement that gave birth to the new entity, called CINCEL Corporation or Consortium for the Access to Electronic Scientific Information.
The starting point
The problem that we tried to address was the profoundly unequal distribution of access to scientific information, which also resulted in scientific output, measured as articles indexed in Web of Science (WoS), being highly concentrated in just a few institutions. For example, in 2002, when CINCEL was created, only CONICYT had online access to WoS. Thus, the consortium's first action was the acquisition of this product for all partners.
For journals, the overall picture was not very promising either. Of the 25 partner universities of CINCEL that in 2005 received public funding, only three had acceptable levels of access to serial publications. Those same three, for the same year, represented 69.8% of the scientific output (or articles indexed as proxy), which points to a correlation between access and productivity.
Both phenomena are illustrated in Figures 1 and 2 below.
When access is so poorly distributed, the lack of information carries unquantified costs with a resulting loss of quality, relevance and timeliness of the research undertaken. Additionally, it is a weak basis for programmes aimed at improving ‘human capital’ (such as scholarships granted by Government for a variety of postgraduate programmes, training or postdoctoral courses), or for policies designed to decentralize scientific activity.
Public policies: design options
The significant asymmetry and inequity in access to information resources was the motivation for designing a public policy to reduce the gap.
CONICYT had several intervention options. In a state like Chile, that is still dominated by an ethos of entitlement from the so-called ‘Washington Consensus’, an obvious option was to subsidize the information demand by ‘voucher’ schemes. Although a very efficient measure to satisfy individual demands, this type of public policy does not generate economies of scale.
“The significant asymmetry and inequity in access to information resources was the motivation for designing a public policy …”
Another option was to create an institution to take charge of the problem. It is a commonly used approach, but a sterile one if it does not have leadership, competence and funding.
An alternative approach was to not undertake major institutional changes, but to make incremental improvements, for example, through performance agreements between the public agency and institutions with serious access problems. As the indicators of access, use and productivity improved, the institution would be ensured a guaranteed budget year after year.
Two further options, preferable in cases of budget scarcity, were to work with the most productive scientific groups – a striking meritocratic public policy because it would achieve early results (but rather than eliminating any gap, it would instead make it wider) – or the inverse approach, to focus resources on the less productive groups. The latter is the most complex option to manage, because there are longer-term results, where public policy is shielded from political cycles that alter the priorities of the authority in charge.
Whichever design option was chosen, there were at least three conditions to be met; public policy had to be:
- technically impeccable (not produce negative consequences or unwanted effects)
- politically viable and manageable (in most cases, the perfect technical solutions are not ‘politically correct’), which means willing to bear the transaction costs while the demand is aggregated and institutional heterogeneity is addressed
- in the public good.
In other words, the final option needed to be the most efficient, effective, equitable and sustainable solution that could be achieved.
The response, in the Chilean case, was a mix between a partial subsidy (because there are co-financing resources available) and the creation of a body to take charge of negotiation, acquisition and management of the so-called National Infrastructure of Scientific Information Access. With a decentralized model of access, it provides a continuous service to electronic information resources.
The main resource is the Electronic Library of Scientific Information Program (BEIC), started in January 2008, and in which 27 universities participated, representing almost 85% of Chilean scientific publication. It is comprised of approximately 5,000 academic journals from eight publishers: AAAS, Nature Publishing Group, Annual Reviews, American Chemical Society, Oxford University Press, Wiley-Blackwell, Springer and Elsevier. Of the titles, 75% are indexed in WoS and 69% in Journal Citation Reports (JCR), 46% indexed in the first and second quartile of JCR. The accumulated cost to 2012 reached 35 million dollars.
However, state involvement in an initiative of this calibre meant that democratic access (with all its positive benefits) should also carry a key outcome: an increase in scientific productivity.
Evaluation limits
The evaluation of BEIC, undertaken by Nicholas Cop Consulting, and designed by the Executive Secretary of CINCEL and the Scientific Information Program of CONICYT, took place between January and November 2011. It included qualitative components such as satisfaction surveys of end users and contract analysis to protect the investment. The quantitative aspects were concentrated in a bibliometric study of citation usage and publication of new articles to determine if the existence of better access brings with it improvements in the productivity of the institutions.
“… state involvement in an initiative of this calibre meant that democratic access … should also carry a key outcome: an increase in scientific productivity.”
When evaluating a public policy instrument such as BEIC, it is important to note the limitations of a study that explains scientific productivity using only one variable – access to information – but ignores others, such as the existence or not of public policies that fund basic and applied research, researchers, infrastructure and equipment distribution, and variables whose impacts are even more difficult to measure, such as the social capital of scientists and their collaborative networks. There is also an important dimension in the measurement of impact related to an improved quality of life, which requires another approach.
In the case of BEIC, impact measurement is very conservative. The eventual use of information (articles) contained in BEIC journals for the publication of new knowledge (new articles and where they are published) is analysed from a bibliometric point of view.
For the analysis, it was considered that publications subject to evaluation would be those indexed in WoS and in JCR, including articles published in national open access journals (SciELO-Chile). This last decision allowed the estimation, in an indirect way, of the value of that collection, which CONICYT has managed since 1998, as part of our scientific communication cycle.
Some findings
Firstly, we have been able to confirm that the collection is well used, so it has not been a wasted investment: the usage of BEIC titles, measured as ‘downloads’, has significantly increased; before 2008, 40.5% of titles present in the collection were downloaded, increasing to 63.3% in the 2008–2010 period (see Table 1).
Elsevier | Wiley-Blackwell | Springer | NPG | OUP | ACS | AAAS | Annual Reviews | Total | |
---|---|---|---|---|---|---|---|---|---|
2008 | 1,211,651 | 268,653 | 175,299 | 100,056 | 74,880 | 56,518 | 38,036 | 29,416 | 1,954,509 |
2009 | 1,469,978 | 236,172 | 201,729 | 122,229 | 89,703 | 73,663 | 40,000 | 30,095 | 2,263,539 |
2010 | 1,525,075 | 232,148 | 186,215 | 136,944 | 83,826 | 81,931 | 36,277 | 28,319 | 2,310,735 |
cumulative variation | 313,424 | −36,505 | 10,916 | 36,888 | 8,946 | 25,413 | −1,759 | −1,097 | 356,226 |
cumulative percentage variation | 26% | −14% | 6% | 37% | 12% | 45% | −5% | −4% | |
If usage is measured as citations, an average increase of 20% has been demonstrated compared with the period before BEIC.
Analysing disciplines, 60.4% of publications in the 2005–2010 period were concentrated in seven fields: Clinical Medicine, Space Science, Plant & Animal Science, Chemistry, Engineering, Social Sciences (general) and Physics. Clinical Medicine is the subject category with the highest concentration of publications (19.11% of all publications) and in the post- BEIC period, its share of publication in international journals increased to 26.5%, which is expected due to the publication rate in the discipline, but also because it is the discipline represented the most in national journals, along with Social Science and Humanities, because of the current specific incentives for publishing in SciELO-Chile. (See Table 2.)
Area (publications) | BEIC publishers | No | SciELO Chile | Total per Journal type | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
AAAS | ACS | AnnReu | Blackweli | Elsev | NPG | OUP | Springer | Wiley | BEIC | No | SciELO | |||
Agricultural Sciences | – | 11,8% | – | 86,4% | 29,3% | – | – | 110,0% | −36,4% | 4,3% | 76,9% | 414 | 333 | 504 |
Biology & Biochemistry | – | −50,0% | – | 8,5% | 20,2% | 100,0% | −11,1% | 47,4% | −3,7% | 5,7% | −34,7% | 611 | 436 | 195 |
Chemistry | – | 17,0% | – | −8,9% | 20,4% | – | – | 32,3% | −18,8% | 58,9% | 0,7% | 1.395 | 422 | 271 |
Clinical Medicine | – | – | – | 19,6% | 26,2% | 100,0% | 29,4% | 33,3% | 3,1% | 48,7% | 8,8% | 1.083 | 1.174 | 3.446 |
Compuier Science | – | – | – | – | 86,7% | – | −33,3% | −12,9% | 300,0% | 47,7% | – | 318 | 436 | |
Economics & Business | – | – | – | 23,8% | 124,0% | – | 166,7% | 19,0% | 50,0% | 83,8% | – | 195 | 193 | |
Engineering | – | −30,8% | – | −40,0% | 58,9% | – | – | 1,8% | 6,8% | 48,1% | 32,4% | 658 | 970 | 251 |
Environment/Ecology | – | −66,7% | – | 64,3% | 24,0% | – | 0,0% | −−9,1% | 57,1% | 42,2% | 159,3% | 652 | 390 | 97 |
Geosciences | – | – | – | 85,7% | 18,1% | – | 100,0% | 71,4% | 100,0% | 28,6% | 29,5% | 514 | 352 | 397 |
Human Sciences | – | – | – | – | – | – | 0,0% | – | – | 178,1% | 50,2% | 5 | 242 | 1.251 |
Immunology | – | – | – | 200,0% | 23,8% | – | −100,0% | 100,0% | – | 31,0% | – | 71 | 67 | |
Materials Science | – | −100,0% | – | −100,0% | 26,4% | – | – | 77,8% | 800,0% | 32,4% | – | 263 | 237 | |
Mathematics | – | – | – | −40,0% | 26,6% | – | −25,0% | 69,0% | 33,3% | 30,7% | – | 562 | 526 | 20 |
Microbiology | – | – | =n | 52,9% | 42,9% | −100,0% | – | 118,2% | −22,2% | 58,6% | – | 147 | 150 | |
Molecular Biology & Genetics | – | −100,0% | − | 100,0% | −16,3% | 0,0% | 266,7% | 14,3% | −1,7% | 45,2% | – | 269 | 179 | |
Multidisciplinary | 40,0% | – | – | – | – | 5,3% | – | 200,0% | – | 2,8% | – | 73 | 73 | |
Neuroscience & Behavior | – | – | – | 9,5% | 1,4% | – | 0,0% | 62,5% | −15,0% | 24,2% | – | 250 | 148 | |
Pharmacology & Toxicology | – | 0,0% | – | 100,0% | −18,0% | – | – | 400,0% | 100,0% | 117,9% | – | 145 | 124 | |
Physics | – | 75,0% | – | – | −9,0% | 200,0% | – | 5,2% | −14,3% | 24,5% | – | 539 | 1.091 | |
Plant & Animal Science | – | 166,7% | – | 34,4% | 41,2% | – | −16,2% | 65,9% | 0,0% | 18,6% | −8,4% | 897 | 988 | 228 |
Psychiatry/Psychology | – | – | – | 166,7% | 41,7% | – | −100,0% | 75,0% | – | 57,0% | −17,1% | 70 | 221 | 150 |
Social Sciences, general | – | – | – | 228,6% | 102,9% | – | 0,0% | 23,1% | −50,0% | 210,1% | 36,7% | 174 | 488 | 1.077 |
Space Science | – | – | 0,0% | 32,2% | −29,4% | – | – | 9,1% | 46,2% | 8,0% | – | 563 | 2.105 | |
All publications | 40,0% | 6,2% | 0,0% | 34,1% | 25,5% | 29,6% | 7,7% | 31,0% | 3,9% | 35,9% | 22,2% | 9.868 | 11.345 | 7.887 |
Table 3 shows the percentage of titles in each collection of BEIC cited in articles by Chilean researchers. For example, ACS and Elsevier represent 46.9% of total titles cited in Chemistry.
The three subject categories in which the usage is better covered by BEIC journals are Immunology, Pharmacology/Toxicology and Multidisciplinary. But there are other subject categories such as Space Science or Social Science that use only 16.8% and 16.3% of BEIC titles, respectively, in its usage pattern.
Regarding the output levels of the 27 institutions of CINCEL analysed in this study, three categories were defined to compare the publication output and focus: high (five institutions with more than 5% of the total publication), medium (six institutions that publish between 2% and 5%) and low (16 institutions with less than 2%).
Figures 3 and 4 show the publication and usage distribution in these three categories for journals indexed in JCR, SciELO-Chile and non-indexed titles.
Area (usage) | BEIC publishers (usage) | No | Total | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
AAAS | ACS | AnnRev | Blackwell | Elsev | NPG | OUP | Springer | Wiley | |||
Agricultural Sciences | 0.3% | 4.3% | 0.3% | 7.1% | 0.5% | 1.4% | 6.2% | 2.3% | 56.9% | 100% | |
Biology & Biochemistry | 14% | 2.2% | 0.8% | 6.7% | 18.2% | 3.1% | 2.3% | 5.7% | 3.6% | 55.9% | 100% |
Chemistry | 0.6% | 19.0% | 0.1% | 1.6% | 27.9% | 0.6% | 0.3% | 3.7% | 7.2% | 38.9% | 100% |
Clinical Medicine | 0.5% | 0.3% | 0.3% | 6.5% | 16.4% | 1.3% | 2.5% | 4.8% | 3.7% | 63.8% | 100% |
Computer Science | 0.2% | 0.2% | 0.0% | 0.6% | 10.6% | 0.2% | 0.5% | 10.8% | 1.3% | 75.5% | 100% |
Economics & Business | 0.1% | 0.0% | 0.1% | 9.4% | 12.7% | 0.1% | 1.2% | 3.0% | 1.5% | 72.0% | 100% |
Engineering | 0.3% | 1.6% | 0.1% | 1.4% | 17.2% | 0.2% | 0.3% | 4.6% | 2.7% | 71.5% | 100% |
Environment/Ecology | 1.5% | 1.2% | 0.3% | 10.7% | 15.6% | 1.4% | 1.3% | 8.1% | 1.0% | 58.9% | 100% |
Geosciences | 1.7% | 0.4% | 0.3% | 3.2% | 19.4% | 1.6% | 1.1% | 4.6% | 1.1% | 66.8% | 100% |
Human Sciences | 0.7% | 0.0% | 0.2% | 4.7% | 6.2% | 0.6% | 0.8% | 3.2% | 0.9% | 82.6% | 100% |
Immunology | 1.2% | 0.3% | 1.6% | 7.7% | 17.6% | 3.9% | 1.7% | 2.6% | 3.8% | 59.6% | 100% |
Materials Science | 1.0% | 4.5% | 0.3% | 2.6% | 27.8% | 0.9% | 0.5% | 5.3% | 3.7% | 53.4% | 100% |
Mathematics | 0.1% | 0.0% | 0.0% | 1.6% | 16.6% | 0.1% | 1.5% | 12.5% | 2.4% | 65.2% | 100% |
Microbiology | 1.0% | 1.0% | 0.8% | 8.1% | 16.1% | 2.1% | 2.8% | 6.7% | 1.7% | 59.8% | 100% |
Molecular Biology & Genetics | 2.0% | 0.6% | 1.1% | 5.0% | 16.1% | 6.0% | 2.8% | 4.2% | 5.0% | 57.3% | 100% |
Multidisciplinary | 3.6% | 0.5% | 1.2% | 7.4% | 10.7% | 8.0% | 2.2% | 4.0% | 1.7% | 60.7% | 100% |
Neuroscience & Behavior | 2.1% | 0.3% | 0.9% | 7.1% | 25.4% | 5.1% | 1.3% | 4.0% | 5.6% | 48.2% | 100% |
Pharmacology & Toxicology | 0.9% | 2.6% | 0.8% | 5.0% | 25.4% | 2.2% | 1.6% | 4.2% | 3.2% | 54.1% | 100% |
Physics | 0.7% | 0.8% | 0.2% | 0.8% | 20.3% | 1.3% | 0.1% | 4.5% | 0.8% | 70.4% | 100% |
Plant & Animal Science | 0.9% | 0.5% | 0.3% | 8.8% | 12.6% | 1.1% | 2.5% | 7.7% | 0.8% | 64.8% | 100% |
Psychiatry/Psychology | 0.4% | 0.2% | 0.4% | 5.7% | 10.6% | 0.5% | 1.1% | 5.0% | 2.5% | 73.6% | 100% |
Social Sciences, general | 0.3% | 0.0% | 0.2% | 4.4% | 7.0% | 0.2% | 1.0% | 2.3% | 1.0% | 83.7% | 100% |
Space Science | 0.5% | 0.0% | 1.1% | 10.9% | 1.6% | 1.5% | 0.0% | 1.0% | 0.2% | 83.2% | 100% |
The institutions with a low publication level have a greater percentage of articles in SciELO-Chile. The other two categories concentrate their publication in the first two quartiles of JCR.
By usage and by JCR quartiles, the concentration is in the first quartile and in non-indexed titles. The institutions with a high publication level use fewer SciELO journals.
Conclusions: some lessons learned
The ‘big deal’ problem
The acquisition of journal packages is based on the economic logic of cost containment. But the global skewing of the scientific communication cycle is on the verge of destroying that model, and there seem to be no promising alternatives. At this stage, for countries like ours – marginal science producers – confronting the big oligopolies forces us to acknowledge that we are ‘piece takers’, as a classic economist would say (although in this market it is an imperfect extreme). At the same time comes the realization that there are few available tools to level the playing field. It is an inescapable fact that a publisher like Elsevier represents 60% of article downloads and 25% of citations in Chile. It is impossible not to include it in the design of any public policy on access to scientific information that aims to meet the needs of end users.
“… confronting the big oligopolies forces us to acknowledge that we are ‘piece takers’ …”
Understanding of the national cycle of scientific communication in order to optimize it
Chile has formed part of the SciELO network since 1998, and CONICYT has designed incentives to promote publication in these journals. The scene is then open to complementary policies, such as the mandate to deposit pre-prints in institutional repositories if the research has been funded with public resources or under a policy of research data access. Discussions in the USA and Europe regarding open access and its several routes (the ‘green’, which is the obvious, the ‘gold’, proposed by the Finch Report, or the ‘blue’ that is the preferred by the medical societies of the USA) are being closely followed, but we know that in our country part of the problem is already resolved because the government, through CONICYT, is the body that finances SciELO-Chile.
Evaluation for better decision-making
In our public system, evaluation is fashionable. If in the past our concern was the input (how much we spent, how many beneficiaries we reached, what forms of delivery were the most efficient), now we are more concerned by output (Is what we do useful? Can we improve? Is it a contribution to people's life quality? Is it the better solution for the problem or will it generate other unexpected consequential problems?).
Evaluation always involves a shake-up of the assumptions on which we base our actions and it is not easy to undertake in hierarchical, bureaucratic organizations which are subject to political control, such as public bodies. However, if we take the role of the public agency in the creation of value seriously, it is impossible to improve without an evaluation process. The result is a strengthening of the impact metrics, making them even more consistent and transparent, because we have to continue proving that access is a crucial element in the improvement of the scientific performance of individuals and institutions. This also includes strengthening the capacity of public agencies to design appropriate studies, including ensuring that staff have sufficient skills to interpret the data and translate them into public policy adjustments that can be implemented by the authority.
“Evaluation always involves a shake-up of the assumptions on which we base our actions …”
Finally, in Chile we still have a long way to go in defining an institutional framework capable of dealing with the current problems of scientific research and how to become even more collaborative and cross-disciplinary, and that has the merit of being conducive to strengthening the steps we are taking today, while ensuring the scientific basis of the future.