An analysis of neuroscience and psychiatry papers published from 2009 and 2019 outlines opportunities for increasing discovery of sex differences -…

Our survey of 3193 papers across six journals in neuroscience and psychiatry revealed some insights into the inclusion, use, and analyses of both sexes in research in the sample years, 2009 and 2019 (Fig.6). Most papers in our sample had studies that used both males and females in 2019, a 30% increase from 2009, irrespective of discipline. On the face of it, this is a positive indication of greater knowledge and awareness on the importance of sex and gender as variables in research. However, we found the majority of papers we sampled in 2019, did not use what we consider an optimal design or analysis for the discovery of possible sex differences. This is concerning, as scientific discovery will lose out on valuable information if researchers are neglecting to embrace the power of studying potential sex differences. Specifically, we determined that out of the total number of papers sampled that reported studies using both males and females, 16.5% reported using an optimal design for discovery of sex differences. Most of the sampled papers that used both sexes (75%) either did not specify sample size, used unequal proportions of the sexes, or used the sexes inconsistently within the studies in the paper and furthermore, 58% of these papers did not include an analysis by sex. Only 6% of the total number of papers reporting studies using both sexes included sex as a discovery variable, and this value was consistent across years and disciplines.

Optimal design refers to relatively based sample size and use of males and females consistently across the experiments whereas optimal analyses refers to the use of sex as a discovery variable. Although the percentage of studies in the sample of neuroscience and psychiatry papers analysed has increased the use of optimal design and analyses has not changed as much and remain at low levels. There are nine times the percentage of male-only compared to female-only studies.

Furthermore, the percentage of papers reporting studies using optimal designs or analyses for discovery of sex differences has not meaningfully shifted between 2009 and 2019 across either discipline, despite the number of recent initiatives such as SABV, SGBA and SAGER14,16,31. It is possible that these percentages will increase with time as these mandates are relatively recent. These findings should serve as a reminder to researchers, funders and publishers, that if we are to harness the wealth of knowledge from studying the sexes, more needs to be done to improve the appropriate application of sex in reporting and analyses for discovery.

As noted, there has been an increase in the reporting of both sexes in both psychiatry and neuroscience papers in our sample to almost 70% in 2019 from 20% in 2009. The neuroscience sample showed a 50% increase in reporting the use of both sexes over the years whereas the increase between 2019 and 2009 was only 10% in the psychiatry sample. This difference between the disciplines is likely driven by the majority of papers in the psychiatry sample using humans as participants, which may be a direct result of an earlier (2001) NIH mandate to include males and females in clinical research. The majority of neuroscience and psychiatry papers in our sample reported studies that used both sexes in 2019, which is encouraging. Our finding of a 50% increase in the 2019 neuroscience sample compared to 2009 sample is higher than previous work22,23, indicating an upward trend over the years sampled. For example, past research sampling neuroscience papers reporting the use of both sexes has found an almost 20% increase between 2010201425 and a 34% increase across the same time points assessed here22. In addition, the 68% of papers reporting studies that included males and females in 2019 in our study is higher than the 52% of neuroscience papers reporting the use of both sexes in 201723, againlikely reflecting an upward trend across years. The large progress made in neuroscience across the 10 years was also noted by Woitowich22 who reported an increase to 63% in 2019 using a sampling of 20 articles from 4 journals, two of which overlapped with ours (Journal of Neuroscience and Nature Neuroscience). In the present work we sampled from 3 journals in neuroscience, similar to Meitzen and colleagues23,27 who sampled neuroscience papers in 6 journals, 3 of which overlapped with the journals we chose (Nature Neuroscience, Neuron, Journal of Neuroscience). Thus, collectively, multiple studies, using different journals and methods of sampling, consistently indicate that there is an increasing trend in papers that include males and females in their work.

Although the use of both males and females in research has been steadily increasing to include a majority of studies, research highlighting or mentioning sex differences is scarce. Why might this be? We examined whether papers were reporting studies that used optimal designs for discovery of possible sex differences. When we accounted for papers in our sample that did not disclose sample size of the sexes, used an unbalanced design or did not use both sexes throughout all the studies within the paper, we found that only 16% of papers used a design that was optimal for discovery of sex differences. Some researchers may argue that investigating both males and females is only important in the first study of the paper and thus the use of both sexes in further experiments, beyond the initial study is not required. However, there are numerous examples where a trait may not show sex differences but the neural mechanisms underlying that trait do show significant differences between males and females32,33,34,35. Thus, using males and females in one experiment does not preclude the fact that sex differences may be seen in further related experiments that uncover mechanism. The use of what we consider to be the most advantageous design for discovery of sex differences was employed in just under 20% of papers sampled in 2019. Thus, although it appears on the face of it that most papers sampled report studies using males and females, the majority of these studies do not incorporate sex in a design that we consider optimal for discovery of possible sex differences.

Our findings also demonstrated that 25% of the papers sampled that report studies using both males and females do not report sample size, consistent with the findings from Woitowich and colleagues22. Perhaps more concerning is that in the neuroscience papers sampled, this trend increased between 2009 to 2019 with almost 50% not reporting the sample size of males and females used in 2019. This trend is troubling as readers are unable to judge how effectively males and females were used in the study.

As others have reported21,22,23,26,27, most publications do not report studies that analyse by sex. In our analysis, only 6% of the papers in our total sample that reported studies that used males and females also used sex as a discovery variable, which did not increase in the 2019 compared to the 2009 sample. This translates into only 4% of all the publications examined in our sample from both years and both disciplines that used sex as a discovery variable, matching a previous estimate21. The most common statistical method for analysing sex that was used by papers in our sample was controlling for sex using a covariate. A covariate removes the linear association of the factor of sex against the dependent variable, removing any linear variation due to sex. In our view, this is in opposition to the intention of SABV or SGBA, as we believe that the goal of these mandates is not to remove the variation due to sex but to determine whether or not sex is a variable that could be causing differences in outcomes. The use of sex as a covariate can result in the reduction of power and the loss of important information when a sex difference is present36. Mersha and colleagues36 show that 26 more single-nucleotide polymorphisms (SNPs) were identified in a sex-stratified analysis compared to when sex was used as a covariate. Put another way, when sex was used as a discovery variable 47 SNPs were identified that were associated with asthma but if sex was used as a covariate only 21 SNPs were identified36. They also found that effect sizes were larger when a sex-stratified analysis was used, contrary to popular opinion that power would be negatively affected with the addition of sex as a discovery variable. Some argue that design and sample sizes are not sufficiently powered to consider sex-stratified analyses, but if the sex effects are large, or in opposing directions, the resulting power with the inclusion of sex as a discovery variable, will improve as others have demonstrated25,26,36,37. Taken together, our survey of the literature suggests that researchers are underestimating the power of using sex as a discovery variable in their research.

Similar to other reports in neuroscience and other biological disciplines23,24,27,37, we found that papers we identified as female-only (i.e., reporting studies that used females only) were a small percentage of the sample (3%)38,39. Our findings are comparable with others showing that 5% of neuroscience papers sampled were female-only in 200937 and in 201723. While the consideration of sex and gender in studies is important, in our view, single-sex studies are still needed. In particular, given the dearth of information on womens health, disparities in diagnosis7, and continued underrepresentation of women in clinical trials13, one could argue that female-only studies are needed more so than male-only studiesor at least that single-sex studies should be conducted and published in equivalent proportions. Indeed, mandates such as SABV and SGBA were instigated in part because of the lack of knowledge of how females differed in their response to treatments and disease17. There are female-specific experiences that affect female health, such as menstruation, hormonal contraceptives, pregnancy and menopause that need to be studied40,41,42,43. Unfortunately, as highlighted by the current study, the percentage of papers that use only females remains consistently low and has not increased in 2019 compared with 2009 in our sample. Funders and researchers should work to correct this imbalance.

Our current study indicated that the rationale for excluding females used most often was to reduce variability. To exclude females based on greater variation than males is not valid, as two studies have found that the within-sex variability, on a variety of measures, is not statistically different between males and females in rats and mice44,45. Moreover, our findings and others37, reveal that it is a common belief that females will have more variability due to their hormones, however it is important to note that both males and females have diurnal fluctuations in cortisol or corticosterone in humans and rodents, respectively46,47. Furthermore, human males have diurnal fluctuations in testosterone levels that vary significantly with age46 Researchers should be encouraged to consider that many hormones vary with diet, age, housing conditions, and experience across both sexes48,49,50. Thus, variability within females should not be considered a limiting factor to the use of females in research51.

There have been calls in the literature for editors and reviewers of manuscripts to ensure that published reports use both males and females and report on outcomes51. SAGER guidelines were developed by the European Association of Science Editors to improve sex and gender in research reporting in 201631, and indeed, some journals have adopted SAGER guidelines including over 500 Elsevier journals52. Among the guidelines, it is recommended that authors include the sex or gender in the title and abstract, background information on sex and gender effects on the variables of interest in the paper and in the results to disaggregate and analyse the data by sex or gender. However, the percentage of journals that have adopted SAGER is low with one study finding under 10% of journals in psychology had adopted the guidelines53 and in those journals the guidelines were only adopted for the title, abstract and methods but not on reporting of analyses or data by sex or gender53. However, as can be seen from the present data, the publishing of this information, particularly with respect to the analyses of sex as a discovery variable is limited, and a more concerted effort needs to be adopted.

We note several limitations to this work. We only examined three journals for each of the two disciplines, however we did a comprehensive search of eligible research papers within each journal, culminating in over 3000 papers reviewed. Previous studies have either surveyed 841 articles across the same 2 years22 or examined 6000 articles across 4 years27. We, like others27, selected journals based on high ranking by ISI, with some overlap in journals chosen. However, our comprehensive search of these six journals gave values that were not appreciably different from those that used fewer papers within more journals, or those that carried out analysis of more publication years. This suggests that the different survey methods used across these bodies of work yield similar results.

A final consideration is that for biomedical research at NIH, the SABV consideration was instituted in 2016, which may not have given enough time to fully realise the potential in the 2019 sample examined. However, the fact that, in the neuroscience journals sampled here, there was a 70% increase in 2019 in the percentage of papers with studies using both males and females, suggests there is greater inclusion of studies using males and females. However, this increased use of both sexes in studies is unfortunately not yet resulting in using sex as a discovery variable in analyses.

Given that there is excellent uptake in the use of both males and females in research, what is driving the lack of optimal design and analyses for discovery of sex differences? It seems possible that researchers themselves are not aware that they are not using best practices, perhaps due to the lack of consensus on how to use sex in analyses and the required sample size in the literature17. In one manuscript on this topic it was reported that three-quarters of researchers say they report the sex in their papers54, and 50% of these researchers said they analysed their findings by sex54. Our results from the literature survey show that although 40% of papers we sampled included analysis by sex in some fashion, only 6% used sex as a discovery variable. Taken together, the findings here, along with prior data5, suggest that researchers may be considering analyses that are suboptimal or not reporting analyses even when they have done them. Thus, it is possible that researchers believe that the addition of both sexes without thorough analyses is enough to satisfy the initiatives.

Researchers themselves may need more training in sex and gender analyses. Qualitative analyses from structured interviews with US-based researchers55 found that while researchers indicated they had a good knowledge of SABV they incorrectly used the terms sex and gender when discussing their views, indicating a lack of knowledge. The misuse of the terms sex and gender has also been noted in grant submissions as well as in the biological literature56,57,58. Gender is a psychosocial construct that includes gender identity and societal expectations for roles and behaviour based on gender identity. Gendered effects can be realised when considering a number of intersectional variables, such as race, ethnicity or age, along with sex and gender identity59. Thus, perhaps more training for researchers may be needed to ensure fruitful addition of sex and gender in research.

One could argue that the mandates do not go far enough and are limited to a few agencies in the EU, Canada, and the US. There are also no repercussions when authors do not publish or analyse by sex. Indeed, NIH funding did not significantly affect the percentage of papers that reported studies that analysed by sex with a net increase of just 3% (to 9%) overall23. Our data indicate that there is a non-significant increase in the sampled papers that reported studies that used sex as a discovery variable from research groups based in the US, Canada, and the EU in the 2019 sample studied compared with 2009, pointing to an overall potential benefit of the current mandates that exist in those countries. However, it is important to underscore that the percentage of these papers was low even when the research groups were based in those countries, and that there are no reporting requirements from these funding agencies.

What can funders do to promote more work on sex differences? One solution is to have funding dedicated specifically for SABV and SGBA proposals and not as a supplement to regular funding. Evidence suggests that this approach has been successful in cardiovascular research. For example, the American Heart Association (US) has dedicated funding for sex differences, and as a result sex and gender-based research and analyses in cardiovascular disease has flourished60. Our view is that funders should make these funds a significant portion of the budget to provide enough incentive to encourage researchers to think deeply about incorporation of sex in research. Dedicated funding would not only generate proposals and knowledge dedicated to the analyses of sex differences, but they would also have the by-product of creating the next generation of researchers that integrate sex into their research. One can also look at how significant funding to amyotrophic lateral sclerosis (ALS) and AIDS has advanced research in these areas. In 2014, the ice bucket challenge raised greater than $115M in the US and this attention leveraged dedicated funding from other sources tripling ALS research budgets in 5 years61. This bolus of funding doubled the number of ALS publications, led to a 50% increase in investigators interested in ALS, and has dramatically accelerated the number of clinical trials in ALS62. Scientific evidence takes time to build, but fruits of discovery with the increased funding are paying off with promising new treatments63. Its hard not to get excited about the possibilities if this type of funding is extended to fill the sex disparities in health research. AIDS research is another success story with dramatic advancements in AIDS research that came with dedicated funding. Worldwide HIV/AIDS research funding more than doubled from 2000 to 2019 to >1B64. With these dedicated funds have come advancements in therapeutics such that individuals with HIV can live relatively full lives65. To make significant progress, funders need to have dedicated funding for SABV, which would have a cascading effect to get more researchers interested in SABV, ensure consideration of sex and gender as discovery variables, increase the number of discoveries and train the next generation of SABV researchers.

What can publishers do to promote publications using sex-based analyses? When journals adopt SAGER guidelines, it is up to the authors, reviewers and editors to ensure the guidelines are met. In over a third of submissions to a neuroendocrinology journal, authors and reviewers failed to notice that neither sex nor gender had been disclosed66. This suggests, not surprisingly, that not every reviewer is prompted to think about the consideration of sex in experimental design and analyses upon reviewing a paper. Training modules from funders or scholarly organisations with an SABV focus may help, but working on a similar premise as above, enticing researchers to explore the influence of sex and gender in their data may be a more fruitful approach. If journals, especially those with higher visibility, adopt calls for papers using sex and gender-based analyses this may serve as a catalyst to ensure more researchers consider possible sex differences and further promote the notion that this research is important to publish.

Lastly, others have found that the presence of inferred-female first or last authors (inferred from names) was associated with the use and analyses of both sexes in research28,29 consistent with our own data. Recently, there have been concerted efforts to promote diversity in science67 and these findings suggest that increasing sex and gender diversity among authors of scientific research is another fruitful path to improve the percentage of papers reporting use of sex as a discovery factor in analyses.

We hope these data are a call to the research community to not only include males and females in their research but to ensure appropriate methods of integration and analyses are used as well. If researchers are merely including a few animals of the opposite sex in one of many experiments this will not allow for discovery of the impact of sex as a biological variable. Nor will the non-robust adoption of sex in experiments harness the additional power that the analyses of sex can afford36. Research shows us that the use of sex as a discovery variable can lead to fruitful knowledge, and can enable conclusions that the different mechanisms between males and females require distinct treatment25. Indeed, inclusion of sex in analyses and design will improve not only the health of females but of males as well68. We lose collectively, not just in knowledge gained, but also in our search of more effective treatments when sex is not considered in the design and analyses of our studies. We call on funders, reviewers and researchers to recognise that sex and gender matter across all disciplines. The community needs to be aware that there are many types of sex differences19,69 and that some sex differences are revealed due to perturbations in environment, genotype, or disease19,70,71 so it is important to continually examine and analyse both sexes throughout the studies. It is imperative that more attention is paid to the appropriate design and analyses of sex and gender in the literature. We need to study how mandates can improve adherence in both study design and dissemination. To ensure precision medicine, we need the community of funders, researchers and publishers to embrace the addition of SABV, SGBA and SAGER to improve the health of women, men and gender-diverse individuals.

View post:
An analysis of neuroscience and psychiatry papers published from 2009 and 2019 outlines opportunities for increasing discovery of sex differences -...

Related Posts