.: Natural Language Processing

  • Natural Language Processing (NLP): part of the solution to the replicability crisis?

    ISIT's Carolyn Meinel is collaborating with researchers at Basis Technology to explore the potential of their new text vectors NLP system, Semantic Search. The figure below is a preliminary architecture diagram of how we hypothesize tha we could achieve high accuracy in predicting which social and behavioral sciences papers could be replicated.

  • Preliminary findings suggest that NLP can reveal the difference between papers that can be replicated vs not papers that will fail replication.

    Pictured below, Dr. Alex Jones' word cloud depiction of his Naive Bayes analysis of the 100 replicated papers vs papers that failed replication from Colin F. Camerer, et al, “Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015,” Nature Human Behaviour, Volume 2, pp. 637–644 (2018). https://www.nature.com/articles/s41562-018-0399-z

    This word cloud depiction shows how obviously different they are on average. The challenge is to narrow down which individucal research papers could almost certainly be replicated vs those that cannot. Dr. Jones' approach correctly identified 80% of these papers. The architecture diagram to the right is how my colleagues and I hypothesize that we might improve on Jones' impressive results, presuming that we can win a contract with DARPA's Systematizing Confidence in Open Research and Evidence (SCORE) program