Cell Reports (Apr 2018)

Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images

  • Joel Saltz,
  • Rajarsi Gupta,
  • Le Hou,
  • Tahsin Kurc,
  • Pankaj Singh,
  • Vu Nguyen,
  • Dimitris Samaras,
  • Kenneth R. Shroyer,
  • Tianhao Zhao,
  • Rebecca Batiste,
  • John Van Arnam,
  • Ilya Shmulevich,
  • Arvind U.K. Rao,
  • Alexander J. Lazar,
  • Ashish Sharma,
  • Vésteinn Thorsson,
  • Samantha J. Caesar-Johnson,
  • John A. Demchok,
  • Ina Felau,
  • Melpomeni Kasapi,
  • Martin L. Ferguson,
  • Carolyn M. Hutter,
  • Heidi J. Sofia,
  • Roy Tarnuzzer,
  • Zhining Wang,
  • Liming Yang,
  • Jean C. Zenklusen,
  • Jiashan (Julia) Zhang,
  • Sudha Chudamani,
  • Jia Liu,
  • Laxmi Lolla,
  • Rashi Naresh,
  • Todd Pihl,
  • Qiang Sun,
  • Yunhu Wan,
  • Ye Wu,
  • Juok Cho,
  • Timothy DeFreitas,
  • Scott Frazer,
  • Nils Gehlenborg,
  • Gad Getz,
  • David I. Heiman,
  • Jaegil Kim,
  • Michael S. Lawrence,
  • Pei Lin,
  • Sam Meier,
  • Michael S. Noble,
  • Gordon Saksena,
  • Doug Voet,
  • Hailei Zhang,
  • Brady Bernard,
  • Nyasha Chambwe,
  • Varsha Dhankani,
  • Theo Knijnenburg,
  • Roger Kramer,
  • Kalle Leinonen,
  • Yuexin Liu,
  • Michael Miller,
  • Sheila Reynolds,
  • Ilya Shmulevich,
  • Vesteinn Thorsson,
  • Wei Zhang,
  • Rehan Akbani,
  • Bradley M. Broom,
  • Apurva M. Hegde,
  • Zhenlin Ju,
  • Rupa S. Kanchi,
  • Anil Korkut,
  • Jun Li,
  • Han Liang,
  • Shiyun Ling,
  • Wenbin Liu,
  • Yiling Lu,
  • Gordon B. Mills,
  • Kwok-Shing Ng,
  • Arvind Rao,
  • Michael Ryan,
  • Jing Wang,
  • John N. Weinstein,
  • Jiexin Zhang,
  • Adam Abeshouse,
  • Joshua Armenia,
  • Debyani Chakravarty,
  • Walid K. Chatila,
  • Ino de Bruijn,
  • Jianjiong Gao,
  • Benjamin E. Gross,
  • Zachary J. Heins,
  • Ritika Kundra,
  • Konnor La,
  • Marc Ladanyi,
  • Augustin Luna,
  • Moriah G. Nissan,
  • Angelica Ochoa,
  • Sarah M. Phillips,
  • Ed Reznik,
  • Francisco Sanchez-Vega,
  • Chris Sander,
  • Nikolaus Schultz,
  • Robert Sheridan,
  • S. Onur Sumer,
  • Yichao Sun,
  • Barry S. Taylor,
  • Jioajiao Wang,
  • Hongxin Zhang,
  • Pavana Anur,
  • Myron Peto,
  • Paul Spellman,
  • Christopher Benz,
  • Joshua M. Stuart,
  • Christopher K. Wong,
  • Christina Yau,
  • D. Neil Hayes,
  • Joel S. Parker,
  • Matthew D. Wilkerson,
  • Adrian Ally,
  • Miruna Balasundaram,
  • Reanne Bowlby,
  • Denise Brooks,
  • Rebecca Carlsen,
  • Eric Chuah,
  • Noreen Dhalla,
  • Robert Holt,
  • Steven J.M. Jones,
  • Katayoon Kasaian,
  • Darlene Lee,
  • Yussanne Ma,
  • Marco A. Marra,
  • Michael Mayo,
  • Richard A. Moore,
  • Andrew J. Mungall,
  • Karen Mungall,
  • A. Gordon Robertson,
  • Sara Sadeghi,
  • Jacqueline E. Schein,
  • Payal Sipahimalani,
  • Angela Tam,
  • Nina Thiessen,
  • Kane Tse,
  • Tina Wong,
  • Ashton C. Berger,
  • Rameen Beroukhim,
  • Andrew D. Cherniack,
  • Carrie Cibulskis,
  • Stacey B. Gabriel,
  • Galen F. Gao,
  • Gavin Ha,
  • Matthew Meyerson,
  • Steven E. Schumacher,
  • Juliann Shih,
  • Melanie H. Kucherlapati,
  • Raju S. Kucherlapati,
  • Stephen Baylin,
  • Leslie Cope,
  • Ludmila Danilova,
  • Moiz S. Bootwalla,
  • Phillip H. Lai,
  • Dennis T. Maglinte,
  • David J. Van Den Berg,
  • Daniel J. Weisenberger,
  • J. Todd Auman,
  • Saianand Balu,
  • Tom Bodenheimer,
  • Cheng Fan,
  • Katherine A. Hoadley,
  • Alan P. Hoyle,
  • Stuart R. Jefferys,
  • Corbin D. Jones,
  • Shaowu Meng,
  • Piotr A. Mieczkowski,
  • Lisle E. Mose,
  • Amy H. Perou,
  • Charles M. Perou,
  • Jeffrey Roach,
  • Yan Shi,
  • Janae V. Simons,
  • Tara Skelly,
  • Matthew G. Soloway,
  • Donghui Tan,
  • Umadevi Veluvolu,
  • Huihui Fan,
  • Toshinori Hinoue,
  • Peter W. Laird,
  • Hui Shen,
  • Wanding Zhou,
  • Michelle Bellair,
  • Kyle Chang,
  • Kyle Covington,
  • Chad J. Creighton,
  • Huyen Dinh,
  • HarshaVardhan Doddapaneni,
  • Lawrence A. Donehower,
  • Jennifer Drummond,
  • Richard A. Gibbs,
  • Robert Glenn,
  • Walker Hale,
  • Yi Han,
  • Jianhong Hu,
  • Viktoriya Korchina,
  • Sandra Lee,
  • Lora Lewis,
  • Wei Li,
  • Xiuping Liu,
  • Margaret Morgan,
  • Donna Morton,
  • Donna Muzny,
  • Jireh Santibanez,
  • Margi Sheth,
  • Eve Shinbrot,
  • Linghua Wang,
  • Min Wang,
  • David A. Wheeler,
  • Liu Xi,
  • Fengmei Zhao,
  • Julian Hess,
  • Elizabeth L. Appelbaum,
  • Matthew Bailey,
  • Matthew G. Cordes,
  • Li Ding,
  • Catrina C. Fronick,
  • Lucinda A. Fulton,
  • Robert S. Fulton,
  • Cyriac Kandoth,
  • Elaine R. Mardis,
  • Michael D. McLellan,
  • Christopher A. Miller,
  • Heather K. Schmidt,
  • Richard K. Wilson,
  • Daniel Crain,
  • Erin Curley,
  • Johanna Gardner,
  • Kevin Lau,
  • David Mallery,
  • Scott Morris,
  • Joseph Paulauskis,
  • Robert Penny,
  • Candace Shelton,
  • Troy Shelton,
  • Mark Sherman,
  • Eric Thompson,
  • Peggy Yena,
  • Jay Bowen,
  • Julie M. Gastier-Foster,
  • Mark Gerken,
  • Kristen M. Leraas,
  • Tara M. Lichtenberg,
  • Nilsa C. Ramirez,
  • Lisa Wise,
  • Erik Zmuda,
  • Niall Corcoran,
  • Tony Costello,
  • Christopher Hovens,
  • Andre L. Carvalho,
  • Ana C. de Carvalho,
  • José H. Fregnani,
  • Adhemar Longatto-Filho,
  • Rui M. Reis,
  • Cristovam Scapulatempo-Neto,
  • Henrique C.S. Silveira,
  • Daniel O. Vidal,
  • Andrew Burnette,
  • Jennifer Eschbacher,
  • Beth Hermes,
  • Ardene Noss,
  • Rosy Singh,
  • Matthew L. Anderson,
  • Patricia D. Castro,
  • Michael Ittmann,
  • David Huntsman,
  • Bernard Kohl,
  • Xuan Le,
  • Richard Thorp,
  • Chris Andry,
  • Elizabeth R. Duffy,
  • Vladimir Lyadov,
  • Oxana Paklina,
  • Galiya Setdikova,
  • Alexey Shabunin,
  • Mikhail Tavobilov,
  • Christopher McPherson,
  • Ronald Warnick,
  • Ross Berkowitz,
  • Daniel Cramer,
  • Colleen Feltmate,
  • Neil Horowitz,
  • Adam Kibel,
  • Michael Muto,
  • Chandrajit P. Raut,
  • Andrei Malykh,
  • Jill S. Barnholtz-Sloan,
  • Wendi Barrett,
  • Karen Devine,
  • Jordonna Fulop,
  • Quinn T. Ostrom,
  • Kristen Shimmel,
  • Yingli Wolinsky,
  • Andrew E. Sloan,
  • Agostino De Rose,
  • Felice Giuliante,
  • Marc Goodman,
  • Beth Y. Karlan,
  • Curt H. Hagedorn,
  • John Eckman,
  • Jodi Harr,
  • Jerome Myers,
  • Kelinda Tucker,
  • Leigh Anne Zach,
  • Brenda Deyarmin,
  • Hai Hu,
  • Leonid Kvecher,
  • Caroline Larson,
  • Richard J. Mural,
  • Stella Somiari,
  • Ales Vicha,
  • Tomas Zelinka,
  • Joseph Bennett,
  • Mary Iacocca,
  • Brenda Rabeno,
  • Patricia Swanson,
  • Mathieu Latour,
  • Louis Lacombe,
  • Bernard Têtu,
  • Alain Bergeron,
  • Mary McGraw,
  • Susan M. Staugaitis,
  • John Chabot,
  • Hanina Hibshoosh,
  • Antonia Sepulveda,
  • Tao Su,
  • Timothy Wang,
  • Olga Potapova,
  • Olga Voronina,
  • Laurence Desjardins,
  • Odette Mariani,
  • Sergio Roman-Roman,
  • Xavier Sastre,
  • Marc-Henri Stern,
  • Feixiong Cheng,
  • Sabina Signoretti,
  • Andrew Berchuck,
  • Darell Bigner,
  • Eric Lipp,
  • Jeffrey Marks,
  • Shannon McCall,
  • Roger McLendon,
  • Angeles Secord,
  • Alexis Sharp,
  • Madhusmita Behera,
  • Daniel J. Brat,
  • Amy Chen,
  • Keith Delman,
  • Seth Force,
  • Fadlo Khuri,
  • Kelly Magliocca,
  • Shishir Maithel,
  • Jeffrey J. Olson,
  • Taofeek Owonikoko,
  • Alan Pickens,
  • Suresh Ramalingam,
  • Dong M. Shin,
  • Gabriel Sica,
  • Erwin G. Van Meir,
  • Hongzheng Zhang,
  • Wil Eijckenboom,
  • Ad Gillis,
  • Esther Korpershoek,
  • Leendert Looijenga,
  • Wolter Oosterhuis,
  • Hans Stoop,
  • Kim E. van Kessel,
  • Ellen C. Zwarthoff,
  • Chiara Calatozzolo,
  • Lucia Cuppini,
  • Stefania Cuzzubbo,
  • Francesco DiMeco,
  • Gaetano Finocchiaro,
  • Luca Mattei,
  • Alessandro Perin,
  • Bianca Pollo,
  • Chu Chen,
  • John Houck,
  • Pawadee Lohavanichbutr,
  • Arndt Hartmann,
  • Christine Stoehr,
  • Robert Stoehr,
  • Helge Taubert,
  • Sven Wach,
  • Bernd Wullich,
  • Witold Kycler,
  • Dawid Murawa,
  • Maciej Wiznerowicz,
  • Ki Chung,
  • W. Jeffrey Edenfield,
  • Julie Martin,
  • Eric Baudin,
  • Glenn Bubley,
  • Raphael Bueno,
  • Assunta De Rienzo,
  • William G. Richards,
  • Steven Kalkanis,
  • Tom Mikkelsen,
  • Houtan Noushmehr,
  • Lisa Scarpace,
  • Nicolas Girard,
  • Marta Aymerich,
  • Elias Campo,
  • Eva Giné,
  • Armando López Guillermo,
  • Nguyen Van Bang,
  • Phan Thi Hanh,
  • Bui Duc Phu,
  • Yufang Tang,
  • Howard Colman,
  • Kimberley Evason,
  • Peter R. Dottino,
  • John A. Martignetti,
  • Hani Gabra,
  • Hartmut Juhl,
  • Teniola Akeredolu,
  • Serghei Stepa,
  • Dave Hoon,
  • Keunsoo Ahn,
  • Koo Jeong Kang,
  • Felix Beuschlein,
  • Anne Breggia,
  • Michael Birrer,
  • Debra Bell,
  • Mitesh Borad,
  • Alan H. Bryce,
  • Erik Castle,
  • Vishal Chandan,
  • John Cheville,
  • John A. Copland,
  • Michael Farnell,
  • Thomas Flotte,
  • Nasra Giama,
  • Thai Ho,
  • Michael Kendrick,
  • Jean-Pierre Kocher,
  • Karla Kopp,
  • Catherine Moser,
  • David Nagorney,
  • Daniel O’Brien,
  • Brian Patrick O’Neill,
  • Tushar Patel,
  • Gloria Petersen,
  • Florencia Que,
  • Michael Rivera,
  • Lewis Roberts,
  • Robert Smallridge,
  • Thomas Smyrk,
  • Melissa Stanton,
  • R. Houston Thompson,
  • Michael Torbenson,
  • Ju Dong Yang,
  • Lizhi Zhang,
  • Fadi Brimo,
  • Jaffer A. Ajani,
  • Ana Maria Angulo Gonzalez,
  • Carmen Behrens,
  • Jolanta Bondaruk,
  • Russell Broaddus,
  • Bogdan Czerniak,
  • Bita Esmaeli,
  • Junya Fujimoto,
  • Jeffrey Gershenwald,
  • Charles Guo,
  • Alexander J. Lazar,
  • Christopher Logothetis,
  • Funda Meric-Bernstam,
  • Cesar Moran,
  • Lois Ramondetta,
  • David Rice,
  • Anil Sood,
  • Pheroze Tamboli,
  • Timothy Thompson,
  • Patricia Troncoso,
  • Anne Tsao,
  • Ignacio Wistuba,
  • Candace Carter,
  • Lauren Haydu,
  • Peter Hersey,
  • Valerie Jakrot,
  • Hojabr Kakavand,
  • Richard Kefford,
  • Kenneth Lee,
  • Georgina Long,
  • Graham Mann,
  • Michael Quinn,
  • Robyn Saw,
  • Richard Scolyer,
  • Kerwin Shannon,
  • Andrew Spillane,
  • onathan Stretch,
  • Maria Synott,
  • John Thompson,
  • James Wilmott,
  • Hikmat Al-Ahmadie,
  • Timothy A. Chan,
  • Ronald Ghossein,
  • Anuradha Gopalan,
  • Douglas A. Levine,
  • Victor Reuter,
  • Samuel Singer,
  • Bhuvanesh Singh,
  • Nguyen Viet Tien,
  • Thomas Broudy,
  • Cyrus Mirsaidi,
  • Praveen Nair,
  • Paul Drwiega,
  • Judy Miller,
  • Jennifer Smith,
  • Howard Zaren,
  • Joong-Won Park,
  • Nguyen Phi Hung,
  • Electron Kebebew,
  • W. Marston Linehan,
  • Adam R. Metwalli,
  • Karel Pacak,
  • Peter A. Pinto,
  • Mark Schiffman,
  • Laura S. Schmidt,
  • Cathy D. Vocke,
  • Nicolas Wentzensen,
  • Robert Worrell,
  • Hannah Yang,
  • Marc Moncrieff,
  • Chandra Goparaju,
  • Jonathan Melamed,
  • Harvey Pass,
  • Natalia Botnariuc,
  • Irina Caraman,
  • Mircea Cernat,
  • Inga Chemencedji,
  • Adrian Clipca,
  • Serghei Doruc,
  • Ghenadie Gorincioi,
  • Sergiu Mura,
  • Maria Pirtac,
  • Irina Stancul,
  • Diana Tcaciuc,
  • Monique Albert,
  • Iakovina Alexopoulou,
  • Angel Arnaout,
  • John Bartlett,
  • Jay Engel,
  • Sebastien Gilbert,
  • Jeremy Parfitt,
  • Harman Sekhon,
  • George Thomas,
  • Doris M. Rassl,
  • Robert C. Rintoul,
  • Carlo Bifulco,
  • Raina Tamakawa,
  • Walter Urba,
  • Nicholas Hayward,
  • Henri Timmers,
  • Anna Antenucci,
  • Francesco Facciolo,
  • Gianluca Grazi,
  • Mirella Marino,
  • Roberta Merola,
  • Ronald de Krijger,
  • Anne-Paule Gimenez-Roqueplo,
  • Alain Piché,
  • Simone Chevalier,
  • Ginette McKercher,
  • Kivanc Birsoy,
  • Gene Barnett,
  • Cathy Brewer,
  • Carol Farver,
  • Theresa Naska,
  • Nathan A. Pennell,
  • Daniel Raymond,
  • Cathy Schilero,
  • Kathy Smolenski,
  • Felicia Williams,
  • Carl Morrison,
  • Jeffrey A. Borgia,
  • Michael J. Liptay,
  • Mark Pool,
  • Christopher W. Seder,
  • Kerstin Junker,
  • Larsson Omberg,
  • Mikhail Dinkin,
  • George Manikhas,
  • Domenico Alvaro,
  • Maria Consiglia Bragazzi,
  • Vincenzo Cardinale,
  • Guido Carpino,
  • Eugenio Gaudio,
  • David Chesla,
  • Sandra Cottingham,
  • Michael Dubina,
  • Fedor Moiseenko,
  • Renumathy Dhanasekaran,
  • Karl-Friedrich Becker,
  • Klaus-Peter Janssen,
  • Julia Slotta-Huspenina,
  • Mohamed H. Abdel-Rahman,
  • Dina Aziz,
  • Sue Bell,
  • Colleen M. Cebulla,
  • Amy Davis,
  • Rebecca Duell,
  • J. Bradley Elder,
  • Joe Hilty,
  • Bahavna Kumar,
  • James Lang,
  • Norman L. Lehman,
  • Randy Mandt,
  • Phuong Nguyen,
  • Robert Pilarski,
  • Karan Rai,
  • Lynn Schoenfield,
  • Kelly Senecal,
  • Paul Wakely,
  • Paul Hansen,
  • Ronald Lechan,
  • James Powers,
  • Arthur Tischler,
  • William E. Grizzle,
  • Katherine C. Sexton,
  • Alison Kastl,
  • Joel Henderson,
  • Sima Porten,
  • Jens Waldmann,
  • Martin Fassnacht,
  • Sylvia L. Asa,
  • Dirk Schadendorf,
  • Marta Couce,
  • Markus Graefen,
  • Hartwig Huland,
  • Guido Sauter,
  • Thorsten Schlomm,
  • Ronald Simon,
  • Pierre Tennstedt,
  • Oluwole Olabode,
  • Mark Nelson,
  • Oliver Bathe,
  • Peter R. Carroll,
  • June M. Chan,
  • Philip Disaia,
  • Pat Glenn,
  • Robin K. Kelley,
  • Charles N. Landen,
  • Joanna Phillips,
  • Michael Prados,
  • Jeffry Simko,
  • Karen Smith-McCune,
  • Scott VandenBerg,
  • Kevin Roggin,
  • Ashley Fehrenbach,
  • Ady Kendler,
  • Suzanne Sifri,
  • Ruth Steele,
  • Antonio Jimeno,
  • Francis Carey,
  • Ian Forgie,
  • Massimo Mannelli,
  • Michael Carney,
  • Brenda Hernandez,
  • Benito Campos,
  • Christel Herold-Mende,
  • Christin Jungk,
  • Andreas Unterberg,
  • Andreas von Deimling,
  • Aaron Bossler,
  • Joseph Galbraith,
  • Laura Jacobus,
  • Michael Knudson,
  • Tina Knutson,
  • Deqin Ma,
  • Mohammed Milhem,
  • Rita Sigmund,
  • Andrew K. Godwin,
  • Rashna Madan,
  • Howard G. Rosenthal,
  • Clement Adebamowo,
  • Sally N. Adebamowo,
  • Alex Boussioutas,
  • David Beer,
  • Thomas Giordano,
  • Anne-Marie Mes-Masson,
  • Fred Saad,
  • Therese Bocklage,
  • Lisa Landrum,
  • Robert Mannel,
  • Kathleen Moore,
  • Katherine Moxley,
  • Russel Postier,
  • Joan Walker,
  • Rosemary Zuna,
  • Michael Feldman,
  • Federico Valdivieso,
  • Rajiv Dhir,
  • James Luketich,
  • Edna M. Mora Pinero,
  • Mario Quintero-Aguilo,
  • Carlos Gilberto Carlotti, Jr.,
  • Jose Sebastião Dos Santos,
  • Rafael Kemp,
  • Ajith Sankarankuty,
  • Daniela Tirapelli,
  • James Catto,
  • Kathy Agnew,
  • Elizabeth Swisher,
  • Jenette Creaney,
  • Bruce Robinson,
  • Carl Simon Shelley,
  • Eryn M. Godwin,
  • Sara Kendall,
  • Cassaundra Shipman,
  • Carol Bradford,
  • Thomas Carey,
  • Andrea Haddad,
  • Jeffey Moyer,
  • Lisa Peterson,
  • Mark Prince,
  • Laura Rozek,
  • Gregory Wolf,
  • Rayleen Bowman,
  • Kwun M. Fong,
  • Ian Yang,
  • Robert Korst,
  • W. Kimryn Rathmell,
  • J. Leigh Fantacone-Campbell,
  • Jeffrey A. Hooke,
  • Albert J. Kovatich,
  • Craig D. Shriver,
  • John DiPersio,
  • Bettina Drake,
  • Ramaswamy Govindan,
  • Sharon Heath,
  • Timothy Ley,
  • Brian Van Tine,
  • Peter Westervelt,
  • Mark A. Rubin,
  • Jung Il Lee,
  • Natália D. Aredes,
  • Armaz Mariamidze

Journal volume & issue
Vol. 23, no. 1
pp. 181 – 193.e7

Abstract

Read online

Summary: Beyond sample curation and basic pathologic characterization, the digitized H&E-stained images of TCGA samples remain underutilized. To highlight this resource, we present mappings of tumor-infiltrating lymphocytes (TILs) based on H&E images from 13 TCGA tumor types. These TIL maps are derived through computational staining using a convolutional neural network trained to classify patches of images. Affinity propagation revealed local spatial structure in TIL patterns and correlation with overall survival. TIL map structural patterns were grouped using standard histopathological parameters. These patterns are enriched in particular T cell subpopulations derived from molecular measures. TIL densities and spatial structure were differentially enriched among tumor types, immune subtypes, and tumor molecular subtypes, implying that spatial infiltrate state could reflect particular tumor cell aberration states. Obtaining spatial lymphocytic patterns linked to the rich genomic characterization of TCGA samples demonstrates one use for the TCGA image archives with insights into the tumor-immune microenvironment. : Tumor-infiltrating lymphocytes (TILs) were identified from standard pathology cancer images by a deep-learning-derived “computational stain” developed by Saltz et al. They processed 5,202 digital images from 13 cancer types. Resulting TIL maps were correlated with TCGA molecular data, relating TIL content to survival, tumor subtypes, and immune profiles. Keywords: digital pathology, immuno-oncology, machine learning, lymphocytes, tumor microenvironment, deep learning, tumor-infiltrating lymphocytes, artificial intelligence, bioinformatics, computer vision