Introduction

Artificial intelligence (AI) has shown promise for diagnosing prostate cancer in biopsies.^1,2,3,4,5 For example, in my Ph.D. research, we have shown that AI can grade prostate cancer at the level of experienced pathologists and can also support them during their review. However, results have generally been limited to individual studies, lacking validation in multinational settings. Competitions (“Challenges”) have been shown to be accelerators for medical imaging innovations, with the CAMELYON challenge being a prime example. Though, the impact of challenges is often hindered by a lack of reproducibility and independent validation.

For the 1.3 million new prostate cancer patients yearly, the Gleason grade of their biopsies is a crucial element for treatment planning. Pathologists characterize tumors into different Gleason growth patterns based on the histologic architecture of the tumor tissue. Based on the distribution of Gleason patterns, biopsy specimens are categorized into one of five groups, with a higher number denoting a worse outlook for the patient. Unfortunately, this process is subjective as pathologists frequently differ in their assessments. This could have significant consequences for a patient, for example leading to undergrading and overgrading of prostate cancer.

With a large team from Radboud University Medical Center, Karolinska Institute, Tampere University, Google Health, and Kaggle, we have organized the PANDA challenge: Prostate cANcer graDe Assessment using the Gleason grading system. We aimed to set the next step in automated Gleason grading for prostate cancer with this new challenge.

For the PANDA challenge, we released all training data of two major studies^4,5 on automated Gleason grading, both previously published in Lancet Oncology. The training set contained almost 11.000 prostate biopsies, with slide-level labels and label masks. During the competition, 1010 teams with a total of 1290 developers from 65 countries joined the challenge. With these numbers, the PANDA challenge is, to the best of our knowledge, the largest challenge organized for pathology to date.

The challenge results are now available online in our paper AI for diagnosis and Gleason grading of prostate cancer: the PANDA challenge as published in Nature Medicine.

Setup of the challenge

The main goal for all challenge participants was to design an algorithm that could automatically assign a Gleason grade group to a biopsy specimen. There was no limitation on the chosen methods. Still, teams had to follow some instructions to make sure they could correctly load the biopsies during evaluations.

Unique to the PANDA challenge was the way the study was set up. We wanted to make sure there was no room for “cheating,” and that results would be translatable to other datasets and settings. In many previous challenges, test data is released, and participants needed to submit their algorithms predictions to a central system. Even with the ground truth labels hidden, there is a risk that teams will tune their algorithm to the test data or even hand labeling the data themselves. Therefore, we asked teams to submit their algorithms instead. The test data was kept private at all times.

Throughout the competition, teams could request evaluations of their algorithm on the public leader board. This was done by submitting a new version of their algorithm. The algorithms were then simultaneously blindly validated on the private set. The algorithms were required to analyze 1000 biopsies within 6 hours for this process, but most algorithms needed less time than that.

After the competition ended, we selected fifteen teams to join for extensive independent validation of their algorithms on new data. The selection was based on the score on the final leaderboard and method description, and scientific contribution. The latter criteria were used to ensure we had a good representation of algorithms for the analysis.

We reproduced these fifteen algorithms fully in separate cloud systems without the original developers. This made sure we evaluated the algorithms as-is, without any additional tuning. We then applied the algorithms to the new data from the EU⁵ and the US². The challenge, combined with these additional evaluations, resulted in a total of 32.137.756 biopsies processed by the algorithms. As with the public/private leaderboard data, we kept the data from the extended validations secret. In fact, the teams did not even know that their algorithms would be analyzed on this data when they designed them. We tried to simulate “AI in the wild” with this setup, as in real-life scenarios, algorithms will also be applied to completely unseen and new data.

Crowdsourced AI

Challenges are often a powerful way of crowdsourcing new AI innovations. The PANDA challenge was no exception: Due to the scale of the competition, after ten days, one of the algorithms was already at the level of the average pathologist. In the remainder of the challenge, many teams caught up and improved further. This speed in development was also driven by extensive sharing of tips and tricks through the challenge forums.

A complete discussion of all results is out of the scope of this blog post; for that, I recommend checking our paper, which is Open Access. In the paper, we compare the algorithms on several datasets and two pathologists panels. We also dive deeper into the question of whether it’s actually feasible to evaluate algorithms across datasets with different reference standards.

Our main results from the validation phase can be summarized as follows: We found that the AI algorithms successfully generalized across different patient populations, laboratories, and reference standards. Although only trained using data from EU pathologists and never tuned to new data, the algorithms retained their performance compared to pathologists from the US. This is an exciting result as it shows that AI is ready for clinical implementation.

In the paper we report more results on all datasets and show how these algorithms hold up compared to different panels of pathologists.

Acessing the PANDA dataset

We made the entire training set of 10,616 digitized de-identified H&E stained prostate biopsies (383GB) publicly available for further non-commercial research. The data can be used under a Creative Commons BY-SA-NC 4.0 license. To adhere to the “Attribution” part of the license, we ask anyone who uses the data to cite the corresponding paper (see Acknowledgements). In addition, you can also cite the two papers^4,5 that describe the original data collection.

The latest information about downloading and using the PANDA dataset is available on the PANDA Challenge website. For now, the quickest way to access the data is through the competition on Kaggle.

Using the PANDA dataset

To quickly get started with the PANDA dataset, we made a Jupyter notebook with some examples. You can find this notebook in our Github repository or try it out directly on Kaggle.

If you want a quick start, you can read a patch from one of the slides with a few lines of Python and OpenSlide:

import os
import openslide
from IPython.display import Image, display

# Open the image (does not yet read the image into memory)
image = openslide.OpenSlide(os.path.join(data_dir, '005e66f06bce9c2e49142536caf2f6ee.tiff'))

# Read a specific region of the image starting at the upper left coordinate (x=17800, y=19500) on level 0 and extract a 256*256 pixel patch.
# At this point image data is read from the file and loaded into memory.
patch = image.read_region((17800,19500), 0, (256, 256))

# Display the image
display(patch)

# Close the opened slide after use
image.close()

After running the snippet above, you should see the following (a single patch):

Want to try a late submission?

It’s still possible to participate in the challenge and evaluate your algorithm on the internal validation set. Please go to the competition page on Kaggle for more information about these late submissions. Note that you will need to submit a working algorithm to receive a score on the test set.

Acknowledgements

What started with a small email in 2019 ended up in a vast project that spanned over two years. The PANDA challenge has been a group effort by researchers, pathologists, technicians, developers, and many others.

Author list: Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F. Steiner, Hester van Boven, Robert Vink, Christina Hulsbergen-van de Kaa, Jeroen van der Laak, Mahul B. Amin, Andrew J. Evans, Theodorus van der Kwast, Robert Allan, Peter A. Humphrey, Henrik Grönberg, Hemamali Samaratunga, Brett Delahunt, Toyonori Tsuzuki, Tomi Häkkinen, Lars Egevad, Maggie Demkin, Sohier Dane, Fraser Tan, Masi Valkonen, Greg S. Corrado, Lily Peng, Craig H. Mermel, Pekka Ruusuvuori, Geert Litjens, Martin Eklund and the PANDA Challenge consortium.

Please use the following to refer to our paper or this post:

Bulten, W., Kartasalo, K., Chen, PH.C. et al. Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge. Nat Med (2022). https://doi.org/10.1038/s41591-021-01620-2

@article{Bulten2022,
  doi = {10.1038/s41591-021-01620-2},
  url = {https://doi.org/10.1038/s41591-021-01620-2},
  year = {2022},
  month = jan,
  publisher = {Springer Science and Business Media {LLC}},
  author = {Wouter Bulten and Kimmo Kartasalo and Po-Hsuan Cameron Chen and Peter Str\"{o}m and Hans Pinckaers and Kunal Nagpal and Yuannan Cai and David F. Steiner and Hester van Boven and Robert Vink and Christina Hulsbergen-van de Kaa and Jeroen van der Laak and Mahul B. Amin and Andrew J. Evans and Theodorus van der Kwast and Robert Allan and Peter A. Humphrey and Henrik Gr\"{o}nberg and Hemamali Samaratunga and Brett Delahunt and Toyonori Tsuzuki and Tomi H\"{a}kkinen and Lars Egevad and Maggie Demkin and Sohier Dane and Fraser Tan and Masi Valkonen and Greg S. Corrado and Lily Peng and Craig H. Mermel and Pekka Ruusuvuori and Geert Litjens and Martin Eklund and {The PANDA challenge consortium}},
  title = {Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the {PANDA} challenge},
  journal = {Nature Medicine}
}

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

• Epithelium segmentation using deep learning and immunohistochemistry

  We developed a new deep learning method to segment epithelial tissue in digitized hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

References

Automated deep learning systems have shown promise in accurately grading prostate cancer^1,2. Several studies have shown that these systems can achieve pathologist-level performance, including my own research on automated Gleason grading. These developments have the potential to assisting pathologists in their diagnostic process. However, A large multi-center evaluation on diagnostic data is still missing.

Do you want to help us to improve prostate cancer diagnosis?

The PANDA challenge

Researchers and staff from Radboud Universiy Medical Center, Karolinska Institutet, Tampere University and Kaggle are organizing the PANDA challenge: Prostate cANcer graDe Assessment using the Gleason grading system. With this challenge we aim to set the next step in automated Gleason grading for prostate cancer.

For this challenge, we are releasing all training data of two major studies^1,2 on automated Gleason grading, both previously published in Lancet Oncology. The training set contains almost 11.000 prostate biopsies, with slide level labels and label masks. All submitted methods are evaluated on a private set of around 500 biopsies, split between public and private leaderboard.

The top three teams are eligble for a cash prize. The total amount of prize money available for the challenge is $25.000. Additionally, we will invite five teams to present their method the challenge workshop at MICCAI 2020.

After the challenge, the data can also be used for other research. We will release the data under a CC BY-NC-SA license. We ask challenge participants to respect an embargo until the challenge paper has been published. After the embargo is lifted, participants are free to publish their own results.

How to participate

To participate in the challenge, please go to the competition page on Kaggle for more information. All challenge data can be downloaded from the Kaggle platform.

Please signup up before July 15th to participate in the challenge. The final submission must be submitted July 22th the latest.

Join the PANDA challenge

MICCAI 2020 Workshop

The PANDA challenge is part of the MICCAI 2020 conference. During MICCAI, we will host a workshop on the challenge and the overall results. Up to five teams will be invited to present their method. We will select teams for a presentations mostly based on ranking, but can deviate to diversify in the methods presented.

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

• Epithelium segmentation using deep learning and immunohistochemistry

  We developed a new deep learning method to segment epithelial tissue in digitized hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

References

Within medicine, there is somewhat of an “AI hype.” Mainly driven by advances in deep learning, more and more studies show the potential of using artificial intelligence as a diagnostic tool. The same trend is present in the subfield I am working in. The field of pathology slowly transitions from being centered around microscopes to assessing tissue using a computer screen (i.e., digital pathology). With this change, computational models assisting the pathologist became within reach. A new field was born: computational pathology.

In recent years many studies have shown that deep learning-based tools can replicate histological tasks. For prostate cancer, the main topic of my Ph.D. project, detection models¹² have been around for some time and are slowly becoming available to the clinic.³ The next step is building AI systems that can perform prognostic tasks, such as grading cancer. The first steps have already been made in this direction. We recently published our work on AI-based grading for prostate cancer in Lancet Oncology.⁴ In the same issue, colleagues from Karolinska (Sweden) showed similar findings.⁵ These studies, and others,⁶ showed that AI systems can achieve pathologist-level performance, or even outperform pathologists (within the limits of the study setup).

Regardless of these results, the question remains how much benefit these systems actually offer to individual patients? Does outperforming pathologists, also mean that it improves the diagnostic process? Despite the merits of deep learning systems, they are also constrained by limitations that can have dramatic effects on the output of such systems. The presence of artifacts, which are very common in the field of pathology, can drastically alter a prediction of a deep learning system. A human observer would not be hindered by, for example, ink on slides or cutting artifacts, and at least flag the case as ungradable. A deep learning system without countermeasures against such artifacts could happily predict high-grade cancer, even if the tissue is benign.

In the end, AI algorithms be applied in the clinic and outside of a controlled research setting. In such applications a pathologist will be in the loop that actively uses the system, or, in the event of a fully automated system, has to sign-off on the cases. In other medical domains, such as radiology, this kind of integration has already been investigated.⁷ Interestingly, within pathology, research on pathologists actually using AI systems is limited. Moreover, most studies focus on the benefits of computer-aided detection, and not on prognostic measures such as grading cancer.

After completing our study on automated Gleason grading for prostate cancer, we were curious to what extend our system could improve the diagnostic process. In a completely new study, now available Open Acccess through Modern Pathology, we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

AI assistance for pathologists

When we developed the deep learning system, one of our design focusses was that the system should give interpretable output. Knowing the final diagnosis for a case is interesting, but it is more useful if a system can also show on what it based its decision. There are several ways to show output of a deep learning system⁸. In our case, we do this by letting the system highlight prostate glands that it finds malignant. Each detected malignant gland is marked either yellow ☗, orange ☗, or red ☗. This would make the inspection of the system’s prediction easy and should benefit the final diagnose, so we assumed.

In our Lancet Oncology publication,⁴ we showed that our deep learning system outperformed the manjority of the panel members. We, however, only compared the AI system to the panel. To investigate the integration of the AI system in the grading workflow of the pathologist, we invited the same panel to participate in a follow-up experiment. In total, 14 pathologists and residents joined our new study. The new experiment consisted of two reads: 1) the initial unassisted read as part of the Lancet Oncology publication, and 2) an AI-assisted read which occurred after a wash-out period.

The unassisted read was already performed as part of the previous study. For the assisted read, we showed our panel of pathologists a set of 160 biopsies through an online viewer. Of these biopsies, 100 were part of the previous study; this way, we could compare performance between assisted and unassisted. The minimal time between reads was three months. The remaining 60 cases were used as controls.

For the pathologists, the difference between the two reads was the AI feedback. While in the unassisted read, panel members graded cases as they would normally. In the assisted read, we showed the output of the AI system next to the original biopsy (see figure, and live example). Additionally, the biopsy-level prediction of the system was also available to the panel (grade group and Gleason score).

For all of these biopsies, the reference standard was set by three uropathologists in consensus. Panel members determined the grade group of the biopsy, which we later compared to the reference standard.

Better performance, lower variability

After the assisted read, the scores of the individual readers were compared across the two runs. As the primary metric, we used quadratically weighted Cohen’s kappa. Whereas in the unassisted read, the panel achieved a median score of 0.79, this increased to 0.87 in the AI-assisted read (an increase of almost 10%). Besides this increase, the variability of the panel members dropped; the interquartile range of the panel’s kappa values dropped from 0.11 to 0.07.

Without assistance, the AI system outperformed 10 out of 14 observers (71%). In the assisted read, this flipped, and 9 out of 14 panel members exceeded the AI. On a group level, the AI-assisted pathologists outperformed not only the unassisted reads but also the AI itself.

When split out on experience level, less experienced pathologists benefitted the most. Most of the pathologists who had more than 15 years of experience, already scored similar or better than the AI system and had less to gain in terms of diagnostic performance. Still, the AI could have helped them grade the cases faster, something we did not explicitly test.

External validation

After the experiment on the internal dataset, we performed an additional experiment to test our hypothesis on external data. It is well known that the performance of deep learning systems can degrade when applied to unseen data. Additionally, by repeating our experiment, we were interested to see if the measured effects could be reproduced.

For this external validation, we made use of the Imagebase dataset,⁹ which consisted of 87 cases graded by 24 international experts in prostate pathology. The deep learning system was applied as-is to the dataset without any normalization of the data. The experiment was repeated in the same fashion: first an unassisted read, and, after a washout period, an assisted read with the AI predictions shown next to the original biopsy.

Of the 14 panel members, 12 joined the second experiment using the Imagebase dataset. In the unassisted read, the median pairwise agreement with the Imagebase panel was 0.73 (quadratically weighted Cohen’s kappa) and 9 out of the 11 panel members (75%) scored lower than the AI system. With AI assistance, the agreement increased significantly to 0.78 (p = 0.003), with the majority of the panel now outperforming the standalone AI system (10 out of 12, 83%). In contrast to the internal experiment, improvements could be seen for all but one of the panel members, with no clear effect of experience level.

Looking beyond diagnostic performance

Besides diagnostic performance, other factors are essential to increase the adoption of AI techniques in the diagnostic process¹⁰. Pathologists are often time-constrained and have high workloads. Any algorithm has to be easy to use and ideally speed up diagnosis. More research is welcome on how we can efficiently embed these new techniques in routine practice, especially in a field that is used to analog examination through microscopes.

In our study, we did not quantitatively measure the use of our system. Though, we did ask all panel members to fill in a survey regarding the grading process and their usage of the system. A summary of some of the findings:

1. The majority of pathologists indicated that AI assistance sped up grading. It would be interesting to investigate the source of this time gain and to measure it quantitatively. Does it speed up finding the tumor? Makes it assessing tumor volumes faster?
2. The gland-level prediction of the system was found most useful. The case-level label the least. This suggests that there is an added benefit of showing AI feedback on the source level. More detailed feedback also allows for more interaction between AI systems and pathologists.
3. The majority of panel members would like to use an AI system during clinical practice.

For the other survey questions and the pathologist’s responses, please see the full paper.

Limitations, conclusion and future outlook

With our research, we aimed to set a new (small) step towards the clinical adoption of AI systems within pathology. For prostate cancer specifically, we showed that AI assistance reduced the observer variability of Gleason grading. Something that is highly desirable, as it could result in a stronger prognostic marker for individual patients and reduces the effect of the diagnosing pathologist on potential treatment decisions.

Of course, there are limitations to our study that open up many avenues of future research. For example, in our main experiment, we tested only two reads with a fixed order. While we were able to reproduce the same effect in a second experiment, additional research with larger datasets is welcome. Specifically, it would be exciting to investigate the effect of AI assistance over a more extended period. Maybe the benefit of AI assistance wears out over time, as pathologists learn from the feedback and apply this knowledge to unassisted reads. It is also plausible that pathologists become faster as they get accustomed to the feedback of the system. Ultimately, systems such as ours should be evaluated on a patient-level, which allows for new approaches such as automatically prioritizing slides.

In any case, the future is exciting, both from the AI perspective as well as on the implementation side. The tools are there to improve cancer diagnosis even further.

Example of AI feedback

Acknowledgements

Author list: Wouter Bulten, Maschenka Balkenhol, Jean-Joël Awoumou Belinga, Américo Brilhante, Aslı Çakır, Lars Egevad, Martin Eklund, Xavier Farré, Katerina Geronatsiou, Vincent Molinié, Guilherme Pereira, Paromita Roy, Günter Saile, Paulo Salles, Ewout Schaafsma, Joëlle Tschui, Anne-Marie Vos, ISUP Pathology Imagebase Expert Panel, Hester van Boven, Robert Vink, Jeroen van der Laak, Christina Hulsbergen-van der Kaa & Geert Litjens

This work was financed by a grant from the Dutch Cancer Society (KWF).

Please use the following to refer to our paper or this post:

Bulten, W., Balkenhol, M., Belinga, J.A. et al. Artificial intelligence assistance significantly improves Gleason grading of prostate biopsies by pathologists. Mod Pathol (2020). https://doi.org/10.1038/s41379-020-0640-y

Or, if you prefer BibTeX:

@article{bulten2020artificial,
  author={Bulten, Wouter and Balkenhol, Maschenka and Belinga, Jean-Jo{\"e}l Awoumou and Brilhante, Am{\'e}rico and {\c{C}}ak{\i}r, Asl{\i} and Egevad, Lars and Eklund, Martin and Farr{\'e}, Xavier and Geronatsiou, Katerina and Molini{\'e}, Vincent and Pereira, Guilherme and Roy, Paromita and Saile, G{\"u}nter and Salles, Paulo and Schaafsma, Ewout and Tschui, Jo{\"e}lle and Vos, Anne-Marie and Delahunt, Brett and Samaratunga, Hemamali and Grignon, David J. and Evans, Andrew J. and Berney, Daniel M. and Pan, Chin-Chen and Kristiansen, Glen and Kench, James G. and Oxley, Jon and Leite, Katia R. M. and McKenney, Jesse K. and Humphrey, Peter A. and Fine, Samson W. and Tsuzuki, Toyonori and Varma, Murali and Zhou, Ming and Comperat, Eva and Bostwick, David G. and Iczkowski, Kenneth A. and Magi-Galluzzi, Cristina and Srigley, John R. and Takahashi, Hiroyuki and van der Kwast, Theo and van Boven, Hester and Vink, Robert and van der Laak, Jeroen and Hulsbergen-van der Kaa, Christina and Litjens, Geert},
  title={Artificial intelligence assistance significantly improves Gleason grading of prostate biopsies by pathologists},
  journal={Modern Pathology},
  year={2020},
  month={Aug},
  day={05},
  abstract={The Gleason score is the most important prognostic marker for prostate cancer patients, but it suffers from significant observer variability. Artificial intelligence (AI) systems based on deep learning can achieve pathologist-level performance at Gleason grading. However, the performance of such systems can degrade in the presence of artifacts, foreign tissue, or other anomalies. Pathologists integrating their expertise with feedback from an AI system could result in a synergy that outperforms both the individual pathologist and the system. Despite the hype around AI assistance, existing literature on this topic within the pathology domain is limited. We investigated the value of AI assistance for grading prostate biopsies. A panel of 14 observers graded 160 biopsies with and without AI assistance. Using AI, the agreement of the panel with an expert reference standard increased significantly (quadratically weighted Cohen's kappa, 0.799 vs. 0.872; p{\thinspace}={\thinspace}0.019). On an external validation set of 87 cases, the panel showed a significant increase in agreement with a panel of international experts in prostate pathology (quadratically weighted Cohen's kappa, 0.733 vs. 0.786; p{\thinspace}={\thinspace}0.003). In both experiments, on a group-level, AI-assisted pathologists outperformed the unassisted pathologists and the standalone AI system. Our results show the potential of AI systems for Gleason grading, but more importantly, show the benefits of pathologist-AI synergy.},
  issn={1530-0285},
  doi={10.1038/s41379-020-0640-y},
  url={https://doi.org/10.1038/s41379-020-0640-y}
}

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• Epithelium segmentation using deep learning and immunohistochemistry

  We developed a new deep learning method to segment epithelial tissue in digitized hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

References

At first, we trained a system using the conventional way: based on human annotations, we trained a U-Net in a simple patch-based approach. Unfortunately, we were hindered by time and the limits of human performance: the system’s performance can at most be as best as the annotations of the data. In the case of prostate cancer, epithelial tissue can express as individual cells laying in groups in the stroma, which makes manual annotating data a time-consuming and challenging task. With this project, we set out to develop a novel method to circumvent the need for elaborate manual annotations.

Data (the PESO dataset)

We have developed our system using a new dataset of 102 prostatectomy tissue blocks. From each block a new section was cut, stained with H&E and scanned. After scanning, the tissue was destained, restained using immunohistochemistry, and scanned again. All slides were scanned at 20x magnification (pixel resolution 0.24 μm).

We used two markers for the immunohistochemistry: CK8/18 (using DAB) to mark all glandular epithelial tissue (benign and malignant), and P63 (using NovaRED) for the basal cell layer, which is normally present in benign glands but not in malignant glands. Restaining, instead of making consecutive slides, results in an H&E and IHC whole-slide image (WSI) pair for each patient that contains the same tissue.

We have released the H&E images used in this project as a public dataset on Zenodo: the PESO dataset. More information on this dataset and how to use it can be found in a separate blog post.

Method

To train our system, we used a two-step approach. First, we trained a convolutional network to segment epithelium in the IHC slides. By applying color deconvolution and subsequent recognition of positively stained pixels, we were able to have extensive training data while preventing the cumbersome and imprecise process of manually annotating epithelial regions. After this first step, we transfered the annotations to the H&E slides and trained the final network. The steps are described in more detail below:

1. On a subset of the IHC dataset, we applied color deconvolution to select the brown color channel. Some binary operations were used to remove small artifacts. The resulting mask contains most of the epithelial tissue but also includes non epithelial tissue that was colored by the stain (such as corpora amylacea).
2. On this subset, we corrected significant errors by hand. Artifacts were marked as such. Note that this task is only a fraction of the work; instead of annotating individual epithelial cells, only errors have to be annotated in a subset of the slides.
3. We trained a first U-Net on the corrected IHC slides. This network could then be used to generate epithelial masks for all the IHC slides (including the non-corrected slides).
4. We registered the H&E and IHC slides. As the original tissue was restained, the masks generated by the IHC network matched the H&E slides perfectly.
5. The final network was trained using the transferred masks.

All hyperparameters for both U-Nets can be found in the paper.

Results

Our system was able to accurately segment epithelial tissue, both in benign and cancerous regions (overall F1 score of 0.893). Even in regions with high grade prostate cancer, the system is able to segment individual cells. Some problems occurred in regions with high inflammation (that can look very similar to epithelial tissue). Correcting the color deconvolution masks increased the performance, but even more important removed consistent misclassifications of non-epithelial regions (like corpora amylacea). For a complete overview of all results, including results on an external dataset, please refer to the paper (Open Access).

The final U-Net was a critical component in the automatic data labeling technique of our project on automated Gleason grading. Using the epithelial masks, we were able to generate precise gland-level outlines of benign and tumorous tissue in prostate biopsies.

More info

Read full paper on Scientific Reports

Download dataset

This work was financed by a grant from the Dutch Cancer Society (KWF). You can use the following reference if you want to cite the paper:

Bulten, Wouter, et al. “Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard.” Scientific reports 9.1 (2019): 864.

Or, if you prefer BibTeX:

@article{bulten2019epithelium,
  title={Epithelium segmentation using deep learning in H\&E-stained prostate specimens with immunohistochemistry as reference standard},
  author={Bulten, Wouter and B{\'a}ndi, P{\'e}ter and Hoven, Jeffrey and van de Loo, Rob and Lotz, Johannes and Weiss, Nick and van der Laak, Jeroen and van Ginneken, Bram and Hulsbergen-van de Kaa, Christina and Litjens, Geert},
  journal={Scientific reports},
  volume={9},
  number={1},
  pages={864},
  year={2019},
  publisher={Nature Publishing Group}
}

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

Download dataset

Contents of the dataset

The PESO dataset consists 102 whole-slide images, split in to in a training and test part. The total dataset is around 140GB. For the training part, the reference standard is included. This set consists of:

• 62 whole-slide images, exported at a pixel resolution of 0.48mu/pixels.
• 62 Raw color deconvolution masks containing the P63&CK8/18 channel of the color deconvolution operation. These masks mark all regions that are stained by either P63 or CK8/18 in the IHC version of the slides.
• 25 color deconvolution masks (N=25) on which manual annotations have been made. Within these regions, stain and other artifacts have been removed.
• 62 training masks (N=62) that have been used to train the main network of our paper. These masks are generated by a trained U-Net on the corresponding IHC slides.

The test set consists of:

• 40 whole-slide images, exported at a pixel resolution of 0.48mu/pixels.
• 40 xml files containing a total of 160 annotations of regions that are used in the original evaluation.
• 160 png files of 2500x2500 pixels, exported at a pixel resolution of 0.48mu/pixels. Each png file corresponds to one test region.
• 160 padded png files of 3500x3500 pixels of the same test regions.
• A mapping file (csv) describing whether a test region contains cancer or only benign tissue.

Using the data

All slides are saved in TIFF format and can be opened with a viewer such as ASAP. To use the images in Python we can use the python binding of ASAP. To do so, make sure that the <ASAP install directory>/bin is in your PYTHONPATH. Then use the following snippet to extract a patch:

# Import the ASAP module
import multiresolutionimageinterface as mir

# Create a new reader to open files
reader = mir.MultiResolutionImageReader()
image = reader.open('path to image file')

# Get a patch at x=1000, y=12000 (coordinates relative to level 0) at level 2
level = 1
patch = image.getUCharPatch(10000, 12000, 1000, 1000, level)

More info / questions?

Background information on the dataset can be found in the paper. For questions please use the contact form.

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

Authors: Wouter Bulten, Hans Pinckaers, Hester van Boven, Robert Vink, Thomas de Bel, Bram van Ginneken, Jeroen van der Laak, Christina Hulsbergen-van de Kaa, Geert Litjens

Article URL: Paper at Lancet Oncology

arXiv URL: Preprint at arXiv

Introduction

Prostate cancer is one of the most common forms of cancer, with more than 1.2 million new cases each year. The diagnosis of prostate cancer is complicated by multiple factors. Patients with low-grade prostate cancer are often better off with a wait-and-see approach than with active treatment due to the side effects of, for example, surgery. High-grade cancers, however, need to be diagnosed as soon as possible not to delay treatment and to increase patient survival. Unfortunately, diagnosis and grading of prostate cancer is a difficult task which suffers from inter- and Intra-observer variability. While export uropathologists have shown better concordance rates, such expertise is not available for every patient. In other words, there is a need for robust and reproducible grading at expert levels. We have developed an artificially intelligent system, using deep learning, that can perform the grading of prostate cancer at a pathologist-level performance.

In this blog post, we describe our approach, show how we tackled the problem of data labeling and, finally, show that our deep learning system operates at a pathologist-level performance. The article is now published at Lancet Oncology.

Background on Gleason grading

Treatment planning for prostate cancer patients is based mainly on the Gleason score of a biopsy. After the biopsy procedure, a pathologist examines the tissue through a microscope. Through the microscopic analysis, the pathologist needs to distinguish between benign and malignant tissue. Any malignant tissue is then classified according to the architectural pattern of the tumor. This classification system is formally described in the Gleason grading system.

In the Gleason grading system, a tumor region is assigned a number between 1 (low-risk) and 5 (high risk), though patterns 1 and 2 are not reported anymore for biopsies. For prostate biopsies, the most common pattern and the second-highest pattern together form the Gleason score, e.g., 3+5=8. To make reporting of prostate cancer more apparent for patients, Gleason scores are mapped to five prognostically different grade groups; with group 1 being the lowest risk, and 5 the highest.

Semi-automatic data labeling

Pathology archives are often full of tissue specimens, but detailed labels of these specimens are lacking. Deep learning, however, requires large annotated datasets to reach its full potential. Due to time and budget restrictions, it is often practically infeasible to hire pathologists to annotate the data.

In the case of Gleason grading, this problem is even more significant. High-grade cancers (Gleason growth pattern 5) can express in the form of (strands of) individual tumor cells. Individually outlining hundreds of these cells in thousands of tissue specimens is no entertaining endeavor.

We developed a novel approach to circumvent the need of manual annotations. First, we sampled cases from our dataset that contained only one Gleason grade (score 3+3, 4+4 or 5+5). Then we employed the following steps to label the data:

1. For each case in our dataset, we used the original pathologist’s report (from diagnostics) to determine the Gleason score of a biopsy.
2. A previously-trained tumor detection system was applied to all these biopsies. This procedure resulted in a rough outline of the tumor bulk. These tumor regions were still very coarse and needed further refinement. A description of this system can be found in the corresponding paper.
3. The tumor masks were refined by filtering out all non-epithelial tissue. For this, we used a deep learning system that was trained using immunohistochemistry to detect epithelial cells. More information about this system can be read in the paper.
4. All detected and filtered tumor cells were assigned with the Gleason grade of the biopsy.
5. We trained an initial network on the automatically annotated data. After training, this segmentation network was able to assign Gleason growth patterns to individual cells.

Network training

The trained network from step 5 (above) could then be used to annotate our complete dataset. This trained network was not perfect in assigning Gleason scores as it was only trained on a subset of our dataset. To annotate the full training set, we applied the network to all cases of our set. We then used the original pathologist’s report to fix any major mistakes. For example, a predicted Gleason 5 region in a biopsy with Gleason score 3+3 would be removed. Tissue that originated from benign biopsies, and was detected as malignant, was classified as “hard negative.”

With our full training set annotated, we could train the final deep learning system. The whole procedure required minimal human effort. Moreover, no pathologists were required to annotate the data; instead, we were able to utilize expert knowledge extracted from the pathologist’s reports. All training and network details can be found in the paper on arXiv.

After the final network was trained, we were able to assign grade groups to biopsies automatically. Our system was trained as a segmentation network and outputted precise outlines of the tumor. Because of this segmentation, we can use a simple method to assign grade groups. Similar to clinical practice, we compute the volumes of each growth pattern. These volumes are then used to determine the primary and secondary Gleason grade, and finally, the grade group. We believe that this method is more interpretable and usable by pathologists in clinical practice. Each grade group prediction of the system can be easily checked by validating the marked glands.

Online examples

Two examples showing the raw output of the algorithm, without any post-processing, overlayed on biopsies. Benign tissue is colored green, Gleason 3 in yellow, Gleason 4 in orange and Gleason 5 in red. The original biopsy is shown on the left, the biopsy with overlay on the right. You can zoom in by scrolling, moving around can be done with click&drag.

Overview of main results

We compared our deep learning system on a test set of 535 biopsies. These biopsies were graded by three experts, and their consensus is used as the reference standard. The deep learning system achieved a high agreement with a quadratic Cohen’s kappa of 0.918.

Furthermore, from the test set, we selected 100 cases to be graded by an external panel. This panel consisted of 15 external raters (13 pathologists and 2 pathologists-in-training) from 10 different countries. In agreement with the reference standard (quadratic Cohen’s kappa) the deep learning system outperforms 10 out of 15 pathologists.

We also evaluated our deep learning system on grouping patients in prognostically relevant groups: (1) benign versus malignant biopsies; (2) using grade group 2 as cut-off; and (3) high versus low cancer. The deep learning system achieves a pathologist-level performance on all groups. Please see the full paper for further results and more detail.

Future challenges

There are still challenges to overcome before a system such as ours can be used in clinical practice. First of all, our system was developed and evaluated using data from a single center. Including data from different centers, with different stain protocols and scanners could increase the generalization ability of the system. Second, while the performance is high, it is not perfect, and the accuracy of the system could be improved. Though, at some point, it is difficult to say what is better given the high inter- and intra-rater agreement within prostate cancer grading.

Given our results, we conclude that there is clear use for automated systems for Gleason grading. Such systems can give feedback to a pathologist at expert levels, both in a first or second reader setting. AI systems and pathologists excel at different things, and, we firmly believe that a union of both has the most potential for the individual prostate cancer patient.

Interested in how this AI system can help pathologists? We have published follow-up research on how our AI system can assist pathologist during grading of prostate biopsies.

Try out the algorithm online

Interested on testing our algorithm on your own data? The algorithm we developed is available to try out online, without any requirements on deep learning hardware.

Try out the algorithm

The online Gleason grading algorithm consists of multiple steps to prepare and analyze the data:

1. For each biopsy, a tissue mask will be generated by a deep learning-based tissue segmentation system.
2. Based on the input and a small set of target files, a CycleGAN-based normalization algorithm is trained.
3. The trained normalization algorithm is applied to all input biopsies to normalize the data.
4. Each normalized biopsy is processed by the automated Gleason grading system. The output of the system consists of an overlay and a set of metrics for each biopsy. The metrics include the biopsy-level grade group, the Gleason score, and predicted volume percentages.

More info

Download paper Download preprint

• More details can be found in the article at Lancet Oncology or in the preprint. Both the article and the preprint have supplementary materials with more details.
• A description of the tumor detection algorithm can be found in the paper on tumor detection.
• The epithelium segmentation system is described in a separate paper. The dataset is public, can be found online through Zenodo and is described in a blog post.
• This research was performed as part of the Computational Pathology Group and the Diagnostic Image Analysis Group of Radboud University Medical Center.

Please use the following to refer to our paper or this post:

Bulten, Wouter; Pinckaers, Hans; van Boven, Hester; Vink, Robert; de Bel, Thomas; van Ginneken, Bram; van der Laak, Jeroen; Hulsbergen-van de Kaa, Christina; Litjens, Geert, “Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study.” The Lancet Oncology (2020).

Or, if you prefer BibTeX:

@article{bulten2020automated,
  title={Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study},
  author={Bulten, Wouter and Pinckaers, Hans and van Boven, Hester and Vink, Robert and de Bel, Thomas and van Ginneken, Bram and van der Laak, Jeroen and Hulsbergen-van de Kaa, Christina and Litjens, Geert},
  journal={The Lancet Oncology},
  year={2020},
  publisher={Elsevier}
}

Acknowledgements

This work was financed by a grant from the Dutch Cancer Society (KWF). We would also like to thank all pathologists that contributed to the observer experiment.

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

“The way to develop a histologic classification was to forget anything I thought I knew about the behavior of prostate cancer and simply look for different histologic pictures … . Then, the pictures would be handed to statisticians and compared with a ‘gold standard’ of clinical tumor behavior (ie, patient survival).” - Donald F. Gleason

As an AI scientist, to me this sounds a lot like pattern recognition; something computers could potentially do better. Currently the grading system consists of 5 distinct classes (of which practically only 3 are used), who says that there are not 10 or 50 relevant morphological classes in the data? That is why I am interested in unsupervised methods that can extract patterns from data and want to apply this to detecting and grading prostate cancer. All information is already present in the data, we ‘just’ need to find methods to retrieve it.

At MIDL2018 I presented a first step towards this goal: an unsupervised method for detecting and grading prostate cancer. The idea behind this project is to cluster prostate tissue in an unsupervised fashion and then later link these clusters to patient prognosis.

Methods overview

Our method is based on adversarial autoencoders and uses these autoencoders to cluster tissue during training: a clustering adversarial autoencoder (CAAE)². As opposed to normal autoencoders, this method does not require post-processing in terms of kMeans, t-SNE or other clustering methods.

Our CAAE consists of four subnetworks (see figure above) and two latent vectors as the embedding. The first latent vector, y (size 50), represents the cluster vector and is regularized by a discriminator to follow a one-hot encoding. This is the vector that is used for the clustering and represent high level structures in the data (for example, in case of MNIST this could represent individual digits). The second latent vector, z (size 20), represents the style of the input patch, and follows a Gaussian distribution (think of writing style in MNIST).

Training the CAAE forces the network to describe high level information in the cluster vector using one of the 50 classes, and low-level reconstruction information in the style vector. The ratio between the length of the two vectors is critical, a too large z and the network will encode all information using the more easy to encode style vector and disregard the cluster vector.

The CAAE is trained in three stages on each minibatch. First, the autoencoder itself is updated to minimize the reconstruction loss. Second, both discriminators are updated to regularize y and z using data from the encoder and the target distribution. Last, the adversarial part is trained by connecting the encoder to the two discriminators separately and maximizing the individual discriminator loss, updating only the encoder’s weights. This forces the latent spaces to follow the target distributions.

Data & Anti-body driven feature learning

We trained our CAAE on patches (128 x 128 pixels) extracted from registered whole slide image (WSI) pairs. Each pair consists of a H&E slide (the staining method that is used for grading prostate cancer), and a slide that was processed using immunohistochemistry (IHC) with an epithelial (CK8/18) and basal cell (P63) marker. The patches were sampled at random as no annotations or information regarding tumor location was available.

The H&E patches were used as training input. We tested two reconstruction targets: using the input patch as the target (H&E to H&E) and using the IHC version of the same patch (H&E to IHC). By using a IHC patch as reconstruction target we force the network to learn which features in H&E correspond to features in the IHC; we hypothesized that this anti-body driven feature learning results in more relevant encodings as the network needs to learn which features in the H&E correspond to features in IHC.

Results

After training the networks can be used to cluster new patches by passing patches through the encoder. The other components of the network can be discarded. By taking the argmax of the cluster vector each patch is assigned to a single cluster. A few examples clusters are shown below. Note that not every cluster is relevant as we enforce no structure or prior knowledge on the clustering.

The best network achieves an F1 score of 0.62 on tumor versus non-tumor (see paper for all experiments and results). This score leaves enough room for improvement, but our network achieves this score without using labeled training data on a very noise dataset. In other words, lots of work still ahead of us but it shows that there is information in the data that we can extract using these autoencoders. Donwsides of this method are that these autoencoders are hard to train, can suffer from mode collapse (i.e. only 1 cluster is formed) and can be very unstable during training.

To determine whether the clustering encodes relevant information we can, for visualisation, apply the network as a sliding window to a larger area. Of course this is not really useful for diagnostics, as it is very coarse, but it gives an overview of what the network ‘sees’. This method results in nice heatmaps for each class:

More info & citation

This work was presented at MIDL 2018. Want to read more? Please refer to the conference abstract as it contains more detail on the data, training procedure and network architectures. Feel free to add any questions as comments to this post.

You can use the following reference if you want to cite my paper:

W. Bulten and G. Litjens. “Unsupervised Prostate Cancer Detection on H&E using Convolutional Adversarial Autoencoders”, in: Medical Imaging with Deep Learning, 2018

Or, if you prefer BibTeX:

@InProceedings{Bulten18,
  author    = {Bulten, Wouter and Litjens, Geert},
  title     = {Unsupervised Prostate Cancer Detection on H\&E using Convolutional Adversarial Autoencoders},
  booktitle = {Medical Imaging with Deep Learning},
  year      = {2018},
  url       = {https://arxiv.org/abs/1804.07098},
  abstract  = {We propose an unsupervised method using self-clustering convolutional adversarial autoencoders to classify prostate tissue as tumor or non-tumor without any labeled training data. The clustering method is integrated into the training of the autoencoder and requires only little post-processing. Our network trains on hematoxylin and eosin (H\&E) input patches and we tested two different reconstruction targets, H&E and immunohistochemistry (IHC). We show that antibody-driven feature learning using IHC helps the network to learn relevant features for the clustering task. Our network achieves a F1 score of 0.62 using only a small set of validation labels to assign classes to clusters.},
}

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

References

This post gives a highlight of our work. Want to have more in depth details? All details on the data, training procedure and network architectures are available in the article itself.

Why epithelium?

Prostate cancer (PCa) is the most common cancer in men in developed countries¹. PCa develops from genetically damaged glandular epithelium and in the case of high-grade tumors, the glandular structure is eventually lost². As PCa originates from epithelial cells, glandular structures within prostate specimens are regions of interest for finding malignant tissue.

Building systems to detect tumor is often hard due to a lack of data. In general, more data equals a higher performance. Typically, deep learning methods that try to detect cancer from scanned tissue specimens use a set of annotated cancer regions as the reference standard for training. As these algorithms learn from their training data, the quality of the annotations directly influences the quality of the output. Tumor annotations made by pathologists are often coarse due to time constraints (see picture below, 2nd image). Outlining all individual tumor cells within PCa is practically infeasible due to the mixture of glandular, stromal and inflammatory components. These coarse annotations limit the potential of these deep learning methods.

We propose a method to automatically refine these coarse tumor annotations by training a system that can automatically filter out irrelevant parts of the data, in this case all non-epithelial tissue. This method results in fine-grained annotations without additional manual effort.

How did we approach this?

As data for our experiments we used 30 digitally scanned H&E whole mount tissue sections from 27 patients that underwent RP treatment. The reported Gleason growth patterns in these sections ranged from 3 to 5. The specimens were randomly split into three sets: 15 slides for training, 5 for validation and 10 for testing.

We compared U-Net³ (which is specifically designed for segmentation) and a general fully convolutional network (FCN). For both types of network we tested differences in layer depth. The networks are trained on patches extracted from digitally scanned prostate tissue slices at 10x magnification.

What are the results?

The most important numerical results can be found in the table below. See the article for a breakdown in cancer/non-cancer.

For both types of networks, the deeper the network the higher the performance. However, a deeper network also increases the parameter complexity, especially in the case of the FCN. Given the comparable performance, we labeled the U-Net as the winner given its lower parameter count.

The best U-Net produces fine grained segmentations and achieves a F1 score of 0.82 and an AUC of 0.97.

Next steps

Room for improvement lays with segmenting glands as a whole. Our models rarely misses complete cancer regions, though with some high-grade PCa only parts of the glands are detected. We suspect that most of the errors are, first of all, caused by a lack of training examples and not due to a limitation of the models.

A logical next step is to improve our dataset. However, manual annotations are flawed due to the difficulty of making correct and precise annotations, especially in tumor areas. A potential solution for this is using immunohistochemistry to assist in making annotations. By using specific stains that highlight epithelial cells, we hope to generate precise and less erroneous annotations of epithelial tissue.

More info & citation

This work was presented at SPIE Medical Imaging 2018. Want to read more? Please refer to the conference paper as it contains more detail on the data, training procedure and network architectures. Feel free to add any questions as comments to this post.

You can use the following reference if you want to cite my paper:

Wouter Bulten, Christina A. Hulsbergen-van de Kaa, Jeroen van der Laak, Geert J. S. Litjens, “Automated segmentation of epithelial tissue in prostatectomy slides using deep learning,” Proc. SPIE 10581, Medical Imaging 2018: Digital Pathology, 105810S (6 March 2018)

Or, if you prefer BibTeX:

@inproceedings{Bulten2018,
  author = {Bulten, Wouter and {Hulsbergen-van de Kaa}, Christina A. and van der Laak, Jeroen and Litjens Geert J S},
  booktitle = {Medical Imaging 2018: Digital Pathology},
  doi = {10.1117/12.2292872},
  isbn = {9781510616516},
  title = {{Automated segmentation of epithelial tissue in prostatectomy slides using deep learning}},
  volume = {10581},
  year = {2018}
}

Series on my PhD in Computational Pathology

This post is part of a series related to my PhD project on prostate cancer, deep learning and computational pathology. In my research I developed AI algorithms to diagnose prostate cancer. Interested in the rest of my research? The three latest post are shown below. For all the posts, you can find all posts tagged with research.

Latest post related to my research

• AI for diagnosis of prostate cancer: the PANDA challenge

  Artificial intelligence (AI) for prostate cancer analysis is ready for clinical implementation, shows a global programming competition, the PANDA challenge.

• Improve prostate cancer diagnosis: participate in the PANDA Gleason grading challenge

  Can you build a deep learning model that can accurately grade protate biopsies? Participate in the PANDA challenge

• The potential of AI in medicine: AI-assistance improves prostate cancer grading

  In a completely new study we investigated the possible benefits of an AI system for pathologists. Instead of focussing on pathologist-versus-AI, we instead look at potential pathologist-AI synergy.

Deep learning posts

Sometimes I write a blog post on deep learning or related techniques. The latest posts are shown below. Interested in reading all my blog posts? You can read them on my tech blog.

• Simple and efficient data augmentations using the Tensorfow tf.Data and Dataset API

  The tf.data API of Tensorflow is a great way to build a pipeline for sending data to the GPU. In this post I give a few examples of augmentations and how to implement them using this API.

• Getting started with GANs Part 2: Colorful MNIST

  In this post we build upon part 1 of 'Getting started with generative adversarial networks' and work with RGB data instead of monochrome. We apply a simple technique to map MNIST images to RGB.

• Getting started with generative adversarial networks (GAN)

  Generative Adversarial Networks (GANs) are one of the hot topics within Deep Learning right now and are applied to various tasks. In this post I'll walk you through the first steps of building your own adversarial network with Keras and MNIST.

References

The indoor localisation problem is more complex than just finding whereabouts of users. Finding positions of users relative to the devices of a smart space is even more important. Unfortunately, configuring such systems manually is a tedious process, requires expert knowledge, and is sensitive to changes in the environment. Moreover, many existing solutions do not take user privacy into account.

We propose a new system, called Simultaneous Localisation and Configuration (SLAC), to address the problem of locating devices and users relative to those devices, and combine this problem into a single estimation problem. The SLAC algorithm, based on FastSLAM, is able to locate devices using the received signal strength indicator (RSSI) of devices and motion data from users.

The presentation can be viewed online for those that are interested in our research and design considerations. As the presentation is built using Reveal.js it can be viewed in any modern browser (both mobile and desktop).

View the presentation full screen (external window) →

Even more important for a system to know where users exactly are, is to know where users are relative to the devices it can control or use to sense the environment. This relation between user and device location is an essential input to these systems. A central question in this field is therefore:

What are the locations of devices in a smart space and what are the current locations of users relative to these devices?

During my graduation project at DoBots I worked on an algorithm, called SLAC : Simultaneous Localization and Configuration, to solve this double localization problem: finding position of users and devices simultaneously. With SLAC we aimed to simultaneously locate both the user and the devices of a system deployed in an indoor environment. The target platform was the Crownstone.

The general idea of the project is to combine characteristics of devices with information from users that walk around in a building. The figure below illustrates this. We start without any knowledge (1), then we let users walk around and gather data (2), eventually we will learn the locations of each device in the building (3).

We use characteristics that are already available in many smart spaces: signal strength measurements (or RSSI) from devices and motion data from smart phones and other portable devices. These two inputs are combined in a system that can locate users and devices, respect individual users’ privacy and perform all estimations in real time. SLAC is based on a common technique from robotics, simultaneous localization and mapping (SLAM), and in particular the FastSLAM algorithm¹.

Algorithm overview

The SLAC algorithm uses two types of inputs: signal strength (RSSI) measurements to determine distances to devices and inertial measurement unit (IMU) data for pose estimations.

The raw RSSI signal contains noise. To filter out large spikes, while trying to retain distance information, a (regular) Kalman filter is used to filter incoming signal strength measurements.

The second input is focussed on modeling motion. Two sensors are used to measure the motion of users: an accelerometer and a compass. These two sensors are present in almost any modern mobile device (including phones, tablets and wearables). The compass returns the current rotation or heading relative to the global north; this does not require any processing and can be used directly as an input. The acceleration is used as input for a pedometer. The pedometer is based on a design by Zhao (2010)² and Ménigot (2014)³.

Given the distance measurements and the motion data, we can map the SLAM problem to the domain of sensor networks and indoor localization. The robot’s controls, which are used for pose sampling, are replaced by our motion estimates. The observations, which are often 2D measurements, are replaced by 1D RSSI measurements similar to the approach of Sun et al. (2009)⁴. The flow and update rate of the SLAC algorithm is controlled by the pedometer: the algorithm is run after a new step has been detected.

As device observations are 1D and signals propagate spherical it is impossible, given a single measurement, to determine the bearing of an observation. We therefore must first make an initial estimate of a beacon location before we can refine it using the default method of FastSLAM. We solve this by implementing a second separate particle filter that focusses solely on finding an initial estimate of a device’s location. After each new measurement we update this filter by computing the importance weight and subsequently resampling the filter. When the variance between particles is low enough (given some threshold), we assume that a a devices’s location has been found.

Our initialization filter is a separate component that uses the current best user estimate as input. So, to improve on our rough initial estimate we move the estimation from the global initalization filter to each individual particle. As it is impractical to update MxN particle filters (one particle filter per landmark per particle) we use an extended Kalman filter (EKF) to estimate a landmark’s location (similar to the original FastSLAM implementation).

When each observation is processed all particles have been updated and contain new importance weights. We then perform resampling. However, it does not make sense to resample after each step; there is just to little information for the resample process. In order to overcome this we utilize Sequential Importance Resampling (SIR).

Implementation

SLAC has been fully implemented in Javascript and more specifically using the ECMAScript version 6/2015 standard. Javascript has been chosen to support a large range of devices on which the algorithm can run; this includes web browsers and mobile phones. The Apache Cordova platform was used to access native API’s of mobile devices such as the Bluetooth radio and the motion sensors.

Simulations

We evaluated the system using simulations first; this granted the opportunity to repeat the experiment and control the environment. In the simulations we emulated the world by building an environment of similar dimensions as one of our real world test beds.

We varied the number of RSSI updates each device broadcasts between each consecutive algorithm step. As the signal strength is used to make range estimates the number of received messages could have an effect on the overall performance. The different settings are: 1, 2, 5, 10, 25, 50 and 100 updates per step. The results are shown in the figure below.

We found that the number of RSSI updates per algorithm step has a very high effect on the performance of the system. This follows directly from our system: given more information the Kalman filter responsible for filtering the raw RSSI signal will be able to give a less noisy estimate of the current distance to a device. These distance estimates are vital in updating device positions and weighing particles. These results suggest that we need a high update rate for real world applications.

In terms of performance, these simulations showed that within controlled environments (i.e. Gaussian noise, fixed paths of users, no obstacles), we can achieve an average localization error below .20m. This means that, after running the algorithm, the estimate of a device’s location will, on average, only be 20 centimeters away from its actual position. When the update frequency of devices is lowered, to a level similar to our live tests, the average localization error increased and resulted in an average error of .69 to 1.25m.

Real world tests

Simulating RSSI values and movement has its drawbacks: noise is predictable and there is less interference from events in the environment. In general it is hard to fully simulate all the factors of a real world environment. In order to asses the performance of the algorithm outside a simulated world, we also tested the algorithm in the wild at the DoBots/Almende building.

While SLAC runs online and in real time, the data for this live test has been recorded and analyzed offline at a later stage. The algorithm did however run during the data collection to give feedback about the process. Each recorded data set consisted of the raw unprocessed and unfiltered motion data (i.e. acceleration and heading) and RSSI measurements. These datasets are played back several times to get an average performance. This is particular important as the algorithm is a random process: using the same input data twice will result in different outcomes.

Interested in a demo? On the project page of SLACjs there is a demo that shows the localization algorithm using real (recorded) data.

The results of our live tests are shown in the figure below (A_1 to A_4). Additionally, the simulations results of the two conditions (Sim 5 & Sim 10) which are comparable to a real world setup are displayed as a reference.

All combined, our live tests showed a localization error of 2.3m, averaged over all devices. This result is good enough for room level accuracy, but there is room for improvement. These results where achieved by letting a single user walk around for one to two minutes (roughly 60 steps). All computations are done locally, i.e. running on users’ devices and without using prior information of the environment.

Demo

A video of a simulation of the algorithm can be seen in the video below. A user is walking inside a building/smart space filled with seven simulated devices (DoBeacons). The blue line describes the ground truth of the user’s path. The green line is the current best estimate of this path. Devices are displayed using squares; true positions are displayed in black and estimates in red.

An online demo can also be found on the website of SLACjs.

Code

The full source code is available online on GitHub and is licensed under the GNU Lesser General Public License v3.

Source

This post was originally written for the DoBots website.

References