Is cell segmentation really necessary ?

A google scholar search on the term “cell segmentation deep learning” produces over 17,000 results. So this is obviously a problem which thousands of people have devoted much effort to solving. CellPose, for example, has provided a very nice Deep Learning system which makes it ‘easy’ for researchers to develop their own cell segmentation models. ImageJ/Fiji have been developed and used extensively for this purpose. Yet everyday we get feedback that biologists have tried these approaches and yet still fail on many applications. Additionally, the use of CellPose or Fiji for cell segmentation is time consuming, have their own learning curves, and are not fully automated pipelines suitable for high content screening.

But why bother with cell segmentation? What is the point of segmenting cells? With the open-ended nature of biological research we can say for sure we will never be able to enumerate the number of applications for cell segmentation that exist now or will in the future.

However, perhaps we can look as some problems that urgently need to be solved today and ask if cell segmentation is really necessary. Cell segmentation is not an end in itself. It is a means to an end. They end is making other measurements on an individual cell basis. It may be tracking the motion of cells, the growth of a population, the fluorescence signal emitted from individuals or a host of other things. All of these factors are changing over time and it is important to measure these changes at multiple time-points.

But is there a more straightforward and direct approach that by-passes the segmentation issue?

So lets consider just two examples in this blog post.

Accurate Cell Death Measurements

Cell death measurements are not easy to make. For example, you have a co-culture of T cells killing a cancer cell population. After 6 hours the T cells have killed some percentage of the cancer cells but the T cell population has expanded. So one cell phenotype has declined in numbers while another has expanded. You could use flow cytometry methods to separate the phenotypes and then count the separated cells. This is a laborious multi-step process that will not allow you to measure the what percentage of the T cells are doing the killing versus the number that are not engaged. The result is clouded by the difficulty in separating dead cells from living. This will not tell you how many target cells one T cell can kill. The precision of these methods are also lacking since the seeding process itself can result in 10% variation in starting cell densities and thus 20% variation in starting cell ratios.

But if the human were able to watch time-lapse movies of the cell interactions under a microscope, we can count the cells, we can see T cells engaging with target cells, we can see target cells becoming apoptotic, we can see target cells become necrotic. We can count the T cells and target cells. But humans cannot do this for for thousands of cells in thousands of time-lapse images. So, can we teach a computer to do the same? Do we need to solve cell segmentation to do make progress on these critical measurements?

Cell Identification and Tracking

Cell migration is an incredibly important biological activity. Its essential in immune response and its also the cause of cancer metastasis.

How does the human count cells and track cells? If you go about the task of tracking cells, you will likely mark each cell in its approximate center. Then you will count the marks or use a computer program to aid in creating time-lapse cell tracks by connecting the human created marks. So as a human observer, you identified individual cells as the first step.

The thing to note is that you will not segment the cells. Its not important to know precisely where one cell wall ends and another begins to locate the cells.

Once you have identified the cells, the process of tracking cell movements is a simple problem, easily automated, of connecting the dots between time points.

How to Avoid Cell Segmentation

MetaVi Labs used Deep Learning models to teach computer to do what humans naturally do when tracking cells or identifying cells. The AI models are trained to find cells – not to segment cells. This opens a plethora of new applications to advance biology at a more rapid pace.

Lets return to the cell death example. If we have engineered CAR T cells killing target B cells, can we measure how many T cells are engaging with the target cells and how many targets one T cell kills? To solve this we can record these interactions on a microscope taking image samples once per 4 minutes or faster. In the resulting movies, we can see the cells migrating, interacting, dying. With the aid of a fluorescence label on the target (B cells) we can identify which are B cells and which are T cells.

These measurements require the following activities from the observer (whether human or machine):

Identifying and locating individual cells
Tracking individual cells
Identifying T cell to B cell interactions
Quantifying the duration and frequency of these interactions
Identifying apoptotic events
Associating cell interactions (synapse formation) with cell death events
Quantifying the resulting data in a meaningful way

Consider this time-lapse movie of CAR T cells killing malignant B cells:

In this video we observe CAR T cells attacking malignant B cells (Nalm6). The Nalm6 we used in this assay carried a red gene (meaning the cells produced a red protein called TdTomato). This was only produced in living cells. As soon as the cell dies, the red protein is degraded. For clarity, the associated fluorescence images are not shown. Both the phase contrast (visible light) and fluorescence images are multiplexed as the input to the Machine Learning inference engine.

Consider this final “tracked” video:

In this video we see that the analysis engine has marked the T cells with a blue ‘+’, the B cells are marked with a green ‘+’ and a green ID number, dead B cells are marked with a Red ‘+’ and Red ID number. Eventually, the B cells degrade to the point of basically disappearing all-together. With this system we are able to observe interactions between cells solely on proximity of cell centers and duration of proximity.

At no time in the process is the AI taught to ‘segment’ cells, only identify their uniqueness, phenotype, and locations.

Benefit to the Researcher

The primary benefit of this system to the researcher is that the MetaVi Lab’s system does all the work, saving the biologist potentially hundred of hours of time. The AI models are pre-trained, the pipelines are pre-configured. The researcher is only responsible for the biological aspects. Images are uploaded to the on-line system and final reports are provided automatically.

The second benefit is the depth of new data provided to the biologist. The automated system is providing new data (such as individual cell-to-cell interaction histograms and dozens of other new metrics never before provided by population based assays – more on this in subsequent blogs).

Open source projects such as CellPose or Fiji may be useful for students in image processing and machine learning, but to maximize research progress perhaps it is better to use professionally developed solutions which provide direct and immediate results.

How can you Benefit?

For academic users, we provide a simple drag-and-drop interface to upload images. With an academic price of $2 per time-lapse sequence, its affordable even on a student’s budget.

For corporate users and microscopy centers we provide a complete high-content automation package which even removes the need for drag-and-drop, images are automatically extracted from the recording system and fully analyzed in a hands-off manner.

To learn more, please checkout our website.

Why Segment Cells? Example 1: Accurate Cell Death Measurements