Size and shape analysis of oysters using computer vision
June 19, 2020
In this blog we are going to look at how Solve Geosolutions helped Shearwater Oysters – an oyster farm located in Northwestern Tasmania – by developing a computer vision-based solution that generates detailed data about their oysters using video collected from a smartphone.
The market value of oysters are determined by two key qualities: size and shape. Repeatable and objective measurements of these characteristics can be a challenge for many oyster farms.
Shearwater have up to seven million oysters on their lease at any one time, so generating detailed data on the morphology of their stock would allow them to harvest from their river more efficiently and better understand the quality of their oysters.
Currently, Shearwater uses a tumble grader to divide their oysters into broad size fractions and manually sort them according to shape before sale (Figure 1).
Figure 1 Example of one of Shearwater’s tumble graders. A large PVC cylinder with holes of a known diameter that rotates. Oysters roll down the tube and fall through the holes of pass through depending on their size
The limitations of this current method are that it is fairly ad-hoc, non-streamlined, varies between operators and provides no detailed data on stock morphology.
“We approached Solve to develop a solution that provides improved understanding of the size, shape and appearance of our oysters. The solution had to be cost-effective and fit into our existing workflows with minimal disturbance.” Daniel Webb – Operations Manager at Shearwater Oysters.
In consultation with the Shearwater team, we decided that leveraging smartphone video would provide the most straight-forward and streamlined solution. This allows us to analyse individual oysters, live, as they are processed.
Our solution utilised a smart phone outward-facing camera filming oysters as they exited the tumble grader. This video was then processed to detect individual oyster segments and measure their morphology.
There were several requirements for our chosen solution including the ability to :
- Run on an Edge device such as a smartphone or tablet,
- Define the exact margins of individual oysters, and
- Track each unique oyster through the video (including occlusion).
Object detection methods – such as YOLO or SSD – will give good results and allow us to count, but don’t provide enough morphological data. Our solution required an instance segmentation approach to predict a polygon mask for each unique oyster. Mask R-CNN was chosen because it adopts the detect-then-segment approach. It first predicts bounding boxes around each oyster and then performs binary segmentation within each bounding box (Figure 2).
As oysters move through the tumble grader and are detected, each oyster has to be tracked between consecutive frames. Simple Online and Real Time Tracking (SORT) was used to assign a unique ID to each bounding box (Figure 2). SORT has gained popularity as the first Multiple Object Tracking (MOT) pipeline to use The Hungarian Algorithm and a Kalman filter (instead of modern CNN-based algorithms) for assigning tracking IDs to objects.
In order to count each tracked instance only once, we identified when an oyster passes an imaginary finish line at the end of the tumble grader before falling into a bucket.
In addition to counting oysters with tracking IDs, we used the oyster masks coming from Mask R-CNN to measure the size and shape of each oyster using OpenCV (Figure 2).
Figure 2 video frames showing initial Mask R CNN detection with SORT tracking along with some of the resultant statistics that are being generated for each individual oyster.
The image processing workflow produces a number of statistics morphological parameters for each individual oyster including length, width, area, roundness along with the count of how many oysters have passed though the tumble grader (Figure 3). Each individual oyster is given a unique ID attached to its cropped image that can be used for further analysis, or as a digital record of the oysters within each batch.
Figure 3 Image showing detailed data for the first 24 oysters to be detected. Predicted masks and derived morphological features are also shown.
The Outcome for Shearwater Oysters
Having access to this workflow enables us to obtain quick, easy summations of each batch of oysters we handle and gives us data we haven’t had previously with regards to oyster shape and size, and more accurate count data that we have historically had to estimate. Being able to quantify important characteristics of the oysters we sell for both our benefit and that of our customers, it will also allow us to maintain a more accurate database and help guide our stock management procedures with more confidence, eventually leading to a higher quality grade of oyster being produced.
Contact us if you want to know more about this solution or if our team of data scientist can assist your organisation.