With new regulations to improve ship efficiency and reduce species invasion, the number of underwater hull inspections and hull reports is expected to increase greatly. But how can we scale them effectively, from in-water inspection to report generation? Notilo Plus has designed a suite of technologies that will enable fast and replicable inspections as well as automatic, consistent hull reports generation. Using AI algorithms for image recognition, it classifies hull images according to their degree of fouling, coating conditions, and generates insights that are compliant with the best standards of the industry. Firstly designed for the Seasam ROV with localization capability of every image on the General Arrangement plan of the hull, it is now available for all service providers, regardless of their inspection method. Any underwater, diving or ROV video can now be turned into actionable recommendations on Notilo Cloud platform. It opens the door for better hull management, optimized cleaning patterns and predictive models for shipowners and ship managers at a fleet scale, thus facilitating compliance with the strictest regulations. Seasam and Notilo Cloud are now used by shipowners and service providers around the globe, and are selected by DNV on Veracity platform.

I. Introduction

 For a ship, hull fouling is responsible of a significatively increased fuel consumption [1] and therefore, unnecessary GreenHouse Gases (GHG) emissions. Furthermore, globalization requires transcontinental transportation. Therefore, ships are passing by very different ecosystems, which develops on their hulls an unnatural fouling mixing up living species from incompatible environments. This last phenomenon jeopardizes wildlife in the areas around the cargo ports. Biofouling is therefore a main issue to tackle to reach sustainable maritime transportation [2]. In the context of globalization where maritime transportation represents almost 3% of GreenHouse Gases emissions [3], in addition of a increasing awareness of ecosystem endangerment, many initiatives have been deployed to tackle these issues and tends toward Green shipping.

II. New standards and regulation

1. BIMCO and local regulations

The launching of the new Baltic and International Maritime Council (BIMCO) standards, coming into effect by 2023, intends to make biofouling detection and cleaning mandatory. This knowledge is already required for some countries (especially in Oceania) such as New Zealand whose Ministry of Primary Industries has edited “GUIDELINES FOR DIVING SERVICE PROVIDERS”. These guidelines share the same purpose as BIMCO standards, and consequently, we will focus on these standards on the following as they are intended to become the global norm.

BIMCO is the largest international shipping association representing shipowners and is accredited as a Non-Governmental Organisation by the United Nations. To promote greener standards across the shipping industry, and limit the potential damage of hull-related invasive species, the council has created an in-water cleaning industry standard documentation [4].

This documentation states the need to perform regular inspections of the hulls, also considering aggravating factors which require an increase in inspections frequency (temperature, salinity, distance from the shore, depth, …). In addition, even if the purpose is before everything the cleaning, inspections will always be required. Indeed, despite a cleaning is already scheduled, the documentation indicates that a pre-cleaning inspection is always mandatory in order to identify the areas of the hull where the effort needs to be concentrated.

This report also requires that “during the inspection of the underwater area (hull and niche areas) of the ship, the following shall be ascertained:

• Types of biofoulings.
• Percentage of biofouling coverage for each type.
• Height of biofouling for hard calcareous types.
• Condition of the AntiFouling Systems (AFS) on the hull and reference areas.”

These key pieces of information are at the center of latest Notilo Plus developments for our Shipping solution as we will explain it in the following.

2. EEXI/CII

The International Maritime Organization (IMO) of the United Nations adopted amendments on MARPOL on June 2021 in order to enforce by 2023 a new regulation centered around two calculations for each ship: Efficiency EXisting ship Index (EEXI) and Carbon Intensity Indicator (CII) in order to cut off GHG shipping emissions.

The EEXI indicator is based on ships specifications, and is estimated from the documentation emitted by the ship manufacturer.

The CII perfectly completes the EEXI as it is an operational carbon emissions indicator. This one will consist of a mark from A to E, with a legal target on the A to C range. It will be based on real emissions measured during the operations performed by the ship. Conforming to the regulation will consequently require an accurate, regular and efficient maintenance of the hull. At the scale of a fleet, it will be a real challenge to rise up for ship owners, and turn-key quick and efficient ship inspections solutions represent a significant asset to prepare for this new regulation, and go beyond the future standards thanks to good practices made easy to implement.

III. Notilo Plus Solution, combining reliable hardware with powerful software

1. State of the art

Nowadays, all these inspections are performed by diving teams or Remotely Operated Vehicles. In the first case, it requires a specific organization because of regulatory requirements to ensure a safe diving team in any port. As a consequence, the inspections take more time and human resources, for a result which is not optimized as it consists of generally poor quality photographs and videos of the hull that are used to interpret a general status. This solution is rather expensive, sometimes complicated to schedule, and requires a long time before the report is ready because all the data collected needs to be processed by experts, but image localization is complicated to pinpoint in post processing.

Hull reports are, in the best case, impossible to merge together to extract useful information such as monitoring of change over time, crossing data with other sources, etc. In the worst case, the reports can be inaccurate and partial due to inadequate survey or difficult post-processing.

With the current context in shipping and the pressure to shift towards Green Shipping for the main actors of the sector, inspections will need to scale. The industry could use a more extensive, more reliable and more informative solution.

2. Seasam ecosystem: a hardware suite

Notilo Plus was historically specialized in drone conception with first automation features with an autonomous diver tracking system. Building on our ability to develop autonomous inspections scenarios, we started with developing a hardware solution to perform hull inspections easily.

The aim was to provide them with an easy-to-use solution, bringing to the surface reliable and high quality data: localized images, steady frames, consistent distance to the hull.

To do so, our Seasam solution is composed of a Seasam drone, a Seasam Navigator, a ground station, a WiFi reel and a touchscreen tablet with our application — Seasam control. It can be used as a Remotely Operated Vehicle after a very short training as the remote control makes it intuitive to pilot. Equipped with a high-definition camera and possibly with powerful lights and extra-sensors (such as acoustic camera), it is the perfect tool to perform inspections on the hull with higher quality data than a diving team, without the danger and for a lower cost [6].

In addition, using the ground station and Seasam Navigator —equipped with a GPS — allows for a live autonomous location of the Seasam drone in relation to the hull thanks to an acoustic system. It means that during the inspection, the precise location (accuracy: few meters) of the Seasam drone is recorded and linked to the data acquired on the hull. The correlation of these pieces of data enables a full exploitation that will be described later. Further developments allow us to propose a fully autonomous inspection, based on live location and hull-servoing to ensure the inspection is always optimized.

With these several options, Seasam is a suite of Hardware that can adapt to the level of simplicity necessary during the inspection, and to the level of precision that is required for the hull reports.

3. Notilo Cloud: the platform to exploit the whole extent of the collected data. Hull reports 

Notilo Cloud is the perfect prolongation of the Seasam suite. Designed to valorise the data collected, the videos recorded with the position are easily uploaded to the platform in order to be analysed. Indeed, we built four classifiers trained with a 25k images dataset enabling us to determine a fouling score for each image, to evaluate the status of the coating, to identify the niche areas and to categorize the images according to their visibility.

These classifiers are Convolutional Neural Networks for which we used transfer learning: it means that they are pre-trained for classification tasks, and we added fully connected layers on top of the network in order to adapt it to our specific problems. The whole network has then been trained again with our datasets in order to adjust the weights and to gain in accuracy. This design allows an accuracy of 90% for coating status (Fig. 1), 90% for visibility evaluation (Fig. 2) and 97% for niche areas identification (Fig. 3). Concerning fouling, the accuracy is limited to 83% but we can observe on Fig. 4 that the mistakes are concentrated on neighbouring scores. After further review, we noticed that these mistakes were made because borders are not strict, even in human scoring as the experts we integrated in our initial qualification process showed a lack of consensus on the concerned frames.

At the end of inspections, the users upload their videos on Notilo Cloud. In the software are several actions available:

• store all previous dives
• create a hull reports
• share with relevant partners or customers.

The hull reports generation consists in itself in several tasks and four different algorithms:

• extract relevant frames from all inspection videos, with a visibility algorithm. Only images actually showing the hull will be selected. After the frame extraction, a preprocessing algorithm adapts the format of the data to make it compatible with the other classifiers.
• precise if the frame consists in a niche area, and in that case, which niche area
• evaluate for each frame the level of fouling from 0 to 3
• determine if any coating defect is present: painting defect or mechanical damage
• Link each frame to the corresponding location on the hull out of the raw dive data

All these actions lead to the creation of a report, with 30 hull sections that have color and information on the status of each area.

This automatic evaluation enables us to better understand the status of the hull after inspection, to identify the areas to prioritize during the next cleaning operation and to save much time because there is no requirement for an expert analysis of the video as the results of our algorithms have already been validated by experts. This first step permits the automatic edition of an hull report in less than 30 minutes — compared to 10 hours for traditional reports — which is far quicker, more cost-effective, and more efficient than traditional inspection.

4. Notilo Cloud for any ROV source: opening data for all

Furthermore, in order to democratize the predictive maintenance of hulls and the understanding of underwater assets, Notilo Plus has opened a stand-alone version of Notilo Cloud for any ROV.

It opens the use of our algorithms and autonomous hull reports tool to anyone who performs underwater inspections with any underwater camera (GoPro, ROV, …). The main difference resides in the need to declare directly on the platform the location of the different images.

Classification and filtering of all images is possible to have a closer look to all relevant frames of the video.

IV. Discussion

Currently, our solution appears as a great help for CII as it helps to understand the origin of any problem on the actual carbon consumption of the ships. Indeed, by indicating an overall fouling status of the hull to the Ship Owners, Notilo Cloud helps them to optimize their cleaning operations in order to keep the carbon (and GHG in general) emissions of their ships under control.

Our fouling algorithm is not yet fully compliant with BIMCO standards or other regulations for invasive species control.

BIMCO standards differentiate many types of fouling and various types of coverage.

The authors have found it difficult to find experts able to converge in their classification of images between various types of macro soft fouling and macro hard fouling. The experts had often diverging opinions on classification.

It has been found that, for now the type (hard or soft, micro or macro) was sufficient, on an operational point of view, to plan maintenance and cleaning action. It is recommended that these official classifications shouldn’t be too precise, to account for the actual capabilities of experts to differentiate between fouling types. Alternatively, a new and more complete dataset labeled by experts should allow us to tackle this challenge.

What’s more, the coverage of fouling in BIMCO standards is a way of classifying images from a human eye perspective: e.g. looking at a frame, how scattered the fouling zones are. From an A.I. perspective, and with the possibility of extensive classification of each and every image from a video inspection, it becomes more relevant to use broader information, such as the actual severity of fouling depending on the frames.

The author propose that international standards for underwater inspections use more precise, AI driven framework for underwater hull inspections.

V. Conclusion

Notilo Cloud has been designed to fit with emerging regulations. It has already been identified by many majors in the sector as a high-end tool to exploit the full extent of all the data that can be collected during inspections. Moreover, combined with the Seasam solution, it offers a turn-key solution for underwater hull inspection. Seasam solution is making possible quick, optimized, easy to set up inspections, at lower cost than traditional inspection solutions. All these features make Notilo Plus’ ecosystem a key set of tools to tend toward Green Shipping through predictive maintenance. In other words, to make a step towards a sustainable activity while making savings.

As a versatile tool, that can be used anywhere in the world, from any data source, Notilo Cloud makes possible intensive hull monitoring and management, bringing together all data sources and creating predictive models for hull efficiency.

The scoring of 83% to 97% accuracy of our algorithm will continue to improve as the number of inspections increase, and we foresee a future where inspectors will never have to spend more than 1 hour to produce a full hull report, with a greater accuracy than any previous method.

This opens the door to efficiently scaling hull inspections, and better protecting our planet and our oceans.

References

[1] HAKIM M. L., UTAMA I K. A. P., NUGROHO B., YUSIM A. K., BAITHAL M. S., SUASTIKA I K., Review of correlation between marine fouling and fuel consumption on a ship, SENTA 2017: 17th Conference on Marine Technology

[2] DAVIDSON I., SCIANNI C., HEWITT C., EVERETT R., HOLM E., TAMBURRI M., RUIZ G. (2016), Mini-review: Assessing the drivers of ship biofouling management – aligning industry and biosecurity goals, Biofouling, 32:4, 411-428

[3] INTERNATIONAL MARITIME ORGANIZATION, Fourth IMO GHG Study 2020, 2021

[4] BIMCO, Industry standard on in-water cleaning with capture

[5] Author conversations with industry players

[6] Pay-as-you-inspect plans are available from 1000€/inspection