Written By
Bianca de Loryn
College/Division
College of Science and Engineering
Publish Date
14 September 2021
Related Study Areas
Using augmented reality to measure a fish
JCU Information Technology Alumni Magnus Persson tells us how he came to move straight from being an undergraduate student to working on a major artificial intelligence (AI) project and about the daily struggles to teach an AI what a fish is.
“I have a big interest in how people interact with software and computers, and in games as well,” Magnus says. “This is why I did a Bachelor of Information Technology, and my minor was human-computer interaction and games design.”
This interest also attracted Magnus to the JCUFish app project that aims to give people the ability to measure and identify a fish with their mobile phone alone. The JCUFish app team is using a technique called ‘augmented reality’. “Essentially, you're building a 3D world within your phone based on the real world,” Magnus says.
The first step is to open the JCUFish app and take a photograph of the fish with your mobile phone. “This way, you're placing your fish in the 3D world, and then it's measuring the distance between two points.” The two points are the tip of the closed mouth of the fish and the end of the tail. “After measuring the fish, we send that picture to an artificial intelligence (AI), and the AI tries to figure out what species it is,” Magnus says.
Learning more about fish species in remote areas
The JCUFish project started in July last year. The idea behind this app was trying to find out which fish species are being sold in markets in remote areas around the world, and in which numbers and sizes. This could help researchers find out which species exist in certain areas of the Pacific, what the average fish size is when they are caught, and if these species are still abundantly available or perhaps in decline. “Definitely, where it's of interest is in the Torres Strait, Vanuatu, Fiji and the Solomon Islands,” Magnus says.
James Cook University's Marine Data Technology Hub (M-DataTech) and the World Wildlife Fund (WWF) are funding and collaborating in this project. “The WWF also helped getting volunteers to use the app because they're in all of those remote areas already,” Magnus says. “They currently have about 15 people going through the fish markets over there. But we have potential for anywhere between 50 to 100 to start getting involved.”
The hard work of training an AI
Magnus says that even though the software is 90 per cent finished, the devil is always in the detail — or in making the last 10 percent work. Choosing the best artificial intelligence (AI) and training it is a long and difficult process, as the images still have to be specifically prepared.
“So, Kurt Schoenhoff, who is also involved in this project, and I, we received hundreds of images of five to seven different fish species. The guys at M-DataTech spent a ridiculous amount of time preparing, rotating and categorising these by species,” Magnus says. “Then what we do is we take that dataset and send it to this artificial intelligence and say, ‘this is the species’.
“The AI goes through and looks at all those images. Then we start giving them training images to ask, ‘what is this?’. If it was accurately predicting it, that's great. But it takes thousands of images to make it an accurate data set.”
Working on the user experience
Another issue that still needs attention is that the JCUFish app currently only works on late model Android phones. But the team wants the software to run on iPhones and older model Android phones as well.
The app also needs to be easier to use - which programmers call the 'user experience'. “The user experience is really important for us. We can't really do that without people testing the app,” Magnus says. “We've coded it all the way up to this point, and now people testing it might be going 'nah, that’s a bit clunky’ or ‘that doesn’t work’ or ‘we don’t understand that’, which we need to fix. That's the hard bit, getting that testing done so people find the app useful and easy to use.”
Getting into IT development for research
Magnus got involved in the project when he was still an undergraduate student. Doing a research project or an internship with a company is an essential part of the Bachelor of Information Technology. “In my case, I decided to work on the JCUFish app instead of trying to find a job with another company,” Magnus says.
This was a good decision, as the JCUFish app project also landed Magnus a job as a research worker on the same project after his graduation. “It was a bit of word of mouth, and I have gotten a bit of a connection with the lecturers. I was constantly bugging them, saying, ‘you know, if you need any workers or you need anyone to get a bit more done, I’d be interested’.”
The future of JCUFish
In the long run, the app should be able to recognise most fish that live in the South Pacific and are caught by commercial fishers. But Magnus knows that the team could take the project even further.
“You could take a photo of the fish, record the location, how big the fish is and what species it is," Magnus says.
"You could take the app anywhere around the world, not just using it in fish markets. You could get a good set of data to determine things like spawning rate and other useful data info."
JCU Alumni and Researcher Magnus Persson
Volunteering for JCUFish
As of September 2021, JCUFish is available for free on Google Play. “It is available to all who would like to measure their fish. But only registered WWF volunteers are able to send their images back to storage right now,” Magnus says. At this stage, the images still need to be prepared for processing by the AI.
Those who are living in a remote area in the Pacific and would like to volunteer can get in contact with Magnus Persson or Michael Bradley from James Cook University’s Marine Data Technology Hub (M-DataTech).