United Kingdom Creative Ideas Ltd. (UKCI) is an international company headquartered in London, which is the registrar and owner of the domain zone. The company's main project is the management of the .art domain zone. In addition, they develop services for artists and collectors, bridging art and technology. UKCI approached us with a request to develop a technology for evaluating works of art using a smartphone.
The concept is straightforward: users can take a photo of a painting, and the app will automatically identify the artist and the period in which it was created.
The concept is straightforward: users can take a photo of a painting, and the app will automatically identify the artist and the period in which it was created.
Tasks
- Develop an algorithm that matches a painting in a photograph with its original from the catalogue of The Metropolitan Museum of Art (The Met).
- Evaluate the quality of the catalogue search, identify the main problems and form a plan to solve them.
Solution
- We integrated with the MetObjects dataset, The Met's comprehensive data warehouse, which contains information on approximately 470,000 art objects.
- Following data analysis, our system can now identify key characteristics of a painting—such as the author, century, and title—in just 2 seconds.
HOW IT WORKS
The system searches for paintings in two ways:
- Through The Met museum API by keywords: e.g. painting, landscape, portrait, etc.
- By keywords in the MetObjects dataset.
The IDs of the paintings found by these methods were combined. In total, about 25,000 objects were found in this way.
We have tested several algorithms for image recognition and acquisition. One of them is embedders.
Embedder is a neural network. Such a model is trained like a classical classifier. The application phase of the model usually uses the last full-link layer, which converts the image into a numerical vector of length n.
SIFT and Classical Computer Vision Algorithms
We tested traditional classifier architectures like ResNet50 and MobileNet, along with architectures incorporating the ArcFace layer. The ArcFace layer enhances training by increasing the distance between embeddings of objects from different classes, improving classification accuracy.
To minimize the risk of overfitting to the training dataset, we utilized various input image formats, including deliberate distortions.
PIPELINE
- Detecting paintings in a photo.
- Matching the paintings selected by the detector.
- Catalogue expansion.
Detecting Pictures in a Photo
We trained a CenterNet ResNet50 model as a detector. To achieve this, we created a training dataset consisting of 367 photographs taken in museums. These images were then labeled to identify paintings and drawings within the photos, enabling precise detection.
Testing
We tested the detector and pipelines on the same samples used for the embedders. However, we faced a problem: the test photos contained only one artwork, while real-world scenarios often include neighboring objects in the frame. To ensure the detector's reliability, we trained it to recognize these additional objects as well.
We tested the detector and pipelines on the same samples used for the embedders. However, we faced a problem: the test photos contained only one artwork, while real-world scenarios often include neighboring objects in the frame. To ensure the detector's reliability, we trained it to recognize these additional objects as well.
Matching the Paintings Selected by the Detector
After the detector has selected all the paintings of interest in the photo, the system starts searching for their matches in the database. There is a nuance here: we use extended validation, which includes not only the main paintings, but also additional paintings that are next to them.
However, for most of these additional paintings, we do not have labelling per class. In other words, there are no paintings in the database to which they can be matched. Therefore, the next task is to expand the catalogue.
Catalog Expansion
We consider several ways to expand the catalogue, the simplest of which is to use the Wikiart dataset.
Originally it contained 81,000 paintings categorised by style. The titles of the images included only the artist's name and the title of the work.
RESULTS
- The matching algorithm works not only with photos of paintings, but also with photos of drawings.
- A working payline has been built to handle the whole process - from the detection of paintings and drawings in photos to their matching.
- The catalogue has been expanded to 90,000 objects, including paintings and drawings, and the pipeline is successfully handling global searches with good results.
TECHNOLOGY USED
- MetObjects
- ResNet50
- MobileNet
- Pytorch
- FAISS
THE TEAM
- Andrei Tatarinov, СТО.
- Alexander Kozlov, teamlead.
- Anna Zakutnyaya, ML-engineer.
