visual art analysis and generic category recognition

Summary: Universal Multi-Expert Metrics for Visual Art


Advances and Existing Challenges in Computer Vision and Image Processing

Recent advances in computer vision and image processing made possible development of automated systems for medical image analysis, robust face recognition in surveillance, security and related applications, reconstruction of 3D objects and scenes from multiple views, as well as for numerous other important problems. While some applications, such as image segmentation and 3D reconstruction, are more developed due to possibility of generic formalization of the problem, generic category recognition is still very difficult task without universal solution. Analyzing a scene and recognizing all of constituent objects remains to be the most challenging problem. While computers can efficiently and accurately reconstruct 3D shape from images taken from different views, they cannot name all the objects and animals in the picture even at the levels of a 2-year-old child.

The most developed techniques in the challenging area of object recognition include face recognition and instance recognition, i.e. identifying a known 2D or 3D object potentially viewed from a novel viewpoint, with different background or partial occlusions. On the other hand, generic category (class) recognition remains to be the most difficult and largely unsolved problem. For example, none of the existing computer vision systems could approach performance level of a 2-year-old child in generic category recognition. The main obstacles for performance improvement in this area include great variability within the same class and occlusions in multiple-object scenes.

Computerized Analysis of Visual Art

As mentioned above, even standard recognition tasks of differentiation among natural classes of objects could encounter significant difficulties. However, the developing field of computerized visual art analysis is facing much greater challenges. Indeed, one of the considered tasks is not to identify objects in the painting and determine their relevance to the known classes, but rather to capture much less obvious and hard-to-quantify features that allow relating considered piece of art to a particular famous painter, school of visual art or epoch. Even the same object or scene can be represented in a wide variety of styles by different painters or schools of art. Thus, variability in such problems could create even more serious challenges than in more standard problems of generic category (class) recognition.

Similar to more developed areas of computer vision, certain problems of computerized visual art analysis are more formalizable and quantifiable which warranties certain level of success. Mainly, these areas include applications dealing with the analysis of certain aspect of style used by artist. For example, fractal analysis was shown to be successful in identifying paintings of Jackson Pollock. Quantitative characterization of brushstrokes in paintings by van Gogh has been shown to be useful for detection of forgeries. Other computer-based systems for analysis of perspective, shading, color and form have also been used for identification of artists and styles.

However, unique style signatures of talented artists cannot be fully captured by a single feature. Typically, personal style of an artist consists of complex combination of features. There is no universal way for optimal choice of such features and for their quantification. A related hard-to-formalize problem is quantitative differentiation between "high art", that is capable of capturing viewer attention on deep psycho-emotional level, and mediocre visual representation of the same scene that lacks such "magical" effect. Intuitively, potential quantitative measures capable to identify such subtle effects of "high art" should be based on optimal combination and interrelation of many different features that characterize style of the true artist.

Such multi-feature measures could be very useful in providing objective quantitative support and interpretation for opinions expressed by art experts who may have intuitive ability to grasp complex multi-level visual information implicitly encoded in the art work. More importantly, such multi-expert quantitative measures could consistently discover effects which could be difficult or impossible to notice even for the most experienced human experts. Computerized system with these capabilities could be especially useful in applications requiring fast processing of large number of images where abilities of human experts are significantly inferior to those of automated systems.

Multi-Expert Solutions

While practical benefits of the multi-expert system for visual art analysis are obvious, there are no universal recipes for the discovery of such multi-feature measures for robust quantification of visual arts. Nevertheless, generic AQSCS framework for multi-expert discovery has demonstrated its effectiveness for many challenging problems in medical diagnostics and monitoring, quantitative finance and systematic investment strategies, quantitative decision support in professional sport, and many others. In particular, our ensemble-based indicators are capable of robust quantitative description of rare and complex physiological states and abnormalities as well as rare market states and tradable events that cannot be handled by other state-of-the-art approaches. Therefore, it is not surprising that our first efforts to apply AQSCS framework for discovery of generic multi-feature measures capable of consistently detecting subtle differences in visual art pieces resulted in very encouraging results.

Our multi-expert solutions for the analysis of visual art (listed below) could be effectively used for automated identification of art pieces (by artist or school), for objective indication of subtle and/or slow style changes of the same artist in time as well as for early detection of emerging talented artists which could be used as investment indicator. In addition, the same generic multi-feature measure could be used for quantitative comparison of images with complex visual content in a variety of applications: rare and complex cases in image-based medical diagnostics, robotic vision, surveillance, and other hard-to-formalize problems of computer vision.


Optimal ensembles of complementary image-recognition models for multi-class classification and identification of visual art To be done
Optimal ensembles of complementary low-level content descriptors for generic representation and comparison of images with complex and hard-to-quantify visual information To be done

Log in form