The contemporaneous development in recent years of deep neural networks, hardware accelerators with large memory capacity and massive training datasets has advanced the state-of-the-art on tasks in fields such as computer vision and natural language processing. Today’s deep learning (DL) systems however remain prone to issues such as poor robustness, inability to adapt to novel task settings, and requiring rigid and inflexible configuration assumptions. This has led researchers to explore the incorporation of ideas from collective intelligence observed in complex systems into DL methods to produce models that are more robust and adaptable and have less rigid environmental assumptions.

In the new paper Collective Intelligence for Deep Learning: A Survey of Recent Developments, a Google Brain research team surveys historical and recent neural network research on complex systems and the incorporation of collective intelligence principles to advance the capabilities of deep neural networks.

Collective intelligence can manifest in complex systems as self-organization, emergent behaviours, swarm optimization, and cellular systems; and such self-organizing behaviours can also naturally arise in artificial neural networks. The paper identifies and explores four DL areas that show close connections with collective intelligence: image processing, deep reinforcement learning, multi-agent learning, and meta-learning.