PhD candidate on Neural Networks as Dynamical Systems
Six cross-theme PhD positions in Computer Science The Informatics Institute is renowned for its research in complex information systems at large with a focus …
- Science Park, Amsterdam, Noord-Holland
- Vast contract
- Uren per week:
- 38 - 38 uur
- € 2325 - € 2972 per maand
Six cross-theme PhD positions in Computer Science
The Informatics Institute is renowned for its research in complex information systems at large with a focus on three themes: artificial intelligence, computational science, and systems and networking. Within these themes, specific topics are selected and pursued through theory and application development, using a broad range of methods from computer science and engineering. We also highly value topics that are crossing the themes. This brings together the different fields of expertise and creates the synergy and creativity needed for innovative approaches. To strengthen collaboration we have opened six cross-theme PhD positions on various scientific challenges.
We seek a multidisciplinary researcher who can study deep neural networks in the context of complex adaptive systems analysis. Specifically, the aim is to reformulate and reinterpret a neural network description as an equivalent description of a dynamical system. Subsequently, using tools from dynamical systems and complex systems you will analyze the dynamical system and thereby gain insights into the original neural network. The goal is to gain insights into the structural and functional properties of the neural network computational graph resulting from the learning process. Techniques that will be employed include dynamical systems theory and iterative maps (chaotic attractors; Lyapunov exponent), information theory (Shannon entropy, mutual information, multivariate measures such as synergistic information (Quax et al., Entropy, 2018)), and network theory.
More about the project
The starting point of this project is the observation that residual connections have a dramatic impact on the accuracy of neural networks and yet their theoretical underpinning remains nefarious. Inspiration is the recent work from (Chen et al, NeurIPS, 2018), who re-interpret a neural network as a first-order linear dynamical system, coining them ‘neural ordinary differential equations’. This is done by a simple arithmetic operation on the guiding mathematical formula. Under the new formulation the neural network can be described and implemented as a dynamical system and thus optimized using standard differential equation solvers like Pontryagin’s adjoint sensitivity method and Runge-Kutta. Similarly, in (Ghodrati, Gavves, BMVC, 2018) we had proposed to view neural network layers as time steps, thus unifying feedforward and recurrent neural networks and arriving at a similar architecture as Chen et al. These works open up a whole new set of tools to study neural networks with respect to their dynamical properties, stability, phases and phase transitions, and so on. This project aims at studying more generally complex neural networks as complex dynamical systems using appropriate re-interpretations and mathematical transformations of guiding equations.
Overarching questions include:
- Can any multi-layer neural network and its optimization be expressed as a dynamical system? Is there a relation between the success of a neural network architecture and its dynamical system formulation?
- How do these (multilayer) networks self-organize to solve a particular task?
- How is information represented in these systems?
- Is there a set of fundamental properties underlying the structure and dynamics of deep neural networks?
This PhD project is a close collaboration between the Computational Science Lab and the Intelligent Sensory Information Systems at the University of Amsterdam; both world-leading labs in vision, learning and computational sciences, situated next to other renowned AI labs like AMLab and ILPS with top AI researchers. This work combines the state-of-the-art research activities of both PIs of the project (Dr Efstratios Gavves and Dr Rick Quax), who will closely supervise you.
What are you going to do?
You are expected to:
- perform novel academic research on the crossroads of deep machine learning, information theory, and dynamical systems. Research will be published in the top related, conferences, including ICLR, NeurIPS, ICML, CVPR, PAMI; as well as aiming for peer-reviewed interdisciplinary journals, such as Entropy, PRX, Nat. Comm., Science Advances, Chaos.
- have strong analytical skills as well as technical skills;
- have an interdisciplinary mindset and an open and proactive personality in interacting with researchers from different disciplines;
- take part in ongoing educational activities, such as assisting in a course and guiding student thesis projects, at the BSc or MSc level;
- have a proactive and independent mindset.
- An MSc degree in Computer Science, Computational Science, AI, (Applied) Mathematics, Statistics, Statistical Physics, Electrical/Computer/Telecommunications Engineering, or a closely related field;
- a solid understanding of Machine Learning, Deep Learning, Dynamical Systems, Information Theory. Familiarity with Shannon information theory (pre), optimization;
- a strong scientific interest in understanding complex theoretical concepts; the concepts of (deep) learning, dynamical systems theory, deterministic chaos, and information theory in particular;
- excellent mathematical foundations. Special emphasis in statistics and probability theory, calculus and differential calculus, linear algebra;
- excellent programming skills. Preferably Python, or other similar languages. Knowledge of a Deep Learning framework such as PyTorch or TensorFlow. Experience with mathematical/computational modelling;
- fluent communication and writing skills, cooperative spirit, excellent command of English.
Creativity and high motivation are greatly appreciated!
A temporary contract for 38 hours per week for the duration of 4 years, preferably starting as soon as possible but no later dan 1 September. The initial appointment will be for a period of 18 months and after satisfactory evaluation it will be extended for a total duration of 4 years and should lead to a dissertation (PhD thesis). We will draft an educational plan that includes attendance of courses and (international) meetings. We also expect you to assist in teaching undergraduates and master students.
The salary will be €2,325 to €2,972 (scale P) gross per month, based on full-time employment (38 hours a week). These amounts are exclusive 8% holiday allowance and 8,3% end-of-year bonus. A favorable tax agreement, the ‘30% ruling’, may apply to non-Dutch applicants. The Collective Labour Agreement of Dutch Universities is applicable.
Are you curious about our extensive package of secondary employment benefits like our excellent opportunities for study and development? Then find out more about working at the Faculty of Science.
Do you have questions about this vacancy? Or do you want to know more about our organisation? Please contact: