Mittagsseminar (in cooperation with J. Lengler, A. Steger, and D. Steurer)

Date and Time: Thursday, August 30, 2018, 12:15 pm

Duration: 30 minutes

Location: OAT S15/S16/S17

Speaker: Sebastian Stich (EPFL)

Communication Efficient SGD through Quantization with Memory

Nowadays machine learning applications require stochastic optimization algorithms that can be implemented on distributed systems. The communication overhead of the algorithms is a key bottleneck that hinders perfect scalability. Various recent works proposed to use quantization or sparsification techniques to reduce the amount of data that needs to be communicated, for instance by only sending the most significant entries of the stochastic gradient (top-k sparsification). Whilst this scheme shows good performance in practice it eluded theoretical analysis so far.

In this work we analyze a variant of Stochastic Gradient Descent (SGD) with k-sparsification (for instance top-k or random-k) and show that this scheme converges at the same rate as vanilla SGD. That is, the communication can be reduced by a factor of the dimension of the whilst still converging at the same rate.

Joint work with Jean-Baptiste Cordonnier and Martin Jaggi

Upcoming talks     |     All previous talks     |     Talks by speaker     |     Upcoming talks in iCal format (beta version!)

Previous talks by year:   2025  2024  2023  2022  2021  2020  2019  2018  2017  2016  2015  2014  2013  2012  2011  2010  2009  2008  2007  2006  2005  2004  2003  2002  2001  2000  1999  1998  1997  1996  

Information for students and suggested topics for student talks

Automatic MiSe System Software Version 1.4803M   |   admin login