Programming for a Large-Scale Infrastructure

Advanced Programming Specialist Group event

Thursday 10th November 2005, 6.00 pm

BCS, Davidson Building, 5 Southampton Street, London WC2. (Southampton Street runs north from The Strand to the Covent Garden Piazza; nearest rail stations - Charing Cross and Covent Garden).

Dr Knut Magne Risvik, Engineering Director, Google.

Having to process petabytes of data sounds hard, but it's easier than you may think, and more affordable too. In this talk we wil describe how we deal with large amounts of data at Google. It starts with compute clusters that are optimised for throughput rather than peak performance, and a software layer that turns these clusters of relatively unreliable machines into a reliable computing platform.

To simplify storage management, the Google File System (GFS) organises a sea of local IDE disks into a convenient and reliable file system optimised for very large files. To simplify programming, the MapReduce framework allows programmers to focus on just the transformations they want to accomplish, freeing them from worrying about parallelisation, scalability, and machine failures. And finally, to allow for effective sharing of a large clusters, a workqueue management system schedules the combined workloads onto the available servers.

Free, but please ask Dr Frank Martin to add your name to the security list by e-mailing him at -: Notification is required individually for each meeting.

CPD Value:
Half unit