So I've stumbled upon an article mentioning that the most powerful supercomputer in 2010 was Tianhe-1 with a peak performance of 2.5 petaFLOPS.

Fast forward, it's 2023. We've got Graphcore IPUs available for free (ref link so I'll use it for paid plans...) at Paperspace.

And they've got... 1 petaFLOPS with FP16 (half-precision floating point format):

Okay, okay, Tianhe-1 has 2.56 on FP32, with the free IPU having just 0.25... But just think about it: 13 years had passed, and we've got 10% of the world's most powerful computing unit power, costing $20M/year in operating expenses and serviced by 200 people worth of staff, for free. And it weighs about 16.5 kgs instead of a datacenter-level size.