Perhaps Incandescence, though it's less about mathematics and more about a society working out the physics of its environment. His website has more maths on it

Perhaps Incandescence, though it's less about mathematics and more about a society working out the physics of its environment. His website has more maths on it than I recall being in the novels.