Mathematics and Computational Geometry

Schneider, Philip. David Eberly. Geometric Tools for Computer Graphics
I don't know why, but this book seems to be overlooked by most. It shouldn't be, because it is a fantastic reference to a large collection of geometrical problems, including (but not limited to) distance, containment, and intersection tests. At over 1,000 pages everyone is likely to find lots of useful stuff in this book!

Mortenson, Michael. Mathematics for Computer Graphics Applications (2nd Edition)
This is a lovely book providing a gentle introduction to points, vectors, matrices, halfspaces, CSG, curves, and surfaces (and more).

O'Rourke, Joseph. Computational Geometry in C (2nd Edition)
de Berg, Mark. et al. Computational Geometry (2nd Edition)
So, I cannot make up my mind as to which one of these two is the best computational geometry book. One day I like O'Rourke's book better because it is more practical. Another day I like the "Dutch book" better because it has a better theoretical presentation. One thing is certain though: computational geometry is quite important and either one of these two books provides a good introduction to the field.

Computer Graphics

Akenine-Möller, Tomas. Eric Haines. Real-Time Rendering (2nd Edition)
It used to be that every graphics programmer had to have a copy of Computer Graphics: Principles and Practice by Foley et al. Well, that was then. Today, this is the new bible for graphics programmers! (And yes, it is a coincidence that my book has half its title in common with this book.)

Blinn, Jim. Jim Blinn's Corner
Blinn, Jim. Jim Blinn's Corner: Dirty Pixels
Blinn, Jim. Jim Blinn's Corner: Notation, Notation, Notation
These three compendium books of Blinn's "Jim Blinn's Corner" columns (from IEEE Computer Graphics and Applications) contain choice tidbits that are not available elsewhere (such as the discussion of line clipping in the first book, just to mention one tidbit). Great writing too!

Physics and Collision Detection

Coutinho, Murilo. Dynamic Simulations of Multibody Systems
This delightful little book reads like the book version of Baraff's physics papers, with additional coverage of some collision detection algorithms (most notably V-Clip and GJK).

Linear and Quadratic Programming

Okay, so linear programming (and quadratic) isn't for everyone. However, if you work on, say, collision detection or physics it doesn't hurt to know a little about these topics. For that reason, I (briefly) cover both subjects in my book, giving what I think is a readable introduction. For more in-depth information on these topics, these are some solid reference books:

Chvátal, Vasek. Linear Programming
Quite a few years old now (the book was published in 1983), Chvátal's book is still one of the best introductions to linear programming. Very readable.

Boyd, Stephen. Lieven Vandenberghe. Convex Optimization
More modern than Chvátal's book, this book is actually available in its entirety on the internet, so you can read it before you decide to buy it. Suffice to say, it is such a good presentation of convex optimization (think of it as a generalization of linear programming) that I bought it outright!

Luenberger, David. Introduction to Linear and Nonlinear Programming (2nd Edition)
This classic covers more material than Chvátal's book. Was out of print, but is now back in a second edition.

General Programming

Warren Jr., Henry. Hacker's Delight
How do you implement xor if no xor instruction is available? Can you do it in three instructions? How do you best count how many one bits are set in a 32-bit word? If you don't know the answers to these questions or are fascinated by obscure bit-twiddling tricks and other programming magic then this is the definitive reference. In fact, I really wish this book didn't exist, because I would have wanted to write it myself!

Dewdney, A. K. The New Turing Omnibus
I read the previous edition of this book, when it was called just "The Turing Omnibus" and I can only assume it has gotten better in the new edition, which would be amazing, considering how good the first edition was. This book is pretty much a condensed, slightly popularized version of a four-year computer science education. Had a hard time understanding big-O notation and algorithm complexity? It's in there. And so is linear programming, the halting problem, Huffman coding, and a gazillion (well, 60-some) other interesting topics. Anything that makes the study of NP-completeness fun must be good. It'll even tell you how to sort numbers using spaghetti, how can you go wrong with that!