General Programming

Warren Jr., Henry. Hacker's Delight
How do you implement xor if no xor instruction is available? Can you do it in three instructions? How do you best count how many one bits are set in a 32-bit word? If you don't know the answers to these questions or are fascinated by obscure bit-twiddling tricks and other programming magic then this is the definitive reference. In fact, I really wish this book didn't exist, because I would have wanted to write it myself!
Dewdney, A. K. The New Turing Omnibus
I read the previous edition of this book, when it was called just "The Turing Omnibus" and I can only assume it has gotten better in the new edition, which would be amazing, considering how good the first edition was. This book is pretty much a condensed, slightly popularized version of a four-year computer science education. Had a hard time understanding big-O notation and algorithm complexity? It's in there. And so is linear programming, the halting problem, Huffman coding, and a gazillion (well, 60-some) other interesting topics. Anything that makes the study of NP-completeness fun must be good. It'll even tell you how to sort numbers using spaghetti, how can you go wrong with that!

Bentley, Jon. Programming Pearls (2nd Ed)
Jon Bentley is one of those rare beasts: an academic who actually knows how to code, and code well. The chapters of the book originally were essays from his column in CACM, so they are nice bite-sized chunks written to both entertain and educate. This makes for excellent bedtime reading, and the topics are so interesting that you're guaranteed to dream happy programmer dreams, and will wake up refreshed wanting to code like never before!
Skiena, Stephen The Algorithm Design Manual and The Algorithm Design Manual (2nd Ed)
This is an excellent book (soon in a second edition) that covers a large number of computer science topics, including graph problems, computational geometry, sorting and searching, numerical problems, combinatorial problems, set and string problems, and many more. Each topic is succinctly covered in just a few pages so it is easy to get a quick overview of the problem, and there are pointers to where further, more detailed information can be found.

Computer Graphics

Akenine-Möller, Tomas. Eric Haines, Naty Hoffman Real-Time Rendering (3rd Ed)
It used to be that every graphics programmer had to have a copy of Computer Graphics: Principles and Practice by Foley et al. Well, that was then. Today, this is the new bible for graphics programmers! (And yes, it is a coincidence that my book has half its title in common with this book.) It is now in a third edition, and is an absolutely must have. The third edition is massive and covers just about every computer graphics topic you can think of, in an easy-to-read manner.
Blinn, Jim. Jim Blinn's Corner
Blinn, Jim. Jim Blinn's Corner: Dirty Pixels
Blinn, Jim. Jim Blinn's Corner: Notation, Notation, Notation
These three compendium books of Blinn's "Jim Blinn's Corner" columns (from IEEE Computer Graphics and Applications) contain choice tidbits that are not available elsewhere (such as the discussion of line clipping in the first book, just to mention one tidbit). Great writing too!

Collision Detection and Physics

Ericson, Christer. Real-Time Collision Detection
Am I allowed to recommend my own book? Well, screw it, I am, because it is good. But don't take my word for it: check the table of contents, read the reviews on Amazon, and see the endorcements page too. Perhaps the important thing to note is that the title is actually quite misleading: my book covers way more stuff than just collision detection! There's lots of information about spatial partitioning (grids, trees, and hierarchies of all kinds) which is relevant for graphics, AI, and other systems that require object databases, a readable primer to math and geometry, unmatched detail about numerical and geometric robustness, and lots of goodness about how to optimize code in general.
Coutinho, Murilo. Dynamic Simulations of Multibody Systems
This delightful little book reads like the book version of Baraff's physics papers, with additional coverage of some collision detection algorithms (most notably V-Clip and GJK).

Mathematics and Computational Geometry

Hausner, Melvin A Vector Space Approach to Geometry
This is a really good book that combines linear algebra and geometry in one. Unlike the Mortenson book (below), this is a proper textbook, but for a textbook it's surprisingly easy to read. Covers vector spaces, area and volume, matrices and linear transformations, and more.
Mortenson, Michael. Mathematics for Computer Graphics Applications (2nd Edition)
This is a lovely book providing a gentle introduction to points, vectors, matrices, halfspaces, polygons and polyhedra, topology, CSG, curves and surfaces (and more). Best thing is that this is not a dry textbook but more of a smörgåsbord of very tasty math and geometry morsels.
Schneider, Philip. David Eberly. Geometric Tools for Computer Graphics
I don't know why, but this book seems to be overlooked by most. It shouldn't be, because it is a fantastic reference to a large collection of geometrical problems, including (but not limited to) distance, containment, and intersection tests. At over 1,000 pages everyone is likely to find lots of useful stuff in this book!
O'Rourke, Joseph. Computational Geometry in C (2nd Edition)
de Berg, Mark. et al. Computational Geometry (2nd Edition)
So, I cannot make up my mind as to which one of these two is the best computational geometry book. One day I like O'Rourke's book better because it is more practical. Another day I like the "Dutch book" better because it has a better theoretical presentation. One thing is certain though: computational geometry is quite important and either one of these two books provides a good introduction to the field.

Linear and Quadratic Programming

Okay, so linear programming (and quadratic) isn't for everyone. However, if you work on, say, collision detection or physics it doesn't hurt to know a little about these topics. For that reason, I (briefly) cover both subjects in my book, giving what I think is a readable introduction. For more in-depth information on these topics, these are some solid reference books:
Boyd, Stephen. Lieven Vandenberghe. Convex Optimization
More modern than Chvátal's book, this book is actually available in its entirety on the internet, so you can read it before you decide to buy it. Suffice to say, it is such a good presentation of convex optimization (think of it as a generalization of linear programming) that I bought it outright!
Chvátal, Vasek. Linear Programming
Quite a few years old now (the book was published in 1983), Chvátal's book is still one of the best introductions to linear programming. Very readable.
Luenberger, David. Introduction to Linear and Nonlinear Programming (2nd Edition)
This classic covers more material than Chvátal's book. Was out of print, but is now back in a second edition.