In addition to script and language specific punctuation and native numbers, the following glyphs are highly recommended for inclusion in fonts.
First four glyphs
TrueType outline fonts should have the following four glyphs at the beginning of a font. These were listed in Apple's original TrueType specification. These glyphs are recommended to allow for the same version of the font to work on both Windows and Macintosh.
Glyph 0 is the .notdef (missing character) glyph.
Glyph 1 is the .null glyph; it has no contours and zero advance width.
Glyph 2 is the nonmarkingreturn glyph; it has no contours and positive advance width.
Glyph 3 is the space (and no-break space) glyph; it has no contours and positive advance width.
*) For the Macintosh platform, all unmapped characters are mapped to the first glyph.
Glyph 2 and 3 should have the same advance width.
The .notdef glyph is very important for providing the user feedback that a glyph is not found in the font. This glyph should not be left without an outline as the user will only see what looks like a space if a glyph is missing and not be aware of the active font's limitation.
It is recommended that the shape of the .notdef glyph be either an empty rectangle, a rectangle with a question mark inside of it, or a rectangle with an "X". Creative shapes, like swirls or other symbols, may not be recognized by users as indicating that a glyph is missing from the font and is not being displayed at that location.
General punctuation and 'Latin' numbers
Quotes and Ellipsis
Smart quotes (also known as curly quotes) are fancy characters which make text look better compared to the straight apostrophe (') and straight quote or inches character ("). Microsoft Word also automatically changes 3 periods to an ellipsis.
If your font does not support these characters, you can either turn the "Smart Quotes" and "Ellipsis" features off in the application (In Microsoft Word it's under Tools -> AutoCorrect) or make sure these glyphs and their mappings are available in the font.
Other suggested glyphs
* nbspace is usually mapped to the space glyph
** bulletoperator is usually mapped to the periodcentered glyph