Reward within a game system is when you recieve positive encouragement, points, a bonus or power-up, or any other manifestation of a positive state change.
Reward can be both diegetic, or strictly conceptual, and both are used within modern designs. Examples of non-tangible, non-gameplay rewards are uncovering more story, seeing a beautiful vista in the game world, or being told by a game character that you've done a good job. The last example can be heavy-handed if not done carefully.
Different types of rewards fit different types of game systems, because of their contextual character. For example, a "health kit", while it is a positive state change, would not be considered a reward in a space shoot'em up because it does not fit the context of the game world. It is thus non-diegetic as well.