When you haven’t activated any widgets for your sidebar* (in Appearance>Widgets), each theme displays its own default set of sidebar items. That’s why when you start a blog you see some items in your sidebar but if you go to Appearance>Widgets you see no widgets listed under “Sidebar” (before the latest update things were even more puzzling to beginners: there was a notice saying “you’re using 0 widgets”).
[* "Sidebar" for brevity: actually (depending on the theme) = the sidebar, or one of the sidebars, or a bottombar, or even the header.]
Once you activate any widget, the default set is cancelled. Beginners are invariably (and justifiably) taken aback by this: they activate a widget, expect to see it in the sidebar in addition to the items already present, and instead end up with that widget alone.
• If you want all the default sidebar items of the theme you’re using and no more than that, you need do nothing.
• If you want to remove some of them, what you must actually do is add the ones you really want by activating the relevant widgets. Same thing if you want some or all of the default ones plus additional ones.
• If you want nothing in a sidebar, then in most themes the way to do it is activate a blank Text widget. In some themes, however, the content of text widgets is enclosed in a frame, so a blank text widget will give you an empty frame in the sidebar instead of an empty sidebar; in that case activate a blank RSS widget instead. (For no sidebar at all instead of blank sidebar: there are a couple of themes in which the sidebar can be deactivated, plus a few themes that don’t show, or can be made not to show, the sidebar on static pages – see here.)
Other things to note:
• Widgets are normally activated (in Appearance>Widgets) by dragging them from the “Available Widgets” module to the “Sidebar” module (when you reach the right spot, a dashed rectangle shows up). They can be arranged in any order you wish, both when you initially activate them and afterwards (again by dragging them above or below others).
• Some users experience difficulties with dragging. If that’s the case, you can click “Screen Options” (while on the Widgets screen) and tick “Enable accessibility mode”: that will produce a more surefire “Add” tab to the right of each widget name.
• The default sidebar items of a theme are artificially generated from the theme’s underlying files: in some cases they are not identical in look and/or function to the standard widgets common to all wp.com themes.
• The default sidebar set varies from theme to theme. So if you haven’t activated any widgets yourself, you may get a different set of items when you switch from one theme to another.
• If you have activated one or more widgets, and switch from a theme to a similarly designed one, the widgets will remain in place; otherwise they may move to “Inactive”, and you must move them back to “Sidebar”.
• Update Nov. 2011: A few of the newer themes are free of the issue analyzed in this post: they have no default sidebar items, and they simply don’t display the sidebar unless you add widgets yourself.