The same setting covers categories, tags and WooCommerce product categories, because their archive pages have addresses too, and on a shop that is where most links point. WordPress's own guess at a missing page only ever looks for a post, so without this a renamed category's old address simply answers "page not found".

With Keep the old address working when a page is renamed on:

Renaming one. Change Shoes to Footwear, or just its slug, and /product-category/shoes/ redirects to /product-category/footwear/. The redirect is marked from a renamed category in the list, and it is a redirect like any other.
Renaming it again. The first redirect is moved to the newest address, so a visitor makes one hop, not two. Rename it back to an address it once had and the redirect for that address is taken away, because the category lives there again.
Moving it under another parent, or renaming a parent. Every category underneath moves too (/product-category/clothes/shirts/ becomes /product-category/apparel/shirts/), and each one gets its own redirect.
Deleting a parent. WordPress moves the categories under it up a level, which changes their addresses, so each of them gets a redirect as well. Delete the ones under it at the same time (ticking them all on the categories screen) and none of them gets a redirect: each goes onto Broken links at the address it had before, as a deleted category does.
Giving an address to another category. Swap two categories' names, rename one to the slug another gave up, or create a new Sale after renaming the old one, and the redirect the earlier rename made for that address is deleted, so the category living there now answers. A redirect of that kind that you have pointed somewhere else yourself is left where it is.

"Category" here means any of the three. Menus, post formats and WooCommerce's hidden product flags are not addresses anybody shares, so they are left alone, and so is a taxonomy another plugin registered without addresses of its own. On a site with plain permalinks, a category's address is ?cat=12, which a rename leaves alone. A tag's (?tag=winter) and a product category's (?product_cat=shoes) do change when their slug changes, and nothing is made for them: a redirect is matched on the part of an address before the ?, and those names live after it. Any other choice under Settings → Permalinks gives them addresses this can keep.

The redirects are written once, as the save finishes. Renaming a category with two hundred under it makes two hundred and one redirects in one go, and past a hundred the page caches are emptied once rather than asked about each address.

A redirect you wrote by hand for the old address is never overwritten, as for a page, and nor is one a rename made that you have pointed somewhere else. One you wrote by hand for the address a category moves to still wins too, so the category cannot be reached there until you delete that redirect. If that redirect of yours sends the new address back to the old one, no redirect is made the other way, because the two would send visitors round in circles: the old address answers "page not found" and appears on Broken links once somebody asks for it.

Deleting a category, tag or product category is different: nothing can work out where its visitors should go. So its address goes onto Broken links straight away, at the top, marked you deleted this category and carrying its name, as a deleted page does. This needs Keep a list of addresses that were not found on, and skips an address you already have a redirect for or have put on Never list these addresses. Renamed and then deleted in the same save, it is listed at the address it had before the rename.

Its older addresses, the ones earlier renames kept working, go onto the list with it, and the redirects made for them are deleted, because they led to the address that is now gone. That leaves one place to decide where all of its visitors should go. If you already have a redirect of your own for its last address, the older ones reach it through that, so they are left alone. With the list switched off, nothing is moved.