Good try! The ? in regex is "zero or one of the preceding character" but the square brackets you referenced could specify the character as being in an allowable set, i.e., it has nothing to do with grouping. Your use of the parenthetical does group and, if followed by a ? will make the entire group optional.
Redirect (almost) everything to one location, eh? Why not:
RewriteRule ^(category/)?item(/|\\.html/)?$ category/item.html [R=301,L]
Okay, that looks a bit complex but it says:
Start with an optional category/, continue with item, continue with EITHER an optional trailing / OR .html/ and ends. The redirection will not loop as category/item.html does not have a trailing /.
WARNING, a redirection like this (many different levels of directories for your request) will force the use of (internal) absolute links in all support files (.css, .js, .gif, .jpg, etc.) and, IMHO, a trailing / is something which is ONLY done for a directory request.