Expected results: On top of the menu the "Editing with:" followed by the list of users that are editing doesn't appear anymore

No, this setting controls the user avatars displayed on the left side of the editing area. I'll enable the hints for the realtime configuration options in order to remove the confusion, but after 16.3RC1 is released.