The concept of “freedom” is so important because it’s multi-faceted. People aren’t entitled to do whatever they want because everyone is also entitled to freedom from certain things. That includes things like slavery (Article 4 of the UDHR), discrimination, and harassment. “Freedom from” places certain restrictions on the “freedom to.” That’s why things like hate speech are not protected rights.