To borrow a stock, you need someone to lend it. Brokerage firms fill this role. With limited exceptions, short sellers are borrowing from brokerage firms (i.e. your brokerage firm may be lending out your securities to another party who is actively hoping to drive down the price of the stock).
While your brokerage firm can’t lend out your stocks without your permission, you likely signed a customer agreement that explicitly allows your broker to do just that. This clause is often buried in your customer agreement and few people pay any attention to it.