With the rapidly increasing demand of data traffic in fifth-generation (5G) wireless systems, various sophisticated techniques have been developed to cope with the demand. New radio-unlicensed (NR-U) technology is one of the most promising techniques to address the exponential growth of data traffic. At the same time, NR base stations are densely deployed in 5G. Therefore, large numbers of NR-U base stations attempt to access the unlicensed spectrum. Although listen before talk (LBT) with random backoff can guarantee fairness, it can also increase the collision probability. For the IEEE 802.11 Wi-Fi networks, a contention-based random access protocol is adopted, which cannot guarantee the Quality of Service (QoS). To overcome these problems, a new LBT protocol, referred to as cooperative LBT, is designed, in which zero forcing (ZF) precoding is applied to suppress the multi-user interference. Based on the new LBT protocol, a (N+3)-state semi-Markovian model is established to characterize the effective capacity of NR-U in unlicensed bands, where N is the number of NR-U base stations. An expression for the effective capacity is obtained, which is a function of QoS, instantaneous transmit rate, the number of Wi-Fi nodes, and NR-U base stations. The influences of the finite backhaul and transmit power on the effective capacity are also analyzed in this paper. The simulation results show that the usage of cooperative communication can reduce the collision probabilities from 0.4928 to 0.1632 and increase the effective capacities by 220.04%. The cooperative LBT protocol has a more compelling advantage for 5G NR ultradense deployment scenarios.