Drilling engineering plays an indispensable role in the modern energy exploration and development. The wellbore stability directly determines the success or the failure of the drilling operation. The traditional wellbore stability model does not allow any degree of wellbore damage, but in the actual drilling operation, the stable wellbore generally allows a limited width of caving area. The design based on the traditional wellbore stability model will lead to a high density of drilling fluid, which not only will increase the drilling cost but also will reduce the drilling efficiency, more likely will cause the loss of drilling fluid or even the formation fracture. According to the characteristics of the brittle shale formation, a wellbore stability model is established. Based on the logging data and the test data, a geomechanical model is established. The quantitative risk analysis method based on the limited collapse width is used to reasonably evaluate the wellbore stability probability and optimize the drilling fluid density. Based on the case study of a certain oilfield in Qinghai Province, this paper uses this method to optimize the density window of drilling fluid in the stable and unstable sections of the wellbore, to some extent, to improve the ROP and reduce the drilling cost. The probability of the wellbore stability can reach 91-98% according to calculation and evaluation. The calculation results are basically consistent with the actual situation, which shows that this method is effective and the results are reliable and practical.