Sorafenib, a potent anticancer drug, has low absorption in the gastrointestinal tract due to its poor aqueous solubility. The main purpose of this investigation was to design sorafenib nanoparticle using a newly developed technique, nanoparticulation using fat and supercritical fluid (NUFS™) to improve the absorption of sorafenib. The quality by design (QbD) tool was adopted to define the optimal formulation variables: hydroxypropyl methyl cellulose (HPMC), polyvinyl pyrrolidone K30 (PVP), and poloxamer. The studied response variables were particle size of nanoparticle, dissolution (5, 60, and 180 min), drug concentration time profile of nanoparticle formulations, and maximum drug concentration. The result of particle size revealed that an increase in concentration of poloxamer and HPMC decreased the particle size of nanoparticles (p < 0.05). Likewise, the concentration of drug release at different time point (5, 60, and 180 min) showed HPMC and poloxamer had positive effects on drug dissolution while PVP had negative effects on it. The design space was built in accordance with the particle size of nanoparticle (target < 500 nm) and dissolution of sorafenib (target > 7 µm/mL), following failure probability analysis using Monte Carlo simulations. In vivo pharmacokinetics studies in beagle dogs demonstrated that optimized formulation of sorafenib (F3 and F4 tablets) exhibited higher blood drug profiles indicating better absorption compared to the reference tablet (Nexavar®). In conclusion, this study showed the importance of systematic formulation design for understanding the effect of formulation variables on the characteristics of nanoparticles of the poorly soluble drug.