To cite randomForest in publications use:

A. Liaw and M. Wiener (2002). Classification and Regression by randomForest. R News 2(3), 18--22.

Corresponding BibTeX entry:

  @Article{,
    title = {Classification and Regression by randomForest},
    author = {Andy Liaw and Matthew Wiener},
    journal = {R News},
    year = {2002},
    volume = {2},
    number = {3},
    pages = {18-22},
    url = {https://CRAN.R-project.org/doc/Rnews/},
  }