The Jamaican sprinter, who beat out American hurdler Aries Merritt and Kenyan runner David Rudisha, had previously won the award in 2008, 2009, and 2011.
"For me this is a great honor to win a fourth time. I really worked hard and I was really focused this year. This season was one of my toughest. I had my ups and downs, even though we don't like to talk about them," said Bolt, who thanked his coach and the fans at the Olympics.
"The London games were magnificent. Fans were there in the stadium from 10 in the morning.
Sprinter Allyson Felix of the United States took home the women's award.
In 2003, Bolt and Felix won the Rising Star awards for the most promising young athletes of the year.
"Nine years ago I was here with Allyson, now I am back, that means I am getting old, which is slightly worrying," said the 26-year-old Bolt. "But I will continue to work hard."
Felix won the 200-meter gold medal in London after settling for silver four years earlier in Beijing.
"My biggest defeats have come at the Olympic Games. After Beijing I was devastated, I had worked as hard as possible but I came up short," she said. "But I decided to rededicate myself even harder and it was a blessing in disguise. I was able to work harder than ever before and finally it all came together."
The other finalists for the women's award were New Zealand shot putter Valerie Adams and British heptathlete Jessica Ennis—both gold medalists in London.
Merritt won the Inspiration Award for his season, which included a gold medal and a world record in the 110-meter hurdles.
Rudisha, who was named Athlete of the Year in 2010, won the Performance of the Year award for his record-setting gold medal run in the 800 meters.
Javelin thrower Keshorn Walcott of Trinidad and Bahamas sprinter Anthonique Strachan won the Rising Star awards.
The awards were announced on the final day of the IAAF's 100th anniversary celebration.