They attend at least four meetings and work about 300 hours a year, but the people who sit on the boards of S&P 500 companies received an average of $251,000 last year.

Financial and technology company Fidelity National Information Services Inc. (FIS) paid the highest compensation due to a $9.5 million retention bonus, while directors of Warren Buffett's Berkshire Hathaway received the lowest, according to data compiled by Bloomberg. Berkshire Hathaway directors received $3,800.

Berkshire Hathaway and FIS did not immediately return requests for comment.

Though duties vary by company and individual director, a director's duties are usually part-time and typically include a meeting each quarter, plus committee meetings, preparation work and the ability to be on-call. Major events like acquisitions or mergers require longer hours.

Although Berkshire Hathaway directors own equity - whether Class A or Class B shares of the company, Andy Restaino, founder of Technical Compensation Advisors, said he also thinks the modest director pay is a philosophical preference stemming from Buffett, also known as "the Oracle of Omaha."

"In general, even Berkshire executives don't get paid very much," Restaino said.

Whether subconsciously delivered or not, Buffett's message is likely: "If you want to play in my sandbox, you've got to play according to my rules," Restaino said.

Buffett, second-richest person in the U.S. after Bill Gates, is awarded compensation of $423,923. He is worth about $53.5 billion, according to Forbes.

See a List of the Highest-Paid CEOs

Donald Keough, one of the directors of Berkshire Hathaway and chairman of Allen and Company Inc., owns about 100 shares of Berkshire's Class A stock, which is trading for around $171,608 a share. He also owns about 60 shares of Class B stock, which is trading at $114.

However, Aaron Boyd, director of governance research for executive compensation research firm Equilar, said directors aren't in the business for the money.

"Pay is not driving their desire. It's a little bit of prestige, the opportunity and challenge to serve as a director and privilege to serve in that role and try to do as good of a job as possible," Boyd said.

Most directors of companies are highly-experienced people, often CEOs of other firms, who have a difficult and highly-coveted job of representing shareholders.

"You can't go to school and get a degree to become a director of a company," Boyd said.

While they do set their own pay, Boyd said directors are very conscious of that power and don't make wild swings in their compensation.

"They want to compensate directors for their pay, but they obviously don't want to look like they are paying themselves a lot of money just for the sake of doing it," he said.